How to Use the Firestore Export API Before it is Released

TL;DR

Check out FirestoreRestore.
It uses a brand new API to backup and restore your Firestore database!

Background

There have been many requests for Google to add backup and restore functionality to their Cloud Firestore offering. Until this week, the only method for backing up a Firestore database has been to run utilities that traverse the tree as an admin client, saving each document's JSON representation on disk.

Discovery

This morning, I opened the Google APIs Explorer and found two new methods listed under Firestore v1beta1: exportDocuments and importDocuments.

Exploration

As the API documentation states, firestore.projects.databases.exportDocuments is able to export Firestore data to Google Cloud Storage.

To test this out, I opened up the Firebase Console and created a new project. In order for there to be data to back up, I created a user document from the console.

In the same project, I opened the Google Cloud Storage Console and created a bucket in which to store my backups.

From the Google APIs Explorer's exportDocuments page, I entered the Firebase project name, as well as a path to the storage bucket, and hit Execute.

The following is the API's response:

{
 "name": "projects/firestore-restore/databases/(default)/operations/<REDACTED>",
 "metadata": {
  "@type": "type.googleapis.com/google.firestore.admin.v1beta1.ExportDocumentsMetadata",
  "startTime": "2018-07-18T23:20:26.535130Z",
  "operationState": "PROCESSING",
  "outputUriPrefix": "gs://firestore-restore/backups/2018-07-18"
 }
}

A few moments later, I opened the bucket. Sure enough, the data was present!

Restore

Now, to test the restore functionality. I deleted the contents of the Firestore database, and ran importDocuments with the same parameters as the export request, and sure enough, the data came back!

Is the Backup Complete?

Unfortunately, there is currently no "documented" way to check the status of a Firestore export in progress. Luckily, with a little URL guessing, I was able to find the endpoint for the operation resource. It was there all along; the name field in the export response is the status path!

A Temporary Tool: FirestoreRestore

As I don't know when Google will officially support and build tooling for backing up and restoring Cloud Firestore databases, I developed a simple command line utility to interact with the service.

It requires the creation of a service account with the "Cloud Datastore Import Export Admin" role. With a JSON key for the service account, this tool is able to perform a backup and restore.

Check out the README for more info 😀

Speculation

Google will likely launch backup and restore functionality within the Firebase UI. It'll probably expose the per-collection backup capability that is present in the API. It'll probably be much easier to use than some command line program you found on the internet.

Until then, enjoy the CLI!

Comments

[masters.🤔]

Thanks for the write up. How hard would it be to leverage this API to do offsite backups (in case you wanted to migrate somewhere else for instance)

[Christian Alexander]

Thanks for the question! It should be trivial to download the export files from the storage bucket and store them elsewhere. I'm not sure if the API would support an external destination directly (S3, for instance), but my guess is that gs:// URLs are the only supported destinations.

Migration may be a bigger issue. It appears that the files generated by this API are in the same format as Google Cloud Datastore entity exports, which appear to be proprietary. The decision to use Firebase comes with a ton of vendor lock-in, so this is not surprising.

If anyone is looking to export their data for use in another system, one of the tree traversal scripts found on NPM would likely be their best bet. Most of them are capable of producing JSON files that could be consumed by other systems.

[Luis Ruiz]

Do you know if it is now possible to download the entire folder created when exporting? to save it on my computer

[Christian Alexander]

The files should be downloadable from the Google Cloud Storage bucket. I'm not sure about the file format or how to use the data after downloading.