How to import data from cloud firestore to the local emulator?

I want to be able to run cloud functions locally and debug against a copy from the production data. Is there a way to copy the data that is online to the local firestore emulator?

This can be accomplished through a set of commands in terminal on the existing project:

1. Login to firebase and Gcloud:

firebase login
gcloud auth login

2. See a list of your projects and connect to one:

firebase projects:list
firebase use your-project-name

gcloud projects list
gcloud config set project your-project-name

3. Export your production data to gcloud bucket with chosen name:

gcloud firestore export gs://your-project-name.appspot.com/your-choosen-folder-name

4. Now copy this folder to your local machine, I do that in functions folder directly:

cd functions
gsutil -m cp -r gs://your-project-name.appspot.com/your-choosen-folder-name .

5. Now we just want to import this folder. This should work with the basic command, thanks to latest update from Firebase team https://github.com/firebase/firebase-tools/pull/2519.

firebase emulators:start --import ./your-choosen-folder-name

Check out my article on Medium about it and a shorthanded script to do the job for you https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-local-emulator-e82ae1c6ed8

Note: Its better to use a different bucket for it, as copying into your project bucket will result in the folder created in your firebase storage.
If you are interested in gsutil arguments like -m, you can see them described by executing gsutil --help.

My method is somewhat manual but it does the trick. I've shared it in this useful Github thread but I'll list the steps I did here if you find them useful:

1. Go to my local Firebase project path.
2. Start the emulators using: firebase emulators:start
3. Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
4. Export this data locally using: emulators:export ./mydirectory
5. About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this: gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test)
6. Download this folder to local drive with: gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this: gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
7. Once downloaded, imitate the local export structure renaming the folder to firestore_export and copying the firebase-export-metadata.json file from the local export folder. Just to be visual, here's the structure I got:

$ tree .
.
├── local_data_export
│   ├── firebase-export-metadata.json
│   └── firestore_export
│       ├── all_namespaces
│       │   └── all_kinds
│       │       ├── all_namespaces_all_kinds.export_metadata
│       │       └── output-0
│       └── firestore_export.overall_export_metadata
└── production_data_export
    ├── firebase-export-metadata.json
    └── firestore_export
        ├── all_namespaces
        │   └── kind_myCollection
        │       ├── all_namespaces_kind_myCollection.export_metadata
        │       ├── output-0
        │       └── output-1
        └── firestore_export.overall_export_metadata

8 directories, 9 files

8. Finally, start the local emulator pointing to this production data to be imported: firebase emulators:start --import=./mock_up_data/production_data_export/
9. You should see the imported data at: http://localhost:4000/firestore/

This should assist readers for now, while we await a more robust solution from the Firebase folks.

You can use the firestore-backup-restore to export and import your production data as JSON files.

I wrote a quick hack to allow for importing these JSON in the Firebase Simulator Firestore instance.

I proposed a pull request and made this npm module in the meantime.

You can use it this way:

const firestoreService = require('@crapougnax/firestore-export-import')
const path = require('path')

// list of JSON files generated with the export service
// Must be in the same folder as this script
const collections = ['languages', 'roles']

// Start your firestore emulator for (at least) firestore
// firebase emulators:start --only firestore

// Initiate Firebase Test App
const db = firestoreService.initializeTestApp('test', {
   uid: 'john',
   email: '[email protected]',
})

// Start importing your data
let promises = []
try {
   collections.map(collection =>
      promises.push(
         firestoreService.fixtures(
            path.resolve(__dirname, `./${collection}.json`),
            [],
            [],
            db,
         ),
      ),
   )
   Promise.all(promises).then(process.exit)
} catch (err) {
   console.error(err)
}

Obviously, since this data won't persist in the emulator, you'll typically inject them in the before() function of your test suite or even before every test.

There is no built-in way to copy data from a cloud project to the local emulator. Since the emulator doesn't persist any data, you will have to re-generate the initial data set on every run.

I was able to make some npm scripts to import from remote to local emulator and vice-versa.

"serve": "yarn build && firebase emulators:start --only functions,firestore --import=./firestore_export",
"db:update-local-from-remote": "yarn db:backup-remote && gsutil -m cp -r gs://my-firebase-bucket.appspot.com/firestore_export .",
"db:update-remote-from-local": "yarn db:backup-local && yarn db:backup-remote && gsutil -m cp -r ./firestore_export gs://my-firebase-bucket.appspot.com && yarn run db:import-remote",
"db:import-remote": "gcloud firestore import gs://my-firebase-bucket.appspot.com/firestore_export",
"db:backup-local": "firebase emulators:export --force .",
"db:rename-remote-backup-folder": "gsutil mv gs://my-firebase-bucket.appspot.com/firestore_export gs://my-firebase-bucket.appspot.com/firestore_export_$(date +%d-%m-%Y-%H-%M)",
"db:backup-remote": "yarn db:rename-remote-backup-folder && gcloud firestore export gs://my-firebase-bucket.appspot.com/firestore_export"

So you can export the local Firestore data to remote with:

npm db:update-remote-from-local

Or to update your local Firestore data with remote one, do:

npm db:update-local-from-remote

These operations will backup the remote Firestore data, making a copy of it and storing it on Firebase Storage.

I was about to go add a cli option to firebase-tools but pretty happy with the node-firestore-import-export package.

yarn add -D node-firestore-import-export

  "scripts": {
    "db:export": "firestore-export -a ./serviceAccountKey.json -b ./data/firestore.json",
    "db:import": "firestore-import -a ./serviceAccountKey.json -b ./data/firestore.json",
    "db:emulator:export": "export FIRESTORE_EMULATOR_HOST=localhost:8080 && yarn db:export",
    "db:emulator:import": "export FIRESTORE_EMULATOR_HOST=localhost:8080 && yarn db:import",
    "db:backup": "cp ./data/firestore.json ./data/firestore-$(date +%d-%m-%Y-%H-%M).json",
    "dev": "firebase emulators:start --import=./data --export-on-exit=./data",
  },

You will need to create a service account in the firebase console.

You can replace the GCLOUD_PROJECT environment variable with hard coded values.

open https://console.firebase.google.com/project/$GCLOUD_PROJECT/settings/serviceaccounts/adminsdk
mv ~/Downloads/myProjectHecticKeyName.json ./serviceAccountKey.json

That being said the gcloud tools are definitely the way to go in production, as you will need s3 backups anyway.

I have created fire-import npm package for importing firebase firestore and firebase storage to the local emulator.

I used gcloud sdk cli under the hood. I know implementation isn't the perfect. But I hope this package will help you import firestore and firebase storage easily. Have fun using the package.

I wrote a little script to able to do that:

const db = admin.firestore();
const collections = ['albums', 'artists'];
let rawData: any;
    
for (const i in collections) {
    rawData = fs.readFileSync(`./${collections[i]}.json`);
    const arr = JSON.parse(rawData);

    for (const j in arr) {
        db.collection(collections[i]).add(arr[j])
        .then(val => console.log(val))
        .catch(err => console.log('ERRO: ', err))
    }
        
}