Export json from Firestore

As we can download json file at Firebase RTDB console, are there any way to export json file of Firestore collection/document data?

One of my main objectives is to compare data before/after updating document.

I just wrote a backup and restore for Firestore. You can have a try on my GitHub.

https://github.com/dalenguyen/firestore-backup-restore

Thanks,

There is not, you'd need to come up with your own process such as querying a collection and looping over everything.

Update

As of August 7th, 2018, we do have a managed export system that allows you to dump your data into a GCS bucket. While this isn't JSON, it is a format that is the same as Cloud Datastore uses, so BigQuery understands it. This means you can then import it into BigQuery.

Google made it harder than it needed to be, so the community found a workaround. If you have npm installed, you can do this:

Export

npx -p node-firestore-import-export firestore-export -a credentials.json -b backup.json

Import

npx -p node-firestore-import-export firestore-import -a credentials.json -b backup.json

Source

I've written a tool that traverses the collections/documents of the database and exports everything into a single json file. Plus, it will import the same structure as well (helpful for cloning/moving Firestore databases). Since I've had a few colleagues use the code, I figured I would publish it as an NPM package. Feel free to try it and give some feedback.

https://www.npmjs.com/package/node-firestore-import-export

If someone wants a solution using Python 2 or 3.

Edit: note that this does not backup the rules

Fork it on https://github.com/RobinManoli/python-firebase-admin-firestore-backup

First install and setup Firebase Admin Python SDK: https://firebase.google.com/docs/admin/setup

Then install it in your python environment:

pip install firebase-admin

Install the Firestore module:

pip install google-cloud-core
pip install google-cloud-firestore

(from https://stackoverflow.com/questions/48264536/importerror-failed-to-import-the-cloud-firestore-library-for-python)

Python Code

# -*- coding: UTF-8 -*-

import firebase_admin
from firebase_admin import credentials, firestore
import json

cred = credentials.Certificate('xxxxx-adminsdk-xxxxx-xxxxxxx.json') # from firebase project settings
default_app = firebase_admin.initialize_app(cred, {
    'databaseURL' : 'https://xxxxx.firebaseio.com'
})

db = firebase_admin.firestore.client()

# add your collections manually
collection_names = ['myFirstCollection', 'mySecondCollection']
collections = dict()
dict4json = dict()
n_documents = 0

for collection in collection_names:
    collections[collection] = db.collection(collection).get()
    dict4json[collection] = {}
    for document in collections[collection]:
        docdict = document.to_dict()
        dict4json[collection][document.id] = docdict
        n_documents += 1

jsonfromdict = json.dumps(dict4json)

path_filename = "/mypath/databases/firestore.json"
print "Downloaded %d collections, %d documents and now writing %d json characters to %s" % ( len(collection_names), n_documents, len(jsonfromdict), path_filename )
with open(path_filename, 'w') as the_file:
    the_file.write(jsonfromdict)

Firestore is still early in its development so please check the docs on backups for any information pertaining to Firestore.

I found this npm package, node-firestore-backup, to be easy and useful.

Note that the --accountCredentials path/to/credentials/file.json is referring to a service account key json file that you can get by following instructions from https://developers.google.com/identity/protocols/application-default-credentials.

> 1. Go to the API Console Credentials page. > 2. From the project drop-down, select your project. > 3. On the Credentials page, select the Create credentials drop-down, then select Service account key. > 4. From the Service account drop-down, select an existing service account or create a new one. > 5. For Key type, select the JSON key option, then select Create. The file automatically downloads to your computer. > 6. Put the *.json file you just downloaded in a directory of your choosing. This directory must be private (you can't let anyone get access to this), but accessible to your web server code.

There is an npm for firestore export / import

Project to export Goto -> project settings -> Service account -> Generate new private key -> save it as exportedDB.json

Project to import Goto -> project settings -> Service account -> Generate new private key -> save it as importedDB.json

run these 2 commands from the folder where u saved the files

Export: npx -p node-firestore-import-export firestore-export -a exportedDB.json -b backup.json

Import: npx -p node-firestore-import-export firestore-import -a importedDB.json -b backup.json

It works for me.

I used Cloud Functions to export all data in Firestore to JSON format. The function that I was used:

exports.exportFirestore2Json = functions.https.onRequest((request, response) => {
    db.collection("data").get().then(function(querySnapshot) {
        const orders = [];
        var order = null

         querySnapshot.forEach(doc => {
             order = doc.data();
             orders.push(order);
         });

         response.send(JSON.stringify(orders))

         return true
    })
    .catch(function(error) {
        console.error("Error adding document: ", error);
        return false
    });
})

Then, go to https://your-project-id.cloudfunctions.net/exportFirestore2Json you will see something like this

Yes you can, you did not need to start billing in your firebase console. There is a great npm package https://www.npmjs.com/package/firestore-export-import with this you can export and import firestore collection and documents easily. Just follow some steps:

-Get your service account key Open Firebase console > Project settings > Service accounts > generate new private key

rename the downloaded file with serviceAccountKey.json

-Now create a new folder and index.js file.

-Paste you servicekey.json in this folder

-Now install this package

npm install firestore-export-import

OR

yarn add firestore-export-import

Exporting data from firebase

const { initializeApp} =  require('firestore-export-import')

const  serviceAccount  =  require('./serviceAccountKey.json')

const  appName  =  '[DEFAULT]'

initializeApp(serviceAccount, appName)

const  fs  =  require('fs');

const { backup } =  require('firestore-export-import')
//backup('collection name')

backup('users').then((data) =>
{
	const  json  =  JSON.stringify(data);

	//where collection.json is your output file name.
	fs.writeFile('collection.json', json, 'utf8',()=>{

	console.log('done');

})

});

Execute node index.js and you should see a new collection.json file with your collection and documents in it. If it looks a little messy pretty format it online with https://codebeautify.org/jsonviewer

This index.js was just a very basic configuration which exports the whole collection with everything in it, read their documentation you could do queries and much more!

Importing data to firebase

const { initializeApp,restore } =  require('firestore-export-import')

const  serviceAccount  =  require('./serviceAccountKey.json')
const  appName  =  '[DEFAULT]'

initializeApp(serviceAccount, appName)
restore('collection.json', {
//where refs is an array of key items
	refs: ['users'],
	//autoParseDates to parse dates if documents have timestamps
	autoParseDates: true,

	},()=>{

console.log('done');
})

After execution you should see your firestore populated with collection users!

1. Create a blank folder (call it firebaseImportExport ) and run npm init
2. Go to the source Firebase project -> Settings -> Service Accounts
3. Click on the Generate new private key button and rename the file as source.json and put it in the firebaseImportExport folder
4. Do the same (step 2 & 3) for the destination project and rename the file as destination.json
5. Install the npm i firebase-admin npm package.
6. Write the following code in the index.js

const firebase = require('firebase-admin');

var serviceAccountSource = require("./source.json");
var serviceAccountDestination = require("./destination.json");

const sourceAdmin = firebase.initializeApp({
    credential: firebase.credential.cert(serviceAccountSource),
    databaseURL: "https://**********.firebaseio.com" // replace with source
});

const destinationAdmin = firebase.initializeApp({
    credential: firebase.credential.cert(serviceAccountDestination),
    databaseURL: "https://$$$$$.firebaseio.com"
  }, "destination");

const collections = [ "books", "authors",   ...]; // replace with your collections

    var source = sourceAdmin.firestore();
    var destination = destinationAdmin.firestore();
   collections.forEach(colName => {
    source.collection(colName).get().then(function(querySnapshot) {
        querySnapshot.forEach(function(doc) {
            destination.collection(colName).doc(doc.id).set({...doc.data()});
        });
    });
   });

Open any of your clientside firebase apps (React, Angular, etc.). Use this code anywhere to log console and copy

const products = await db
  .collection("collectionName")
  .where("time", ">", new Date("2020-09-01"))
  .get()


const json = JSON.stringify(products.docs.map((doc) => ({ ...doc.data() })))
console.log(json)