I'd like to make a copy of a collection in Firestore upon an event using Cloud Functions
I already have this code that iterates over the collection and copies each document
const firestore = admin.firestore()
firestore.collection("products").get().then(query => {
query.forEach(function(doc) {
var promise = firestore.collection(uid).doc(doc.data().barcode).set(doc.data());
});
});
is there a shorter version? to just copy the whole collection at once?
I wrote a small nodejs snippet for this.
const firebaseAdmin = require('firebase-admin');
const serviceAccount = '../../firebase-service-account-key.json';
const firebaseUrl = 'https://my-app.firebaseio.com';
firebaseAdmin.initializeApp({
credential: firebaseAdmin.credential.cert(require(serviceAccount)),
databaseURL: firebaseUrl
});
const firestore = firebaseAdmin.firestore();
async function copyCollection(srcCollectionName, destCollectionName) {
const documents = await firestore.collection(srcCollectionName).get();
let writeBatch = firebaseAdmin.firestore().batch();
const destCollection = firestore.collection(destCollectionName);
let i = 0;
for (const doc of documents.docs) {
writeBatch.set(destCollection.doc(doc.id), doc.data());
i++;
if (i > 400) { // write batch only allows maximum 500 writes per batch
i = 0;
console.log('Intermediate committing of batch operation');
await writeBatch.commit();
writeBatch = firebaseAdmin.firestore().batch();
}
}
if (i > 0) {
console.log('Firebase batch operation completed. Doing final committing of batch operation.');
await writeBatch.commit();
} else {
console.log('Firebase batch operation completed.');
}
}
copyCollection('customers', 'customers_backup').then(() => console.log('copy complete')).catch(error => console.log('copy failed. ' + error));
Currently, no. Looping through each document using Cloud Functions and then setting a new document to a different collection with the specified data is the only way to do this. Perhaps this would make a good feature request.
How many documents are we talking about? For something like 10,000 it should only take a few minutes, tops.
This is the method i use to copy data to another collection, I used it to shift data (like sells or something) from an active collection to a 'sells feed' or 'sells history' collection.
At the top i reference the documents, at the bottom is the quite compact code.
You can simply add a for loop on top for more than 1 operation.
Hope it helps somebody :)
DocumentReference copyFrom = FirebaseFirestore.instance.collection('curSells').doc('0001');
DocumentReference copyTo = FirebaseFirestore.instance.collection('sellFeed').doc('0001');
copyFrom.get().then((value) => {
copyTo.set(value.data())
});
There is no fast way at the moment. I recommend you rewrite your code like this though:
import { firestore } from "firebase-admin";
async function copyCollection() {
const products = await firestore().collection("products").get();
products.forEach(async (doc)=> {
await firestore().collection(uid).doc(doc.get('barcode')).set(doc.data());
})
}
Related
Transactions and batched writes can be used to write multiple documents by means of an atomic operation.
Documentation says that Using the Cloud Firestore client libraries, you can group multiple operations into a single transaction.
I cannot understand what is the meaning of client libraries here and if it's correct to use Transactions and batched writes within a Cloud Function.
Example given: suppose in the database I have 3 elements (which doc IDs are A, B, C). Now I need to insert 3 more elements (which doc IDs are C, D, E). The Cloud Function should add just the latest ones and send a Push Notification to the user telling him that 2 new documents are available.
The doc ID could be the same but since I need to calculate how many documents are new (the ones that will be inserted) I need a way to read the doc ID first and check for its existence. Hence, I'm wondering if Transactions fit Cloud Functions or not.
Also, each transaction or batch of writes can write to a maximum of 500 documents. Is there any other way to overcome this limit within a Cloud Function?
Firestore Transaction behaviour is different between the Clients SDKs (JS SDK, iOS SDK, Android SDK , ...) and the Admin SDK (a set of server libraries), which is the SDK we use in a Cloud Function. More explanations on the differences here in the documentation.
Because of the type of data contention used in the Admin SDK you can, with the getAll() method, retrieve multiple documents from Firestore and hold a pessimistic lock on all returned documents.
So this is exactly the method you need to call in your transaction: you use getAll() for fetching documents C, D & E and you detect that only C is existing so you know that you need to only add D and E.
Concretely, it could be something along the following lines:
const db = admin.firestore();
exports.lorenzoFunction = functions
.region('europe-west1')
.firestore
.document('tempo/{docId}') //Just a way to trigger the test Cloud Function!!
.onCreate(async (snap, context) => {
const c = db.doc('coltest/C');
const d = db.doc('coltest/D');
const e = db.doc('coltest/E');
const docRefsArray = [c, d, e]
return db.runTransaction(transaction => {
return transaction.getAll(...docRefsArray).then(snapsArray => {
let counter = 0;
snapsArray.forEach(snap => {
if (!snap.exists) {
counter++;
transaction.set(snap.ref, { foo: "bar" });
} else {
console.log(snap.id + " exists")
}
});
console.log(counter);
return;
});
});
});
To test it: Create one of the C, D or E doc in the coltest collection, then create a doc in the tempo collection (Just a simple way to trigger this test Cloud Function): the CF is triggered. Then look at the coltest collection: the two missing docs were created; and look a the CF log: counter = 2.
Also, each transaction or batch of writes can write to a maximum of
500 documents. Is there any other way to overcome this limit within a
Cloud Function?
AFAIK the answer is no.
There used to also be a one second delay required as well between 500 record chunks. I wrote this a couple of years ago. The script below reads the CSV file line by line, creating and setting a new batch object for each line. A counter creates a new batch write per 500 objects and finally asynch/await is used to rate limit the writes to 1 per second. Last, we notify the user of the write progress with console logging. I had published an article on this here >> https://hightekk.com/articles/firebase-admin-sdk-bulk-import
NOTE: In my case I am reading a huge flat text file (a manufacturers part number catalog) for import. You can use this as a working template though and modify to suit your data source. Also, you may need to increase the memory allocated to node for this to run:
node --max_old_space_size=8000 app.js
The script looks like:
var admin = require("firebase-admin");
var serviceAccount = require("./your-firebase-project-service-account-key.json");
var fs = require('fs');
var csvFile = "./my-huge-file.csv"
var parse = require('csv-parse');
require('should');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://your-project.firebaseio.com"
});
var firestore = admin.firestore();
var thisRef;
var obj = {};
var counter = 0;
var commitCounter = 0;
var batches = [];
batches[commitCounter] = firestore.batch();
fs.createReadStream(csvFile).pipe(
parse({delimiter: '|',relax_column_count:true,quote: ''})
).on('data', function(csvrow) {
if(counter <= 498){
if(csvrow[1]){
obj.family = csvrow[1];
}
if(csvrow[2]){
obj.series = csvrow[2];
}
if(csvrow[3]){
obj.sku = csvrow[3];
}
if(csvrow[4]){
obj.description = csvrow[4];
}
if(csvrow[6]){
obj.price = csvrow[6];
}
thisRef = firestore.collection("your-collection-name").doc();
batches[commitCounter].set(thisRef, obj);
counter = counter + 1;
} else {
counter = 0;
commitCounter = commitCounter + 1;
batches[commitCounter] = firestore.batch();
}
}).on('end',function() {
writeToDb(batches);
});
function oneSecond() {
return new Promise(resolve => {
setTimeout(() => {
resolve('resolved');
}, 1010);
});
}
async function writeToDb(arr) {
console.log("beginning write");
for (var i = 0; i < arr.length; i++) {
await oneSecond();
arr[i].commit().then(function () {
console.log("wrote batch " + i);
});
}
console.log("done.");
}
I've been struggling for a while to find out how exactly to use the getAll function on the first param of the callback function on runTransation function.
The doc firebase doc only shows how to use the get function to retrieve a single doc, but I want to retrieve multiple docs based on multiple where statements from a collection.
the question is how to use getAll function bellow?
export const match = functions.firestore
.document('waitingList/{userId}').onCreate(async (snapshot, context) => {
app.firestore().runTransaction(async transaction => {
transaction.getAll( //?); // ???
});
});
UPDATE 1 | 23/4/2022
I figured out a way (but I'm not fully satisfied with it because of duplication in reading from firestore).
The solution is as follows
app.firestore().runTransaction(async transaction => {
const docRefs: FirebaseFirestore.DocumentReference<any>[] = [];
(await firestore
.collection('collectionName')
.limit(100)
.get())
.forEach((doc) => {
docRefs.push(doc.ref);
}); //This is reading the Database for 100 docs
const users = await transaction.getAll(...docRefs); // This also is reading the database for the same 100 docs (But in a transaction context)
});
if someone knows how to read the docs only once in the transaction please do provide the solution.
Something like that:
transaction.getAll(...docRefs).then((docs) => {
docs.forEach((doc) => {/*...*/}
}
As #Abobker already stated, I agree that for now this will be the best way to retrieve multiple docs based on multiple where statements from a collection:
app.firestore().runTransaction(async transaction => {
const docRefs: FirebaseFirestore.DocumentReference<any>[] = [];
(await firestore
.collection('collectionName')
.limit(100)
.get())
.forEach((doc) => {
docRefs.push(doc.ref);
}); //This is reading the Database for 100 docs
const users = await transaction.getAll(...docRefs); // This also is reading the database for the same 100 docs (But in a transaction context)
});
Although is not the most efficient solution, It does address the problem.
I'm using Firebase as backend to my iOS app and can't figure out how to construct a batch write through their Cloud Functions.
I have two collections in my Firestore, drinks and customers. Each new drink and each new customer is assigned a userId property that corresponds to the uid of the currently logged in user. This userId is used with a query to the Firestore to fetch only the drinks and customers connected to the logged in user, like so: Firestore.firestore().collection("customers").whereField("userId", isEqualTo: Auth.auth().currentUser.uid)
Users are able to log in anonymously and also subscribe while anonymous. The problem is if they log out there's no way to log back in to the same anonymous uid. The uid is also stored as an appUserID with the RevenueCat SDK so I can still access it, but since I can't log the user back in to their anonymous account using the uid the only way to help a user access their data in case of a restoring of purchases is to update the userId field of their data from the old uid to the new uid. This is where the need for a batch write comes in.
I'm relatively new to programming in general but I'm super fresh when it comes to Cloud Functions, JavaScript and Node.js. I dove around the web though and thought I found a solution where I make a callable Cloud Function and send both old and new userID with the data object, query the collections for documents with the old userID and update their userId fields to the new. Unfortunately it's not working and I can't figure out why.
Here's what my code looks like:
// Cloud Function
exports.transferData = functions.https.onCall((data, context) => {
const firestore = admin.firestore();
const customerQuery = firestore.collection('customers').where('userId', '==', `${data.oldUser}`);
const drinkQuery = firestore.collection('drinks').where('userId', '==', `${data.oldUser}`);
const customerSnapshot = customerQuery.get();
const drinkSnapshot = drinkQuery.get();
const batch = firestore.batch();
for (const documentSnapshot of customerSnapshot.docs) {
batch.update(documentSnapshot.ref, { 'userId': `${data.newUser}` });
};
for (const documentSnapshot of drinkSnapshot.docs) {
batch.update(documentSnapshot.ref, { 'userId': `${data.newUser}` });
};
return batch.commit();
});
// Call from app
func transferData(from oldUser: String, to newUser: String) {
let functions = Functions.functions()
functions.httpsCallable("transferData").call(["oldUser": oldUser, "newUser": newUser]) { _, error in
if let error = error as NSError? {
if error.domain == FunctionsErrorDomain {
let code = FunctionsErrorCode(rawValue: error.code)
let message = error.localizedDescription
let details = error.userInfo[FunctionsErrorDetailsKey]
print(code)
print(message)
print(details)
}
}
}
}
This is the error message from the Cloud Functions log:
Unhandled error TypeError: customerSnapshot.docs is not iterable
at /workspace/index.js:22:51
at fixedLen (/workspace/node_modules/firebase-functions/lib/providers/https.js:66:41)
at /workspace/node_modules/firebase-functions/lib/common/providers/https.js:385:32
at processTicksAndRejections (internal/process/task_queues.js:95:5)
From what I understand customerSnapshot is something called a Promise which I'm guessing is why I can't iterate over it. By now I'm in way too deep for my sparse knowledge and don't know how to handle these Promises returned by the queries.
I guess I could just force users to create a login before they subscribe but that feels like a cowards way out now that I've come this far. I'd rather have both options available and make a decision instead of going down a forced path. Plus, I'll learn some more JavaScript if I figure this out!
Any and all help is greatly appreciated!
EDIT:
Solution:
// Cloud Function
exports.transferData = functions.https.onCall(async(data, context) => {
const firestore = admin.firestore();
const customerQuery = firestore.collection('customers').where('userId', '==', `${data.oldUser}`);
const drinkQuery = firestore.collection('drinks').where('userId', '==', `${data.oldUser}`);
const customerSnapshot = await customerQuery.get();
const drinkSnapshot = await drinkQuery.get();
const batch = firestore.batch();
for (const documentSnapshot of customerSnapshot.docs.concat(drinkSnapshot.docs)) {
batch.update(documentSnapshot.ref, { 'userId': `${data.newUser}` });
};
return batch.commit();
});
As you already guessed, the call customerQuery.get() returns a promise.
In order to understand what you need, you should first get familiar with the concept of promises here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
For your use case, you will probably end up with either using the then callback:
customerQuery.get().then((result) => {
// now you can access the result
}
or by making the method call synchronous, by using the await statement:
const result = await customerQuery.get()
// now you can access the result
I'm new to the "async/await" aspect of JS and I'm trying to learn how it works.
The error I'm getting is Line 10 of the following code. I have created a firestore database and am trying to listen for and get a certain document from the Collection 'rooms'. I am trying to get the data from the doc 'joiner' and use that data to update the innerHTML of other elements.
// References and Variables
const db = firebase.firestore();
const roomRef = await db.collection('rooms');
const remoteNameDOM = document.getElementById('remoteName');
const chatNameDOM = document.getElementById('title');
let remoteUser;
// Snapshot Listener
roomRef.onSnapshot(snapshot => {
snapshot.docChanges().forEach(async change => {
if (roomId != null){
if (role == "creator"){
const usersInfo = await roomRef.doc(roomId).collection('userInfo');
usersInfo.doc('joiner').get().then(async (doc) => {
remoteUser = await doc.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
}
}
})
})
})
However, I am getting the error:
Uncaught (in promise) TypeError: Cannot read property 'joinerName' of undefined
Similarly if I change the lines 10-12 to:
remoteUser = await doc.data();
remoteNameDOM.innerHTML = `${remoteUser.joinerName} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser.joinerName}`;
I get the same error.
My current understanding is that await will wait for the line/function to finish before moving forward, and so remoteUser shouldn't be null before trying to call it. I will mention that sometimes the code works fine, and the DOM elements are updated and there are no console errors.
My questions: Am I thinking about async/await calls incorrectly? Is this not how I should be getting documents from Firestore? And most importantly, why does it seem to work only sometimes?
Edit: Here are screenshots of the Firestore database as requested by #Dharmaraj. I appreciate the advice.
You are mixing the use of async/await and then(), which is not recommended. I propose below a solution based on Promise.all() which helps understanding the different arrays that are involved in the code. You can adapt it with async/await and a for-of loop as #Dharmaraj proposed.
roomRef.onSnapshot((snapshot) => {
// snapshot.docChanges() Returns an array of the documents changes since the last snapshot.
// you may check the type of the change. I guess you maybe don’t want to treat deletions
const promises = [];
snapshot.docChanges().forEach(docChange => {
// No need to use a roomId, you get the doc via docChange.doc
// see https://firebase.google.com/docs/reference/js/firebase.firestore.DocumentChange
if (role == "creator") { // It is not clear from where you get the value of role...
const joinerRef = docChange.doc.collection('userInfo').doc('joiner');
promises.push(joinerRef.get());
}
});
Promise.all(promises)
.then(docSnapshotArray => {
// docSnapshotArray is an Array of all the docSnapshots
// corresponding to all the joiner docs corresponding to all
// the rooms that changed when the listener was triggered
docSnapshotArray.forEach(docSnapshot => {
remoteUser = docSnapshot.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
});
});
However, what is not clear to me is how you differentiate the different elements of the "first" snapshot (i.e. roomRef.onSnapshot((snapshot) => {...}))). If several rooms change, the snapshot.docChanges() Array will contain several changes and, at the end, you will overwrite the remoteNameDOM and chatNameDOM elements in the last loop.
Or you know upfront that this "first" snapshot will ALWAYS contain a single doc (because of the architecture of your app) and then you could simplify the code by just treating the first and unique element as follows:
roomRef.onSnapshot((snapshot) => {
const roomDoc = snapshot.docChanges()[0];
// ...
});
There are few mistakes in this:
db.collection() does not return a promise and hence await is not necessary there
forEach ignores promises so you can't actually use await inside of forEach. for-of is preferred in that case.
Please try the following code:
const db = firebase.firestore();
const roomRef = db.collection('rooms');
const remoteNameDOM = document.getElementById('remoteName');
const chatNameDOM = document.getElementById('title');
let remoteUser;
// Snapshot Listener
roomRef.onSnapshot(async (snapshot) => {
for (const change of snapshot.docChanges()) {
if (roomId != null){
if (role == "creator"){
const usersInfo = roomRef.doc(roomId).collection('userInfo').doc("joiner");
usersInfo.doc('joiner').get().then(async (doc) => {
remoteUser = doc.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
}
}
}
})
I am using Firebase in my React Native app. I have a collection of users and in that, I have documents and the ids of the documents are custom (using users' phone numbers as ids) and in that, I have blocked users array field in which I will add the users who are blocked by the user. I want to show the list where user can only see the users who are not blocked.
I am getting all the users list and I want to filter them and fetch only the people not blocked by the user.
var getUsersList = async() => {
const findUser = await firestore().collection('users').get();
if (findUser.docs[0] != undefined && findUser.docs[0]._exists){
setUserList(findUser.docs)
}
}
I understand that your firestore collection is similar to this one:
If that is the case, then I have structured your requirement in the three functions below:
1.readBlockedNumbers to return the array of blocked numbers that the user has.
2.show_nonBlockedUsersthat receives the array of the blocked numbers from the previous method and displays the users who are not in this array.
3. test to coordinate the execution of the above two methods.
const admin = require('firebase-admin');
const serviceAccount = require('/home/keys.json');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount)
});
const db = admin.firestore();
async function readBlockedNumbers(docId){
const userRef = db.collection('users').doc(docId);
const doc = await userRef.get();
if (!doc.exists) {
console.log('No such document!');
return [];
}
else
{
return doc.data()['blocked_numbers'];
}
async function show_nonBlockedUsers(blocked_users){
console.log('Array length:', blocked_users.length);
if(blocked_users.length == 0)
return;
const userRef = db.collection('users');
const snapshot = await userRef
.where(admin.firestore.FieldPath.documentId(), 'not-in', blocked_users)
.get();
if (snapshot.empty) {
console.log('No matching documents.');
return;
}
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
}
async function test(){
const docId = '202-555-0146';
//'202-555-0102'
const blocked_users = await readBlockedNumbers(docId);
await show_nonBlockedUsers(blocked_users);
}
Important here is how to use the not-in operator and the method admin.firestore.FieldPath.documentId().
I have found the not-in operator here and the method firebase.firestore.FieldPath.documentId() referenced in this other stackoverflow question since the id cannot be passed like the other document fields in the where clause.
Please refer to the firebase documentation as well for the limitations of the not-in operator.
I hope you find this useful.