Transactions and batched writes can be used to write multiple documents by means of an atomic operation.
Documentation says that Using the Cloud Firestore client libraries, you can group multiple operations into a single transaction.
I cannot understand what is the meaning of client libraries here and if it's correct to use Transactions and batched writes within a Cloud Function.
Example given: suppose in the database I have 3 elements (which doc IDs are A, B, C). Now I need to insert 3 more elements (which doc IDs are C, D, E). The Cloud Function should add just the latest ones and send a Push Notification to the user telling him that 2 new documents are available.
The doc ID could be the same but since I need to calculate how many documents are new (the ones that will be inserted) I need a way to read the doc ID first and check for its existence. Hence, I'm wondering if Transactions fit Cloud Functions or not.
Also, each transaction or batch of writes can write to a maximum of 500 documents. Is there any other way to overcome this limit within a Cloud Function?
Firestore Transaction behaviour is different between the Clients SDKs (JS SDK, iOS SDK, Android SDK , ...) and the Admin SDK (a set of server libraries), which is the SDK we use in a Cloud Function. More explanations on the differences here in the documentation.
Because of the type of data contention used in the Admin SDK you can, with the getAll() method, retrieve multiple documents from Firestore and hold a pessimistic lock on all returned documents.
So this is exactly the method you need to call in your transaction: you use getAll() for fetching documents C, D & E and you detect that only C is existing so you know that you need to only add D and E.
Concretely, it could be something along the following lines:
const db = admin.firestore();
exports.lorenzoFunction = functions
.region('europe-west1')
.firestore
.document('tempo/{docId}') //Just a way to trigger the test Cloud Function!!
.onCreate(async (snap, context) => {
const c = db.doc('coltest/C');
const d = db.doc('coltest/D');
const e = db.doc('coltest/E');
const docRefsArray = [c, d, e]
return db.runTransaction(transaction => {
return transaction.getAll(...docRefsArray).then(snapsArray => {
let counter = 0;
snapsArray.forEach(snap => {
if (!snap.exists) {
counter++;
transaction.set(snap.ref, { foo: "bar" });
} else {
console.log(snap.id + " exists")
}
});
console.log(counter);
return;
});
});
});
To test it: Create one of the C, D or E doc in the coltest collection, then create a doc in the tempo collection (Just a simple way to trigger this test Cloud Function): the CF is triggered. Then look at the coltest collection: the two missing docs were created; and look a the CF log: counter = 2.
Also, each transaction or batch of writes can write to a maximum of
500 documents. Is there any other way to overcome this limit within a
Cloud Function?
AFAIK the answer is no.
There used to also be a one second delay required as well between 500 record chunks. I wrote this a couple of years ago. The script below reads the CSV file line by line, creating and setting a new batch object for each line. A counter creates a new batch write per 500 objects and finally asynch/await is used to rate limit the writes to 1 per second. Last, we notify the user of the write progress with console logging. I had published an article on this here >> https://hightekk.com/articles/firebase-admin-sdk-bulk-import
NOTE: In my case I am reading a huge flat text file (a manufacturers part number catalog) for import. You can use this as a working template though and modify to suit your data source. Also, you may need to increase the memory allocated to node for this to run:
node --max_old_space_size=8000 app.js
The script looks like:
var admin = require("firebase-admin");
var serviceAccount = require("./your-firebase-project-service-account-key.json");
var fs = require('fs');
var csvFile = "./my-huge-file.csv"
var parse = require('csv-parse');
require('should');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://your-project.firebaseio.com"
});
var firestore = admin.firestore();
var thisRef;
var obj = {};
var counter = 0;
var commitCounter = 0;
var batches = [];
batches[commitCounter] = firestore.batch();
fs.createReadStream(csvFile).pipe(
parse({delimiter: '|',relax_column_count:true,quote: ''})
).on('data', function(csvrow) {
if(counter <= 498){
if(csvrow[1]){
obj.family = csvrow[1];
}
if(csvrow[2]){
obj.series = csvrow[2];
}
if(csvrow[3]){
obj.sku = csvrow[3];
}
if(csvrow[4]){
obj.description = csvrow[4];
}
if(csvrow[6]){
obj.price = csvrow[6];
}
thisRef = firestore.collection("your-collection-name").doc();
batches[commitCounter].set(thisRef, obj);
counter = counter + 1;
} else {
counter = 0;
commitCounter = commitCounter + 1;
batches[commitCounter] = firestore.batch();
}
}).on('end',function() {
writeToDb(batches);
});
function oneSecond() {
return new Promise(resolve => {
setTimeout(() => {
resolve('resolved');
}, 1010);
});
}
async function writeToDb(arr) {
console.log("beginning write");
for (var i = 0; i < arr.length; i++) {
await oneSecond();
arr[i].commit().then(function () {
console.log("wrote batch " + i);
});
}
console.log("done.");
}
Related
I'm trying to make some query from different databases, which I cannot merge them to one forAll, the following query probably fails due to async call which I cannot manage to solve.
var someImageIds = ["111111111111111111"]
use "databaseA"
var transactions = db.transactions.find({"data.transactionImaginaryId": {$in: someImageIds}}) // uses index
use "databaseB"
transactions.forEach(transaction => {
var report = db.reports.find({"metadata.companyId" : parseInt(transaction.data.companyId) , "metadata.originReportId": transaction.data.reportId}).project({}) // uses index
var expenses = db.expenses.find({"metadata.reportId": report._id}) // uses index
var assets = db.assets.find({"_id": report.assets[0].imaginaryId}) // uses index
print(`report with status: ${"report.reportFlow.value"}, ${expenses.count()} expenses, ${assets.count()} assets for ${transaction.data.matchType} transaction _id: ${transaction._id.valueOf()}`)
})
The problem is that
var report = db.reports.find({"metadata.companyId" : parseInt(transaction.data.companyId) , "metadata.originReportId": transaction.data.reportId}).project({})
returns value of undefined and I cannot continue with the query since the next line is using this line data.
Any ideas on how to solve that?
I'm using NoSqlBooster v6.2.8, mongo4, and written in NoSqlBooster console.
Thanks!
Thanks to #Jeremy Thille I managed to write the following WORKING code:
var someImageIds = ["111111111111111111"]
use "databaseA"
var transactions = db.transactions.find({"data.transactionImaginaryId": {$in: someImageIds}}) // uses index
use "databaseB"
transactions.forEach((transaction)=> {
const report = await(db.reports.find({ "metadata.companyId": parseInt(transaction.data.companyId), "metadata.originReportId": transaction.data.reportId }).toArray()) // uses index
const expenses = await(db.expenses.find({ "metadata.reportId": report[0]._id }).toArray()) // uses index
const assets = await(db.assets.find({ "_id": report[0].assets[0].imaginaryId }).toArray()) // uses index
print(`report with status: ${report[0].reportFlow.value}, ${expenses.length} expenses, ${assets.length} assets for ${transaction.data.matchType} transaction _id: ${transaction._id.valueOf()}`)
});
Unfortunately, databases (and HTTP requests, and many other things) are not instantaneous. They need some time to perform an operation. So you need to await them, which can't be done in a .forEach() loop, but can be in a for loop :
const someFunctionName = async () => { // needs async
for (let transaction of transactions) {
const report = await db.reports.find({ "metadata.companyId": parseInt(transaction.data.companyId), "metadata.originReportId": transaction.data.reportId }).project({}) // uses index
const expenses = await db.expenses.find({ "metadata.reportId": report._id }) // uses index
const assets = await db.assets.find({ "_id": report.assets[0].imaginaryId }) // uses index
print(`report with status: ${"report.reportFlow.value"}, ${expenses.count()} expenses, ${assets.count()} assets for ${transaction.data.matchType} transaction _id: ${transaction._id.valueOf()}`)
}
}
The Problem:
I have been unable to use Firebase (Google) Cloud Functions to collect and utilize device tokens for the cloud messaging feature.
Context:
I am a self-taught android-Java developer and have no JavaScript experience. Despite that, I believe I have code that should work and am not sure what the problem is. To my understanding, it could be one of three things:
Somehow my Firebase Realtime Database references are being called incorrectly and I am not retrieving data as expected.
I may need to use Promises to wait for all calls to be made before proceeding, however I don't really understand how I would incorporate that into the code I have.
I may be using multiple return statements incorrectly (which I am also fuzzy on).
My error message on the Firebase Realtime Database console is as follows:
#firebase/database: FIREBASE WARNING: Exception was thrown by user callback. Error: Registration token(s) provided to sendToDevice() must be a non-empty string or a non-empty array.
at FirebaseMessagingError.FirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:42:28)
at FirebaseMessagingError.PrefixedFirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:88:28)
at new FirebaseMessagingError (/srv/node_modules/firebase-admin/lib/utils/error.js:254:16)
at Messaging.validateRegistrationTokensType (/srv/node_modules/firebase-admin/lib/messaging/messaging.js:729:19)
at Messaging.sendToDevice (/srv/node_modules/firebase-admin/lib/messaging/messaging.js:328:14)
at admin.database.ref.once.snapshot (/srv/index.js:84:12)
at onceCallback (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:4933:51)
at /srv/node_modules/#firebase/database/dist/index.node.cjs.js:4549:22
at exceptionGuard (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:698:9)
at EventList.raise (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:9684:17)
The above indicates I am not retrieving data either at all or by the time the return is called. My JavaScript function code is:
'use strict';
const admin = require('firebase-admin');
admin.initializeApp();
exports.pushNotification = functions.database.ref('/Chat Messages/{chatId}/{pushID}').onCreate((snapshot, context) => {
const valueObject = snapshot.after.val();
return admin.database().ref(`/Chat Basics/${valueObject.chatKey}/Chat Users`).once('value', statusSnapshot => {
var index = 0;
var totalkeys = statusSnapshot.numChildren();
var msgIDs = [];
statusSnapshot.forEach(msg=>{
msgIDs.push(msg.key.toString());
if(index === totalkeys - 1){
const payload = {
notification : {
title: valueObject.userName,
body: valueObject.message,
sound: "default"
}
}
sendNotificationPayload(valueObject.uid, payload);
}
index++;
});
});
});
function sendNotificationPayload(uid, payload){
admin.database()
.ref(`/User Token Data/${uid}`)
.once('value', snapshot=> {
var tokens = [];
//if(!snapshot.exists())return;
snapshot.forEach(item =>{
tokens.push(item.val())
});
admin.messaging()
.sendToDevice(tokens, payload)
.then(res => {
return console.log('Notification sent')
})
.catch(err => {
return console.log('Error in sending notification = '+err)
});
});
}
This code is mostly inspired by what was said to be a working example here from another Stack Overflow question here. I have successfully tested sending a notification to a single device by manually copying a device token into my function, so the function does run to completion. My Java code seems to be irrelevant to the problem, so I have not added it (please ask in the comments if you would like it added for further context).
What I Have Tried:
I have tried implementing promises into my code, but I don't think I was doing it properly. My main reference for this was here. I have also looked at the documentation for literally everything related to this topic, however my knowledge of JS is not sufficient to really apply barebones examples to my code.
My Firebase Realtime Database Nodes:
#1: Loop through chat members to collect user IDs:
"Chat Basics" : {
"1607801501690_TQY41wIfArhHDxEisyupZxwyHya2" : {
"Chat Users" : {
"JXrclZuu1aOwEpCe6KW8vSDea9h2" : true,
"TQY41wIfArhHDxEisyupZxwyHya2" : true
},
#2: Collect user tokens from collected IDs (ignore that tokens are matching):
"User Token Data" : {
"JXrclZuu1aOwEpCe6KW8vSDea9h2" : "duDR3KH3i3I:APA91bH_LCeslZlqL8akYw-LrM9Dv__nx4nU1TquCS0j6bGF1tlIARcheREuNdX1FheC92eelatBC8LO4t6gt8liRdFHV-NDuNLa13oHYxKgl3JBPPlrMo5rB5XhH7viTo4vfYOMftRi",
"TQY41wIfArhHDxEisyupZxwyHya2" : "duDR3KH3i3I:APA91bH_LCeslZlqL8akYw-LrM9Dv__nx4nU1TquCS0j6bGF1tlIARcheREuNdX1FheC92eelatBC8LO4t6gt8liRdFHV-NDuNLa13oHYxKgl3JBPPlrMo5rB5XhH7viTo4vfYOMftRi"
}
Conclusion:
Concrete examples would be much appreciated, especially since I am crunching right now. Thanks for your time and help!
Update:
After some more testing, it looks like the problem is definitely due to my lack of understanding of promises in two areas. Firstly, only one user is collected before the final return is called. Secondly, the final return is called before the 2nd forEach() loop can store snapshot data to an array.
For this code then, how may I modify (or rebuild) it so that it collects all keys before proceeding to retrieve token data from all keys - ultimately before returning the notification?
Just as with every question I post, I managed to figure out how to do it (tentatively) a few hours later. Below is a full example of how to send a notification to chat users based on a message sent (although it does not yet exclude the sender) to a given chat. The order of operations are as such:
User message is saved and triggers event. Relevant data the message contains are:
username, chat key, message
These are retrieved, with (username + message) as the (title + body) of the
notification respectively, and the chat key is used for user id reference.
Loop through chat user keys + collect.
Loop through array of chat user keys to collect array of device tokens.
Send notification when complete.
The code:
//Use firebase functions:log to see log
exports.pushNotification = functions.database.ref('/Chat Messages/{chatId}/{pushId}').onWrite((change, context) => {
const valueObject = change.after.val();
return admin.database().ref(`/Chat Basics/${valueObject.chatKey}/Chat Users`).once('value', statusSnapshot => {
var index = 0;
var totalkeys = statusSnapshot.numChildren();
var msgIDs = [];
statusSnapshot.forEach(msg=>{
msgIDs.push(msg.key.toString());
if(index === totalkeys - 1){
const payload = {
notification : {
title: valueObject.userName,
body: valueObject.message,
sound: "default"
}
}
let promises = [];
var tokens = [];
for(let i=0; i < msgIDs.length; i++){
let userId = msgIDs[i];
let promise = admin.database().ref(`/User Token Data/${userId}`).once('value', snapshot=> {
tokens.push(snapshot.val());
})
promises.push(promise);
}
return Promise.all(promises).then(() => {
return admin.messaging().sendToDevice(tokens, payload);
});
}
index++;
return false;
});
});
});
In my model.js (using mongoose) , I am initially creating 40 objects in model.js which are to be used in the entire program. No other function in any file creates more objects but only updates the existing ones.
My model.js
var TicketSchema = mongoose.model('Tickets', TicketSchema);
for(let i = 1;i<=40;i++)
{
var new_ticket = new TicketSchema({ticket_number:i});
new_ticket.save(function(err, ticket) {
});
}
Problem is I noticed there were much more objects than 40 after some time. I wanted to know if model.js runs more than once during execution or is it just due to repeated calling of npm run start and then closing the server?
Also is there way better way of creating objects initially which are to be used for the entire program?
It will create new 40 documents every time you start the server. You can use this function to avoid creating if the records already exist by checking count.
const TicketModel = mongoose.model('Tickets', TicketSchema);
const insertTicketNumber = async () => {
try {
const count = await TicketModel.countDocuments({});
if (count) return;
await TicketModel.create(
[...Array(40).keys()]
.map(i => i + 1)
.map(number => ({ ticket_number: number }))
);
} catch (error) {
console.log(error.message);
}
};
I'd like to make a copy of a collection in Firestore upon an event using Cloud Functions
I already have this code that iterates over the collection and copies each document
const firestore = admin.firestore()
firestore.collection("products").get().then(query => {
query.forEach(function(doc) {
var promise = firestore.collection(uid).doc(doc.data().barcode).set(doc.data());
});
});
is there a shorter version? to just copy the whole collection at once?
I wrote a small nodejs snippet for this.
const firebaseAdmin = require('firebase-admin');
const serviceAccount = '../../firebase-service-account-key.json';
const firebaseUrl = 'https://my-app.firebaseio.com';
firebaseAdmin.initializeApp({
credential: firebaseAdmin.credential.cert(require(serviceAccount)),
databaseURL: firebaseUrl
});
const firestore = firebaseAdmin.firestore();
async function copyCollection(srcCollectionName, destCollectionName) {
const documents = await firestore.collection(srcCollectionName).get();
let writeBatch = firebaseAdmin.firestore().batch();
const destCollection = firestore.collection(destCollectionName);
let i = 0;
for (const doc of documents.docs) {
writeBatch.set(destCollection.doc(doc.id), doc.data());
i++;
if (i > 400) { // write batch only allows maximum 500 writes per batch
i = 0;
console.log('Intermediate committing of batch operation');
await writeBatch.commit();
writeBatch = firebaseAdmin.firestore().batch();
}
}
if (i > 0) {
console.log('Firebase batch operation completed. Doing final committing of batch operation.');
await writeBatch.commit();
} else {
console.log('Firebase batch operation completed.');
}
}
copyCollection('customers', 'customers_backup').then(() => console.log('copy complete')).catch(error => console.log('copy failed. ' + error));
Currently, no. Looping through each document using Cloud Functions and then setting a new document to a different collection with the specified data is the only way to do this. Perhaps this would make a good feature request.
How many documents are we talking about? For something like 10,000 it should only take a few minutes, tops.
This is the method i use to copy data to another collection, I used it to shift data (like sells or something) from an active collection to a 'sells feed' or 'sells history' collection.
At the top i reference the documents, at the bottom is the quite compact code.
You can simply add a for loop on top for more than 1 operation.
Hope it helps somebody :)
DocumentReference copyFrom = FirebaseFirestore.instance.collection('curSells').doc('0001');
DocumentReference copyTo = FirebaseFirestore.instance.collection('sellFeed').doc('0001');
copyFrom.get().then((value) => {
copyTo.set(value.data())
});
There is no fast way at the moment. I recommend you rewrite your code like this though:
import { firestore } from "firebase-admin";
async function copyCollection() {
const products = await firestore().collection("products").get();
products.forEach(async (doc)=> {
await firestore().collection(uid).doc(doc.get('barcode')).set(doc.data());
})
}
For context: I have a cron-job.org that fires an https function in my firebase project.
In this function, I have to go through all docs inside a collection and update a counter (each doc might have a different counter value). If the counter reaches a limit, I'll update another collection (independent from the first one), and delete the doc entry that reached the limit. If the counter is not beyond the limit, I simply update the doc entry with the updated counter value.
I tried adapting examples from the documentation, tried using transactions, batch, but I'm not sure how to proceed. According to transactions' description, that's the way to go, but examples only show how to edit a single doc.
This is what I have (tried adapting a realtime db sample):
function updateCounter() {
var ref = db.collection('my_collection_of_counters');
return ref.get().then(snapshot => {
const updates = {};
snapshot.forEach(child => {
var docData = child.data();
var newCounter = docData.counter+1;
if (newCounter == 10) {
// TO-DO: add to stock
updates[child.key] = null;
} else {
docData.counter = newCounter;
updates[child.key] = docData;
}
});
// execute all updates in one go and return the result to end the function
return ref.update(updates);
});
}
It doesn't work, collections don't have an update method. What is the best approach to updating each doc in a collection? One-by-one? Transaction? Is there an example?
PS: updateCounter is a function being called by the https trigger. Cron+trigger is working fine.
EDIT
When an item reaches the threshold, I want to update another collection, independent from the counter one. Is nested transactions a good solution?
Modified code:
function updateCounter() {
var ref = db.collection('my_collection_of_counters');
var transaction = db.runTransaction(t => {
return t.get(ref)
.then(snapshot => {
let docs = snapshot.docs;
for (let doc of docs) {
var item = doc.data();
var newCounter = item.counter + 1;
if (newCounter == 10) {
console.log("Update my_stock");
// ADD item.quantity to stock collection
}else{
t.update(doc.ref, {counter: newCounter});
}
}
});
})
.then(result => {
console.log('Transaction success');
})
.catch(err => {
console.log('Transaction failure:', err);
});
}
As you already noted yourself, you'll want to do this in a transaction to ensure that you can update the current counter value in a single operation. You can also create the new document, and delete the existing one, in that same transaction once your counter reaches its threshold. I don't see any benefit of doing this for all documents in a single transaction, since the operation on each doc seems unrelated to the others.
In a Firestore transaction, you perform the operations on a Transaction object as shown in the documentation. In your case you'd:
Get the current document with transaction.get().
Get the counter from the document.
Increment the counter.
If the new value is below your threshold:
Call transaction.update() to write the new counter value into the database
If the new value if above your threshold:
Call transaction.create on the new collection to create the document there.
Call transaction.delete on the existing document, to delete it.
For more, I recommend scanning the reference documentation for the Transaction class.