We Keep Coding

Spawning NodeJS worker thread inside of async function doesn't respect promise resolution

I've got an async function that launches a NodeJS worker thread like so:
encode : async (config) => {
if (isMainThread) {
const encode_worker = new Worker(`./service-encode.js`, { workerData: config });
encode_worker.on('message', (transcode_data) => {
log.info("%o", transcode_data);
return transcode_data;
});
encode_worker.on('error', (err) => { log.error(err)});
encode_worker.on('exit', (code) => {
if (code !== 0)
throw new Error(`Encoding stopped with exit code [ ${code} ]`);
console.log("* * * EXITED ENCODER WORKER * * *")
});
}
},
In the serivce-encode.js file I've got the following code which uses async functions. Note that I am using postMessage to signal that it is done.
var transcoder = require('./transcoder');
const {Worker, isMainThread, parentPort, workerData} = require('worker_threads');
console.log("* * * STARTING ENCODE THREAD * * *\n");
console.log(workerData);
transcoder.run(workerData)
.then((results) => {
transcode_data = results;
parentPort.postMessage(transcode_data);
})
.catch(err => { throw err });
Then, I use the following example code but the code in the 'message' event from above fires off immediately. That is, it doesn't seem to wait until it's done:
encode(conf).then((encode_data) => { console.log("Encode Data :", encode_data);
The encode function works fine, but the console.log statement executes immediately when calling encode() function — also the encode_data var is undefined. Since the return statement in the encode is in the message event, shouldn't the promise of the async function be resolved at that time?

So, NOTHING about the code inside your async function supports promises. You can't just throw random asynchronous (but not promise-based) code inside an async function and expect anything to work. An async function will work just fine with promise-based asynchronous functions that you await. Otherwise, it knows nothing about your asynchronous operations in there. That's why calling encode() returns immediately without waiting for anything to complete.
In addition, return transcode_data is inside an asynchronous callback. Returning inside that callback just goes back into the system code that called the callback and is dutifully ignored. You're not returning anything there from the async function itself. You're returning to that callback.
Since your operation is not promise-based, to solve this, you will have to make it promise-based by wrapping it in a promise and then manually resolved or rejecting that promise when needed with the proper values. You can do that like this:
encode: (config) => {
if (isMainThread) {
return new Promise((resolve, reject) => {
const encode_worker = new Worker(`./service-encode.js`, { workerData: config });
encode_worker.on('message', (transcode_data) => {
log.info("%o", transcode_data);
resolve(transcode_data);
});
encode_worker.on('error', (err) => {
log.error(err)
reject(err);
});
encode_worker.on('exit', (code) => {
if (code !== 0) {
reject(new Error(`Encoding stopped with exit code [ ${code} ]`));
}
console.log("* * * EXITED ENCODER WORKER * * *")
});
});
} else {
// should return a promise from all calling paths
return Promise.reject(new Error("Can only call encode() from main thread"));
}
},
FYI, this code assumes that the "result" you're looking for here from the promise is the transcode_data you get with the first message from this worker.

Update a field from all documents in Firebase

I am very new to Firebase and I am not sure how to update the same field in all documents in one collection.
return db.collection('collection').get().then((snapshot) => {
return snapshot.forEach((doc) => {
doc.update({'field': 1});
});
})
.catch((err) => {
console.log('Error getting documents', err);
});
I am running this code in a firebase function. There are no errors, but nothing is happening. I can't see anything in the documentation about this specific problem. Does anyone know what's wrong?

You'll need to return the result of all the update() calls to the caller, as otherwise it can't know when all asynchronous operations have completed:
return db.collection('collection').get().then((snapshot) => {
return Promise.all(snapshot.documents.map((doc) => {
return doc.ref.update({'field': 1});
}));
});
The changes:
return the promise from update() so that it can be bubbled up to the Cloud Functions container.
Use map so that we get an array of promises.
Then use Promise.all so that we only complete (and thus terminate the Cloud Function) when all the updates have completed.

This approach did the thing for me... Don't know if this is a proper way (as said I am a beginner)
exports.scheduledFunction2 = functions.pubsub.schedule('every 1 minutes').onRun((context) => {
return db.collection("collection")
.get()
.then(querySnapshot => {
if (!querySnapshot.empty) {
for(let i =0;i<querySnapshot.size;i++){
var queryDocumentSnapshotTest = querySnapshot.docs[i];
queryDocumentSnapshotTest.ref.update({'counter':1});
}
return null;
} else {
// do nothing
console.log("No document corresponding to the query!");
return null;
}
});
});

Cloud function not executing ref.once on realtime db

My cloud function was working fine this morning. But now it suddenly stopped working. So I added a few more logs and I noticed that rootRef.once was not executing. Why could this be happening? The function keeps on timing out.
exports.assignTeams = functions.https.onCall((data, context) => {
const judgeId = data.judgeId;
console.log("Assigning teams to judge ", judgeId);
var db = admin.database();
var rootRef = db.ref("/");
rootRef.once(
"value",
function(snapshot) {
console.log("Passed value ", snapshot);
processAssignedQueue(snapshot, judgeId);
},
function(error) {
console.log("ERROR:\n" + error);
return { teams: [] };
}
);
});
The console logs from Firebase:

The problem is that you're not returning a promise from the function that resolves with the data to send to the caller. Right now, your function is actually returning nothing, because once() is asynchronous and returns immediately before the query is complete. When a callable function returns without a promise, it shuts down immediately, an any asynchronous work will not complete.
Also, you should note that the return statements inside the callbacks you passed to once() are only return values from those individual function callbacks, not the entire function.
What you should do instead is make use of the promise returned by once(), and use that to determine what to return to the client. Here's a simple example:
return rootRef.once("value")
.then(snapshot => {
console.log("Passed value ", snapshot);
return { data: snapshot.val() }
})
.catch(error => {
console.log("ERROR:\n" + error);
return { teams: [] };
});
In order to write Cloud Functions effectively, you will need to fully understand how javascript promises work. If you don't work with them correctly, your functions will fail in mysterious ways.

Why is my Firestore call not resolving before going further with the next .then()?

I thought I had a simple function:
database change trigger (.onUpdate)
find out which change is possibly important for a notification (prepareNotifications(change))
ask firebase if there are records that want a notification about that change (getDatabaseNotifications(changeWithNotification))
sent notifications (sentNotifications(changeWithNotification))
I'm stuck for a couple off days now on how to resolve the Firebase call before moving on.
tried to Promise.all([getDatabaseNotifications()])
tried to chain this function like this:
changes
then firebase call
then sent notifiactions
What is happening:
I get the changes,
The call to Firebase is done,
But before waiting for the result it moves on to sending notifications.
It finds no notifications in Firebase (but there are notifications!)
It's gathering the notifications (array [])
... here I push a test notification ...
It's sending 1 notification (the test notification)
Then it resolves the Firebase notifications (this should be resolved before).
Then the function stops without doing anything anymore.
This is how my function looks now. Can someone explain how I can wait on Firebase?
exports.showUpdate = functions.firestore
.document('shows/{showId}')
.onUpdate((change, context) => {
return prepareNotifications(change) // check if and which notifications to get out of the firebase database
.then(changes => {
console.log('changes');
console.log(changes);
if(changes) {
const gatherData = [];
changes.forEach(change => {
console.log('change');
console.log(change);
getDatabaseNotifications(change.showId, change.ring, change.ringNumber) // firebase call
.then(data => {
gatherData.push([...gatherData, ...data]);
console.log('gatherData');
console.log(gatherData);
})
.catch(err => {
console.log(err);
})
})
return gatherData;
}
return null;
})
.then(notifications => {
console.log('notifications');
console.log(notifications);
notifications.push(testData); // add some test notifications
if (notifications && notifications.length > 0) {
sentNotifications(notifications); // sent notifications
return 'sending notifications';
}
return 'no notifications to sent';
})
.catch(err => {
Sentry.captureException(new Error(`Showupdate sending notifications not ok. Error message: ${err.message}`));
})
});
Updated code which works! thanks to your examples.
exports.showUpdate = functions.firestore
.document('shows/{showId}')
.onUpdate((change, context) => {
return prepareNotifications(change) // check if and which notifications to get out of the firebase database
.then(changes => {
if(changes) {
return getDbRecords(changes);
}
})
.then(notifications => {
if (notifications && notifications.length > 0) {
sentNotifications(notifications); // sent notifications
return 'sending notifications';
}
return 'no notifications to sent';
})
.catch(err => {
Sentry.captureException(new Error(`Showupdate sending notifications not ok. Error message: ${err.message}`));
})
});
function getDbRecords(changes) {
const gatherData = [];
const gatherDataPromises = [];
changes.forEach(change => {
gatherDataPromises.push(
getDatabaseNotifications(change.showId, change.ring, change.ringNumber) // firebase call
.then(data => {
gatherData.push(...data);
})
);
});
return Promise.all(gatherDataPromises)
.then(() => { return gatherData }
);
}

This section of your code doesn't handle promises properly, it creates a bunch of work but then will return gatherData before any of it has happened, which is why you don't see any notifications:
if(changes) {
const gatherData = [];
changes.forEach(change => {
console.log('change');
console.log(change);
getDatabaseNotifications(change.showId, change.ring, change.ringNumber) // firebase call
.then(data => {
gatherData.push([...gatherData, ...data]);
console.log('gatherData');
console.log(gatherData);
})
.catch(err => {
console.log(err);
})
})
return gatherData;
}
Notably, you probably want that return gatherData to be chained off the set of promises that are generated by the entire set of calls to getDatabaseNotifications.
Something like:
if(changes) {
const gatherData = [];
const gatherDataPromises = [];
changes.forEach(change => {
console.log('change');
console.log(change);
gatherDataPromises.push(
getDatabaseNotifications(change.showId, change.ring, change.ringNumber) // firebase call
.then(data => {
gatherData.push([...gatherData, ...data]);
console.log('gatherData');
console.log(gatherData);
})
);
});
return Promise.all(gatherDataPromises)
.then(() => { return gatherData });
}
I removed the catch statement to allow the error to bubble up to the top level catch.
Caution: I have not tested this, as I don't have sample data or the code for getDatabaseNotifications, but the general approach should solve your current problem. Likewise, it allows all the calls to getDatabaseNotifications to run in parallel, which should be significantly faster than just awaiting on them in sequence.
That said, you do have other problems in this code -- for example, the return null just below the block I am discussing will likely lead you into trouble when you try to use notifications.push() in the following then() (but this also appears to be test code).

I think it's because of the async nature of the methods. So, instead of waiting "getDatabaseNotifications()" to finish it's job, it jumps into ".then(notifications =>{}" and in this case gatherData returns empty.
putting await before calling the method might work.
await getDatabaseNotifications(change.showId, change.ring, change.ringNumber)

Different variables for different request in nodejs

Say, I have the following code (Assume all undefined functions as async tasks which return promise):
app.post('/abc', (req, res) => {
var project;
getProject(req.body.projectId)
.then((_projectData) => {
//this data is required by other functions in this promise chain
project = _projectData;
})
.then(()=>{
return doSomething1(project)
})
.then(()=>{
return doSomething2(project)
})
.then(()=>{
return doSomething3(project)
})
.then(()=>{
return res.status(200).send('Everything done.')
})
.catch((err) => {
return res.status(500).send('Error occured.')
})
});
Now, when a request came, the project will be set to the project data related to that request. And doSomething1() will be called. Say, doSomething1() is taking a lot of time. Meanwhile second request came and getProject() called, and then the project variable would change. Now when 2nd request's doSomething1() is executing, 1st request's doSomething1() is completed and doSomething2() get called, but with the different project variable.
How to avoid this situation?
What is the best case for such cases?
(One way could be to break this promise chain into 2 promise chains, and put all the dependents in one chain, so that each request has different scope of project variable and all other functions are called in that scope:
app.post('/abc', (req, res) => {
getProject(req.body.projectId)
.then((_projectData) => {
//this data is required by other functions in this promise chain
var project = _projectData;
doSomething1(project).
.then(()=>{
return doSomething2(project)
})
.then(()=>{
return doSomething3(project)
})
.then(()=>{
return res.status(200).send('Everything done.')
})
.catch((err) => {
return res.status(500).send('Error occured.')
})
})
.catch((err) => {
return res.status(500).send('Error occured.')
})
});
But this doesn't seems good, as if the promise chain is big, and there are more dependencies (like output of doSomething1() as input of doSomething3()), the promise chains won't work as they are designed to.
Another way could be to pass the project variable via each promise resolve, but it would not be possible, as it might be 3rd party module function, or even changing the return value of a function for that purpose would not justify the function re-usability.)
What am I missing here?