We Keep Coding

Spawning NodeJS worker thread inside of async function doesn't respect promise resolution

I've got an async function that launches a NodeJS worker thread like so:
encode : async (config) => {
if (isMainThread) {
const encode_worker = new Worker(`./service-encode.js`, { workerData: config });
encode_worker.on('message', (transcode_data) => {
log.info("%o", transcode_data);
return transcode_data;
});
encode_worker.on('error', (err) => { log.error(err)});
encode_worker.on('exit', (code) => {
if (code !== 0)
throw new Error(`Encoding stopped with exit code [ ${code} ]`);
console.log("* * * EXITED ENCODER WORKER * * *")
});
}
},
In the serivce-encode.js file I've got the following code which uses async functions. Note that I am using postMessage to signal that it is done.
var transcoder = require('./transcoder');
const {Worker, isMainThread, parentPort, workerData} = require('worker_threads');
console.log("* * * STARTING ENCODE THREAD * * *\n");
console.log(workerData);
transcoder.run(workerData)
.then((results) => {
transcode_data = results;
parentPort.postMessage(transcode_data);
})
.catch(err => { throw err });
Then, I use the following example code but the code in the 'message' event from above fires off immediately. That is, it doesn't seem to wait until it's done:
encode(conf).then((encode_data) => { console.log("Encode Data :", encode_data);
The encode function works fine, but the console.log statement executes immediately when calling encode() function — also the encode_data var is undefined. Since the return statement in the encode is in the message event, shouldn't the promise of the async function be resolved at that time?

So, NOTHING about the code inside your async function supports promises. You can't just throw random asynchronous (but not promise-based) code inside an async function and expect anything to work. An async function will work just fine with promise-based asynchronous functions that you await. Otherwise, it knows nothing about your asynchronous operations in there. That's why calling encode() returns immediately without waiting for anything to complete.
In addition, return transcode_data is inside an asynchronous callback. Returning inside that callback just goes back into the system code that called the callback and is dutifully ignored. You're not returning anything there from the async function itself. You're returning to that callback.
Since your operation is not promise-based, to solve this, you will have to make it promise-based by wrapping it in a promise and then manually resolved or rejecting that promise when needed with the proper values. You can do that like this:
encode: (config) => {
if (isMainThread) {
return new Promise((resolve, reject) => {
const encode_worker = new Worker(`./service-encode.js`, { workerData: config });
encode_worker.on('message', (transcode_data) => {
log.info("%o", transcode_data);
resolve(transcode_data);
});
encode_worker.on('error', (err) => {
log.error(err)
reject(err);
});
encode_worker.on('exit', (code) => {
if (code !== 0) {
reject(new Error(`Encoding stopped with exit code [ ${code} ]`));
}
console.log("* * * EXITED ENCODER WORKER * * *")
});
});
} else {
// should return a promise from all calling paths
return Promise.reject(new Error("Can only call encode() from main thread"));
}
},
FYI, this code assumes that the "result" you're looking for here from the promise is the transcode_data you get with the first message from this worker.

Method does not wait for Promise to be resolved

I have a function that I'm trying to call and basically force it to wait for a response before continuing onto the next thing.
I have two functions, both are asynchronous.
The first one looks something like this, with all parameters that begin with an '_' to be used as callbacks:
async function formatJson(input, _sendToThirdParty, _handleLogs, _setDimensions)
{
...do some work here to format the payload
if(onlineConnectionRequired)
{
_setDimensions(itemToUpdate, object);
}
else {
// Do non-online based transformations here
}
...do more work after the above
}
Basically from that, I'm trying to call this method setDimensions which looks like this:
async function setDimensions(itemToUpdate, object) {
try
{
if(itemToUpdate != null)
{
console.log("Loading dimensions");
await Promise.resolve(function() {
ns.get(`inventoryItem/${object['Item ID']}?expandSubResources=true`)
.then((res) => {
console.log("Inventory Item Loaded. Updating dimensions...");
itemToUpdate.consignments.push(
{
consignmentID: object.conID,
barcode: object.barcode,
itemID: '', // leaving as empty for now
width : res.data.custitem_width,
length : res.data.custitem_length,
height : res.data.custitem_height,
weight : res.data.custitem_weight,
fragile: object.fragile === 'T' ? 1 : 0,
description: object.description
}
);
console.log("Dimensions Finalised");
})
});
}
}
catch(err)
{
console.log(err);
const message = `Error attempting to set the dimensions for ${object['Item ID']}`;
console.log(message);
throw new Error(message);
}
}
The problems I'm coming across are:
The code from the first method continues on before waiting for the promise to resolve, but I need it to wait so I can fully finish building up the payload before it continues on doing the next bits
If I try and include the await keyword before my call to _setDimensions(...) in the first method, I get an error "SyntaxError: await is only valid in async function", but I would've thought that it was an async function?
If anyone could help, that would be incredibly appreciated! Thank you!!

The correct design of your functions are below:
formatJson(input, (err, value) => {
if(err) {
// Error handler goes here
//console.log(err);
throw err;
} else {
// Implementation for working with returned value
console.log(value);
}
});
function formatJson(input, callback)
{
//...do some work here to format the payload
if(onlineConnectionRequired)
{
setDimensions(itemToUpdate, object)
.then((updatedItem) => {
// Implement anything here to work with your
// result coming from setDimensions() function
//console.log(updatedItem);
// Callback with null error and updatedItem as value
callback(null, updatedItem);
})
.catch((err) => {
// Callback with err object and null value
callback(err, null);
});
}
else {
// Do non-online based transformations here
}
//...do more work after the above
}
function setDimensions(itemToUpdate, object) {
try
{
if(inventoryItemID != null)
{
console.log("Loading dimensions");
return new Promise(function(resolve, reject) {
ns.get(`inventoryItem/${object['Item ID']}?expandSubResources=true`)
.then((res) => {
console.log("Inventory Item Loaded. Updating dimensions...");
itemToUpdate.consignments.push(
{
consignmentID: object.conID,
barcode: object.barcode,
itemID: '', // leaving as empty for now
width : res.data.custitem_width,
length : res.data.custitem_length,
height : res.data.custitem_height,
weight : res.data.custitem_weight,
fragile: object.fragile === 'T' ? 1 : 0,
description: object.description
}
);
console.log("Dimensions Finalised");
resolve(itemToUpdate);
})
.catch((err) => reject(err));
});
}
}
catch(err)
{
console.log(err);
const message = `Error attempting to set the dimensions for ${object['Item ID']}`;
console.log(message);
throw new Error(message);
}
}
Mistakes in your code:
Your formatJson function had async keyword but your formatJson function had callback functions named _sendToThirdParty, _handleLogs, _setDimensions. There are 3 types of implementation to create asynchronous codes. You can use callbacks, Promises or async/await. But Promises and async/await are the same except their uses cases and syntaxses. When you define a function as async fn() {...} it basically return a new Promise, so it is equal saying fn() { return new Promise(); }. Functions with callbacks have a shape like function(params, callback) { callback(cbParams); } you can use call callback function in a several branches in your function. But you only have single callback function, your code had 3 callback functions. Also note that functions with callback don't have async keyword. This is not valid because as I mentioned earlier, an async function will return a Promise. So you should not (but you can) define a function as async function(params, callback) like you did in your first method. This is definition is not wrong and it can work but it's not valid.
Your second method was an async function which was returning nothing. So I have changed it to normal function with returning a Promise.

Is the formatJson method beeing called inside an async method? It need to be, and before _setDimensions you need to add a await keyword.
And, as Daniel said, use the promise constructor.

Mongoose pass data out of withTransaction helper

Introduction
Hey there,
I am trying to pass out data from the mongoose withTransaction callback. Right now, I am using the following code which implements callbacks:
const transactionSession = await mongoose.startSession()
await transactionSession.withTransaction(async (tSession) => {
try {
// MARK Transaction writes & reads removed for brevity
console.log("Successfully performed transaction!")
cb(null, "Any test data")
return Promise.resolve()
} catch (error) {
console.log("Transaction aborted due to error:", error)
cb(error)
return Promise.reject()
}
})
} catch (error) {
console.log(error)
return cb(error)
}
A more detailed snippet of the withTransaction helper in use can be found here.
A link to the official Mongoose documentation regarding the withTransaction helper can be found here.
At the moment, I am using a callback to pass out data from the withTransactioncallback:
cb(null, "Any test data")
However, the problem is that naturally the callback is executed first, before the Promise.resolve() is returned. This means, that (in my case) a success response is sent back to the client before any necessary database writes are committed:
// this is executed first - the callback will send back a response to the client
cb(null, "Any test data")
// only now, after the response already got sent to the client, the transaction is committed.
return Promise.resolve()
Why I think this is a problem:
Honestly, I am not sure. It just doesn't feel right to send back a success-response to the client, if there hasn't been any database write at that time. Does anybody know the appropriate way to deal with this specific use-case?
I thought about passing data out of the withTransaction helper using something like this:
const transactionResult = await transactionSession.withTransaction({...})
I've tried it, and the response is a CommandResult of MongoDB, which does not include any of the data I included in the resolved promise.
Summary
Is it a problem, if a success response is sent back to the client before the transaction is committed? If so, what is the appropriate way to pass out data from the withTransaction helper and thereby committing the transaction before sending back a response?
I would be thankful for any advice I get.

It looks like there is some confusion here as to how to correctly use Promises, on several levels.
Callback and Promise are being used incorrectly
If the function is supposed to accept a callback, don't return a Promise. If the function is supposed to return a Promise, use the callback given by the Promise:
const transactionSession = await mongoose.startSession()
await transactionSession.withTransaction( (tSession) => {
return new Promise( (resolve, reject) => {
//using Node-style callback
doSomethingAsync( (err, testData) => {
if(err) {
reject(err);
} else {
resolve(testData); //this is the equivalent of cb(null, "Any test data")
}
});
})
Let's look at this in more detail:
return new Promise( (resolve, reject) => { This creates a new Promise, and the Promise is giving you two callbacks to use. resolve is a callback to indicate success. You pass it the object you'd like to return. Note that I've removed the async keyword (more on this later).
For example:
const a = new Promise( (resolve, reject) => resolve(5) );
a.then( (result) => result == 5 ); //true
(err, testData) => { This function is used to map the Node-style cb(err, result) to the Promise's callbacks.
Try/catch are being used incorrectly.
Try/catch can only be used for synchronous statements. Let's compare a synchronous call, a Node-style (i.e. cb(err, result)) asynchronous callback, a Promise, and using await:
Synchronous:
try {
let a = doSomethingSync();
} catch(err) {
handle(err);
}
Async:
doSomethingAsync( (err, result) => {
if (err) {
handle(err);
} else {
let a = result;
}
});
Promise:
doSomethingPromisified()
.then( (result) => {
let a = result;
})
.catch( (err) => {
handle(err);
});
Await. Await can be used with any function that returns a Promise, and lets you handle the code as if it were synchronous:
try {
let a = await doSomethingPromisified();
} catch(err) {
handle(err);
}
Additional Info
Promise.resolve()
Promise.resolve() creates a new Promise and resolves that Promise with an undefined value. This is shorthand for:
new Promise( (resolve, reject) => resolve(undefined) );
The callback equivalent of this would be:
cb(err, undefined);
async
async goes with await. If you are using await in a function, that function must be declared to be async.
Just as await unwraps a Promise (resolve into a value, and reject into an exception), async wraps code into a Promise. A return value statement gets translated into Promise.resolve(value), and a thrown exception throw e gets translated into Promise.reject(e).
Consider the following code
async () => {
return doSomethingSync();
}
The code above is equivalent to this:
() => {
const p = new Promise(resolve, reject);
try {
const value = doSomethingSync();
p.resolve(value);
} catch(e) {
p.reject(e);
}
return p;
}
If you call either of the above functions without await, you will get back a Promise. If you await either of them, you will be returned a value, or an exception will be thrown.

JavaScript Promise inside async/await function resolve final array of responses

I'm quite a newbie in JavaScript and in Promises.
I'm trying to build an array of objects that I get from an API.
To do so, I've build two functions in a file MyFile.js.
The first one returns a promise when an axios promise is resolved. It's
function get_items (url) {
return new Promise((resolve, reject) => {
let options = {
baseURL: url,
method: 'get'
}
axios(options)
.then(response => {
resolve(response.data)
})
.catch(error => {
reject(error.stack)
})
})
}
The second one looks like this:
let output = []
let next_url = 'https://some_url.com/api/data'
async function get_data () {
try {
let promise = new Promise((resolve, reject) => {
if (next_url) {
get_items(next_url)
.then(response => {
output.push(...response.results)
if (response.next) {
next_url = response.next
console.log('NEXT_URL HERE', next_url)
get_data()
} else {
console.log('else')
next_url = false
get_data()
}
})
.catch(error => {
reject(error.stack)
})
} else {
console.log('before resolve')
resolve(output)
}
})
return await promise
} catch(e) {
console.log(e)
}
}
It's where I'm grinding my teeth.
What I think I understand of this function, is that:
it's returning the value of a promise (that's what I understand return await promise is doing)
it's a recursive function. So, if there is a next_url, the function continues on. But if there is not, it gets called one last time to go into the else part where it resolves the array output which contains the results (values not state) of all the promises. At least, when I execute it, and check for my sanity checks with the console.log I wrote, it works.
So, output is filled with data and that's great.
But, when I call this function from another file MyOtherFile.js, like this:
final_output = []
MyFile.get_data()
.then(result => {
console.log('getting data')
final_output.push(...result)
})
it never gets into the then part. And when I console.log MyFile.get_data(), it's a pending promise.
So, what I would like to do, is be able to make get_data() wait for all the promises result (without using Promise.all(), to have calls in serie, not in parallel, that would be great for performances, I guess?) and then be able to retrieve that response in the then part when calling this function from anywhere else.
Keep in mind that I'm really a newbie in promises and JavaScript in general (I'm more of a Python guy).
Let me know if my question isn't clear enough.
I've been scratching my head for two days now and it feels like I'm running in circle.
Thanks for being an awesome community!

This is a bit untested
const api_url = 'https://some_url.com/api/data';
get_data(api_url).then((results) => {
console.log(results);
}).catch((error) => {
// console.error(error);
});
function get_items (url) {
const options = {
baseURL: url,
method: 'get'
};
return axios(options).then((response) => response.data);
}
async function get_data(next_url) {
const output = [];
while (next_url) {
const { results, next } = await get_items(next_url);
output.push(...results);
next_url = next;
}
return output;
}
Basically it makes things a bit neater. I suggest to look at more examples with Promises and the advantage and when to ease await/async. One thing to keep in mind, if you return a Promise, it will follow the entire then chain, and it will always return a Promise with a value of the last then.. if that makes sense :)

There are a few problems. One is that you never resolve the initial Promise unless the else block is entered. Another is that you should return the recursive get_data call every time, so that it can be properly chained with the initial Promise. You may also consider avoiding the explicit promise construction antipattern - get_items already returns a Promise, so there's no need to construct another one (same for the inside of get_items, axios calls return Promises too).
You might consider a plain while loop, reassigning the next_url string until it's falsey:
function get_items (baseURL) {
const options = {
baseURL: url,
method: 'get'
}
// return the axios call, handle errors in the consumer instead:
return axios(options)
.then(res => res.data)
}
async function get_data() {
const output = []
let next_url = 'https://some_url.com/api/data'
try {
while (next_url) {
const response = await get_items(next_url);
output.push(...response.results)
next_url = response.next;
}
} catch (e) {
// handle errors *here*, perhaps
console.log(e)
}
return output;
}
Note that .catch will result in a Promise being converted from a rejected Promise to a resolved one - you don't want to .catch everywhere, because that will make it difficult for the caller to detect errors.

Another way of doing it is to not use async at all and just recursively return a promise:
const getItems = (url) =>
axios({
baseURL: url,
method: 'get',
}).then((response) => response.data);
const getData = (initialUrl) => {
const recur = (result, nextUrl) =>
!nextUrl
? Promise.resolve(result)
: getItems(nextUrl).then((data) =>
recur(result.concat([data.results]), data.next),
);
return recur([],initialUrl)
.catch(e=>Promise.reject(e.stack));//reject with error stack
};
As CertainPerformance noted; you don't need to catch at every level, if you want getData to reject with error.stack you only need to catch it once.
However; if you had 100 next urls and 99 of them were fine but only the last one failed would you like to reject in a way that keeps the results so far so you can try again?
If you do then the code could look something like this:
const getData = (initialUrl) => {
const recur = (result, nextUrl) =>
!nextUrl
? Promise.resolve(result)
: getItems(nextUrl)
.catch(e=>Promise.reject([e,result]))//reject with error and result so far
.then((data) =>
recur(result.concat([data.results]), data.next),
);
return recur([],initialUrl);//do not catch here, just let it reject with error and result
};

node.js put event right after another event

Let's say I have two async events, both need to i/o with remote exchange.
placeOrder()
cancelOrder()
Both events fire in async way, which means cancelOrder can be called before placeOrder return. Tricky part is I need the placeOrder to return an Order ID first otherwise there is no way to call cancelOrder, so I need some way to block the cancelOrder event right until placeOrder returns, and the blockage cannot be too long otherwise the Order may be executed, so loop/timeout/frequent checking doesn't work here.
Any idea?

You would use a Promise for that. If your functions already return a promise, you can simply chain the both functions using then()
placeOrder().then(val => cancelOrder(val));
If they do not, you can put them inside a new Promise
function foo() {
return new Promise((resolve, reject) => {
// do stuff
resolve('<result of placeOrder here>');
});
}
function bar(val) {
return new Promise((resolve, reject) => {
// do stuff
resolve('whatever')
})
}
and call
foo()
.then(value => bar(value))
.then(console.log);

If you are able to use ES2017, the you can use async functions. For example, I'm going to assume that your functions perform some sort of request to the database using fetch or axios since you haven't specified. Then you can write placeOrder and cancelOrder like so:
const placeOrder = async () => {
try {
const response = await fetch('/place_order');
// Do something with the response
} catch (err) {
// Handle error
}
};
const cancelOrder = async () => {
try {
const response = await fetch('/cancel_order');
// Do something with the response
} catch (err) {
// Handle error
}
};
const someFunction = async () => {
await placeOrder();
await cancelOrder();
};