We Keep Coding

Promise.all does not await for then

First of all I am a newbie in nodejs, please be indulgent :)
I have a problem with some async functions. In my case, a first API call is made and then a sub call is made on every item returned. this is my code after the first call is made :
const countryStatesCalls: Promise<any>[] = [];
countries.forEach((country, index) => {
countries[index] = convertToCountry(country);
const countryStatesCall = (countryId: number) => {
return new Promise((resolve, reject) => {
const countryStatesList = getStateCountries(countryId);
if (countryStatesList) {
if (Object.prototype.toString.call(countryStatesList) !== '[object Array]') {
resolve([]);
} else {
resolve(countryStatesList);
}
} else {
reject([]);
}
});
};
countryStatesCalls.push(
countryStatesCall(countries[index].id).then(countryStates => {
countries[index].states = countryStates;
}).catch(countryStatesType => {
countries[index].states = countryStatesType;
})
);
});
await Promise.all(countryStatesCalls).then(_ => {
res.json(countries)
});
the following code
res.json(countries)
is supposed to be executed after all promises execution has been finished.
EDIT : the getStateCountries async function :
const getStateCountries = async (countryId: number, lang?: string) => {
const listState = await odoo.searchRead(
'res.country.state',
{
filters: [
{
field: 'country_id',
operator: '=',
value: countryId,
},
],
fields: fieldState,
},
{ lang }
);
if (!listState.length) {
return [];
} else {
listState.forEach((state, index) => {
listState[index] = convertToState(state);
});
return listState;
}
};
the main function is an express controller
I think my mistake may be obvious but can you help me please, I read many topics and I cannot see what am I doing wrong.
Thank you in advance
[SOLUTION]
As pointed out by comments, I was doing it the wrong way around, my call on promise called another promise: the solution was to write :
const countryStatesCalls: Promise<any>[] = [];
for (let index = 0; index < countries.length; index++) {
countries[index] = convertToCountry(countries[index]);
if (withSates) {
countryStatesCalls.push(
getStateCountries(countries[index].id)
);
} else {
countries[index].states = [];
}
}
if (withSates) {
await Promise.all(countryStatesCalls).then((countriesStates) => {
countries.forEach((country, index) => {
country.states = [];
for (const countryStates of countriesStates) {
if (countryStates.length
&& countryStates[0].country_id.code === country.id) {
countries[index].states = countryStates;
}
}
});
res.json(countries);
});
} else {
res.json(countries);
}
thank you everyone

I can't exactly tell why it doesn't work, but this looks a bit like the Promise constructor antipattern. The getStateCountries function is async and already creates a promise, so you need to await it or chain .then() to it. The new Promise is not necessary.
Also I'd recommend to avoid forEach+push, rather just use map, and get rid of the countryStatesCall helper:
const countryStatesCalls: Promise<any>[] = countries.map((country, index) => {
countries[index] = convertToCountry(country);
return getStateCountries(countries[index].id).then(countryStatesList => {
if (countryStatesList) {
if (Object.prototype.toString.call(countryStatesList) !== '[object Array]') {
return [];
} else {
return countryStatesList;
}
} else {
throw new Error("getStateCountries() resolved to nothing");
}
}).then(countryStates => {
countries[index].states = countryStates;
});
});
await Promise.all(countryStatesCalls).then(_ => {
res.json(countries)
});

Collect promises generated within promises in forEach

I want to display the progress of a migration operation to mongodb.
The script looks like:
let promises = [];
mylist.forEach(idx => {
myCollection.find({id: idx}).toArray().then(msgs => {
promises.push(myCollection2.insertMany(msgs.map(msg => ({
msg: msg,
otherFields: others
}))))
})
});
// function to display the progress:
allProgress(promises,
(p) => {
console.log(`% Done = ${p.toFixed(2)}`);
});
function allProgress(proms, progress_cb) {
let d = 0;
progress_cb(0);
proms.forEach((p) => {
p.then(()=> {
d ++;
progress_cb( (d * 100) / proms.length );
});
});
return Promise.all(proms);
}
This won't work because promises is empty when allProgress() is called.
How can I correctly collect all promises before calling allProgress()?
UPDATE
In the process of producing a MCVE, I came up with
let promises = [];
[1,2,3].forEach(idx => {
test(1000).then(promises.push(test(10000)));
});
console.log(promises.length);
// function to display the progress:
allProgress(promises,
(p) => {
console.log(`% Done = ${p.toFixed(2)}`);
});
function test(ms) {
return new Promise((resolve) => {
setTimeout(() => {
console.log(`Waited ${ms}`);
resolve();
}, ms);
});
}
function allProgress(proms, progress_cb) {
let d = 0;
progress_cb(0);
proms.forEach((p) => {
p.then(() => {
d++;
progress_cb((d * 100) / proms.length);
});
});
return Promise.all(proms);
}
This script, to my surprise, works... Why isn't equivalent to my original script?
UPDATE2
[1,2,3].forEach(idx => {
test(1000).then(_ => {
promises.push(test(10000))
});
});
This should be the MCVE, which does not work.

The .find() function is async so while you are still in the process of finding elements the forEach loop itself moves on. In the end you end up waiting for your .find().
What you could do is inside of the .then() callback, check the index of the current forEach item, if you are at the last item then we know all promises have been returned. So call your allProgress function there.
This should allow enough time for waiting for everything to come together. Additionally, by checking against the index, we know that we will only call your allPromises function at completion. Not multiple times as each forEach loop occurs.
let promises = [];
mylist.forEach((idx, index) => {
myCollection.find({id: idx}).toArray().then(msgs => {
promises.push(myCollection2.insertMany(msgs.map(msg => ({
msg: msg,
otherFields: others
}))));
if((index + 1) === mylist.length){
// function to display the progress:
allProgress(promises, (p) => {
console.log(`% Done = ${p.toFixed(2)}`);
});
}
})
});
function allProgress(proms, progress_cb) {
let d = 0;
progress_cb(0);
proms.forEach((p) => {
p.then(()=> {
d ++;
progress_cb( (d * 100) / proms.length );
});
});
return Promise.all(proms);
}
Edit:
Your MCVE (latest edit) is failing for the exact same reason. Your requests are async which is allowing the loop to progress without waiting. Once again, check the index and call when all done.
let promises = [];
let entries = [1, 2, 3]
entries.forEach((idx, index) => {
test(1000).then(_ => {
promises.push(test(10000))
if((index + 1) === entries.length) {
console.log(promises.length);
// function to display the progress:
allProgress(promises,
(p) => {
console.log(`% Done = ${p.toFixed(2)}`);
});
}
});
});
function test(ms) {
return new Promise((resolve) => {
setTimeout(() => {
console.log(`Waited ${ms}`);
resolve();
}, ms);
});
}
function allProgress(proms, progress_cb) {
let d = 0;
progress_cb(0);
proms.forEach((p) => {
p.then(() => {
d++;
progress_cb((d * 100) / proms.length);
});
});
return Promise.all(proms);
}

myCollection.find({id: idx}) is async operation.
so you can like this:
let promises = [];
mylist.forEach(idx => {
myCollection.find({id: idx}).toArray().then(msgs => {
promises.push(myCollection2.insertMany(msgs.map(msg => ({
msg: msg,
otherFields: others
}))))
allProgress(promises,
(p) => {
console.log(`% Done = ${p.toFixed(2)}`);
});
})
});
function allProgress(proms, progress_cb) {
let d = 0;
progress_cb(0);
proms.forEach((p) => {
p.then(()=> {
d ++;
progress_cb( (d * 100) / proms.length );
});
});
return Promise.all(proms);
}

async issues with js generator and promises not returning result

I'm having yet another async issue where I'm lost and have no idea where or how to fix it. Forgive my bad naming.
api call to twitch api and returns an array its results.
exports.batchPromiseWrapper = function(arr) {
const filteredMungedDataArr = [];
let promiseBatachArray = arr.map(vod_id => {
var url = `https://api.twitch.tv/kraken/videos/${vod_id.id}/markers`;
var params = { api_version: 5 };
return axios
.get(url, {
params: params,
headers: {
"Client-ID": "xxxxxxxxxxxxxxx"
}
})
.then(res => {
return res.data;
})
.catch(function(error) {
console.log(error);
});
});
return Promise.all(promiseBatachArray)
.then(markers => {
if (markers !== null) {
markers.map(markerObj => {
if (markerObj.markers.game_changes !== null) {
markerObj.markers.game_changes.forEach(gameName => {
if (gameName.label === "Fortnite") {
filteredMungedDataArr.push(markerObj);
}
});
}
});
return filteredMungedDataArr;
}
})
.catch(err => {
if (err.status === 500 || err.status === 404) {
console.log("error: ", err, err.message);
}
});
};
The data looks like this:
[[1,2,3,4,5],[1,2,3,4,5]], generator will yield and make a promise.all call of 5 before pausing 5sec and continuing to the next batch of 5.
exports.batchFetchingGeneratorWrapper = function(generator, batchArray) {
let evalNextValue = generator.next();
let delay = (v, t) => {
return new Promise(resolve => {
setTimeout(resolve.bind(null, v), t);
});
};
if (!evalNextValue.done) {
exports.batchPromiseWrapper(evalNextValue.value).then(data => {
let newBatchArray = batchArray;
if (data !== undefined) {
newBatchArray = batchArray.concat(data);
}
delay(5000).then(() => {
exports.batchFetchingGeneratorWrapper(generator, newBatchArray);
});
});
} else {
console.log("yay done!", batchArray);
return batchArray;
}
};
I'm able to console the results in batchArray from batchFetchingGeneratorWrapper, but I unable to act on it and I know it has something to do with async and how it has yet to be resolved.
promiseDataWrapper
.then(data => {
return gatherData.cleanUpVODData(data);
})
.then(data => {
function* batchFetching(batchArray) {
for (let i = 0; i < batchArray.length; i++) {
yield batchArray[i];
}
}
let batchArrResult = [];
let g = batchFetching(data);
new Promise((resolve, reject) => {
gatherData.batchFetchingGeneratorWrapper(g, batchArrResult);
if (g.done) { // i dont think this works
console.log("batchArrResult 1: ", batchArrResult);
resolve(batchArrResult);
}
}).then(result => console.log("asdfasdf", batchArrResult)); // empty array is returned
});

As far as I can tell, the problem lies chiefly in batchFetchingGeneratorWrapper().
It should be a matter of :
fixing delay()
making appropriate returns to make the recursion work
ensuring that the function returns Promise.
Almost undoubtedly (syntactically) simpler with async/await but here it is with old-fashioned thens :
exports.batchFetchingGeneratorWrapper = function(generator, batchArray) {
let evalNextValue = generator.next();
let delay = (t) => {
return new Promise(resolve => {
setTimeout(resolve, t);
});
};
if (!evalNextValue.done) {
return exports.batchPromiseWrapper(evalNextValue.value).then(data => {
return delay(5000).then(() => {
return exports.batchFetchingGeneratorWrapper(generator, batchArray.concat(data || []));
});
});
} else {
console.log("yay done!", batchArray);
return Promise.resolve(batchArray); // <<< promise wrapped to ensure that batchFetchingGeneratorWrapper() returns Promise
}
};
And chain the batchFetchingGeneratorWrapper() call appropriately :
promiseDataWrapper
.then(data => gatherData.cleanUpVODData(data))
.then(data => {
function* batchFetching(batchArray) {
for (let i = 0; i < batchArray.length; i++) {
yield batchArray[i];
}
}
return gatherData.batchFetchingGeneratorWrapper(batchFetching(data), []).then(batchArrResult => {
console.log('batchArrResult: ', batchArrResult);
return batchArrResult;
});
}).catch(error => {
console.log(error);
});

Parallel processing of async/await functions queue

I am trying to process a queue, in parallel. I have an object array, and I need to apply function func to each element, in parallel. I am making the parallelism level of chains as below:
async function() {
const pickUpNextTask = () => {
if (this.internalQueue.length) {
return this.func(this.internalQueue.shift())
}
}
const startChain = () => {
return Promise.resolve().then(function next() {
console.log('before then(next)')
return pickUpNextTask().then(next)
})
}
let chains = []
for (let k = 0; k < this.parallelism; k += 1) {
chains.push(startChain())
}
await Promise.all(chains)
this.drain()
}
It is not working like I want to. The pickUpNextTask() ends-up returning undefined when the queue is empty, so there is no then.
How does one deal with this?

Turned out, it was easy enough:
const pickUpNextTask = () => {
if (this.internalQueue.length) {
return this.func(this.internalQueue.shift())
} else {
return Promise.reject()
}
}
const startChain = () => {
return Promise.resolve()
.then(function next() {
return pickUpNextTask()
.then(next)
.catch(() => {
return Promise.resolve()
})
})
}

Use promise to delay mocha

I'm using mocha and zombie.
My goal is to load data from website <a> tags from website then query name and href. So far this is working:
const Browser = require('zombie')
const browser = new Browser()
browser.visit('page', () => {
let temp = browser.document.querySelectorAll('a selector')
for (let i = 0; i < temp.length; i++) {
browsers.push({ name: temp[i].innerHTML, url: temp[i].href })
}
delete temp
})
My next step would be using this data in testing.
const data = [{ name: 'name', url: 'test' }]
describe('test with for', () => {
for (let i = 0; i < data.length; i++) {
it(data[i].name, () => {
// test code with url of data[i]
})
}
})
The only problem I have is that data array is filled asynchronously and it has to be present at time of invoking it function.
I tryed using before but i saw that it does not make sence here (because of invoking async function don't stop code to execute and only function pased in it can depend on async data)
So I tryed to do this using Promise but it also failed.
const data = []
new Promise((resolve, reject) => {
// filing data
resolve()
}).then(() => {
describe('...', () => {
for (let i = 0; i < data.length; i++) {
it (data[i].name, () => {})
}
})
})
I looked at mocha documentation and didn't find solution :/
How should I solve this problem?

Ok i found solution, problem was my mistake i forgot to remove describe inwith my browser reading was contained that's why --delay did not work.
Running code:
const { describe, it } = require('mocha')
const { assert, expect } = chai = require('chai')
const Browser = require('zombie')
const browser = new Browser()
const browsers = []
new Promise((resolve, reject) => {
browser.visit('site', () => {
let temp = browser.document.querySelectorAll('selector')
for (let i = 0; i < temp.length; i++) {
browsers.push({ name: temp[i].innerHTML, url: temp[i].href })
}
delete temp
resolve()
})
}).then(() => {
describe('Get browser name', () => {
for (let i = 0; i < browsers.length; i++) {
it(browsers[i].name, () => {
})
}
})
run()
})