This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 5 years ago.
I'm trying to create a "synchronized like loop" with async await but I get the strange behavior of getting all the results after the first promise instead.
here is my test case. See how you get all together instead of print 1 by 1
const p = () => new Promise((resolve) =>{
setTimeout(()=>{
resolve(Math.floor(Date.now() / 1000))
},1000)
})
const test = () =>{
[1,2,3,4,5,6,7,8,9,10].map(async () =>{
const e = await p();
console.log(e)
});
}
test();
Like I mention in comments, map is not Promise aware. There are alternatives like Bluebirds.map,.. But the easy option is to use a for loop,..
Keeping to your example were your wanting to loop an array, you can use the for of variant too. Changing to for (let a = 1; a <= 10; a ++) will work too.
eg.
const p = () => new Promise((resolve) =>{
setTimeout(()=>{
resolve(Math.floor(Date.now() / 1000))
},1000)
})
const test = async () =>{
for (const a of [1,2,3,4,5,6,7,8,9,10]) {
const e = await p();
console.log(e)
}
}
test();
It's also worth pointing out bluebirds map has another trick up it's sleeve, it has a concurrency option. This is very handy to prevent thrashing that can happen with Promise.all, you basically say you want to do x promises at a time, eg. you have 10 promises to do, but you find it's more optimal to do 3 concurrently.
Related
This question already has answers here:
Resolve promises one after another (i.e. in sequence)?
(36 answers)
Is Node.js native Promise.all processing in parallel or sequentially?
(14 answers)
Closed 6 months ago.
I asked a question that was closed due to similar questions but even after reading those and seeing that map is not Promise aware, I have tried to use await Promise.all() around my mapping.
let foo = [1,2,3];
(async() => {
await Promise.all(foo.map(x => {
return new Promise(async function(resolve) {
await setTimeout(() => {
console.log("X")
resolve()
}, 3000)
});
}))
})()
instead of sequentially getting a "X" log every 3 seconds, it's just waiting 3 seconds and then logging them all. I am not sure what I am doing wrong at this point.
You are setting all the timeouts together at once. You need to increase the timeout duration in each iteration, maybe like this:
let foo = [1,2,3];
(async() => {
await Promise.all(foo.map((x, i) => {
return new Promise(async function(resolve) {
await setTimeout(() => {
console.log("X")
resolve()
}, 3000 * (i + 1))
});
}))
})()
This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 1 year ago.
I have an array of IDs, for each ID I want to make an axios.post call to create something in the database. I've done something like this.
const promises = ids.map(async (id) =>
await methodToCallAxiosPost(params, id)
);
await Promise.all(promises);
// do stuff
now I know that Promise.all runs when all of the promises are resolved which is useful to me to update the UI, but I also want to wait for each promise inside, something like this
const data1 = await methodToCallAxiosPost(params, id1)
const data2 = await methodToCallAxiosPost(params, id2)
const data3 = await methodToCallAxiosPost(params, id3)
Because I am getting potential lock conflicts in the database and I sometimes face issues.
I have tried using for await of but it seemed similar to Promise.all or I might be using it incorrectly.
You can't. They aren't promise aware methods. Use a regular for loop instead.
const results = [];
for (let i = 0; i < ids.length; i++) {
const result = await methodToCallAxiosPost(params, ids[i]);
results[i] = result;
}
My question is basically a combination of
What is the best way to limit concurrency?
Wait until all promises complete even if some rejected
I'm aware of Promise.allSettled, but I'm failing to find a good way to also limit concurrency.
What I have so far:
Idea 1 using p-limit:
const pLimit = require('p-limit');
const limit = pLimit(10);
let promises = files.map(pair => {
var formData = {
'file1': fs.createReadStream(pair[0]),
'file2': fs.createReadStream(pair[1])
};
return limit(() => uploadForm(formData));
});
(async () => {
const result = await Promise.allSettled(promises).then(body => {
body.forEach(value => {
if(value.status == "rejected")
file.write(value.reason + '\n---\n');
});
});
})();
My problem with this solution is, that I have to create all promises in the first place and in doing so opening two file streams for each promise and I'll hit the limit of open files.
Idea 2 using p-queue:
I tried around with generator functions to create and add new promises in the queue.on 'next' event, but I couldn't get it to work properly and this is probably not the right tool for this job.
Idea 3 using a PromisePool:
This looked very promising in the beginning. Some of them support a generator function to create the promises for the pool, but I couldn't find one, who explicitly stated to behave like Promise.allSettled.
I implemented es6-promise-pool only to find out that it will stop after the first promise rejection.
It's simple enough to implement it yourself - make an array of functions that, when called, return the Promise. Then implement a limiter function that takes functions from that array and calls them, and once finished, recursively calls the limiter again until the array is empty:
const request = (file) => new Promise((res, rej) => {
console.log('requesting', file);
setTimeout(() => {
if (Math.random() < 0.5) {
console.log('resolving', file);
res(file);
} else {
console.log('rejecting', file);
rej(file);
}
}, 1000 + Math.random() * 1000);
});
const files = [1, 2, 3, 4, 5, 6];
const makeRequests = files.map(file => () => request(file));
const results = [];
let started = 0;
const recurse = () => {
const i = started++;
const makeRequest = makeRequests.shift();
return !makeRequest ? null : Promise.allSettled([makeRequest()])
.then(result => {
results[i] = result[0];
return recurse();
})
};
const limit = 2;
Promise.all(Array.from({ length: limit }, recurse))
.then(() => {
console.log(results);
});
If the order of the results doesn't matter, it can be simplified by removing the started and i variables.
The accepted answer works more or less like p-limit.
You were having issues with p-limit because the streams were declared outside the limit callback.
This would have solved your problem:
let promises = files.map(pair => {
return limit(() => uploadForm({
'file1': fs.createReadStream(pair[0]),
'file2': fs.createReadStream(pair[1])
}));
});
This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 2 years ago.
The below JavaScript is returning a strange version of an array. The below forEach logs nothing to console. I'm probably missing something quite fundamental here, but I know for a fact that the array is populated...unless it's not populated until later due to being async and Chrome just evaluates the console.log after the fact?
let damageRollStringArr = []
monster.damage_types.forEach(async (damageTypeName) => {
const damageType = await checkResponseAndReturnValue(await DamageTypesService.findDamageType(damageTypeName.trim()))
if (damageType !== null) {
damageRollStringArr.push(damageType.damage)
}
})
damageRollStringArr.forEach(el => console.log(el))
// Was returning an empty array[] with objects inside?
return damageRollStringArr
Thanks
demageRollStringArr.forEach (2nd foreach) wont wait for monster.demage_types.forEach (1st foreach) to execute although there is async await in the first forEach
To achieve what you want to do, try this,
let damageRollStringArr = []
const promises = monster.damage_types.map(async (damageTypeName) => {
const damageType = await checkResponseAndReturnValue(await DamageTypesService.findDamageType(damageTypeName.trim()))
if (damageType !== null) {
damageRollStringArr.push(damageType.damage)
}
})
await Promise.all(promises)
damageRollStringArr.forEach(el => console.log(el))
return damageRollStringArr
Or, you can use normal for() loop to replace 1st foreach
let damageRollStringArr = []
for(let i = 0; i < monster.demage_types.length; i++) {
const damageType = await checkResponseAndReturnValue(await DamageTypesService.findDamageType(damageTypeName.trim()))
if (damageType !== null) {
damageRollStringArr.push(damageType.damage)
}
}
damageRollStringArr.forEach(el => console.log(el))
return damageRollStringArr
1st solution will run the loop parallel,
2nd solution will run the loop sequential
I have a certain promise chain in my code that looks like this:
myPromise()
.then(getStuffFromDb)
.then(manipulateResultSet)
.then(manipulateWithAsync)
.then(returnStuffToCaller)
Now, in my manipulateWithAsync I'm trying to enhance my result set by calling the DB again, but it's not working as I expected, since while debugging i figured out the control moves to the next function which is the returnStuffToCaller
here's an idea of what's into my manipulateWithAsync function:
function manipulateWithAsync(rs) {
return rs.map( async function whoCares(singleRecord) {
let smthUseful = await getMoreData(singleRecord.someField);
singleRecord.enhancedField = smthUseful;
return singleRecord;
})
}
I get the point of this behaviour: the map function does work as expected and the promise chain doesn't give a duck about it since it's not working with the awaits.
Is there a way to allow my returnStuffToCaller function to wait till the async function did his job?
I also use bluebird and i tried to use coo-routine, so if you thing that it's a good solution I'll post my bluebird coo-routine failing code :)
Thanks!
The problem is in using async/await with Array.map
This answer should help: https://stackoverflow.com/a/40140562/5783272
rs.map iterator jumps to the next element without waiting in each separate iteration.
You need something like asyncMap
You can use - https://github.com/caolan/async
or either implement yourself
async function asyncMap(array, cb) {
for (let index = 0; index < array.length; index++) {
return await cb(array[index], index, array);
}
}
*cb function must be async one
Wrap your map with Promise.all return the Promise then await for the results wherever you call the manipulateWithAsync.
// MOCKS FOR DEMO
// Test data used as input for manipulateWithAsync
const testData = [
{ recordNumber: 1 },
{ recordNumber: 2 },
{ recordNumber: 3 }
];
// Mock function which returns Promises which resolve after random delay ranging from 1 - 3 seconds
const getMoreData = () =>
new Promise(resolve => {
const calledAt = Date.now();
setTimeout(() => {
resolve({
usefulData: `Promise called at ${calledAt}`
});
}, Math.floor(Math.random() * 3000) + 1000);
});
// SOLUTION / ANSWER
const manipulateWithAsync = async rs =>
Promise.all(
rs.map(async singleRecord => {
const smthUseful = await getMoreData(singleRecord.someField);
// Instead of manipulating original data,
// which might cause some unwanted side effects going forward,
// instead return new objects
return { ...singleRecord, enhancedField: smthUseful };
})
);
await manipulateWithAsync(testData);