Using Async/Await vs Promise.all with for-loops - javascript

Here is how I understand things currently:
async/await in a for-loop should pause execution until the promise resolves, meaning that the next iteration of the loop won't occur until that line finishes.
Consider the following array:
const data = [];
for (let i = 0; i <= 100000; i++) {
data.push(i);
}
Method 1: Awaiting Promises in a for-loop
async function method1() {
const startTime = new Date();
console.log('start:', startTime);
for (const item in data) {
await new Promise(resolve => {
if (item % 3 === 0) {
resolve({});
} else {
resolve(item)
}
});
}
const endTime = new Date();
console.log('finish:', endTime);
console.log('total time:', endTime-startTime);
}
Because it requires sequential execution of promises to continue through the loop, I figured using Promise.all would be a performance enhancement that leverages asynchronous processing to a greater degree:
Method 2: Promise.all following a for-loop
async function method2() {
const promises = [];
const startTime = new Date();
console.log('start:', startTime);
for (const item in data) {
const promise = new Promise(resolve => {
if (item % 3 === 0) {
resolve({});
} else {
resolve(item)
}
});
promises.push(promise);
}
await Promise.all(promises);
const endTime = new Date();
console.log('finish:', endTime);
console.log('total time:', endTime-startTime);
}
My reasoning: the loop would continue to bang out new Promises while each previously created Promise attempts resolve. So in my head, method 1 = blocking... while method 2 = less blocking.
When I run these both on repl.it, I found that method method 1 is actually quicker, almost by a factor of 2. Can somebody explain why this is? Shouldn't it be the other way around?

My guess is that you resolve too fast, so the looping dominates the execution time, since method2 has two loops: one to push to the array and one for promise.all to complete while method1 only has one loop, so your feeling "almost a factor of two" is actually theoretically correct.
Try faking an actual async operation like new Promise(resolve => setTimeout(resolve, 200)) and you should get what you expected

To add further to what Drake said. The only reason why method 2 is slower without timeouts is because of the number of loops it will have to encounter, but in a real scenario where request or asynchronous operations have delays. Method 2 will most of the time be faster
Here is the modified snippet with SetTimeout.
const data = [];
for (let i = 0; i <= 1000; i++) {
data.push(i);
}
async function method1() {
const startTime = new Date();
console.log('start 1:', startTime);
for (const item in data) {
await new Promise(resolve => {
setTimeout(()=>{
if (item % 3 === 0) {
resolve({});
} else {
resolve(item)
}
},1)
});
}
const endTime = new Date();
console.log('finish 1:', endTime);
console.log('total time:', endTime-startTime);
}
async function method2() {
const promises = [];
const startTime = new Date();
console.log('start 2:', startTime);
for (const item in data) {
const promise = new Promise(resolve => {
setTimeout(()=>{
if (item % 3 === 0) {
resolve({});
} else {
resolve(item)
}
},1)
});
promises.push(promise);
}
await Promise.all(promises);
const endTime = new Date();
console.log('finish 2:', endTime);
console.log('total time:', endTime-startTime);
}
method1()
method2()

Related

Can someone tell me why my promise queue don't work?

I'm trying to write a simple promise queue function, the function will process 50 tasks with 10 concurrent promises:
const sendTasks = () => {
let tasks = 50;
const concurrentCount = 10;
let promisePool = 0;
while (tasks > 0) {
console.log(`current tasks: ${tasks}`);
while (promisePool < concurrentCount && task > 0) {
console.log("create promise");
tasks--;
promisePool++;
const promise = new Promise((resolve, reject) => {
setTimeout(() => {
resolve("haha");
}, Math.floor(Math.random() * 3) * 1000);
});
promise.then((value) => {
console.log(value);
promisePool--;
});
console.log(`current promisePool: ${promisePool}`);
}
}
return "done";
};
But when I execute it, the promises seems never resolve, then keep stuck in the task>0 while loop. Can someone explain to me why the promises never resolve?
A while loop inside a synchronous function will never yield control flow to Promise .thens, or to anything else. You will need to restructure the code to wait for the promises to resolve without fully terminating the sendTasks function and also without blocking the engine.
One approach is to push each Promise to an array, then await a Promise.any on that array. Have the Promises remove themselves from the array when they're finished, and recursively push more Promises to the array. Then return when no more Promises exist in the array.
const sendTasks = async () => {
let tasks = 50;
const concurrentCount = 10;
let promisePool = 0;
let promises = [];
const enqueueNext = () => {
if (!tasks) return;
// creating this variable just for the sake of logging
const thisTask = tasks--;
console.log("create promise", thisTask);
const prom = new Promise((resolve) => {
setTimeout(() => {
promises = promises.filter(p => p !== prom);
console.log('resolving', thisTask);
resolve("haha");
// recursive asynchronous call; init the next promise, if there is one
enqueueNext();
}, Math.floor(Math.random() * 3) * 1000);
});
promises.push(prom);
};
for (let i = 0; i < concurrentCount; i++) {
enqueueNext();
}
while (promises.length) {
await Promise.any(promises);
}
console.log("done");
};
sendTasks();
JavaScript is single-threaded except when it explicitly isn’t (web workers, Node multiprocessing, … – not promises), so your while (tasks > 0) { is a busy loop that never returns control to whatever event loop and gives timers the chance to fire.
You need to give a function (.then) or continuation (async/await) to your promises so that you can yield back to the event loop and be told when to continue processing.
const sendTasks = async () => {
let tasks = 50;
const concurrentCount = 10;
const promisePool = new Set();
while (tasks > 0) {
console.log(`current tasks: ${tasks}`);
while (promisePool.size < concurrentCount) {
console.log("create promise");
tasks--;
const promise = new Promise((resolve, reject) => {
setTimeout(() => {
resolve("haha");
}, Math.floor(Math.random() * 3) * 1000);
});
promise.then((value) => {
console.log(value);
promisePool.delete(promise);
});
promisePool.add(promise);
console.log(`current promisePool: ${[...promisePool]}`);
}
await Promise.race(promisePool);
}
// all tasks have been created here, but not necessarily completed
await Promise.all(promisePool);
return "done";
};
sendTasks().then(console.log);

How to run "x" promises in parallel Javascript

I have a function "promiseFunction" which returns a promise, which is resolved at a later time.
I need to call this function many times, however I only want a set number of executions of this function to happen at once.
The function calls some external single threaded c code on my computer, if I call too many instances of this code at once I crash my system, but if I call it sequentially 1 at a time it's very slow as only one thread of my cpu is doing any work.
So I came up with the code below, however it doesn't work. It will call the first 10 promises in parallel, but slowly it starts to call less and less promises at once, until it's only calling 1 promise at a time.
var totalNumberOfPromises = // total number times to run promiseFunction;
var promiseCounter = 0; // keep track of which promise call this is
for(w=0;w<10;w++){ // run 10 promises at a time
promiseFunction().then(function(resolve) {
loadNewPromise();
})
promiseCounter++;
}
function loadNewPromise(){
if(promiseCounter<totalNumberOfPromises){
promiseFunction().then(function(resolve) { loadNewPromise(); });
}else{
alert("Finished");
}
promiseCounter++;
}
Is there anything wrong with the code above that causes this behavior? And is there a standard way of doing this?
Here's a function I prepared earlier (I've used this for a few years now for just such a thing
const multiQueue = length => {
length = (isNaN(length) || length < 1) ? 1 : length;
const q = Array.from({length}, () => Promise.resolve());
let index = 0;
const add = cb => {
index = (index + 1) % length;
return (q[index] = q[index].then(() => cb()));
};
return add;
};
// demo usage
const q = multiQueue(10);
let inFlight = 0;
let maxInFlight = 0;
const promiseFunction = (i) => {
inFlight++;
maxInFlight = Math.max(inFlight, maxInFlight);
const obj = {inFlight, maxInFlight, i};
return new Promise(resolve => {
setTimeout(() => {
inFlight--;
resolve(Object.assign(obj, {t:performance.now()}));
}, 10 );
})
};
for (let i = 0; i < 40; i++) {
q(() => promiseFunction(i)).then(v => console.log(JSON.stringify(v)));
}
You can see that at most there are 10 "inFlight" requests
How about something like this? If you construct your queue which is and Array of functions that return a Promise you can splice chunks out of it and process each with a Promise.all.
const fakePromise = (id) => {
return new Promise((resolve) => {
setTimeout(() => {
console.log(`Resolving promise ${id}`)
resolve(id)
}, 100)
})
}
const queue = Array(100).fill().map((_, i) => {
return () => fakePromise(i)
})
const batchProcessPromises = (promises, batchSize) => {
if (promises && promises.length) {
return Promise.all(
promises.splice(0, batchSize)
.map(promise => promise())
)
.then(() => {
console.log('Batch complete')
return batchProcessPromises(promises, batchSize)
})
}
console.log('Batch complete')
return Promise.resolve()
}
batchProcessPromises(queue, 10)
.then(() => {
console.log('Time to get one with my day')
})
How do you plan to construct all your promises? This function effects the original queue so you would need to ensure that the array being passed into batchProcessPromises isn't shared. To get around this you could potentially use the spread operator like so
batchProcessPromises([...queue], 10)
.then(() => {
console.log('Time to get one with my day', queue)
})
Fiddle here https://jsfiddle.net/stwilz/2bpdcxo6/24/

How to guarantee sequential order with angular http rest api in for loop?

I'm trying to create a form that allows you to create multiple resources in sequential order.
Example below
Floor 1
Floor 2
Floor 3
...
Floor 9
The problem with the code is that the order is not guarantee.
My code below
let startAt = this.addAreasForm.controls['startAt'].value
const name = this.addAreasForm.controls['name'].value
const newArea = {name: name}
for (let i = 1; i < (amount + 1); i++) {
newArea.name = name + ' ' + startAt
startAt++
this.areasService.createArea(newArea, parentId)
.subscribe(
area => this.added.emit(area)
)
}
Can come back like
Floor 2
Floor 3
Floor 1
Floor 5
Floor 4
How do you handle async api calls to guarantee sequential order?
You can use async / await for that purpose with the Promise resolve:
for (let i = 1; i < (amount + 1); i++) {
await new Promise(resolve => {
newArea.name = name + ' ' + startAt
startAt++
this.areasService.createArea(newArea, parentId)
.subscribe(
area => {
this.added.emit(area);
resolve();
});
});
}
Remember to put async before your function. See this demo on StackBlitz.
You can try something like this, I don't exactly all your code from your services, but the main idea is this: In order to execute async code in order, you can build an array of promises and then to use Promise.all to take each result in the same order from the creation:
Promise.all
let startAt = this.addAreasForm.controls['startAt'].value;
const name = this.addAreasForm.controls['name'].value;
const newArea = {name: name};
Keep your services into variables I don't know from where your context comes.
const areasService = this.areasService,
added = this.added;
Make a function that create a promise for your subscribe:
function createAreaPromise(newArea, parentId) {
return new Promise((resolve, reject) => {
areasService.createArea(newArea, parentId)
.subscribe(area => resolve(area));
});
}
Than another function to build multiple an array of promises:
function buildPromises() {
let promises = [];
for (let i = 1; i < (amount + 1); i++) {
newArea.name = name + ' ' + startAt
startAt++
promises.push(createAreaPromise(newArea, parentId));
}
return promises;
}
Then solve them with Promise.all, to obtain the same order from creation
let promises = buildPromises();
Promise.all(promises)
.then(results => {
results.forEach(result => added.emit(result));
});
Here a live example:
function random() {
return Math.floor(Math.random() * 5);
}
function makePromise(index) {
return new Promise((resolve) => {
setTimeout(() => {
resolve(index);
}, random() * 1000);
});
}
function buildPromises() {
let promises = [];
for(let i = 0; i < 5; i++) {
promises.push(makePromise(i));
}
return promises;
}
let promises = buildPromises();
Promise.all(promises)
.then(results => {
results.forEach(result => {
console.log(result);
});
});

Best way to wait for .forEach() to complete

Sometimes I need to wait for a .forEach() method to finish, mostly on 'loader' functions. This is the way I do that:
$q.when(array.forEach(function(item){
//iterate on something
})).then(function(){
//continue with processing
});
I can't help but feel that this isn't the best way to wait for a .forEach() to finish. What is the best way to do this?
If there is no asynchronous code inside the forEach, forEach is not asynchronous, for example in this code:
array.forEach(function(item){
//iterate on something
});
alert("Foreach DONE !");
you will see the alert after forEach finished.
Otherwise (You have something asynchronous inside), you can wrap the forEach loop in a Promise:
var bar = new Promise((resolve, reject) => {
foo.forEach((value, index, array) => {
console.log(value);
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
});
Credit: #rolando-benjamin-vaz-ferreira
The quickest way to make this work using ES6 would be just to use a for..of loop.
const myAsyncLoopFunction = async (array) => {
const allAsyncResults = []
for (const item of array) {
const asyncResult = await asyncFunction(item)
allAsyncResults.push(asyncResult)
}
return allAsyncResults
}
Or you could loop over all these async requests in parallel using Promise.all() like this:
const myAsyncLoopFunction = async (array) => {
const promises = array.map(asyncFunction)
await Promise.all(promises)
console.log(`All async tasks complete!`)
}
var foo = [1,2,3,4,5,6,7,8,9,10];
If you're actually doing async stuff inside the loop, you can wrap it in a promise ...
var bar = new Promise((resolve, reject) => {
foo.forEach((value, index, array) => {
console.log(value);
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
});
If you have an async task inside a loop and you want to wait. you can use for await
for await (const i of images) {
let img = await uploadDoc(i);
};
let x = 10; //this executes after
Use for of instead of forEach. Like this:
for (const item of array) {
//do something
}
console.log("finished");
"finished" will be logged after finishing the loop.
forEach() doesn't return anything, so a better practice would be map() + Promise.all()
var arr = [1, 2, 3, 4, 5, 6]
var doublify = (ele) => {
return new Promise((res, rej) => {
setTimeout(() => {
res(ele * 2)
}, Math.random() ); // Math.random returns a random number from 0~1
})
}
var promises = arr.map(async (ele) => {
// do some operation on ele
// ex: var result = await some_async_function_that_return_a_promise(ele)
// In the below I use doublify() to be such an async function
var result = await doublify(ele)
return new Promise((res, rej) => {res(result)})
})
Promise.all(promises)
.then((results) => {
// do what you want on the results
console.log(results)
})
A universal solution for making sure that all forEach() elements finished execution.
const testArray = [1,2,3,4]
let count = 0
await new Promise( (resolve) => {
testArray.forEach( (num) => {
try {
//some real logic
num = num * 2
} catch (e) {
// error handling
console.log(e)
} fanally {
// most important is here
count += 1
if (count == testArray.length) {
resolve()
}
}
})
})
The idea is same with the answer using index to count. But in real case, if error happened, the index way cannot count correctly. So the solution is more robust.
Thx
const array = [1, 2, 3];
const results = [];
let done = 0;
const asyncFunction = (item, callback) =>
setTimeout(() => callback(item * 10), 100 - item * 10);
new Promise((resolve, reject) => {
array.forEach((item) => {
asyncFunction(item, (result) => {
results.push(result);
done++;
if (done === array.length) resolve();
});
});
}).then(() => {
console.log(results); // [30, 20, 10]
});
// or
// promise = new Promise(...);
// ...
// promise.then(...);
The order of results in the "results" array can be different than the order of items in the original array, depending on the time when the asyncFunction() finishes for each of the items.
Alter and check a counter at the end of every possible unique branch of code, including callbacks. Example:
const fs = require('fs');
/**
* #description Delete files older than 1 day
* #param {String} directory - The directory to purge
* #return {Promise}
*/
async function purgeFiles(directory) {
const maxAge = 24*3600000;
const now = Date.now();
const cutoff = now-maxAge;
let filesPurged = 0;
let filesProcessed = 0;
let purgedSize = 0;
await new Promise( (resolve, reject) => {
fs.readdir(directory, (err, files) => {
if (err) {
return reject(err);
}
if (!files.length) {
return resolve();
}
files.forEach( file => {
const path = `${directory}/${file}`;
fs.stat(path, (err, stats)=> {
if (err) {
console.log(err);
if (++filesProcessed === files.length) resolve();
}
else if (stats.isFile() && stats.birthtimeMs < cutoff) {
const ageSeconds = parseInt((now-stats.birthtimeMs)/1000);
fs.unlink(path, error => {
if (error) {
console.log(`Deleting file failed: ${path} ${error}`);
}
else {
++filesPurged;
purgedSize += stats.size;
console.log(`Deleted file with age ${ageSeconds} seconds: ${path}`);
}
if (++filesProcessed === files.length) resolve();
});
}
else if (++filesProcessed === files.length) resolve();
});
});
});
});
console.log(JSON.stringify({
directory,
filesProcessed,
filesPurged,
purgedSize,
}));
}
// !!DANGER!! Change this line! (intentional syntax error in ,')
const directory = ,'/tmp'; // !!DANGER!! Changeme
purgeFiles(directory).catch(error=>console.log(error));
I'm not sure of the efficiency of this version compared to others, but I used this recently when I had an asynchronous function inside of my forEach(). It does not use promises, mapping, or for-of loops:
// n'th triangular number recursion (aka factorial addition)
function triangularNumber(n) {
if (n <= 1) {
return n
} else {
return n + triangularNumber(n-1)
}
}
// Example function that waits for each forEach() iteraction to complete
function testFunction() {
// Example array with values 0 to USER_INPUT
var USER_INPUT = 100;
var EXAMPLE_ARRAY = Array.apply(null, {length: USER_INPUT}).map(Number.call, Number) // [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, n_final... ] where n_final = USER_INPUT-1
// Actual function used with whatever actual array you have
var arrayLength = EXAMPLE_ARRAY.length
var countMax = triangularNumber(arrayLength);
var counter = 0;
EXAMPLE_ARRAY.forEach(function(entry, index) {
console.log(index+1); // show index for example (which can sometimes return asynchrounous results)
counter += 1;
if (triangularNumber(counter) == countMax) {
// function called after forEach() is complete here
completionFunction();
} else {
// example just to print counting values when max not reached
// else would typically be excluded
console.log("Counter index: "+counter);
console.log("Count value: "+triangularNumber(counter));
console.log("Count max: "+countMax);
}
});
}
testFunction();
function completionFunction() {
console.log("COUNT MAX REACHED");
}
I had to deal with the same problem (forEach using multiple promises inside) and none of the solutions presented at the current date were helpful for me. So I implemented a check array, were each promise updates its complete status. We have a general promise that wraps the process. We only resolve the general promise when each promise completed. Snippet code:
function WaitForEachToResolve(fields){
var checked_fields = new Array(fields.length).fill(0);
const reducer = (accumulator, currentValue) => accumulator + currentValue;
return new Promise((resolve, reject) => {
Object.keys(fields).forEach((key, index, array) => {
SomeAsyncFunc(key)
.then((result) => {
// some result post process
checked_fields[index] = 1;
if (checked_fields.reduce(reducer) === checked_fields.length)
resolve();
})
.catch((err) => {
reject(err);
});
}
)}
}
I like to use async-await instead of .then() syntax so for asynchronous processing of data, modified the answer of #Ronaldo this way -
let finalData = [];
var bar = new Promise(resolve => {
foo.forEach((value, index) => {
const dataToGet = await abcService.getXyzData(value);
finalData[index].someKey = dataToGet.thatOtherKey;
// any other processing here
if (finalData[dataToGet.length - 1].someKey) resolve();
});
});
await Promise.all([bar]);
console.log(`finalData: ${finalData}`);
NOTE: I've modified the if condition where it resolves the promise to meet my conditions. You can do the same in your case.
You can use this, because we are using async/await inside the forEach loop. You can use your own logic inside the loop.
let bar = new Promise((resolve, reject) => {
snapshot.forEach(async (doc) => {
"""Write your own custom logic and can use async/await
"""
const result = await something()
resolve(result);
});
});
let test = []
test.push(bar)
let concepts = await Promise.all(test);
console.log(concepts);
For simple compare code i like use for statement.
doit();
function doit() {
for (var i = 0; i < $('span').length; i++) {
console.log(i,$('span').eq(i).text() );
if ( $('span').eq(i).text() == "Share a link to this question" ) { // span number 59
return;
}
}
alert('never execute');
}
If there are async (observable) method calls inside the for loop, you could use the following method:
await players.reduce(async (a, player) => {
// Wait for the previous item to finish processing
await a;
// Process this item
await givePrizeToPlayer(player);
}, Promise.resolve());
Check here: https://gist.github.com/joeytwiddle/37d2085425c049629b80956d3c618971
I've been using this and it works best .forEach()
//count
var expecting = myArray.length;
myArray.forEach(function(item){
//do logic here
var item = item
//when iteration done
if (--expecting === 0) {
console.log('all done!');
}
})

How to call a JavaScript function sequentially in a for loop?

I want to pass each item into a function that take times.
But seems that the JS function is asynchronized.
How can I call the function sequentially ? (Pass next item to function after the previous done)
function main() {
for (var i = 0; i < n ; i++) {
doSomething(myArray[i]);
}
}
function doSomething(item) {
// do something take time
}
My solution is call the function recusively.
But I want to know is there a different way to solve this issue ? Thanks.
function main() {
doSomething(myArray, 0);
}
function doSomething(item, i) {
// do something take time
doSomething(myArray, i + 1);
}
In JavaScript, as of 2020, the for-loop is async/await aware. You can return a promise and then await that promise inside of the for loop. This causes the loop to execute in a series, even for long running operations.
function randomSleep(ms, seed=10) {
ms = ms * Math.random() * seed;
return new Promise((resolve, reject)=>setTimeout(resolve, ms));
}
async function doSomething(idx) {
// long running operations
const outcome = await randomSleep(500);
return idx;
}
const arrItems = ['a','b','c','d'];
for(let i=0,len=arrItems.length;i<len;i++) {
const result = await doSomething(i);
console.log("result is", result)
}
Read more about async await in for loops and forEach loops here https://zellwk.com/blog/async-await-in-loops/
if you want to pass next item to function after the previous done, you can try to use promise, just like this
var Q = require('q');
var promise;
main();
function main() {
promise = doSomethingPromise(0)
for (var i = 1; i < 10 ; i++) {
(function (i) {
promise = promise.then(function (res) {
return doSomethingPromise(res + ' ' + i)
});
})(i)
}
}
function doSomethingPromise (item) {
var d = Q.defer();
d.resolve(doSomething(item));
return d.promise;
}
function doSomething(item) {
// do something take time
console.log('doSomething', item);
return item;
}
it can make you function to be called by order.

Categories