Related
If you were given an array of async functions and the task is to create a class that takes this array and runs the functions as fast as possible with a constraint that only 15 functions can run at the same time, what would be a way to do that?
If there wasn't a constraint for 15 functions, I believe Promise.all would be the way to go.
Using just async/await and waiting for one function to resolve to add the next one is very slow as we must have to wait for 1 function to resolve until we can add another one and we can thus have a bottleneck function.
Adding 15 functions to array and running them with Promise.all and after that resolves, adding another 15 or the rest of them, is again, not very efficient as what we want to do is to call another function as soon as one of the functions resolves.
Any ideas?
Let's create a stack that has an async popAsync method:
const createAsyncStack = () => {
const stack = [];
const waitingConsumers = [];
const push = (v) => {
if (waitingConsumers.length > 0) {
const resolver = waitingConsumers.shift();
resolver && resolver(v);
}
else {
stack.push(v);
}
};
const popAsync = () => {
if (stack.length > 0) {
const queueItem = stack.pop();
return typeof queueItem !== 'undefined'
? Promise.resolve(queueItem)
: Promise.reject(Error('unexpected'));
}
else {
return new Promise((resolve) => waitingConsumers.push(resolve));
}
};
return [push, popAsync];
};
This means that any consumer calling popAsync will be returned a Promise that only completes if / when an item is available in the stack.
We can now use this stack as a "gatekeeper" for a simple higher-order function (i.e. a function that returns a function).
Say we only want to allow maxDOP (maximum degrees-of-parallelism) concurrent invocations of an async function, we push maxDOP tokens into the stack (here, I've used empty objects as the tokens), then require that in order to proceed, it is necessary to acquire a token from this stack. When our function call is finished, we return our token to the stack (using push), where that token can then be consumed by any waiting consumers.
const withMaxDOP = (f, maxDop) => {
const [push, popAsync] = createAsyncStack();
for (let x = 0; x < maxDop; ++x) {
push({});
}
return async (...args) => {
const token = await popAsync();
try {
return await f(...args);
}
finally {
push(token);
}
};
};
The function returns a new function that can be called in exactly the same way as the function that is supplied to it (i.e. is has the same signature).
Now, let's create a function that simply calls a supplied function with the supplied arguments:
const runAsync = (asyncFn, ...args) => asyncFn(...args);
and wrap it using the higher-order withMaxDOP function, which will return a new function with an identical signature to the wrapped function:
const limitedRunAsync = withMaxDOP(runAsync, 15);
Now we can use this function to call the functions in our array:
Promise.all(asyncFns.map(f => limitedRunAsync(f)))
.then((returnValues) => console.log("all finished", returnValues));
which will ensure that there are only ever 15 "in-flight" invocations ever permitted at one time.
See this runnable snippet for a full example:
const createAsyncStack = () => {
const stack = [];
const waitingConsumers = [];
const push = (v) => {
if (waitingConsumers.length > 0) {
const resolver = waitingConsumers.shift();
resolver && resolver(v);
} else {
stack.push(v);
}
};
const popAsync = () => {
if (stack.length > 0) {
const queueItem = stack.pop();
return typeof queueItem !== 'undefined' ? Promise.resolve(queueItem) : Promise.reject(Error('unexpected'));
} else {
return new Promise((resolve) => waitingConsumers.push(resolve));
}
};
return [push, popAsync];
};
const withMaxDOP = (f, maxDop) => {
const [push, popAsync] = createAsyncStack();
for (let x = 0; x < maxDop; ++x) {
push({});
}
return async(...args) => {
const token = await popAsync();
try {
return await f(...args);
} finally {
push(token);
}
};
};
const runAsync = (asyncFn, ...args) => asyncFn(...args);
const limitedRunAsync = withMaxDOP(runAsync, 15);
// set up an array of async functions
const delay = (durationMS) => new Promise((resolve) => setTimeout(() => resolve(), durationMS));
const asyncFns = [...Array(50)].map((_, i) => () => {
console.log("starting " + i);
return delay(Math.random() * 5000).then(v => {
console.log("finished " + i);
return i;
});
});
// ...then wrap and call them all at once
Promise.all(asyncFns.map(f => limitedRunAsync(f))).then((returnValues) => console.log("all finished", returnValues));
...and see this TypeScript Playground Link for a fully type-annotated version of the same code.
Here's something I whipped up in the last 20 minutes that should do the job
I'm sure if I thought about it I could probably do it without the Promise constructor, but ... 20 minutes is 20 minutes :p
Please, if someone can rewrite this without the Promise constructor, I'd love to see it - because in the back of my mind, I'm sure there is a way
Note, this will run regardless of rejections
Results will be either
result: actualResult
or
error: rejectionReason
So you can process results/rejections
function runPromises(arrayOfFunctions, maxLength) {
return new Promise(resolve => {
const queue = arrayOfFunctions.map((fn, index) => ({fn, index}));
const results = new Array(arrayOfFunctions.length);
let finished = 0;
const doQ = () => {
++finished;
if (queue.length) {
const {fn, index} = queue.shift();
fn()
.then(result => results[index] = {result})
.catch(error => results[index] = {error})
.finally(doQ);
} else {
if (finished === arrayOfFunctions.length) {
resolve(results);
}
}
};
queue.splice(0, maxLength).forEach(({fn, index}) => fn()
.then(result => results[index] = {result})
.catch(error => results[index] = {error})
.finally(doQ)
);
});
}
//
// demo and show that maximum 15 inflight requests
//
let inFlight = 0;
let maxInFlight = 0;
const fns = Array.from({length:50}, (_, i) => {
return () => new Promise(resolve => {
++inFlight;
maxInFlight = Math.max(inFlight, maxInFlight);
setTimeout(() => {
--inFlight;
resolve(i);
}, Math.random() * 200 + 100,)
});
});
runPromises(fns, 15).then(results => console.log(maxInFlight, JSON.stringify(results)));
Suppose I have some async iterable objects like this:
const a = {
[Symbol.asyncIterator]: async function * () {
yield 'a';
await sleep(1000);
yield 'b';
await sleep(2000);
yield 'c';
},
};
const b = {
[Symbol.asyncIterator]: async function * () {
await sleep(6000);
yield 'i';
yield 'j';
await sleep(2000);
yield 'k';
},
};
const c = {
[Symbol.asyncIterator]: async function * () {
yield 'x';
await sleep(2000);
yield 'y';
await sleep(8000);
yield 'z';
await sleep(10000);
throw new Error('You have gone too far! ');
},
};
And for completeness:
// Promisified sleep function
const sleep = ms => new Promise((resolve, reject) => {
setTimeout(() => resolve(ms), ms);
});
Now, suppose I can concat them like this:
const abcs = async function * () {
yield * a;
yield * b;
yield * c;
};
The (first 9) items yielded will be:
(async () => {
const limit = 9;
let i = 0;
const xs = [];
for await (const x of abcs()) {
xs.push(x);
i++;
if (i === limit) {
break;
}
}
console.log(xs);
})().catch(error => console.error(error));
// [ 'a', 'b', 'c', 'i', 'j', 'k', 'x', 'y', 'z' ]
But imagine that I do not care about the order, that a, b and c yield at different speeds, and that I want to yield as quickly as possible.
How can I rewrite this loop so that xs are yielded as soon as possible, ignoring order?
It is also possible that a, b or c are infinite sequences, so the solution must not require all elements to be buffered into an array.
There is no way to write this with a loop statement. async/await code always executes sequentially, to do things concurrently you need to use promise combinators directly. For plain promises, there's Promise.all, for async iterators there is nothing (yet) so we need to write it on our own:
async function* combine(iterable) {
const asyncIterators = Array.from(iterable, o => o[Symbol.asyncIterator]());
const results = [];
let count = asyncIterators.length;
const never = new Promise(() => {});
function getNext(asyncIterator, index) {
return asyncIterator.next().then(result => ({
index,
result,
}));
}
const nextPromises = asyncIterators.map(getNext);
try {
while (count) {
const {index, result} = await Promise.race(nextPromises);
if (result.done) {
nextPromises[index] = never;
results[index] = result.value;
count--;
} else {
nextPromises[index] = getNext(asyncIterators[index], index);
yield result.value;
}
}
} finally {
for (const [index, iterator] of asyncIterators.entries())
if (nextPromises[index] != never && iterator.return != null)
iterator.return();
// no await here - see https://github.com/tc39/proposal-async-iteration/issues/126
}
return results;
}
Notice that combine does not support passing values into next or cancellation through .throw or .return.
You can call it like
(async () => {
for await (const x of combine([a, b, c])) {
console.log(x);
}
})().catch(console.error);
If I change abcs to accept the generators to process, I come up with this, see inline comments:
const abcs = async function * (...gens) {
// Worker function to queue up the next result
const queueNext = async (e) => {
e.result = null; // Release previous one as soon as possible
e.result = await e.it.next();
return e;
};
// Map the generators to source objects in a map, get and start their
// first iteration
const sources = new Map(gens.map(gen => [
gen,
queueNext({
key: gen,
it: gen[Symbol.asyncIterator]()
})
]));
// While we still have any sources, race the current promise of
// the sources we have left
while (sources.size) {
const winner = await Promise.race(sources.values());
// Completed the sequence?
if (winner.result.done) {
// Yes, drop it from sources
sources.delete(winner.key);
} else {
// No, grab the value to yield and queue up the next
// Then yield the value
const {value} = winner.result;
sources.set(winner.key, queueNext(winner));
yield value;
}
}
};
Live Example:
// Promisified sleep function
const sleep = ms => new Promise((resolve, reject) => {
setTimeout(() => resolve(ms), ms);
});
const a = {
[Symbol.asyncIterator]: async function * () {
yield 'a';
await sleep(1000);
yield 'b';
await sleep(2000);
yield 'c';
},
};
const b = {
[Symbol.asyncIterator]: async function * () {
await sleep(6000);
yield 'i';
yield 'j';
await sleep(2000);
yield 'k';
},
};
const c = {
[Symbol.asyncIterator]: async function * () {
yield 'x';
await sleep(2000);
yield 'y';
await sleep(8000);
yield 'z';
},
};
const abcs = async function * (...gens) {
// Worker function to queue up the next result
const queueNext = async (e) => {
e.result = null; // Release previous one as soon as possible
e.result = await e.it.next();
return e;
};
// Map the generators to source objects in a map, get and start their
// first iteration
const sources = new Map(gens.map(gen => [
gen,
queueNext({
key: gen,
it: gen[Symbol.asyncIterator]()
})
]));
// While we still have any sources, race the current promise of
// the sources we have left
while (sources.size) {
const winner = await Promise.race(sources.values());
// Completed the sequence?
if (winner.result.done) {
// Yes, drop it from sources
sources.delete(winner.key);
} else {
// No, grab the value to yield and queue up the next
// Then yield the value
const {value} = winner.result;
sources.set(winner.key, queueNext(winner));
yield value;
}
}
};
(async () => {
console.log("start");
for await (const x of abcs(a, b, c)) {
console.log(x);
}
console.log("done");
})().catch(error => console.error(error));
.as-console-wrapper {
max-height: 100% !important;
}
This is a complicated task, so I’m going to break it up into individual parts:
Step 1: logging each value from each async iterable to the console
Before we even think about creating an async iterator we should first consider the task of simply logging each value from each iterator to the console as they arrive. As with most concurrent tasks in javascript, this involves calling multiple async functions and awaiting their results with Promise.all.
function merge(iterables) {
return Promise.all(
Array.from(iterables).map(async (iter) => {
for await (const value of iter) {
console.log(value);
}
}),
);
}
// a, b and c are the async iterables defined in the question
merge([a, b, c]); // a, x, b, y, c, i, j, k, z, Error: you have gone too far!
CodeSandbox link: https://codesandbox.io/s/tender-ives-4hijy?fontsize=14
The merge function logs values from each iterator, but is mostly useless; it returns a promise which fulfills to an array of undefined when all iterators finish.
Step 2: Replacing the merge function with a merge async generator
The next step is to replace console.log calls with calls to a function which pushes to a parent async iterator. To do this with an async generator, we need a little bit more code, because the only way to “push” a value onto an async generator is with the yield operator, which can’t be used in child function scopes. The solution is to create two queues, a push queue and a pull queue. Next, we define a push function which either pushes to the push queue if there are no pending pulls, or enqueues a value to be pulled later. Finally, we have to perpetually yield either values from the push queue if it has values, or promises which enqueue a resolve function to be called by push later. Here’s the code:
async function *merge(iterables) {
// pushQueue and pullQueue will never both contain values at the same time.
const pushQueue = [];
const pullQueue = [];
function push(value) {
if (pullQueue.length) {
pullQueue.pop()(value);
} else {
pushQueue.unshift(value);
}
}
// the merge code from step 1
const finishP = Promise.all(
Array.from(iterables).map(async (iter) => {
for await (const value of iter) {
push(value);
}
}),
);
while (true) {
if (pushQueue.length) {
yield pushQueue.pop();
} else {
// important to note that yield in an async generator implicitly awaits promises.
yield new Promise((resolve) => {
pullQueue.unshift(resolve);
});
}
}
}
// code from the question
(async () => {
const limit = 9;
let i = 0;
const xs = [];
for await (const x of merge([a, b, c])) {
xs.push(x);
console.log(x);
i++;
if (i === limit) {
break;
}
}
console.log(xs); // ["a", "x", "b", "y", "c", "i", "j", "k", "z"]
})().catch(error => console.error(error));
CodeSandbox link: https://codesandbox.io/s/misty-cookies-du1eg
This almost works! If you run the code, you’ll notice that the xs is correctly printed, but the break statement is not respected, and values continue to be pulled from child iterators, causing the error thrown in c to be thrown, resulting in an unhandled promise rejection. Also note that we don’t do anything with the result of the Promise.all call. Ideally, when the finishP promise settles, the generator should be returned. We need just a little bit more code to make sure that 1. the child iterators are returned when the parent iterator is returned (with a break statement in a for await loop, for instance), and 2. the parent iterator is returned when all child iterators return.
Step 3: stopping each child iterator when the parent iterator is returned, and the parent iterator when every child has returned.
To make sure each child async iterable is correctly returned when the parent async generator is returned, we can use a finally block to listen for the completion of the parent async generator. And to make sure the parent generator is returned when the child iterators return, we can race yielded promises against the finishP promise.
async function *merge(iterables) {
const pushQueue = [];
const pullQueue = [];
function push(value) {
if (pullQueue.length) {
pullQueue.pop()(value);
} else {
pushQueue.unshift(value);
}
}
// we create a promise to race calls to iter.next
let stop;
const stopP = new Promise((resolve) => (stop = resolve));
let finished = false;
const finishP = Promise.all(
Array.from(iterables).map(async (iter) => {
// we use the iterator interface rather than the iterable interface
iter = iter[Symbol.asyncIterator]();
try {
while (true) {
// because we can’t race promises with for await, we have to call iter.next manually
const result = await Promise.race([stopP, iter.next()]);
if (!result || result.done) {
return;
}
push(result.value);
}
} finally {
// we should be a good citizen and return child iterators
await iter.return && iter.return();
}
}),
).finally(() => (finished = true));
try {
while (!finished) {
if (pushQueue.length) {
yield pushQueue.pop();
} else {
const value = await Promise.race([
new Promise((resolve) => {
pullQueue.unshift(resolve);
}),
finishP,
]);
if (!finished) {
yield value;
}
}
}
// we await finishP to make the iterator catch any promise rejections
await finishP;
} finally {
stop();
}
}
CodeSandbox link: https://codesandbox.io/s/vigilant-leavitt-h247u
There are some things we still need to do before this code is production ready. For instance, values are pulled from the child iterators continuously, without waiting for the parent iterator to pull them. This, combined with the fact that pushQueue is an unbounded array, can cause memory leaks if the parent iterator pulls values at a slower pace than the child iterators produces them.
Additionally, the merge iterator returns undefined as its final value, but you might want the final value to be the final value from the last-completing child iterator.
If you’re looking for a small, focused library which has a merge function like the one above which covers some more use-cases and edge-cases, check out Repeater.js, which I wrote. It defines the static method Repeater.merge, which does what I described above. It also provides a clean API for turning callback-based APIs into promises and other combinator static methods to combine async iterators in other ways.
In case anyone finds it useful, here's a typescript version of the currently accepted answer:
const combineAsyncIterables = async function* <T>(
asyncIterables: AsyncIterable<T>[],
): AsyncGenerator<T> {
const asyncIterators = Array.from(asyncIterables, (o) =>
o[Symbol.asyncIterator](),
);
const results = [];
let count = asyncIterators.length;
const never: Promise<never> = new Promise(noOp);
const getNext = (asyncIterator: AsyncIterator<T>, index: number) =>
asyncIterator.next().then((result) => ({ index, result }));
const nextPromises = asyncIterators.map(getNext);
try {
while (count) {
const { index, result } = await Promise.race(nextPromises);
if (result.done) {
nextPromises[index] = never;
results[index] = result.value;
count--;
} else {
nextPromises[index] = getNext(asyncIterators[index], index);
yield result.value;
}
}
} finally {
for (const [index, iterator] of asyncIterators.entries()) {
if (nextPromises[index] != never && iterator.return != null) {
// no await here - see https://github.com/tc39/proposal-async-iteration/issues/126
void iterator.return();
}
}
}
return results;
};
I solved this using async generators. (I wish I'd find this question a few days ago, would save me some time)
Will gladly hear opinion and criticism.
async function* mergen(...gens) {
const promises = gens.map((gen, index) =>
gen.next().then(p => ({...p, gen}))
);
while (promises.length > 0) {
yield race(promises).then(({index, value: {value, done, gen}}) => {
promises.splice(index, 1);
if (!done)
promises.push(
gen.next().then(({value: newVal, done: newDone}) => ({
value: newVal,
done: newDone,
gen
}))
);
return value;
});
}
};
// Needed to implement race to provide index of resolved promise
function race(promises) {
return new Promise(resolve =>
promises.forEach((p, index) => {
p.then(value => {
resolve({index, value});
});
})
);
}
It took me a bunch of time to find and I got so excited I put it in a npm package :) https://www.npmjs.com/package/mergen
Solution: IxJS
We can use The Interactive Extensions for JavaScript (IxJS) (docs) to easily achieve that:
import { merge } from 'ix/asynciterable'
const d = merge(a, b, c)
for await (const i of d) {
console.info('merged:', i)
}
Will get the result:
$ ./src/t.ts
merged a
merged x
merged b
merged y
merged c
merged i
merged j
merged k
merged z
Error: You have gone too far!
at Object.[Symbol.asyncIterator]
Full Code Example
const sleep = ms => new Promise((resolve) => {
setTimeout(() => resolve(ms), ms);
});
const a = {
[Symbol.asyncIterator]: async function * () {
yield 'a';
await sleep(1000);
yield 'b';
await sleep(2000);
yield 'c';
},
};
const b = {
[Symbol.asyncIterator]: async function * () {
await sleep(6000);
yield 'i';
yield 'j';
await sleep(2000);
yield 'k';
},
};
const c = {
[Symbol.asyncIterator]: async function * () {
yield 'x';
await sleep(2000);
yield 'y';
await sleep(8000);
yield 'z';
await sleep(10000);
throw new Error('You have gone too far! ');
},
};
const d = IxAsynciterable.merge(a, b, c)
async function main () {
for await (const i of d) {
console.info('merged', i)
}
}
main().catch(console.error)
<script src="https://unpkg.com/ix#4.5.2/Ix.dom.asynciterable.es2015.min.js"></script>
I hope I understood your question correctly, here's how I'd approach it:
let results = [];
Promise.all([ a, b, c ].map(async function(source) {
for await (let item of source) {
results.push(item);
}
}))
.then(() => console.log(results));
I tried it with three normal arrays:
var a = [ 1, 2, 3 ];
var b = [ 4, 5, 6 ];
var c = [ 7, 8, 9 ];
And it resulted in [1, 4, 7, 2, 5, 8, 3, 6, 9].
How can I have a JavaScript function let's say piper() which takes several functions as its arguments and it returns a new function that will pass its argument to the first function, then pass the result to the second, then
pass the result of the second to the third, and so on, finally returning the output of the last function.
Something like piper(foo, fee, faa)(10, 20, 30) would be equivalent to calling faa(fee(foo(10,20,30))).
ps:
It was a part of an interview, that I did few days ago.
For an arbritrary number of functions you could use this ES6 function:
function piper(...fs) {
return (...args) => fs.reduce((args,f) => [f.apply(this,args)],args)[0];
}
// Example call:
var result = piper(Math.min, Math.abs, Math.sqrt)(16, -9, 0)
// Output result:
console.log(result);
The same in ES5 syntax:
function piper(/* functions */) {
var fs = [].slice.apply(arguments);
return function (/* arguments */) {
return fs.reduce(function (args,f) {
return [f.apply(this,args)];
}.bind(this), [].slice.apply(arguments))[0];
}.bind(this);
}
// Example call:
var result = piper(Math.min, Math.abs, Math.sqrt)(16, -9, 0)
// Output result:
console.log(result);
Enjoy. Pure ES5 solution. Preserves this.
function piper(){
var i = arguments.length,
piped = arguments[ --i ];
while( --i >= 0 ){
piped = pipeTwo( arguments[ i ], piped );
}
return piped;
}
function pipeTwo( a, b ){
return function(){
return a.call( this, b.apply( this, arguments ) );
}
}
Or, if you want the fancy solution.
function piperES6( ...args ){
return args.reverse().reduce( pipeTwo );
}
Loops can be reversed depending on the desired direction.
Very similar to #trincot's answer (preserves context), but composes in the correct order and is marginally faster since it does not create intermediary arrays:
const piper = (...steps) => function(...arguments) {
let value = steps[0].apply(this, arguments);
for (let i = 1; i < steps.length; ++i) {
value = steps[i].call(this, value);
}
return value;
};
// Usage:
let p = piper(
x => x + 1,
x => x * 2,
x => x - 1
);
console.log(p(2)); // 5
Here is an alternative answer involving method chaining. I shall use ES6, though of course this can be transpiled to ES5. On benefit of this solution is that is has a very succinct TypeScript counterpart with perfect typeability.
class Pipe {
constructor(value) {
this.value = value;
}
then(f) {
return new Pipe(f(this.value));
}
}
const pipe = value => new Pipe(value);
// Example
const double = x => 2 * x;
pipe(42).then(double).then(console.log); // 84
const result = pipe(42).then(double).then(double).value;
console.log(result); // 168
A simple solution based on JS higher-order functions usage:
function pipe(...rest) {
return x => rest.reduce((y, f) => f(y), x);
}
Usage:
pipe((a) => a + 1, (a) => a * 2)(3) // 8
pipe((a) => a + 1, (a) => a * 2)(2) // 2
function f(f1, f2, f3){
return (args => f3(f2(f1(args))));
}
I think what you are trying to do is chaining.
var funct={
total:0,
add:function(a) {
console.log(funct.total,funct.total+a);
funct.total+=a;
return funct;
}
};
funct.add(5).add(6).add(9);
Sometimes I need to wait for a .forEach() method to finish, mostly on 'loader' functions. This is the way I do that:
$q.when(array.forEach(function(item){
//iterate on something
})).then(function(){
//continue with processing
});
I can't help but feel that this isn't the best way to wait for a .forEach() to finish. What is the best way to do this?
If there is no asynchronous code inside the forEach, forEach is not asynchronous, for example in this code:
array.forEach(function(item){
//iterate on something
});
alert("Foreach DONE !");
you will see the alert after forEach finished.
Otherwise (You have something asynchronous inside), you can wrap the forEach loop in a Promise:
var bar = new Promise((resolve, reject) => {
foo.forEach((value, index, array) => {
console.log(value);
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
});
Credit: #rolando-benjamin-vaz-ferreira
The quickest way to make this work using ES6 would be just to use a for..of loop.
const myAsyncLoopFunction = async (array) => {
const allAsyncResults = []
for (const item of array) {
const asyncResult = await asyncFunction(item)
allAsyncResults.push(asyncResult)
}
return allAsyncResults
}
Or you could loop over all these async requests in parallel using Promise.all() like this:
const myAsyncLoopFunction = async (array) => {
const promises = array.map(asyncFunction)
await Promise.all(promises)
console.log(`All async tasks complete!`)
}
var foo = [1,2,3,4,5,6,7,8,9,10];
If you're actually doing async stuff inside the loop, you can wrap it in a promise ...
var bar = new Promise((resolve, reject) => {
foo.forEach((value, index, array) => {
console.log(value);
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
});
If you have an async task inside a loop and you want to wait. you can use for await
for await (const i of images) {
let img = await uploadDoc(i);
};
let x = 10; //this executes after
Use for of instead of forEach. Like this:
for (const item of array) {
//do something
}
console.log("finished");
"finished" will be logged after finishing the loop.
forEach() doesn't return anything, so a better practice would be map() + Promise.all()
var arr = [1, 2, 3, 4, 5, 6]
var doublify = (ele) => {
return new Promise((res, rej) => {
setTimeout(() => {
res(ele * 2)
}, Math.random() ); // Math.random returns a random number from 0~1
})
}
var promises = arr.map(async (ele) => {
// do some operation on ele
// ex: var result = await some_async_function_that_return_a_promise(ele)
// In the below I use doublify() to be such an async function
var result = await doublify(ele)
return new Promise((res, rej) => {res(result)})
})
Promise.all(promises)
.then((results) => {
// do what you want on the results
console.log(results)
})
A universal solution for making sure that all forEach() elements finished execution.
const testArray = [1,2,3,4]
let count = 0
await new Promise( (resolve) => {
testArray.forEach( (num) => {
try {
//some real logic
num = num * 2
} catch (e) {
// error handling
console.log(e)
} fanally {
// most important is here
count += 1
if (count == testArray.length) {
resolve()
}
}
})
})
The idea is same with the answer using index to count. But in real case, if error happened, the index way cannot count correctly. So the solution is more robust.
Thx
const array = [1, 2, 3];
const results = [];
let done = 0;
const asyncFunction = (item, callback) =>
setTimeout(() => callback(item * 10), 100 - item * 10);
new Promise((resolve, reject) => {
array.forEach((item) => {
asyncFunction(item, (result) => {
results.push(result);
done++;
if (done === array.length) resolve();
});
});
}).then(() => {
console.log(results); // [30, 20, 10]
});
// or
// promise = new Promise(...);
// ...
// promise.then(...);
The order of results in the "results" array can be different than the order of items in the original array, depending on the time when the asyncFunction() finishes for each of the items.
Alter and check a counter at the end of every possible unique branch of code, including callbacks. Example:
const fs = require('fs');
/**
* #description Delete files older than 1 day
* #param {String} directory - The directory to purge
* #return {Promise}
*/
async function purgeFiles(directory) {
const maxAge = 24*3600000;
const now = Date.now();
const cutoff = now-maxAge;
let filesPurged = 0;
let filesProcessed = 0;
let purgedSize = 0;
await new Promise( (resolve, reject) => {
fs.readdir(directory, (err, files) => {
if (err) {
return reject(err);
}
if (!files.length) {
return resolve();
}
files.forEach( file => {
const path = `${directory}/${file}`;
fs.stat(path, (err, stats)=> {
if (err) {
console.log(err);
if (++filesProcessed === files.length) resolve();
}
else if (stats.isFile() && stats.birthtimeMs < cutoff) {
const ageSeconds = parseInt((now-stats.birthtimeMs)/1000);
fs.unlink(path, error => {
if (error) {
console.log(`Deleting file failed: ${path} ${error}`);
}
else {
++filesPurged;
purgedSize += stats.size;
console.log(`Deleted file with age ${ageSeconds} seconds: ${path}`);
}
if (++filesProcessed === files.length) resolve();
});
}
else if (++filesProcessed === files.length) resolve();
});
});
});
});
console.log(JSON.stringify({
directory,
filesProcessed,
filesPurged,
purgedSize,
}));
}
// !!DANGER!! Change this line! (intentional syntax error in ,')
const directory = ,'/tmp'; // !!DANGER!! Changeme
purgeFiles(directory).catch(error=>console.log(error));
I'm not sure of the efficiency of this version compared to others, but I used this recently when I had an asynchronous function inside of my forEach(). It does not use promises, mapping, or for-of loops:
// n'th triangular number recursion (aka factorial addition)
function triangularNumber(n) {
if (n <= 1) {
return n
} else {
return n + triangularNumber(n-1)
}
}
// Example function that waits for each forEach() iteraction to complete
function testFunction() {
// Example array with values 0 to USER_INPUT
var USER_INPUT = 100;
var EXAMPLE_ARRAY = Array.apply(null, {length: USER_INPUT}).map(Number.call, Number) // [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, n_final... ] where n_final = USER_INPUT-1
// Actual function used with whatever actual array you have
var arrayLength = EXAMPLE_ARRAY.length
var countMax = triangularNumber(arrayLength);
var counter = 0;
EXAMPLE_ARRAY.forEach(function(entry, index) {
console.log(index+1); // show index for example (which can sometimes return asynchrounous results)
counter += 1;
if (triangularNumber(counter) == countMax) {
// function called after forEach() is complete here
completionFunction();
} else {
// example just to print counting values when max not reached
// else would typically be excluded
console.log("Counter index: "+counter);
console.log("Count value: "+triangularNumber(counter));
console.log("Count max: "+countMax);
}
});
}
testFunction();
function completionFunction() {
console.log("COUNT MAX REACHED");
}
I had to deal with the same problem (forEach using multiple promises inside) and none of the solutions presented at the current date were helpful for me. So I implemented a check array, were each promise updates its complete status. We have a general promise that wraps the process. We only resolve the general promise when each promise completed. Snippet code:
function WaitForEachToResolve(fields){
var checked_fields = new Array(fields.length).fill(0);
const reducer = (accumulator, currentValue) => accumulator + currentValue;
return new Promise((resolve, reject) => {
Object.keys(fields).forEach((key, index, array) => {
SomeAsyncFunc(key)
.then((result) => {
// some result post process
checked_fields[index] = 1;
if (checked_fields.reduce(reducer) === checked_fields.length)
resolve();
})
.catch((err) => {
reject(err);
});
}
)}
}
I like to use async-await instead of .then() syntax so for asynchronous processing of data, modified the answer of #Ronaldo this way -
let finalData = [];
var bar = new Promise(resolve => {
foo.forEach((value, index) => {
const dataToGet = await abcService.getXyzData(value);
finalData[index].someKey = dataToGet.thatOtherKey;
// any other processing here
if (finalData[dataToGet.length - 1].someKey) resolve();
});
});
await Promise.all([bar]);
console.log(`finalData: ${finalData}`);
NOTE: I've modified the if condition where it resolves the promise to meet my conditions. You can do the same in your case.
You can use this, because we are using async/await inside the forEach loop. You can use your own logic inside the loop.
let bar = new Promise((resolve, reject) => {
snapshot.forEach(async (doc) => {
"""Write your own custom logic and can use async/await
"""
const result = await something()
resolve(result);
});
});
let test = []
test.push(bar)
let concepts = await Promise.all(test);
console.log(concepts);
For simple compare code i like use for statement.
doit();
function doit() {
for (var i = 0; i < $('span').length; i++) {
console.log(i,$('span').eq(i).text() );
if ( $('span').eq(i).text() == "Share a link to this question" ) { // span number 59
return;
}
}
alert('never execute');
}
If there are async (observable) method calls inside the for loop, you could use the following method:
await players.reduce(async (a, player) => {
// Wait for the previous item to finish processing
await a;
// Process this item
await givePrizeToPlayer(player);
}, Promise.resolve());
Check here: https://gist.github.com/joeytwiddle/37d2085425c049629b80956d3c618971
I've been using this and it works best .forEach()
//count
var expecting = myArray.length;
myArray.forEach(function(item){
//do logic here
var item = item
//when iteration done
if (--expecting === 0) {
console.log('all done!');
}
})
I want to pass each item into a function that take times.
But seems that the JS function is asynchronized.
How can I call the function sequentially ? (Pass next item to function after the previous done)
function main() {
for (var i = 0; i < n ; i++) {
doSomething(myArray[i]);
}
}
function doSomething(item) {
// do something take time
}
My solution is call the function recusively.
But I want to know is there a different way to solve this issue ? Thanks.
function main() {
doSomething(myArray, 0);
}
function doSomething(item, i) {
// do something take time
doSomething(myArray, i + 1);
}
In JavaScript, as of 2020, the for-loop is async/await aware. You can return a promise and then await that promise inside of the for loop. This causes the loop to execute in a series, even for long running operations.
function randomSleep(ms, seed=10) {
ms = ms * Math.random() * seed;
return new Promise((resolve, reject)=>setTimeout(resolve, ms));
}
async function doSomething(idx) {
// long running operations
const outcome = await randomSleep(500);
return idx;
}
const arrItems = ['a','b','c','d'];
for(let i=0,len=arrItems.length;i<len;i++) {
const result = await doSomething(i);
console.log("result is", result)
}
Read more about async await in for loops and forEach loops here https://zellwk.com/blog/async-await-in-loops/
if you want to pass next item to function after the previous done, you can try to use promise, just like this
var Q = require('q');
var promise;
main();
function main() {
promise = doSomethingPromise(0)
for (var i = 1; i < 10 ; i++) {
(function (i) {
promise = promise.then(function (res) {
return doSomethingPromise(res + ' ' + i)
});
})(i)
}
}
function doSomethingPromise (item) {
var d = Q.defer();
d.resolve(doSomething(item));
return d.promise;
}
function doSomething(item) {
// do something take time
console.log('doSomething', item);
return item;
}
it can make you function to be called by order.