I was able to fix this by changing the contents of the .push() of allTeams (second to last instance of .push() in the code)
The final return statement should return an array of objects. (allObjectTeams, right after await browser.close()), BUT it returns undefined
Placing the variable right above the return statement still does not solve the issue.
Commenting the biggest loop (the one that iterates over blocks.length) makes that final return statement work but I don't know what part of said loop is messing it up, Commenting out anything that does an action to this said variable doesn't solve the issue
makeTeam() is an instance of recursion but it looks fine to me. Essentially taking a random set of elements from the people array and placing it as a property of an object, then to add than object to an array,
The function also removes the specific elements that were chosen from that array so if there are still more, It calls itself again. A heads-up but I don’t think it does anything.
I am using Puppeteer.
Code (JS):
const start = async () => {
let teamSize = 2
let allObjectTeams = [] //This is the variable I am trying to return
let allTeams = []
const browser = await require("puppeteer").launch()
const page = await browser.newPage()
await page.goto("https://www.when2meet.com/?16521246-5ySm2")
const blocks = await page.$$("#GroupGridSlots > div")
for (let i = 0; i < blocks.length; i++) { // This is the loop I'm talking about, blocks.length is finite. This loop does not go on forever for the return statement never to be called.
await blocks[i].hover()
const team = await page.$eval("#Available", ((availableList, teamSize) => {
const people = availableList.innerHTML.split("<br>")
people.pop()
function getMultipleRandom(arr, num) { // Grabs random elements from array
const shuffled = [...arr].sort(() => 0.5 - Math.random());
return shuffled.slice(0, num);
}
if (people.length == teamSize) {
const availableTeam = {}
availableTeam.people = people
return availableTeam
} else if (people.length > teamSize) {
const teamArray = []
const makeTeam = (currentLength) => {
const availableTeam = {}
const teamPeople = getMultipleRandom(people, teamSize)
availableTeam.people = teamPeople
teamArray.push(availableTeam)
if ((currentLength - teamSize) > teamSize) {
teamArray[teamArray.length - 1].people.forEach(person => {
people.forEach(availablePerson => {
if (availablePerson == person) {
people.splice(people.indexOf(person), 1)
}
})
})
makeTeam(currentLength - teamSize)
}
}
makeTeam(people.length)
return teamArray // returns array of Objects, But THIS is not the one
} else {
return
}
}), teamSize)
if (!team) {
} else if (!Array.isArray(team) && allTeams.includes(team.people.toString())) {
return
} else if (Array.isArray(team)) {
team.forEach(t => {
allTeams.forEach(registeredTeam => {
if (t == registeredTeam) {
team.splice(team.indexOf(t), 1)
}
})
})
for (let i = 0; i < team.length; i++) {
allTeams.push(team[i].toString())
const availableDate = await page.$eval("#AvailableDate", (availableDate => {
return availableDate.textContent.split("Sunday ").pop()
}))
team[i].time = availableDate
allObjectTeams.push(team[i])
}
} else if (team.hasOwnProperty("people")) {
allTeams.push(team.people.toString()) // this was changed to allTeams.push(team.toString())
const availableDate = await page.$eval("#AvailableDate", (availableDate => {
return availableDate.textContent.split("Sunday ").pop()
}))
team.time = availableDate
allObjectTeams.push(team)
}
}
await browser.close()
return allObjectTeams // This return statement returns undefined. THE OTHERS DO NOT
}
start().then((arr) => {
console.log(arr)
})
If you were given an array of async functions and the task is to create a class that takes this array and runs the functions as fast as possible with a constraint that only 15 functions can run at the same time, what would be a way to do that?
If there wasn't a constraint for 15 functions, I believe Promise.all would be the way to go.
Using just async/await and waiting for one function to resolve to add the next one is very slow as we must have to wait for 1 function to resolve until we can add another one and we can thus have a bottleneck function.
Adding 15 functions to array and running them with Promise.all and after that resolves, adding another 15 or the rest of them, is again, not very efficient as what we want to do is to call another function as soon as one of the functions resolves.
Any ideas?
Let's create a stack that has an async popAsync method:
const createAsyncStack = () => {
const stack = [];
const waitingConsumers = [];
const push = (v) => {
if (waitingConsumers.length > 0) {
const resolver = waitingConsumers.shift();
resolver && resolver(v);
}
else {
stack.push(v);
}
};
const popAsync = () => {
if (stack.length > 0) {
const queueItem = stack.pop();
return typeof queueItem !== 'undefined'
? Promise.resolve(queueItem)
: Promise.reject(Error('unexpected'));
}
else {
return new Promise((resolve) => waitingConsumers.push(resolve));
}
};
return [push, popAsync];
};
This means that any consumer calling popAsync will be returned a Promise that only completes if / when an item is available in the stack.
We can now use this stack as a "gatekeeper" for a simple higher-order function (i.e. a function that returns a function).
Say we only want to allow maxDOP (maximum degrees-of-parallelism) concurrent invocations of an async function, we push maxDOP tokens into the stack (here, I've used empty objects as the tokens), then require that in order to proceed, it is necessary to acquire a token from this stack. When our function call is finished, we return our token to the stack (using push), where that token can then be consumed by any waiting consumers.
const withMaxDOP = (f, maxDop) => {
const [push, popAsync] = createAsyncStack();
for (let x = 0; x < maxDop; ++x) {
push({});
}
return async (...args) => {
const token = await popAsync();
try {
return await f(...args);
}
finally {
push(token);
}
};
};
The function returns a new function that can be called in exactly the same way as the function that is supplied to it (i.e. is has the same signature).
Now, let's create a function that simply calls a supplied function with the supplied arguments:
const runAsync = (asyncFn, ...args) => asyncFn(...args);
and wrap it using the higher-order withMaxDOP function, which will return a new function with an identical signature to the wrapped function:
const limitedRunAsync = withMaxDOP(runAsync, 15);
Now we can use this function to call the functions in our array:
Promise.all(asyncFns.map(f => limitedRunAsync(f)))
.then((returnValues) => console.log("all finished", returnValues));
which will ensure that there are only ever 15 "in-flight" invocations ever permitted at one time.
See this runnable snippet for a full example:
const createAsyncStack = () => {
const stack = [];
const waitingConsumers = [];
const push = (v) => {
if (waitingConsumers.length > 0) {
const resolver = waitingConsumers.shift();
resolver && resolver(v);
} else {
stack.push(v);
}
};
const popAsync = () => {
if (stack.length > 0) {
const queueItem = stack.pop();
return typeof queueItem !== 'undefined' ? Promise.resolve(queueItem) : Promise.reject(Error('unexpected'));
} else {
return new Promise((resolve) => waitingConsumers.push(resolve));
}
};
return [push, popAsync];
};
const withMaxDOP = (f, maxDop) => {
const [push, popAsync] = createAsyncStack();
for (let x = 0; x < maxDop; ++x) {
push({});
}
return async(...args) => {
const token = await popAsync();
try {
return await f(...args);
} finally {
push(token);
}
};
};
const runAsync = (asyncFn, ...args) => asyncFn(...args);
const limitedRunAsync = withMaxDOP(runAsync, 15);
// set up an array of async functions
const delay = (durationMS) => new Promise((resolve) => setTimeout(() => resolve(), durationMS));
const asyncFns = [...Array(50)].map((_, i) => () => {
console.log("starting " + i);
return delay(Math.random() * 5000).then(v => {
console.log("finished " + i);
return i;
});
});
// ...then wrap and call them all at once
Promise.all(asyncFns.map(f => limitedRunAsync(f))).then((returnValues) => console.log("all finished", returnValues));
...and see this TypeScript Playground Link for a fully type-annotated version of the same code.
Here's something I whipped up in the last 20 minutes that should do the job
I'm sure if I thought about it I could probably do it without the Promise constructor, but ... 20 minutes is 20 minutes :p
Please, if someone can rewrite this without the Promise constructor, I'd love to see it - because in the back of my mind, I'm sure there is a way
Note, this will run regardless of rejections
Results will be either
result: actualResult
or
error: rejectionReason
So you can process results/rejections
function runPromises(arrayOfFunctions, maxLength) {
return new Promise(resolve => {
const queue = arrayOfFunctions.map((fn, index) => ({fn, index}));
const results = new Array(arrayOfFunctions.length);
let finished = 0;
const doQ = () => {
++finished;
if (queue.length) {
const {fn, index} = queue.shift();
fn()
.then(result => results[index] = {result})
.catch(error => results[index] = {error})
.finally(doQ);
} else {
if (finished === arrayOfFunctions.length) {
resolve(results);
}
}
};
queue.splice(0, maxLength).forEach(({fn, index}) => fn()
.then(result => results[index] = {result})
.catch(error => results[index] = {error})
.finally(doQ)
);
});
}
//
// demo and show that maximum 15 inflight requests
//
let inFlight = 0;
let maxInFlight = 0;
const fns = Array.from({length:50}, (_, i) => {
return () => new Promise(resolve => {
++inFlight;
maxInFlight = Math.max(inFlight, maxInFlight);
setTimeout(() => {
--inFlight;
resolve(i);
}, Math.random() * 200 + 100,)
});
});
runPromises(fns, 15).then(results => console.log(maxInFlight, JSON.stringify(results)));
The program description
Create 3 functions:
FuncA – will receive a string and returns it’s length
FuncB – will receive an array of strings and returns their total lengths (using
funcA) after 2 seconds.
FuncC - will receive an array of arrays of strings and returns their total lengths
(using FuncB)
My solution is
function funcA(s)
{
return s.length
}
function funcB(arr)
{
return new Promise(resolve =>
{
setTimeout(() =>
{
let total = 0;
arr.forEach(element => {
total += funcA(element)
});
resolve(total)
},2000)
})
}
function funcC(arr)
{
return new Promise(resolve =>
{
let isFirst = true
//a <=> total
let total = arr.reduce(async (a,b) =>
{
if(isFirst) {
isFirst = false
return (await funcB(a) + await funcB(b))
}
else {//a <=> total
return (a + await funcB(b))
}
})
resolve(total)
})
}
The running is:
funcC([["aa","bbb","tyui"],["ccc"],["dfghj","aedtfr"]]).then(x => console.log(x))
The result is:
[object Promise]11
What is the problem?
This is really convoluted.
Don't put business logic in the setTimeout callback. Only resolve the promise, then do the work in a promise then callback or after an await.
Always pass an initial value to reduce! This will make it work with empty arrays, and it will obviate the need for that really weird isFirst logic.
total already is a promise. Don't unnecessarily wrap it in a new Promise!
These suggestions will lead to
function funcA(s) { return s.length }
function funcB(arr) {
return new Promise(resolve => {
setTimeout(resolve, 2000);
}).then(() => {
let total = 0;
arr.forEach(element => {
total += funcA(element)
});
return total;
});
}
function funcC(arr) {
return arr.reduce(async (a,b) => {
return await a + await funcB(b)
}, Promise.resolve(0))
}
However, reduce is not really suited for asynchronous work. You should rather use the looping approach in funcC, and use reduce in funcB where it fits much better:
async function funcB(arr) {
await new Promise(resolve => {
setTimeout(resolve, 2000);
});
return arr.reduce((total, element) => total + funcA(element), 0);
}
async function funcC(arr) {
let total = 0;
for (const b of arr) {
total += funcB(b);
}
return total;
}
[fixed the answer]
you shall do await a instead of a everywhere
function funcC(arr)
{
return new Promise(resolve =>
{
let isFirst = true
//a <=> total
let total = arr.reduce(async (a,b) =>
{
if(isFirst) {
isFirst = false
return (await funcB(await a) + await funcB(b))
}
else {//a <=> total
return (await a + await funcB(b))
}
})
resolve(total)
})
}
I have a function "promiseFunction" which returns a promise, which is resolved at a later time.
I need to call this function many times, however I only want a set number of executions of this function to happen at once.
The function calls some external single threaded c code on my computer, if I call too many instances of this code at once I crash my system, but if I call it sequentially 1 at a time it's very slow as only one thread of my cpu is doing any work.
So I came up with the code below, however it doesn't work. It will call the first 10 promises in parallel, but slowly it starts to call less and less promises at once, until it's only calling 1 promise at a time.
var totalNumberOfPromises = // total number times to run promiseFunction;
var promiseCounter = 0; // keep track of which promise call this is
for(w=0;w<10;w++){ // run 10 promises at a time
promiseFunction().then(function(resolve) {
loadNewPromise();
})
promiseCounter++;
}
function loadNewPromise(){
if(promiseCounter<totalNumberOfPromises){
promiseFunction().then(function(resolve) { loadNewPromise(); });
}else{
alert("Finished");
}
promiseCounter++;
}
Is there anything wrong with the code above that causes this behavior? And is there a standard way of doing this?
Here's a function I prepared earlier (I've used this for a few years now for just such a thing
const multiQueue = length => {
length = (isNaN(length) || length < 1) ? 1 : length;
const q = Array.from({length}, () => Promise.resolve());
let index = 0;
const add = cb => {
index = (index + 1) % length;
return (q[index] = q[index].then(() => cb()));
};
return add;
};
// demo usage
const q = multiQueue(10);
let inFlight = 0;
let maxInFlight = 0;
const promiseFunction = (i) => {
inFlight++;
maxInFlight = Math.max(inFlight, maxInFlight);
const obj = {inFlight, maxInFlight, i};
return new Promise(resolve => {
setTimeout(() => {
inFlight--;
resolve(Object.assign(obj, {t:performance.now()}));
}, 10 );
})
};
for (let i = 0; i < 40; i++) {
q(() => promiseFunction(i)).then(v => console.log(JSON.stringify(v)));
}
You can see that at most there are 10 "inFlight" requests
How about something like this? If you construct your queue which is and Array of functions that return a Promise you can splice chunks out of it and process each with a Promise.all.
const fakePromise = (id) => {
return new Promise((resolve) => {
setTimeout(() => {
console.log(`Resolving promise ${id}`)
resolve(id)
}, 100)
})
}
const queue = Array(100).fill().map((_, i) => {
return () => fakePromise(i)
})
const batchProcessPromises = (promises, batchSize) => {
if (promises && promises.length) {
return Promise.all(
promises.splice(0, batchSize)
.map(promise => promise())
)
.then(() => {
console.log('Batch complete')
return batchProcessPromises(promises, batchSize)
})
}
console.log('Batch complete')
return Promise.resolve()
}
batchProcessPromises(queue, 10)
.then(() => {
console.log('Time to get one with my day')
})
How do you plan to construct all your promises? This function effects the original queue so you would need to ensure that the array being passed into batchProcessPromises isn't shared. To get around this you could potentially use the spread operator like so
batchProcessPromises([...queue], 10)
.then(() => {
console.log('Time to get one with my day', queue)
})
Fiddle here https://jsfiddle.net/stwilz/2bpdcxo6/24/
Sometimes I need to wait for a .forEach() method to finish, mostly on 'loader' functions. This is the way I do that:
$q.when(array.forEach(function(item){
//iterate on something
})).then(function(){
//continue with processing
});
I can't help but feel that this isn't the best way to wait for a .forEach() to finish. What is the best way to do this?
If there is no asynchronous code inside the forEach, forEach is not asynchronous, for example in this code:
array.forEach(function(item){
//iterate on something
});
alert("Foreach DONE !");
you will see the alert after forEach finished.
Otherwise (You have something asynchronous inside), you can wrap the forEach loop in a Promise:
var bar = new Promise((resolve, reject) => {
foo.forEach((value, index, array) => {
console.log(value);
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
});
Credit: #rolando-benjamin-vaz-ferreira
The quickest way to make this work using ES6 would be just to use a for..of loop.
const myAsyncLoopFunction = async (array) => {
const allAsyncResults = []
for (const item of array) {
const asyncResult = await asyncFunction(item)
allAsyncResults.push(asyncResult)
}
return allAsyncResults
}
Or you could loop over all these async requests in parallel using Promise.all() like this:
const myAsyncLoopFunction = async (array) => {
const promises = array.map(asyncFunction)
await Promise.all(promises)
console.log(`All async tasks complete!`)
}
var foo = [1,2,3,4,5,6,7,8,9,10];
If you're actually doing async stuff inside the loop, you can wrap it in a promise ...
var bar = new Promise((resolve, reject) => {
foo.forEach((value, index, array) => {
console.log(value);
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
});
If you have an async task inside a loop and you want to wait. you can use for await
for await (const i of images) {
let img = await uploadDoc(i);
};
let x = 10; //this executes after
Use for of instead of forEach. Like this:
for (const item of array) {
//do something
}
console.log("finished");
"finished" will be logged after finishing the loop.
forEach() doesn't return anything, so a better practice would be map() + Promise.all()
var arr = [1, 2, 3, 4, 5, 6]
var doublify = (ele) => {
return new Promise((res, rej) => {
setTimeout(() => {
res(ele * 2)
}, Math.random() ); // Math.random returns a random number from 0~1
})
}
var promises = arr.map(async (ele) => {
// do some operation on ele
// ex: var result = await some_async_function_that_return_a_promise(ele)
// In the below I use doublify() to be such an async function
var result = await doublify(ele)
return new Promise((res, rej) => {res(result)})
})
Promise.all(promises)
.then((results) => {
// do what you want on the results
console.log(results)
})
A universal solution for making sure that all forEach() elements finished execution.
const testArray = [1,2,3,4]
let count = 0
await new Promise( (resolve) => {
testArray.forEach( (num) => {
try {
//some real logic
num = num * 2
} catch (e) {
// error handling
console.log(e)
} fanally {
// most important is here
count += 1
if (count == testArray.length) {
resolve()
}
}
})
})
The idea is same with the answer using index to count. But in real case, if error happened, the index way cannot count correctly. So the solution is more robust.
Thx
const array = [1, 2, 3];
const results = [];
let done = 0;
const asyncFunction = (item, callback) =>
setTimeout(() => callback(item * 10), 100 - item * 10);
new Promise((resolve, reject) => {
array.forEach((item) => {
asyncFunction(item, (result) => {
results.push(result);
done++;
if (done === array.length) resolve();
});
});
}).then(() => {
console.log(results); // [30, 20, 10]
});
// or
// promise = new Promise(...);
// ...
// promise.then(...);
The order of results in the "results" array can be different than the order of items in the original array, depending on the time when the asyncFunction() finishes for each of the items.
Alter and check a counter at the end of every possible unique branch of code, including callbacks. Example:
const fs = require('fs');
/**
* #description Delete files older than 1 day
* #param {String} directory - The directory to purge
* #return {Promise}
*/
async function purgeFiles(directory) {
const maxAge = 24*3600000;
const now = Date.now();
const cutoff = now-maxAge;
let filesPurged = 0;
let filesProcessed = 0;
let purgedSize = 0;
await new Promise( (resolve, reject) => {
fs.readdir(directory, (err, files) => {
if (err) {
return reject(err);
}
if (!files.length) {
return resolve();
}
files.forEach( file => {
const path = `${directory}/${file}`;
fs.stat(path, (err, stats)=> {
if (err) {
console.log(err);
if (++filesProcessed === files.length) resolve();
}
else if (stats.isFile() && stats.birthtimeMs < cutoff) {
const ageSeconds = parseInt((now-stats.birthtimeMs)/1000);
fs.unlink(path, error => {
if (error) {
console.log(`Deleting file failed: ${path} ${error}`);
}
else {
++filesPurged;
purgedSize += stats.size;
console.log(`Deleted file with age ${ageSeconds} seconds: ${path}`);
}
if (++filesProcessed === files.length) resolve();
});
}
else if (++filesProcessed === files.length) resolve();
});
});
});
});
console.log(JSON.stringify({
directory,
filesProcessed,
filesPurged,
purgedSize,
}));
}
// !!DANGER!! Change this line! (intentional syntax error in ,')
const directory = ,'/tmp'; // !!DANGER!! Changeme
purgeFiles(directory).catch(error=>console.log(error));
I'm not sure of the efficiency of this version compared to others, but I used this recently when I had an asynchronous function inside of my forEach(). It does not use promises, mapping, or for-of loops:
// n'th triangular number recursion (aka factorial addition)
function triangularNumber(n) {
if (n <= 1) {
return n
} else {
return n + triangularNumber(n-1)
}
}
// Example function that waits for each forEach() iteraction to complete
function testFunction() {
// Example array with values 0 to USER_INPUT
var USER_INPUT = 100;
var EXAMPLE_ARRAY = Array.apply(null, {length: USER_INPUT}).map(Number.call, Number) // [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, n_final... ] where n_final = USER_INPUT-1
// Actual function used with whatever actual array you have
var arrayLength = EXAMPLE_ARRAY.length
var countMax = triangularNumber(arrayLength);
var counter = 0;
EXAMPLE_ARRAY.forEach(function(entry, index) {
console.log(index+1); // show index for example (which can sometimes return asynchrounous results)
counter += 1;
if (triangularNumber(counter) == countMax) {
// function called after forEach() is complete here
completionFunction();
} else {
// example just to print counting values when max not reached
// else would typically be excluded
console.log("Counter index: "+counter);
console.log("Count value: "+triangularNumber(counter));
console.log("Count max: "+countMax);
}
});
}
testFunction();
function completionFunction() {
console.log("COUNT MAX REACHED");
}
I had to deal with the same problem (forEach using multiple promises inside) and none of the solutions presented at the current date were helpful for me. So I implemented a check array, were each promise updates its complete status. We have a general promise that wraps the process. We only resolve the general promise when each promise completed. Snippet code:
function WaitForEachToResolve(fields){
var checked_fields = new Array(fields.length).fill(0);
const reducer = (accumulator, currentValue) => accumulator + currentValue;
return new Promise((resolve, reject) => {
Object.keys(fields).forEach((key, index, array) => {
SomeAsyncFunc(key)
.then((result) => {
// some result post process
checked_fields[index] = 1;
if (checked_fields.reduce(reducer) === checked_fields.length)
resolve();
})
.catch((err) => {
reject(err);
});
}
)}
}
I like to use async-await instead of .then() syntax so for asynchronous processing of data, modified the answer of #Ronaldo this way -
let finalData = [];
var bar = new Promise(resolve => {
foo.forEach((value, index) => {
const dataToGet = await abcService.getXyzData(value);
finalData[index].someKey = dataToGet.thatOtherKey;
// any other processing here
if (finalData[dataToGet.length - 1].someKey) resolve();
});
});
await Promise.all([bar]);
console.log(`finalData: ${finalData}`);
NOTE: I've modified the if condition where it resolves the promise to meet my conditions. You can do the same in your case.
You can use this, because we are using async/await inside the forEach loop. You can use your own logic inside the loop.
let bar = new Promise((resolve, reject) => {
snapshot.forEach(async (doc) => {
"""Write your own custom logic and can use async/await
"""
const result = await something()
resolve(result);
});
});
let test = []
test.push(bar)
let concepts = await Promise.all(test);
console.log(concepts);
For simple compare code i like use for statement.
doit();
function doit() {
for (var i = 0; i < $('span').length; i++) {
console.log(i,$('span').eq(i).text() );
if ( $('span').eq(i).text() == "Share a link to this question" ) { // span number 59
return;
}
}
alert('never execute');
}
If there are async (observable) method calls inside the for loop, you could use the following method:
await players.reduce(async (a, player) => {
// Wait for the previous item to finish processing
await a;
// Process this item
await givePrizeToPlayer(player);
}, Promise.resolve());
Check here: https://gist.github.com/joeytwiddle/37d2085425c049629b80956d3c618971
I've been using this and it works best .forEach()
//count
var expecting = myArray.length;
myArray.forEach(function(item){
//do logic here
var item = item
//when iteration done
if (--expecting === 0) {
console.log('all done!');
}
})