I have a nodejs function processReviews(workflow) that when called, is supposed to push multiple promises to an array promises[], then run them all with promises.all() after the for loop.
function examplePromiseFunc(){
return new Promise((resolve, reject) => {
console.log("examplePromiseFunc() INSIDE ")
resolve('done')
})
}
async function processReviews(workflow){
//get objects from s3
let allObjects = await getAllObjects(workflow);
allObjects = allObjects.filter(obj => obj.Key.includes('output.json'))
console.log(`found ${allObjects.length} .json files.`)
const promises = [];
for (let i = 0; i < allObjects.length; i++) {
console.log('i=',i,' pushing to promises[]')
promises.push( examplePromiseFunc() )
}
const result = await Promise.all(promises)
console.log('running, result = ', result);
}
But when I run my code, the output looks like this:
found 697 .json files.
i= 0 pushing to promises[]
examplePromiseFunc() INSIDE
i= 1 pushing to promises[]
examplePromiseFunc() INSIDE
i= 2 pushing to promises[]
examplePromiseFunc() INSIDE
i= 3 pushing to promises[]
examplePromiseFunc() INSIDE
...
Which means that each time I push a promise to my promises[] array (promises.push( await examplePromiseFunc() )) the function examplePromiseFunc() is getting called instantly and is not waiting.
I want my function to only get called when I run await Promise.all(promises) at the end, is there something I'm missing? are my async functions causing a problem? I've been reading up on javascript promises.all and this seems like a fine implementation.
The issue is that you are already using await inside your loop, which means the loop will "wait" and process the items sequentially.
Instead, you should just add the promises to the array, then await all of them at the end like you had:
async function processReviews(workflow) {
//get objects from s3
const allObjects = await getAllObjects(workflow);
const promises = [];
for (let i = 0; i < allObjects.length; i++) {
// Don't await the promise here, just start it and add it to the array.
promises.push(examplePromiseFunc(allObjects[i]));
}
const result = await Promise.all(promises)
console.log(result);
}
There is a fundamental misunderstanding here of of how the Promise constructor works.
The constructor takes a single function argument known as the executor:
new Promise( executor)
The executor function is called synchronously during construction with two function arguments commonly called resolve and reject:
executor( resolve, reject)
The promise executor is responsible for initiating an asynchronous operation (usually) and make the resolve and reject functions available to code that handles operation completion and error handling, often within a call back function.
Hence the code
for (let i = 0; i < allObjects.length; i++) {
console.log('i=',i,' pushing to promises[]')
promises.push( examplePromiseFunc() )
}
calls examplePromiseFunct multiple times and within that function, the executor for the promise returned is called synchronously (by Promise) during construction. Hence the log is as one would expect: a log of "examplePromiseFunc() INSIDE" each time examplePromiseFunc is called.
This misunderstanding probably leads to the second one:
Promise.all does not "run" promises - promises are passive objects that respond in deterministic ways to calling their associated resolve and reject functions, by calling either fulfilled or rejected handlers attached to them - or chaining off another promise if resolved with a promise.
Promise.all simply returns a promise that is fulfilled with an array of the fulfilled results of its argument promises, or is rejected with the rejection reason or the first promise in its argument array that becomes rejected. It has an executor in native code that effectively attaches then handlers to promises in its argument array and then passively waits (i.e. it returns to the event loop) until argument promises are settled one at a time.
that's because promises work this way only. When you are pushing a promise into an array you are waiting for it already(inside the loop) i.e if you would not wait for them to execute then also they would execute, with or without await promise.all, Also it is possible that all the promises already resolved before you pass that array in promise.all.
The below function would also resolve all promises without the promise all.
async function processReviews(workflow){
//get objects from s3
let allObjects = await getAllObjects(workflow);
allObjects = allObjects.filter(obj => obj.Key.includes('output.json'))
console.log(`found ${allObjects.length} .json files.`)
const promises = [];
for (let i = 0; i < allObjects.length; i++) {
console.log('i=',i,' pushing to promises[]')
promises.push( examplePromiseFunc() )
}
}
Also, you should not use promise.all without limit as it might reach limitations of your hardware.
Using a map with a limit can reduce your problem.
async function processReviews(workflow) {
//get objects from s3
let allObjects = await getAllObjects(workflow);
allObjects = allObjects.filter(obj => obj.Key.includes("output.json"));
console.log(`found ${allObjects.length} .json files.`);
for (let i = 0; i < allObjects.length; i = i + PROMISE_LIMIT) {
const objects = allObjects.slice(i, i + PROMISE_LIMIT);
const result = await Promise.all(objects.map(elem => examplePromiseFunc()));
console.log("running, result = ", result);
}
}
Related
This is my first time using promises so i'm sure there are pretty dumb mistakes here. What im trying to do, is send a http request thats in a for loop.
When doing this without the promise, i can run this loop fine and everything works properly. however, when I do this with the promise, it only returns one object (should be multiple, as its sending multiple requests)
This is my code
function run(o,initialvalue){
const test = new Promise( (resolve,reject) => {
const collectionCountUrl = 'https://x.io/rpc/Query?q=%7B%22%24match%22%3A%7B%22collectionSymbol%22%3A%22'+o.name+'%22%7D%2C%22%24sort%22%3A%7B%22takerAmount%22%3A1%2C%22createdAt%22%3A-1%7D%2C%22%24skip%22%3A'+initialvalue+'%2C%22%24limit%22%3A500%7D'
promises = []
for(i=0; i < o.count(/*in this case 60, so 3 requests*/); i+= 20){
$.get(collectionCountUrl).success(resolve).fail(reject) // This should be sending multiple of the requests above, correct? ^
}
})
test.then(function(data){
console.log(data) // this should return the data from each request? im not sure
})
}
I've tried to check out this post to see if I was setting up the for loop wrong but I dont think I am.
Promise for-loop with Ajax requests
One promise is for a single result. You have to wrap a each request in a seperate promise (move the constructor call into the loop), and store the promises in an array.
Then, you can use aggregator methods like Promise.all (others being Promise.allSettled, Promise.race and Promise.any) to get a single promise of an array of results, that you can wait for.
Like this:
function run(o,initialvalue){
const collectionCountUrl = 'https://x.io/rpc/Query?q=%7B%22%24match%22%3A%7B%22collectionSymbol%22%3A%22'+o.name+'%22%7D%2C%22%24sort%22%3A%7B%22takerAmount%22%3A1%2C%22createdAt%22%3A-1%7D%2C%22%24skip%22%3A'+initialvalue+'%2C%22%24limit%22%3A500%7D'
const promises = []
// ^^^^^----+--- don't forget to declare all your variables
// vvv--+
for(let i=0; i < o.count(/*in this case 60, so 3 requests*/); i+= 20){
promises.push(new Promise( (resolve,reject) => {
$.get(collectionCountUrl).success(resolve).fail(reject)
}))
}
//You've got an array of promises, let's convert it to a single one that resolves when everything's done:
const test = Promise.all(promises)
test.then(function(data){
//This will run when all the requests have succeeded
//`data` is an array of results
console.log(data)
})
}
This will work, but there's an even better way!
jQuery's $.Deferred objects (like the one you're getting from $.get) also have a .then() method like promises do, and while it's not a real promise, you can use it like if it was. Such objects are called thenables.
So, instead of this thing...
promises.push(new Promise( (resolve,reject) => {
$.get(collectionCountUrl).success(resolve).fail(reject)
}))
...you can just do this:
promises.push( $.get(collectionCountUrl) )
So, here's the final code:
function run(o,initialvalue){
const collectionCountUrl = 'https://x.io/rpc/Query?q=%7B%22%24match%22%3A%7B%22collectionSymbol%22%3A%22'+o.name+'%22%7D%2C%22%24sort%22%3A%7B%22takerAmount%22%3A1%2C%22createdAt%22%3A-1%7D%2C%22%24skip%22%3A'+initialvalue+'%2C%22%24limit%22%3A500%7D'
const promises = []
for(let i=0; i < o.count(/*in this case 60, so 3 requests*/); i+= 20){
promises.push( $.get(collectionCountUrl) )
}
//You've got an array of promises, let's convert it to a single one that resolves when everything's done:
const test = Promise.all(promises)
test.then(function(data){
//This will run when all the requests have succeeded
//`data` is an array of results
console.log(data)
})
}
let currentProduct;
for (let i = 0; i < products.length; i++) {
currentProduct = products[i];
subscription.getAll(products[i]._id)
.then((subs) => {
update(subs, currentProduct);
});
}
I'm using bluebird, the methods getAll and update return promises. How can I say "Wait until the two promises return, then update the currentProduct value"? I'm quite new to JS...
This will be straightforward if you can use async/await:
// Make sure that this code is inside a function declared using
// the `async` keyword.
let currentProduct;
for (let i = 0; i < products.length; i++) {
currentProduct = products[i];
// By using await, the code will halt here until
// the promise resolves, then it will go to the
// next iteration...
await subscription.getAll(products[i]._id)
.then((subs) => {
// Make sure to return your promise here...
return update(subs, currentProduct);
});
// You could also avoid the .then by using two awaits:
/*
const subs = await subscription.getAll(products[i]._id);
await update(subs, currentProduct);
*/
}
Or if you can only use plain promises, you can loop through all your products, and put each promise in the .then of the last loop. In that way, it will only advance to the next when the previous has resolved (even though it will have iterated the whole loop first):
let currentProduct;
let promiseChain = Promise.resolve();
for (let i = 0; i < products.length; i++) {
currentProduct = products[i];
// Note that there is a scoping issue here, since
// none of the .then code runs till the loop completes,
// you need to pass the current value of `currentProduct`
// into the chain manually, to avoid having its value
// changed before the .then code accesses it.
const makeNextPromise = (currentProduct) => () => {
// Make sure to return your promise here.
return subscription.getAll(products[i]._id)
.then((subs) => {
// Make sure to return your promise here.
return update(subs, currentProduct);
});
}
// Note that we pass the value of `currentProduct` into the
// function to avoid it changing as the loop iterates.
promiseChain = promiseChain.then(makeNextPromise(currentProduct))
}
In the second snippet, the loop just sets up the entire chain, but doesn't execute the code inside the .then immediately. Your getAll functions won't run until each prior one has resolved in turn (which is what you want).
Here is how I'd do it:
for (let product of products) {
let subs = await subscription.getAll(product._id);
await update(subs, product);
}
No need to manually chain promises or iterate arrays by index :)
You may want to keep track of what products you've processed because when one fails you have no idea how many succeeded and you don't know what to correct (if roll back) or retry.
The async "loop" could be a recursive function:
const updateProducts = /* add async */async (products,processed=[]) => {
try{
if(products.length===0){
return processed;
}
const subs = await subscription.getAll(products[0]._id)
await update(subs, product);
processed.push(product[0]._id);
}catch(err){
throw [err,processed];
}
return await updateProducts(products.slice(1),processed);
}
Without async you can use recursion or reduce:
//using reduce
const updateProducts = (products) => {
//keep track of processed id's
const processed = [];
return products.reduce(
(acc,product)=>
acc
.then(_=>subscription.getAll(product._id))
.then(subs=>update(subs, product))
//add product id to processed product ids
.then(_=>processed.push(product._id)),
Promise.resolve()
)
//resolve with processed product id's
.then(_=>processed)
//when rejecting include the processed items
.catch(err=>Promise.reject([err,processed]));
}
//using recursion
const updateProducts = (products,processed=[]) =>
(products.length!==0)
? subscription.getAll(products[0]._id)
.then(subs=>update(subs, product))
//add product id to processed
.then(_=>processed.push(products[0]._id))
//reject with error and id's of processed products
.catch(err=>Promise.reject([err,processed]))
.then(_=>updateProducts(products.slice(1),processed))
: processed//resolve with array of processed product ids
Here is how you'd call updateProducts:
updateProducts(products)
.then(processed=>console.log("Following products are updated.",processed))
.catch(([err,processed])=>
console.error(
"something went wrong:",err,
"following were processed until something went wrong:",
processed
)
)
let currentProduct;
for (let i = 0; i < products.length; i++) {
currentProduct = products[i];
subscription.getAll(products[i]._id)
.then((subs) => {
update(subs, currentProduct);
});
}
I'm using bluebird, the methods getAll and update return promises. How can I say "Wait until the two promises return, then update the currentProduct value"? I'm quite new to JS...
This will be straightforward if you can use async/await:
// Make sure that this code is inside a function declared using
// the `async` keyword.
let currentProduct;
for (let i = 0; i < products.length; i++) {
currentProduct = products[i];
// By using await, the code will halt here until
// the promise resolves, then it will go to the
// next iteration...
await subscription.getAll(products[i]._id)
.then((subs) => {
// Make sure to return your promise here...
return update(subs, currentProduct);
});
// You could also avoid the .then by using two awaits:
/*
const subs = await subscription.getAll(products[i]._id);
await update(subs, currentProduct);
*/
}
Or if you can only use plain promises, you can loop through all your products, and put each promise in the .then of the last loop. In that way, it will only advance to the next when the previous has resolved (even though it will have iterated the whole loop first):
let currentProduct;
let promiseChain = Promise.resolve();
for (let i = 0; i < products.length; i++) {
currentProduct = products[i];
// Note that there is a scoping issue here, since
// none of the .then code runs till the loop completes,
// you need to pass the current value of `currentProduct`
// into the chain manually, to avoid having its value
// changed before the .then code accesses it.
const makeNextPromise = (currentProduct) => () => {
// Make sure to return your promise here.
return subscription.getAll(products[i]._id)
.then((subs) => {
// Make sure to return your promise here.
return update(subs, currentProduct);
});
}
// Note that we pass the value of `currentProduct` into the
// function to avoid it changing as the loop iterates.
promiseChain = promiseChain.then(makeNextPromise(currentProduct))
}
In the second snippet, the loop just sets up the entire chain, but doesn't execute the code inside the .then immediately. Your getAll functions won't run until each prior one has resolved in turn (which is what you want).
Here is how I'd do it:
for (let product of products) {
let subs = await subscription.getAll(product._id);
await update(subs, product);
}
No need to manually chain promises or iterate arrays by index :)
You may want to keep track of what products you've processed because when one fails you have no idea how many succeeded and you don't know what to correct (if roll back) or retry.
The async "loop" could be a recursive function:
const updateProducts = /* add async */async (products,processed=[]) => {
try{
if(products.length===0){
return processed;
}
const subs = await subscription.getAll(products[0]._id)
await update(subs, product);
processed.push(product[0]._id);
}catch(err){
throw [err,processed];
}
return await updateProducts(products.slice(1),processed);
}
Without async you can use recursion or reduce:
//using reduce
const updateProducts = (products) => {
//keep track of processed id's
const processed = [];
return products.reduce(
(acc,product)=>
acc
.then(_=>subscription.getAll(product._id))
.then(subs=>update(subs, product))
//add product id to processed product ids
.then(_=>processed.push(product._id)),
Promise.resolve()
)
//resolve with processed product id's
.then(_=>processed)
//when rejecting include the processed items
.catch(err=>Promise.reject([err,processed]));
}
//using recursion
const updateProducts = (products,processed=[]) =>
(products.length!==0)
? subscription.getAll(products[0]._id)
.then(subs=>update(subs, product))
//add product id to processed
.then(_=>processed.push(products[0]._id))
//reject with error and id's of processed products
.catch(err=>Promise.reject([err,processed]))
.then(_=>updateProducts(products.slice(1),processed))
: processed//resolve with array of processed product ids
Here is how you'd call updateProducts:
updateProducts(products)
.then(processed=>console.log("Following products are updated.",processed))
.catch(([err,processed])=>
console.error(
"something went wrong:",err,
"following were processed until something went wrong:",
processed
)
)
so i have a promise that collects data from a server but only collects 50 responses at a time. i have 250 responses to collect.
i could just concate promises together like below
new Promise((resolve, reject) => {
resolve(getResults.get())
})
.then((results) => {
totalResults.concat(results)
return getResults.get()
})
.then((results) => {
totalResults.concat(results)
return getResults.get()
}).then((results) => {
totalResults.concat(results)
return getResults.get()
})
In this instance i only need 250 results so this seems a managable solution but is there a way of concating promises in a loop. so i run a loop 5 times and each time run the next promise.
Sorry i am new to promises and if this were callbacks this is what i would do.
If you want to loop and serialise the promises, not executing any other get calls once one fails, then try this loop:
async function getAllResults() { // returns a promise for 250 results
let totalResults = [];
try {
for (let i = 0; i < 5; i++) {
totalResults.push(...await getResults.get());
}
} catch(e) {};
return totalResults;
}
This uses the EcmaScript2017 async and await syntax. When not available, chain the promises with then:
function getAllResults() {
let totalResults = [];
let prom = Promise.resolve([]);
for (let i = 0; i < 5; i++) {
prom = prom.then(results => {
totalResults = totalResults.concat(results);
return getResults.get();
});
}
return prom.then(results => totalResults.concat(results));
}
Note that you should avoid the promise construction anti-pattern. It is not necessary to use new Promise here.
Also consider adding a .catch() call on the promise returned by the above function, to deal with error conditions.
Finally, be aware that concat does not modify the array you call it on. It returns the concatenated array, so you need to assign that return value. In your code you don't assign the return value, so the call has no effect.
See also JavaScript ES6 promise for loop.
Probably you just need Promise.all method.
For every request you should create a promise and put it in an array, then you wrap everything in all method and you're done.
Example (assuming that getResults.get returns a promise):
let promiseChain = [];
for(let i = 0; i <5; i++){
promiseChain.push(getResults.get());
}
Promise.all(promiseChain)
.then(callback)
You can read more about this method here:
Promise.all at MDN
EDIT
You can access data returned by the promises this way:
function callback(data){
doSomething(data[0]) //data from the first promise in the chain
...
doEventuallySomethingElse(data[4]) //data from the last promise
}
the challenge:
We want to make N parallel ajax requests for an item's children.
Upon returning, we want to process them in sequential order (1...N)
We do NOT want to wait for all promises to return, but we want to process them IN ORDER as they come back.
For example:
Even if 2,3,5 come back before 1, we should hold onto the results of 2,3,5, and upon 1's return, process 1,2,3 in order (and wait for 4 to come back before 5)
Tools: Q + ES6 generators
Create array of N-1 length with placeholder variables
EG when N = 3:
let [N1,N2,N3] = yield [ Promise1, Promise2, Promise3 ]
//process items sequentially:
console.log(N1)
console.log(N2)
console.log(N3)
However, populating an array of empty variables doesn't seem to work of course because the reference doesn't know where to find the var declaration
for(var i = 0; i< 3; i++) {
res.push("some empty var")
}
Given the constraints of sticking to the tools provided, how could we parallelize calls, but process their returns sequentially?
You can use Promise.all(), .then()
javascript at Answer returns exact results described at Question
We want to make N parallel ajax requests for an item's children.
Upon returning, we want to process them in sequential order (1...N)
We do NOT want to wait for all promises to return, but we want to process them IN ORDER as they come back.
how could we parallelize calls, but process their returns
sequentially?
You can use .then() chained to original function which returns a promise or Promise object itself to process promise before returning value to be processed in sequential order of parameters passed to Promise.all() at .then() chained to Promise.all()
var n = 0;
var fn = function() {
return new Promise(function(resolve, reject) {
console.log("promise " + ++n + " called");
setTimeout(function(i) {
resolve(i)
}, Math.random() * 2500, n)
})
// handle requirement 3. here
.then(function(res) {
console.log(res);
return res
})
}
Promise.all([fn(), fn(), fn()]) // handle requirement 1. here
// handle requirement 2. here
.then(function(data) {
let [N1, N2, N3] = data;
console.log(N1, N2, N3);
})
You can do that by waiting for the next promise inside the loop:
const promises = […]; // or created programmatically
for (const promise of promises) {
const result = yield promise; // await them sequentially
console.log(result);
}