I'm not looking for any solution in any particular language, I just want to understand what are the good practices in the area of running async tasks from a loop.
This is what I have thought of so far:
var callAcc = 0;
var resultArr = [];
for (var k = 0; k < 20; k++) {
callAcc ++;
asyncFunction("variable")
.then(function (result) {
resultArr.push(result);
if (callAcc === resultArr.length) {
// do something with the resultArr
}
})
.catch(function (err) {
console.warn(err);
});
}
At this point, I'm just using a sync variable that will only let me proceed once I have all of the async tasks complete. However, this feels hacky and I was wondering if there was some kind of design pattern to execute aync tasks from a loop
EDIT:
Based on the proposed solutions, this is what i ended up using
var promises = [];
for (var i = 0; i < 20; i ++) {
promises.push(asyncFunction("param"));
}
Promise.all(promises)
.then(function (result) {
console.log(result);
})
.catch(function (err) {
console.warn(err);
});
Thank you everyone!
Instead of running your async tasks within a loop, you can abstract away the task of waiting on all your promises by using the Promise.all function, which will take an iterable of your promises and create a new promise which will return an array with all of their results in the same order as the original list of promises as soon as all of them resolve, or fail if any of them fails. Most languages which have Promises have a function like that, and if there isn't, it shouldn't be too hard to define.
If you need all of your promises to run even if any of them fails, you can write a function that does that too, you'd just need to define what to do with the failed Promises (discard them, return the errors somehow...).
Anyway, the gist of it is that the best way of dealing with orchestration of multiple Promises is to define a function which will take all the Promises you need to deal with and return a new Promise which will handle the orchestration.
Something like:
orchestratePromises(promiseList) {
return new Promise((resolve, reject) => {
// set up
for (let promise of promiseList) {
// define how to handle your promises when they resolve
// asynchronously resolve or reject
}
}
}
Related
I found that I cannot resolve the promises one by one.
This is my code
const promises = mkt.map((marketItem) => {
return context.program.account.chain
.fetch(marketItem[0].instrument)
.then((instrumentRes) => {
return Test(context, marketItem[0]).then(
testResult => {
return Promise.all([
functionA(),
functionB(),
]).then((result) => {
return result
});
}
);
});
});
console.log(promises)
for (let i=0; i < promises.length; i++) {
const val = await promises[i]();
console.log(val);
}
error
promises[i]() is not a function
Why is that?How can I solve it?
promises[i]() is not a function
Correct, this is because promises is an array of Promise objects, not functions.
You can dump this promises array into Promise.all and wait for them all to resolve.
Promise.all(promises)
.then(resolvedPromises => {
// handle array of resolved promises
});
but I am working on some real time update stuff. If I use Promise.all
it will throw error Too many request, so I want to do it one by one to
see the effect
For this then I think you want to iterate on the mkt array and wait for each created Promise to resolve. Refactor the mapping callback into a standalone function that you can manually invoke within the for-loop, awaiting the the Promise to settle.
const request = (marketItem) => {
return context.program.account.chain
.fetch(marketItem[0].instrument)
.then((instrumentRes) => Test(context, marketItem[0]))
.then(testResult => Promise.all([functionA(), functionB()]))
.then((result) => result);
}
for (let i=0; i < mkt.length; i++) {
try {
const val = await request(mkt[i]);
console.log(val);
} catch(error) {
// handle any error
}
}
Just removing the () from the await call should already work.
const val = await promises[i];
However keep in mind that all the promises have already been "started" (I don't know a better word for that) at that point and you're just retrieving the results or an error with the await call.
so i have a promise that collects data from a server but only collects 50 responses at a time. i have 250 responses to collect.
i could just concate promises together like below
new Promise((resolve, reject) => {
resolve(getResults.get())
})
.then((results) => {
totalResults.concat(results)
return getResults.get()
})
.then((results) => {
totalResults.concat(results)
return getResults.get()
}).then((results) => {
totalResults.concat(results)
return getResults.get()
})
In this instance i only need 250 results so this seems a managable solution but is there a way of concating promises in a loop. so i run a loop 5 times and each time run the next promise.
Sorry i am new to promises and if this were callbacks this is what i would do.
If you want to loop and serialise the promises, not executing any other get calls once one fails, then try this loop:
async function getAllResults() { // returns a promise for 250 results
let totalResults = [];
try {
for (let i = 0; i < 5; i++) {
totalResults.push(...await getResults.get());
}
} catch(e) {};
return totalResults;
}
This uses the EcmaScript2017 async and await syntax. When not available, chain the promises with then:
function getAllResults() {
let totalResults = [];
let prom = Promise.resolve([]);
for (let i = 0; i < 5; i++) {
prom = prom.then(results => {
totalResults = totalResults.concat(results);
return getResults.get();
});
}
return prom.then(results => totalResults.concat(results));
}
Note that you should avoid the promise construction anti-pattern. It is not necessary to use new Promise here.
Also consider adding a .catch() call on the promise returned by the above function, to deal with error conditions.
Finally, be aware that concat does not modify the array you call it on. It returns the concatenated array, so you need to assign that return value. In your code you don't assign the return value, so the call has no effect.
See also JavaScript ES6 promise for loop.
Probably you just need Promise.all method.
For every request you should create a promise and put it in an array, then you wrap everything in all method and you're done.
Example (assuming that getResults.get returns a promise):
let promiseChain = [];
for(let i = 0; i <5; i++){
promiseChain.push(getResults.get());
}
Promise.all(promiseChain)
.then(callback)
You can read more about this method here:
Promise.all at MDN
EDIT
You can access data returned by the promises this way:
function callback(data){
doSomething(data[0]) //data from the first promise in the chain
...
doEventuallySomethingElse(data[4]) //data from the last promise
}
I have some code that I cant get my head around, I am trying to return an array of object using a callback, I have a function that is returning the values and then pushing them into an array but I cant access this outside of the function, I am doing something stupid here but can't tell what ( I am very new to Node.JS )
for (var index in res.response.result) {
var marketArray = [];
(function () {
var market = res.response.result[index];
createOrUpdateMarket(market, eventObj , function (err, marketObj) {
marketArray.push(marketObj)
console.log('The Array is %s',marketArray.length) //Returns The Array is 1.2.3..etc
});
console.log('The Array is %s',marketArray.length) // Returns The Array is 0
})();
}
You have multiple issues going on here. A core issue is to gain an understanding of how asynchronous responses work and which code executes when. But, in addition to that you also have to learn how to manage multiple async responses in a loop and how to know when all the responses are done and how to get the results in order and what tools can best be used in node.js to do that.
Your core issue is a matter of timing. The createOrUpdateMarket() function is probably asynchronous. That means that it starts its operation when the function is called, then calls its callback sometime in the future. Meanwhile the rest of your code continues to run. Thus, you are trying to access the array BEFORE the callback has been called.
Because you cannot know exactly when that callback will be called, the only place you can reliably use the callback data is inside the callback or in something that is called from within the callback.
You can read more about the details of the async/callback issue here: Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
To know when a whole series of these createOrUpdateMarket() operations are all done, you will have to code especially to know when all of them are done and you cannot rely on a simple for loop. The modern way to do that is to use promises which offer tools for helping you manage the timing of one or more asynchronous operations.
In addition, if you want to accumulate results from your for loop in marketArray, you have to declare and initialize that before your for loop, not inside your for loop. Here are several solutions:
Manually Coded Solution
var len = res.response.result.length;
var marketArray = new Array(len), cntr = 0;
for (var index = 0, index < len; index++) {
(function(i) {
createOrUpdateMarket(res.response.result[i], eventObj , function (err, marketObj) {
++cntr;
if (err) {
// need error handling here
}
marketArray[i] = marketObj;
// if last response has just finished
if (cntr === len) {
// here the marketArray is fully populated and all responses are done
// put your code to process the marketArray here
}
});
})(index);
}
Standard Promises Built Into Node.js
// make a version of createOrUpdateMarket that returns a promise
function createOrUpdateMarketAsync(a, b) {
return new Promise(function(resolve, reject) {
createOrUpdateMarket(a, b, function(err, marketObj) {
if (err) {
reject(err);
return;
}
resolve(marketObj);
});
});
}
var promises = [];
for (var i = 0; i < res.response.result.length; i++) {
promises.push(createorUpdateMarketAsync(res.response.result[i], eventObj));
}
Promise.all(promises).then(function(marketArray) {
// all results done here, results in marketArray
}, function(err) {
// an error occurred
});
Enhanced Promises with the Bluebird Promise library
The bluebird promise library offers Promise.map() which will iterate over your array of data and produce an array of asynchronously obtained results.
// make a version of createOrUpdateMarket that returns a promise
var Promise = require('bluebird');
var createOrUpdateMarketAsync = Promise.promisify(createOrUpdateMarket);
// iterate the res.response.result array and run an operation on each item
Promise.map(res.response.result, function(item) {
return createOrUpdateMarketAsync(item, eventObj);
}).then(function(marketArray) {
// all results done here, results in marketArray
}, function(err) {
// an error occurred
});
Async Library
You can also use the async library to help manage multiple async operations. In this case, you can use async.map() which will create an array of results.
var async = require('async');
async.map(res.response.result, function(item, done) {
createOrUpdateMarker(item, eventObj, function(err, marketObj) {
if (err) {
done(err);
} else {
done(marketObj);
}
});
}, function(err, results) {
if (err) {
// an error occurred
} else {
// results array contains all the async results
}
});
I understand using promises in simple scenarios but currently really confused on how to implement something when using a for loop and some updates to local sqlite database.
Code is as follows
surveyDataLayer.getSurveysToUpload().then(function(surveys) {
var q = $q.defer();
for (var item in surveys) {
var survey = surveys[item];
// created as a closure so i can pass in the current item due to async process
(function(survey) {
ajaxserviceAPI.postSurvey(survey).then(function(response) {
//from response update local database
surveyDataLayer.setLocalSurveyServerId(survey, response.result).then(function() {
q.resolve; // resolve promise - tried only doing this when last record also
})
});
})(survey) //pass in current survey used to pass in item into closure
}
return q.promise;
}).then(function() {
alert('Done'); // This never gets run
});
Any help or assistance would be appreciated. I'm probably struggling on how best to do async calls within loop which does another async call to update and then continue once completed.
at least promises have got me out of callback hell.
Cheers
This answer will get you laid at JS conferences (no guarantees though)
surveyDataLayer.getSurveysToUpload().then(function(surveys) {
return Promise.all(Object.keys(surveys).map(function(key) {
var survey = surveys[key];
return ajaxserviceAPI.postSurvey(survey).then(function(response){
return surveyDataLayer.setLocalSurveyServerId(survey, response.result);
});
}));
}).then(function() {
alert('Done');
});
This should work (explanations in comments):
surveyDataLayer.getSurveysToUpload().then(function(surveys) {
// array to store promises
var promises = [];
for (var item in surveys) {
var survey = surveys[item];
// created as a closure so i can pass in the current item due to async process
(function(survey) {
var promise = ajaxserviceAPI.postSurvey(survey).then(function(response){
//returning this promise (I hope it's a promise) will replace the promise created by *then*
return surveyDataLayer.setLocalSurveyServerId(survey, response.result);
});
promises.push(promise);
})(survey); //pass in current survey used to pass in item into closure
}
// wait for all promises to resolve. If one fails nothing resolves.
return $q.all(promises);
}).then(function() {
alert('Done');
});
Awesome tutorial: http://ponyfoo.com/articles/es6-promises-in-depth
You basically want to wait for all of them to finish before resolving getSurveysToUpload, yes? In that case, you can return $q.all() in your getSurveysToUpload().then()
For example (not guaranteed working code, but you should get an idea):
surveyDataLayer.getSurveysToUpload().then(function(surveys) {
var promises = [];
// This type of loop will not work in older IEs, if that's of any consideration to you
for (var item in surveys) {
var survey = surveys[item];
promises.push(ajaxserviceAPI.postSurvey(survey));
}
var allPromise = $q.all(promises)
.then(function(responses) {
// Again, we want to wait for the completion of all setLocalSurveyServerId calls
var promises = [];
for (var index = 0; index < responses.length; index++) {
var response = responses[index];
promises.push(surveyDataLayer.setLocalSurveyServerId(survey, response.result));
}
return $q.all(promises);
});
return allPromise;
}).then(function() {
alert('Done'); // This never gets run
});
I am attempting to loop over a bunch of records, perform a lookup for each, then perform some permutations on the inner lookup (0->1, 0->2, 0->3, 1->2, 1->3, 2->3) and write the permutations to a table. I've got a working synchronous script but I can't figure out how best to use bluebird and the loopback persisted model methods.
If I run it asynchronously with my script, it takes too long to process, and sometimes I run out of memory--and the whole thing is a memory hog to begin with: nothing is written to the DB until all permutations have been looped over. This might be because of PersistedModel so it may be useful to go straight to node-mysql with promisifyAll().
Eventually would love to parallelize this to write all the permutations per trip at the same time.
First I wrapped the PersistedModel.create() and .find() calls in promises:
function createRoute(newRoute) {
return new Promise(function(resolve, reject) {
Route.create(newRoute, function(err, route) {
if (!err) {
resolve(route);
} else {
reject(err);
}
});
});
}
function getHopsFor(trip) {
return new Promise(function(resolve, reject) {
Schedule.find({
where: {
name: trip.name,
tripCode: trip.tripCode,
companyCode: trip.companyCode
},
order: 'eta ASC',
}, function(err, hops) {
if(err) {
reject(err);
}
resolve(hops);
});
});
}
Yes, this is an anti-pattern but I can't use .promisifyAll() on the library (as a whole or explicitly on .create()) since it's a bit incompatible. I think native promises are on the way but not quite ready yet (correct me if I'm wrong, so I can use the promise I already have ;)).
My main question is how would I loop over the above .find() and .create() promises and save an array of promises, so I could then use Promise.settle() to only exit when all new routes have been written to the DB. And chain the whole thing with .then()s properly.
In conjunction with the create a single sailing wrapper and a few other promise wrappers, the main execution loop is something like this:
getTrips().then(function(trips){
for(t = 0; t < trips.length; t++) {
getHopsFor(trips[t]).then(function(hops) {
getRoutesFor(hops).then(function(routes) {
for(r = 0; r < routes.length; r++) {
routes.push((createRoute(route));
}
Promise.settle(routes).then(function(results){
console.log("finished trip " + t)
});
});
});
}
});
My initial script used nested callbacks and was sort of manageable but wanted to try a Bluebird version to wrap my head around Promises.all() and .settle(). Need to use less memory and get the execution time down.
ES5 + Promise
function createRoutesForTrips () {
return getTrips().then(function(trips) {
return Promise.all(trips.map(function(trip) {
return getHopsFor(trip).then(function(hops) {
return getRoutesFor(hops);
}).then(function(routes) {
return Promise.all(routes.map(function(route) {
return createRoute(route);
}));
});
}));
});
}
ES6 fat arrow
var createRoutesForTrips = () =>
getTrips().then((trips) =>
Promise.all(trips.map((trip) =>
getHopsFor(trip).then((hops) =>
getRoutesFor(hops)
).then((routes) =>
Promise.all(routes.map((route) =>
createRoute(route)
))
)
))
)
ES7 async function
async function createRoutesForTrips () {
let trips = await getTrips()
for (let trip of trips) {
let hops = await getHopsFor(trip)
let routes = await getRoutesFor(hops)
for (let route of routes) {
await createRoute(route)
}
}
}