I have this code:
var queue = [];
var allParserd = [];
_.each(webs, function (web) {
queue.push(function () {
WebsitesUtils.parseWebsite(web, function (err, parsed) {
allParserd.push(parsed);
});
});
});
Promise.all(queue).then(function (data) {
console.log(allParserd);
});
Basically I need to fetch all my webs and be sure to give the result after that every parsing is done. the function parseWebsite return the correct data, but in this way is not called and allParsed return just as an empty array. I'm sure that I miss some things, I've started to use the promises just from some days.
If you need some more information just tell me.
P.s.
I want that all the functions to start at the same time; I don't want to wait for each one response for going forward.
Tagged with Bluebird so let's use it:
First, let's convert your callback API to promises:
Promise.promisifyAll(WebsitesUtils);
Now, let's use .map to map every item in webs to it being parsed parseWebsite:
Promise.map(webs, function(item){
return WebsitesUtils.parseWebsiteAsync(item); // note the suffix
}).then(function(results){
// all the results are here.
}).catch(function(err){
// handle any errors
});
As you can see - this is trivial to do with Bluebird.
Promise.all doesn't take a queue of functions to execute. It expects an array of promises which represent the results of the many concurrently running (still pending) requests.
The first step is to have a function that actually returns a promise, instead of only executing a callback. We can use
function parseWebsite(web) {
return new Promise(function(fulfill, reject) {
WebsitesUtils.parseWebsite(web, function (err, parsed) {
if (err)
reject(err);
else
fulfill(parsed);
});
});
}
or simply use promisification that does this generically:
var parseWebsite = Promise.promisify(WebsitesUtils.parseWebsite, WebsitesUtils);
Now we can go to construct our array of promises by calling that function for each site:
var promises = [];
_.each(webs, function (web) {
promises.push(parseWebsite(web));
});
or just
var promises = _.map(webs, parseWebsite);
so that in the end we can use Promise.all, and get back our allParsed array (which even is in the same order as webs was!):
Promise.all(promises).then(function(allParsed) {
console.log(allParsed);
});
Bluebird even provides a shortcut function so you don't need promises:
Promise.map(webs, parseWebsite).then(function(allParsed) {
console.log(allParsed);
});
Here's how might do it with async:
var async = require('async');
var webs = ...
async.map(webs, function(web, callback) {
WebsitesUtils.parseWebsite(web, callback);
}, function(err, results) {
if (err) throw err; // TODO: handle errors better
// `results` contains all parsed results
});
and if parseWebsite() isn't a prototype method dependent on WebsitesUtils then you could simplify it further:
async.map(webs, WebsitesUtils.parseWebsite, function(err, results) {
if (err) throw err; // TODO: handle errors better
// `results` contains all parsed results
});
Related
I have a tricky situation that needs to collect keys that belongs to certain types (types in a given array), then filter the collected keys and pass to a deletion function.
The collection process calls shell codes and process the results in a callback within a loop. I will need to wait until the whole loop of callback finishes then pass to the deletion function.
I am using shelljs in the node codes, basically look like the below:
var del_arr = [];
for (var i in types) {
shell.exec(somecode with
var [i], {
silent: true
},
function(code, stdout, stderr) {
somecode-processing/filtering stdout and pushes the results to del_arr;
});
//loop through array types[] and need to wait for all shell codes' callbacks to finish;
}
//then pass del_arr to deletion function
I wasn't able to build a async function in this format b/s of the shelljs callback. I also don't know how to use promise in this situation.
Can you tell me how to achieve this non-blocking process?
Thanks
Turn child_process.exec into a promise:
function execWrapper(command, options) {
return new Promise((resolve, reject) => {
shell.exec(command, options, (error, out, err) => {
if (error) return reject(error);
resolve({out: out, err: err});
})
})
}
Then you can iterate over types and map each one to a promise:
const promises = types.map(type => execWrapper(type, {slient: true}));
Now wait for each promise to resolve, or for one to reject:
Promise.all(promises).then((del_arr) => {
// del_arr is now a array of objects with the stdout and stderr of each type.
//
})
A good implementation of this case :
async function myAsyncFunction() {
const promises = types.map((type) => myAsyncRequest(type));
let del_arr = Promise.all(promises);
}
A good article that explains this :
https://medium.freecodecamp.org/avoiding-the-async-await-hell-c77a0fb71c4c
Try to convert shell.exec to Promise like
function shellPromise(command,option) {
return Promise((resolv,reject)=>{
shell.exec(command,option,(code,stdout,stderr)=>
resolv({code:code,stdout:stdout,stderr:stderr})
);
};
};
Then you can use something like
for (var i in types){
var result=await shellPromise(somecode with var[i], {silent:true});
// somecode-processing/filtering stdout and pushes the results to del_arr;
}
You can also use async package in npm. It provides a function eachSeries that might come handy in your situation, without useing promises and dealing with callbacks only.
async.eachSeries(hugeArray, function iteratee(item, callback) {
if (inCache(item)) {
callback(null, cache[item]); // if many items are cached, you'll overflow
} else {
doSomeIO(item, callback);
}
}, function done() {
//...
});
For more details on how to use this function: https://caolan.github.io/async/
I'm trying to iterate threw a list of item and do some actions on them by calling an API like this example :
for (i = 0; i < arr.length; i++) {
if (arr[i].id == 42) {
api.requestAction(arr[i].id, function(error, response){ });
}
}
Problem is the loop obviously ended before all the requests are done and the program exits. What should I do to manage it ? I saw the "Promise" method but don't really know how I can use it in this case or maybe there's an other solution.
Thank you by advance !
With node-fetch (a promisify http api) you can together with async/await halt the for loop until it's done but this requires node v6+ with --harmony-async-await flag added
const fetch = require('node-fetch')
async function foo() {
for (let item of arr) {
if (item.id == 42) {
let res = await fetch(url)
let body = await res.text()
console.log(body)
}
}
console.log('done (after request)')
}
now every time you add the async keyword in front of a function it will always return a promise that resolve/rejects when everything is done
foo().then(done, fail)
alternetive you can just wrap you api fn in a promise if you don't want to install node-fetch
await new Promise((rs, rj) => {
api.requestAction(arr[i].id, function(error, response){
error ? rj(error) : rs(response)
})
})
Install bluebird
npm install bluebird --save
Code
//require npm
var Promise = require("bluebird");
//code
//"promisify" converts traditional callback function into a Promise based function
var _requestAction = Promise.promisify(api.requestAction);
//loop over array
Promise.map(arr, function (value) {
if (value.id == 42) {
//async request
return _requestAction(value.id).then(function (_result) {
//success
console.log(_result);
}).catch(function (e) {
//error
console.error(e);
});
}
});
You could use async.js. It's an asyncronous control flow library which provides control flows for things like sequential loops, looping in parralel, and many other common flow control mechanism, check it out.
See code below, the code assumes that you're variable 'arr' is defined somewhere in scope.
npm install async
var async = require("async");
//Loop through each item, waiting for your
//asyncronous function to finish before continuing
//to move onto the next item in the array
//NOTE: This does not loop sequentially, if you want that function with asyncjs then user eachSeries
async.each(arr,
//Item is the current item being iterated over,
//callback is the callback you call to finish the current iteration, it accepts an error and result parameter callback(error, result);
function (item, callback) {
api.requestAction(item.id, function(error, response){
//Check for any errors...
if (error) return callback(error);
callback(null);
});
},
function (err, result) {
//You've now finished the loop
if (err) {
//Do something, you passed an error object to
//in one of the loop's iterations
}
//No errors, move on with your code..
});
Use Bluebird Promises:
var Promise = require('bluebird');
Promise.map(arrayOfIds, function(item){
return api.requestAction(item);
})
.then(function(response){
// all the requests are resolved here
})
if u want sequential execution of the ids then use Promise.mapSeries (is slow as it waits for task to finish)
Let's say I have some code that looks like this:
var doSomething = function(parameter){
//send some data to the other function
return when.promise(function(resolveCallback, rejectCallback) {
var other = doAnotherThing(parameter);
//how do I check and make sure that other has resolved
//go out and get more information after the above resolves and display
});
};
var doAnotherThing = function(paramers){
return when.promise(function(resolveCallback, rejectCallback) {
//go to a url and grab some data, then resolve it
var s = "some data I got from the url";
resolveCallback({
data: s
});
});
};
How do I ensure that var other has completely resolved before finishing and resolving the first doSomething() function? I'm still wrapping my head around Nodes Async characteristic
I really didn't know how else to explain this, so I hope this makes sense! Any help is greatly appreciated
EDIT: In this example, I am deleting things from an external resource, then when that is done, going out the external resource and grabbing a fresh list of the items.
UPDATED CODE
var doSomething = function(parameter){
//send some data to the other function
doAnotherThing(parameter).then(function(){
//now we can go out and retrieve the information
});
};
var doAnotherThing = function(paramers){
return when.promise(function(resolveCallback, rejectCallback) {
//go to a url and grab some data, then resolve it
var s = "some data I got from the url";
resolveCallback({
data: s
});
});
};
The return of doAnotherThing appears to be a promise. You can simply chain a then and put your callback to utilize other. then also already returns a promise. You can return that instead.
// Do stuff
function doSomething(){
return doAnotherThing(parameter).then(function(other){
// Do more stuff
return other
});
}
// Usage
doSomething().then(function(other){
// other
});
Below is how to accomplish what you're trying to do with bluebird.
You can use Promise.resolve() and Promise.reject() within any function to return data in a Promise that can be used directly in your promise chain. Essentially, by returning with these methods wrapping your result data, you can make any function usable within a Promise chain.
var Promise = require('bluebird');
var doSomething = function(parameter) {
// Call our Promise returning function
return doAnotherThing()
.then(function(value) {
// Handle value returned by a successful doAnotherThing call
})
.catch(function(err) {
// if doAnotherThing() had a Promise.reject() in it
// then you would handle whatever is returned by it here
});
}
function doAnotherThing(parameter) {
var s = 'some data I got from the url';
// Return s wrapped in a Promise
return Promise.resolve(s);
}
You can use the async module and its waterfall method to chain the functions together:
var async = require('async');
async.waterfall([
function(parameter, callback) {
doSomething(parameter, function(err, other) {
if (err) throw err;
callback(null, other); // callback with null error and `other` object
});
},
function(other, callback) { // pass `other` into next function in chain
doAnotherThing(other, function(err, result) {
if (err) throw err;
callback(null, result);
})
}
], function(err, result) {
if (err) return next(err);
res.send(result); // send the result when the chain completes
});
Makes it a little easier to wrap your head around the series of promises, in my opinion. See the documentation for explanation.
I have some code that I cant get my head around, I am trying to return an array of object using a callback, I have a function that is returning the values and then pushing them into an array but I cant access this outside of the function, I am doing something stupid here but can't tell what ( I am very new to Node.JS )
for (var index in res.response.result) {
var marketArray = [];
(function () {
var market = res.response.result[index];
createOrUpdateMarket(market, eventObj , function (err, marketObj) {
marketArray.push(marketObj)
console.log('The Array is %s',marketArray.length) //Returns The Array is 1.2.3..etc
});
console.log('The Array is %s',marketArray.length) // Returns The Array is 0
})();
}
You have multiple issues going on here. A core issue is to gain an understanding of how asynchronous responses work and which code executes when. But, in addition to that you also have to learn how to manage multiple async responses in a loop and how to know when all the responses are done and how to get the results in order and what tools can best be used in node.js to do that.
Your core issue is a matter of timing. The createOrUpdateMarket() function is probably asynchronous. That means that it starts its operation when the function is called, then calls its callback sometime in the future. Meanwhile the rest of your code continues to run. Thus, you are trying to access the array BEFORE the callback has been called.
Because you cannot know exactly when that callback will be called, the only place you can reliably use the callback data is inside the callback or in something that is called from within the callback.
You can read more about the details of the async/callback issue here: Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
To know when a whole series of these createOrUpdateMarket() operations are all done, you will have to code especially to know when all of them are done and you cannot rely on a simple for loop. The modern way to do that is to use promises which offer tools for helping you manage the timing of one or more asynchronous operations.
In addition, if you want to accumulate results from your for loop in marketArray, you have to declare and initialize that before your for loop, not inside your for loop. Here are several solutions:
Manually Coded Solution
var len = res.response.result.length;
var marketArray = new Array(len), cntr = 0;
for (var index = 0, index < len; index++) {
(function(i) {
createOrUpdateMarket(res.response.result[i], eventObj , function (err, marketObj) {
++cntr;
if (err) {
// need error handling here
}
marketArray[i] = marketObj;
// if last response has just finished
if (cntr === len) {
// here the marketArray is fully populated and all responses are done
// put your code to process the marketArray here
}
});
})(index);
}
Standard Promises Built Into Node.js
// make a version of createOrUpdateMarket that returns a promise
function createOrUpdateMarketAsync(a, b) {
return new Promise(function(resolve, reject) {
createOrUpdateMarket(a, b, function(err, marketObj) {
if (err) {
reject(err);
return;
}
resolve(marketObj);
});
});
}
var promises = [];
for (var i = 0; i < res.response.result.length; i++) {
promises.push(createorUpdateMarketAsync(res.response.result[i], eventObj));
}
Promise.all(promises).then(function(marketArray) {
// all results done here, results in marketArray
}, function(err) {
// an error occurred
});
Enhanced Promises with the Bluebird Promise library
The bluebird promise library offers Promise.map() which will iterate over your array of data and produce an array of asynchronously obtained results.
// make a version of createOrUpdateMarket that returns a promise
var Promise = require('bluebird');
var createOrUpdateMarketAsync = Promise.promisify(createOrUpdateMarket);
// iterate the res.response.result array and run an operation on each item
Promise.map(res.response.result, function(item) {
return createOrUpdateMarketAsync(item, eventObj);
}).then(function(marketArray) {
// all results done here, results in marketArray
}, function(err) {
// an error occurred
});
Async Library
You can also use the async library to help manage multiple async operations. In this case, you can use async.map() which will create an array of results.
var async = require('async');
async.map(res.response.result, function(item, done) {
createOrUpdateMarker(item, eventObj, function(err, marketObj) {
if (err) {
done(err);
} else {
done(marketObj);
}
});
}, function(err, results) {
if (err) {
// an error occurred
} else {
// results array contains all the async results
}
});
I'm still struggling with promises, but making some progress thanks to the community here.
I have a simple JS function which queries a Parse database. It's supposed to return the array of results, but obviously due to the asynchronous nature of the query (hence the promises), the function returns before the results, leaving me with an undefined array.
What do I need to do to make this function wait for the result of the promise?
Here's my code:
function resultsByName(name)
{
var Card = Parse.Object.extend("Card");
var query = new Parse.Query(Card);
query.equalTo("name", name.toString());
var resultsArray = [];
var promise = query.find({
success: function(results) {
// results is an array of Parse.Object.
console.log(results);
//resultsArray = results;
return results;
},
error: function(error) {
// error is an instance of Parse.Error.
console.log("Error");
}
});
}
Instead of returning a resultsArray you return a promise for a results array and then then that on the call site - this has the added benefit of the caller knowing the function is performing asynchronous I/O. Coding concurrency in JavaScript is based on that - you might want to read this question to get a broader idea:
function resultsByName(name)
{
var Card = Parse.Object.extend("Card");
var query = new Parse.Query(Card);
query.equalTo("name", name.toString());
var resultsArray = [];
return query.find({});
}
// later
resultsByName("Some Name").then(function(results){
// access results here by chaining to the returned promise
});
You can see more examples of using parse promises with queries in Parse's own blog post about it.
What do I need to do to make this function wait for the result of the promise?
Use async/await (NOT Part of ECMA6, but
available for Chrome, Edge, Firefox and Safari since end of 2017, see canIuse)
MDN
async function waitForPromise() {
// let result = await any Promise, like:
let result = await Promise.resolve('this is a sample promise');
}
Added due to comment:
An async function always returns a Promise, and in TypeScript it would look like:
async function waitForPromise() {
// let result = await any Promise, like:
let result: Promise<string> = await Promise.resolve('this is a sample promise');
}
You're not actually using promises here. Parse lets you use callbacks or promises; your choice.
To use promises, do the following:
query.find().then(function() {
console.log("success!");
}, function() {
console.log("error");
});
Now, to execute stuff after the promise is complete, you can just execute it inside the promise callback inside the then() call. So far this would be exactly the same as regular callbacks.
To actually make good use of promises is when you chain them, like this:
query.find().then(function() {
console.log("success!");
return new Parse.Query(Obj).get("sOmE_oBjEcT");
}, function() {
console.log("error");
}).then(function() {
console.log("success on second callback!");
}, function() {
console.log("error on second callback");
});
You don't want to make the function wait, because JavaScript is intended to be non-blocking.
Rather return the promise at the end of the function, then the calling function can use the promise to get the server response.
var promise = query.find();
return promise;
//Or return query.find();