I am using async.js for my node.js app. I need help solving the following problem.
Let's say I have a following aysnc js series function.
async.waterfall([
function getDataFromDB(request, response, callback){ ... },
function someOperationOnDBData(dbData, response, callback){ ... },
function renderSomeFlow(evaluatedData, response, callback){ ... }
]);
I have three functions called in the order mentioned above. I am getting data from getDataFromDB and passing to someOperationOnDBData and so on.
Suppose I need one more operation in between getDataFromDB and someOperationOnDBData but still pass DBData forward. Eg:
async.waterfall([
function getDataFromDB(request, response, callback){ ... },
function extraOperation(dbData, response, callback) {...}
function someOperationOnDBData(dbData, extraOperationData, response, callback){ ... },
function renderSomeFlow(evaluatedData, response, callback){ ... }
]);
Here adding a single step in the middle changes the function definitions and also I need to pass dbData in extraOperation to just forward it to someOperationOnDBData.
Also, if I am calling a different module in the middle, it might not be possible to change it's parameter to forward some data.
How to solve this problem of passing data between functions in async.js without forwarding data in middle functions? Refactoring functions every time a new step is included is not possible. What is the design pattern for solving this type of problem?
Using waterfall you have to pass the data through - you can't stop a waterfall in the middle :). There are alternatives that you might like to use, for example, auto which lets you specify which functions depend on the results of other functions and async will determine the best order to run them.
I find the syntax of auto a little awkward but it does what you need. Here is an example:
async.auto({
db: function getDataFromDB(callback){ ... },
extra: ['db', function extraOperation(results, callback) {...}],
some: ['db', function someOperationOnDBData(results, callback){ ... }],
render: ['some', 'db', function renderSomeFlow(results, callback){ ... }]
});
I've found that currying your functions with arrow functions is a way arround this problem:
const someOperationOnDBData = extraOperationData => (dbData, response, callback) => {
// do your function work
};
async.waterfall([
function getDataFromDB(request, response, callback){ ... },
function extraOperation(dbData, response, callback) {...}
someOperationOnDBData(extraOperationData)(dbData, response, callback),
function renderSomeFlow(evaluatedData, response, callback){ ... }
]);
Many good examples on how to curry javascript functions on Medium.com
Related
I tried to execute a series of events in my Node.js project with async as follows:
server.js:
var express = require('express');
var async = require('async');
...
app.get('/my_page', function (req, res) {
async.series([
function() { console.log("calling foo()"); },
function() { foo(); },
function() { console.log("foo() done"); },
]);
res.render('my_page', {});
}
but I only get the first console output and it's stuck.. the function()s were function(callback)s before. I thought it was waiting a value to be returned but removing them doesn't change the situation..
What am I doing wrong? (I'm newbie to Node.js)
Thanks for any help,
Please read the documentation. Each function should get callback as a parameter, and should call callback once it's done, thus telling async it's time to move on to the next function. So you should have something like:
function(callback) {
console.log("calling foo()");
callback();
},
If there was an error in one of the functions, call callback with the error as the 1st parameter. If you want res.render('my_page', {}); to be executed only after the last function is executed, wrap it in a function and put it as the 2nd parameter to async.series.
And of course, if non of the functions is asynchronous, you should consider not using async.
Firstly, I'm not sure why you're using async here, since none of the functions you've provided to the async.series function are asynchronous.
async.series is useful for calling multiple asynchronous functions in a row and avoiding callback hell.
var async = require("async");
function asyncA(callback){
// do something
callback(null, "value-a");
}
function asyncB(callback){
// do something else
callback(null, "value-b");
}
asyncA(function(err, valueA){
console.log(valueA); // "value-a"
asyncB(function(err, valueB){
console.log(valueB); // "value-b"
});
});
async.series([asyncA, asyncB], function(err, results){
console.log(results); // ["value-a", "value-b"]
}
I suggest doing some reading on asynchronous functions and callbacks and consider whether you really need the async library. I recommend the You Don't Know JS chapter on asynchronous javascript and the series as a whole :)
I'm using node.js and the async package.
Here's the code I have:
async.waterfall(
[
function(callback) {
var data = getSomeData();
callback(null, data);
},
function(data, callback) {
someFunctionThatNeedsData(data);
callback(null, 'done');
}
],
function(err, result) {
}
);
getSomeData has an asynchronous HTTP request that grabs some data from a web service. I'd like to wait until I get a response, and then return that data and pass it to someFunctionThatNeedsData.
What I expected was that getSomeData -- including the callback inside of it -- would have to complete before moving on to invoke someFunctionThatNeedsData.
The problem is that, despite using the waterfall function here, data is undefined by the time it gets to someFunctionThatNeedsData.
Additionally, from console.log I can see that the end of getSomeData is reached before the callback inside of getSomeData even begins.
Am I using waterfall incorrectly, or is it just not the right tool here? If it's just not right, what can I use to achieve the desired effect?
Or do I have to resign to having deeply nested callbacks (which, with future work, I will) and have to just mitigate it by extracting inline code into named functions?
getSomeData() has an asynchronous http request that grabs some data from a web service.
This is the issue. The execution flow already continued to the callback and executed it. This is how asynchronous functions work!
You have to pass the callback to getSomeData, which calls it once the HTTP request finished. So yes: You may need to nest the callbacks.
If you have async operation. You don't necessary to use async.waterfall. You could just do that in a promise chain style.
getSomeData().then(function(data)
{
var changeData = changeYourData(data);
return changeData;
}).then(function(changedData)
{
// some more stuff with it. You can keep on forwarding to the next `then`
}).catch(function(err)
{
// if any error throw at any point will get catch here
}).finally(function()
{
// this one will guarantee get call no matter what,
// exactly the same like async.waterfall end of chain callback
});
This example will work with Q, When, and any promise lib that follow standard.
If you need to use async.waterfall (because you could drive it with an Array.map)
You just need to callback in your then
async.waterfall(
[
function(callback) {
// A
getSomeData().then(function(data)
{
callback(null, data);
});
// B - just throw the whole thing in
callback(null , getSomeData());
},
function(data, callback) {
// A
someFunctionThatNeedsData(data);
// B
data.then(function(resolvedData)
{
someFunctionThatNeedsData(resolvedData);
callback(null, 'done');
});
}
],
function(err, result) {
});
Hope this help.
I have doubts on selecting async.js method for probing several alternatives and stop when the first of them is successful.
For example:
async.probeSeries([
function (callback) {
// try something and call callback
// without arguments - probing fails
callback();
},
function (callback) {
// try something and call callback
// with arguments - probing successful
callback(null, ok);
},
function (callback) {
// will be not executed, because
// the second one is successful
callback();
}
], function (err, result) {
// handle the result returned by the second probe
});
I think that using series and return the result as error way may be a workaround but is there a better way?
Maybe you are looking for detectSeries? It works a littlebit different than your example as it checks values from an array with the same function and then callbacks with one of these values, but maybe you can apply it to your problem.
Btw, this looks like the perfect use case for a promise library like Q, where you'd write
probe1().fail(probe2).fail(probe3).done(resulthandler, errhandler);
Probably asked before, but after the serious searching I'm still not able to find a proper solution. Please consider something like this:
function compute() {
asyncCall(args, function(err, result) {
});
/* 'join thread here' */
}
Even though asyncCall is asynchronous I'd like to use the result and return it from the function compute synchronously. asyncCall is a library call and I can't modify it in any way.
How to wait properly for the asynchronous result without setTimeout and watching a conditional variable? This is possible but suboptimal.
not sure how you can really use something that doesn't exist yet, but it's easy enough to return a slot where the result will be:
function compute() {
var rez=[];
asyncCall(args, function(err, result) {
rez[0]=result;
if(rez.onchange){ rez.onchange(result); }
});
/* 'join thread here' */
return rez;
}
now, you can refer to the [0] property of the return, and once the callback comes in, compute()[0] will have the result. It will also fire an event handler you can attach to the returned array that will fire when the data updates inside the callback.
i would use something more formal like a promise or secondary callback, but that's me...
EDIT: how to integrate a callback upstream:
// sync (old and busted):
function render(){
var myView=compute();
mainDiv.innerHTML=myView;
}
//async using my re-modified compute():
function render(){
var that=compute();
that.onchange=function(e){ mainDiv.innerHTML=e; }
}
see how making it wait only added a single wrapper in the render function?
There's no await syntax in browsers that is widely available. Your options are generally limited to Callback patterns or Promises.
NodeJS follows a callback pattern for most async methods.
function someAsyncMethod(options, callback) {
//callback = function(error, data)
// when there is an error, it is the first parameter, otherwise use null
doSomethingAsync(function(){
callback(null, response);
});
}
....
someAsyncMethod({...}, function(err, data) {
if (err) return alert("OMG! FAilZ!");
// use data
});
Another common implementation is promises, such as jQuery's .ajax() method...
var px = $.ajax({...});
px.data(function(data, xhr, status){
//runs when data returns.
});
px.fail(function(err,xhr, status){
//runs when an error occurs
});
Promises are similar to events...
Of the two methods above, the callback syntax tends to be easier to implement and follow, but can lead to deeply nested callback trees, though you can use utility patterns, methods like async to overcome this.
What's the best way/library for handling multiple asynchronous callbacks? Right now, I have something like this:
_.each(stuff, function(thing){
async(thing, callback);
});
I need to execute some code after the callback has been fired for each element in stuff.
What's the cleanest way to do this? I'm open to using libraries.
Since you're already using Underscore you might look at _.after. It does exactly what you're asking for. From the docs:
after _.after(count, function)
Creates a version of the function that will only be run after first being called count times. Useful for grouping asynchronous responses, where you want to be sure that all the async calls have finished, before proceeding.
There is a great library called Async.js that helps solve problems like this with many async & flow control helpers. It provides several forEach functions that can help you run callbacks for every item in a an array/object.
Check out:
https://github.com/caolan/async#forEach
// will print 1,2,3,4,5,6,7,all done
var arr = [1,2,3,4,5,6,7];
function doSomething(item, done) {
setTimeout(function() {
console.log(item);
done(); // call this when you're done with whatever you're doing
}, 50);
}
async.forEach(arr, doSomething, function(err) {
console.log("all done");
});
I recommend https://github.com/caolan/async for this. You can use async.parallel to do this.
function stuffDoer(thing) {
return function (callback) {
//Do stuff here with thing
callback(null, thing);
}
}
var work = _.map(stuff, stuffDoer)
async.parallel(work, function (error, results) {
//error will be defined if anything passed an error to the callback
//results will be an unordered array of whatever return value if any
//the worker functions passed to the callback
}
async.parallel() / async.series should suit your requirement. You can provide with a final callback that gets executed when all the REST calls succeed.
async.parallel([
function(){ ... },
function(){ ... }
], callback);
async.series([
function(){ ... },
function(){ ... }
], callback);
Have a counter, say async_count. Increase it by one every time you start a request (inside you loop) and have the callback reduce it by one and check if zero has been reached - if so, all the callbacks have returned.
EDIT: Although, if I were the one writing this, I would chain the requests rather than running them in parallel - in other words, I have a queue of requests and have the callback check the queue for the next request to make.
See my response to a similar question:
Coordinating parallel execution in node.js
My fork() function maintains the counter internally and automatically.