I am kind of confused with the logic of results which go from one task to the other task in async.auto. For example in the following code logic I added some data to models in task1, which is initially an output from initialtask and in finalTask added data to models from task1 is reflected in results.initialTask1 as well. Similarly added data in task2 is reflected in results.initialTask1 in finalTask.
To sum up all of results.initialTask1, results.task1[0], results.task2[0], results.task3[0] are identical in finalTask. Is this the logic of async.auto? Or is it something like reference by pointer in C++ which causes whatever changes for models in task1, it reflects in models in initialTask as well?
async.auto({
initialTask: function(callback) {
//Do some operations
callback(null, name, initialModels);
},
task1: ['initialTask', function(callback, results) {
var models = results.initialTask[1];
//Add some more data to models
callback(null, models);
}],
task2: ['initialTask', function(callback, results) {
var models = results.initialTask[1];
//Add some more data to models
callback(null, models);
}],
task3: ['initialTask', function(callback, results) {
var models = results.initialTask[1];
//Add some more data to models
callback(null, models);
}],
finalTask: ['task1', 'task2', 'task3', function(callback, results) {
//Here the followings are the same: results.initialTask[1], results.task1[0], results.task2[0], results.task3[0]
}]
});
I'm looking for any answer which helps me make sure that is the logic or not? I'm not necessarily looking for any official documents or ...
This is expected behavior. Basically async.auto will execute all the functions in the order it deems necessary. So in your case initialTask will be called first. Then task1, task2, and task3 will be called in parallel. Finally finalTask will be called with the results. The reason all the values are the same is related to JavaScript's call-by-sharing, meaning if you change a function parameter itself, then it won't affect the item that was fed into the parameter. If you change the internals of the parameter, it will carry up to the item.
More info here.
Example:
async.auto({
// this function will just be passed a callback
readData: async.apply(fs.readFile, 'data.txt', 'utf-8'),
showData: ['readData', function(results, cb) {
// results.readData is the file's contents
// ...
}]
}, callback);
async.auto({
get_data: function(callback) {
console.log('in get_data');
// async code to get some data
callback(null, 'data', 'converted to array');
},
make_folder: function(callback) {
console.log('in make_folder');
// async code to create a directory to store a file in
// this is run at the same time as getting the data
callback(null, 'folder');
},
write_file: ['get_data', 'make_folder', function(results, callback) {
console.log('in write_file', JSON.stringify(results));
// once there is some data and the directory exists,
// write the data to a file in the directory
callback(null, 'filename');
}],
email_link: ['write_file', function(results, callback) {
console.log('in email_link', JSON.stringify(results));
// once the file is written let's email a link to it...
// results.write_file contains the filename returned by write_file.
callback(null, {'file':results.write_file,
'email':'user#example.com'});
}]
}, function(err, results) {
console.log('err = ', err);
console.log('results = ', results);
});
async.auto is very useful and powerful function which is provided by Async Lib .it have 3 fields
1-task
2- concurrency
3-callback
In Async.auto, Each function depends on its parent function except the first function, if any function will get any error during execution .then their child function or say .. their below-defined function will not get executed further, an error will occur with callback and the main callback will immediately return with an error
1- Task :- an Object
2- concurrency :- An optional integer for determining the maximum number of tasks that can be run in parallel. By default, as many as possible.
3- callback:- return the response
exapmle-
AnyService.prototype.forgetPassword = function (res, email, isMobile, callback) {
Logger.info("In AnyService service forgetPassword email...", email);
db.User.findOne({
email: email.toLowerCase(),
deleted: false
}, function (err, user) {
if (!user) {
configurationHolder.responseHandler(res, null, configurationHolder.LoginMessage.registerFirst, true, 403)
} else {
async.auto({
token: function (next, results) {
return gereratePasswordToken(next, email, user, isMobile);
},
sendMail: ['token', function (next, result) {
return SendMailService.prototype.forgetPasswordMail(next, result.token, email, user.fullName);
}]
}, function (err, result) {
if (err == null && result != null) {
configurationHolder.ResponseUtil.responseHandler(res, null, configurationHolder.LoginMessage.forgotPassword, false, 200)
} else {
callback(new Error(configurationHolder.errorMessage.oops))
}
})
}
});
}
Related
I'm writing a function for my Node.js server (using express as a framework, MongoDB as the database). I have to call two mongoose queries:
(1) Find data filtered by parameters of one collection
(2) Aggregate data of another collection (needs result from the first query because one attribute gets appended to these results)
I now use the async.waterfall a function that makes it possible to ensure that the second operation only gets triggered after the first one has finished.
The variable resultResponse (at the end of my code snippet) has the right value when I run the function.
The problem I am struggling with:
The server throws the error TypeError: callback is not a function for the first callback.
This type error is also the answer I receive via postman calling /getRoutes.
So res.send() doesn't send the variable resultResponse back (which is correct), but the error.
Does anyone know, how I can solve this problem? Thanks a lot!
let async = require('async');
app.get('/getRoutes', function (req, res) {
var param = req.query;
let resultResponse;
async.waterfall([
(1)
function (param, callback) {
let dataCallback;
// first mongoose operation
Route.find(param).lean().exec(function (err, data) {
if (err)
throw err;
dataCallback = data;
}
.then(callback(null, dataCallback)));
},
TypeError: callback is not a function
(2)
function (aRoutes, callback) {
let oRoutes = [];
for (let i in aRoutes) {
// second mongoose operation
Rating.aggregate([{$match: {route: aRoutes[i]._id}}
, {
$group:
{_id: null, rating: {$avg: '$rating'}}
}
]).then(function (response) {
let oneRoute = aRoutes[i];
let avgRating = response[0].rating;
oneRoute.avg_rating = avgRating;
oRoutes.push(oneRoute);
if (i == (aRoutes.length - 1)) {
callback(null, oRoutes);
}
});
}
}],
function (err, result) {
if (err) throw err;
resultResponse = result; //console.log -> right result here
});
res.send(resultResponse); });
Your first function will be seeking callback function as the first parameter. Check here for more details. param is defined globally so you don't need to pass.
function (callback) {
let dataCallback;
// first mongoose operation
Route.find(param).lean().exec(function (err, data) {
if (err)
throw err;
dataCallback = data;
}).then(function(){
callback(null, dataCallback);
});
}
OR
Simply return data insteadof creating undefined object.
function (callback) {
// first mongoose operation
Route.find(param).lean().exec(function (err, data) {
if (err)
throw err;
return data;
}).then(function(dataCallback){
callback(null, dataCallback);
});
}
High level
I'm new to JS and Node. I'm working on an API endpoint that should return a status value. To compute the status value I need to make two sequential mongo queries where the second set of queries depend on the first query. The second set of queries will give me a status for each value found in the first query, of which I will pick one based on some logic. What is the best way to do it in NodeJS?
Specifics
Here are parts of my first attempt.
function getItemStatus(key, value, callback) {
MongoClient.connect(mongo_url, function(err, db) {
if (err) { return console.dir(err); }
db.collection('status', function(err, coll) {
if (err) { return console.dir(err); }
coll.distinct("_id.metric", function(err, metrics) {
if (err) { return console.dir(err); }
console.log('metrics : ', metrics);
_.foreach(metrics, function(metric) {
var query = {"_id": {
"$gte" : {key: key, value: value, created: new Date("1800-01-01T00:00:00"), metric : metric},
"$lte" : {key: key, value: value, created: new Date("2100-01-01T00:00:00"), metric : metric}}};
coll.find(query, {sort: {"_id.created": -1}, limit: 1})
I make a connection, query for a set of metric values using a distinct query. For each metric I then want to ask for the latest status. Ideally I'd like to have the entire set of statuses so that I could write a function taking this set and deciding on which status will be returned. My problem is passing the statuses back "up the chain" so that I can process the set of statuses.
In a synchronous situation I would simply write something like this
val metrics = getDistinctMetrics(key, value)
val statuses = metrics.map(getStatusForMetric)
val status = filterStatuses(statuses)
How can I accomplish this in JavaScript/NodeJS?
UPDATED to highlight the fact that the first queries will trigger several queries in the second step, i.e. one for each result found by the first query.
As I understand your question you want to execute queries parallel or in a waterfall mode and do some logic on the final result. You should look into a library allowing parallel/waterfall execution. Like this
Waterfall: Waterfall
async.waterfall([
function(callback) {
callback(null, 'one', 'two');
},
function(arg1, arg2, callback) {
// arg1 now equals 'one' and arg2 now equals 'two'
callback(null, 'three');
},
function(arg1, callback) {
// arg1 now equals 'three'
callback(null, 'done');
}
], function (err, result) {
// result now equals 'done'
});
Parallel: Parallel
async.parallel({
collectionOne: function (callback) {
collectionOne.find(query, function (err, result) {
if (err) {
return handleError(res, err);
}
callback(null, result);
})
},
collectionTwo: function (callback) {
collectionTwo.find(query, function (err, result) {
if (err) {
return handleError(res, err);
}
callback(null, result);
})
},
collectionThree: function (callback) {
collectionThree.find(query, function (err, result) {
if (err) {
return handleError(res, err);
}
callback(null, result);
})
},
collectionFour: function (callback) {
collectionFour.find(query, function (err, result) {
if (err) {
return handleError(res, err);
}
callback(null, result);
})
},
}, function (err, results) {
return res.status(200).json(results);
});
And in the final callback you can doo some logic or return response.
In your sample code, you are making network calls inside another network calls, which can lead to callback hell which can lead to misbehave of queries, in order to overcome that, you can use promises.
This will help you in avoiding callback hell as well as your query will also be resolved.
Sample code:-
new Promise (function(resolve, reject){
return db.collection.find(function).exec()
.then(function(result_of_first_query){
return db.collection.findOne(function).exec() //in this yopu can user the result of first query
}).then(function(result_of_second_query){
resolve(result_of_second_query);
})
})
You can add more queries with each .then
I'm working with NodeJS and the Async api to create an api function that returns a list of stories. The get can be shallow (only contains Object ID's referencing other objects) or deep (all ID references are dereferenced by replacing the ID with the object referenced by it). The shallow get works fine, however when I run the deep copy, it hangs. You can see in my callbacks I placed console.log(#) to log which callback is fired, but none are fired.
I feel like the issue lies within if I'm mistaking how async handles the callback function parameter for the .each, .serial and .parallel functions. I need a function that will be fired once async completes all of its tasks, but the callback function is instead called after every operation each, serial or parallel completed.
router.get('/stories', function(req, res, next) {
var db = req.db,
options = {
deep : req.query.deep != null ? parseInt(req.query.deep) : false,
offset : req.query.offset || 0,
limit : req.query.limit || 0
};
Story.listStories(db, options, function(err, stories){
if (!options.deep){
res.json(new Response(err, stories));
res.end();
}else{
if (err || stories == null){
res.json(new Response(err, null));
res.end();
return;
}
async.each(stories,
function(story, cb1){
var articles = [],
galleries = [];
async.series([
function(cb2){
async.parallel([
//Extract the story's articles and their outlets
function(cb3){
async.each(story.articles,
function(article_id, cb4){
Article.getArticle(db, article_id, function(err, article){
if (err){
cb4(err);
return;
}
Outlet.getOutlet(db, article.outlet, function(err, outlet){
if (err){
cb4(err);
return;
}
article.outlet = outlet;
articles.push(article);
});
});
},
function(err){console.log(4);
if (err)
cb3(err);
});
}
],
function(err){console.log(3); //Parallel callback
if (err)
cb1(err);
});
},
function(cb2){
story.articles = articles;
}
],
function(err){console.log(2);
if (err)
cb1(err);
});
},
function(err){console.log(1);
res.json(new Response(err, stories));
res.end();
}
);
}
});
});
You're calling those async callbacks (cb1, cb2, cb3, cb4, and etc) only for error cases. you need to call for non-error cases also. Example:
if (err) {
return cb1(err);
}
cb1(null); // or cb1()
I need to request data from two web servers. The tasks are independent; therefore, I am using aync.parallel. Now I am only writing 'abc', 'xyz', and 'Done' to the body of my web page.
Since tasks are performed at the same time, can I run into a strange output? E.g.,
xab
cyz
The code.
var async = require('async');
function onRequest(req, res) {
res.writeHead(200, {
"Content-Type" : "text/plain"
});
async.parallel([ function(callback) {
res.write('a');
res.write('b');
res.write('c\n');
callback();
}, function(callback) {
res.write('x');
res.write('y');
res.write('z\n');
callback();
} ], function done(err, results) {
if (err) {
throw err;
}
res.end("\nDone!");
});
}
var server = require('http').createServer(onRequest);
server.listen(9000);
If you want to be absolutely certain in the order in which the results are printed, you should pass your data (abc\n and xyz\n) through the callbacks (first parameter is the error) and handle/write them in the final async.parallel callback's results argument.
async.parallel({
one: function(callback) {
callback(null, 'abc\n');
},
two: function(callback) {
callback(null, 'xyz\n');
}
}, function(err, results) {
// results now equals to: results.one: 'abc\n', results.two: 'xyz\n'
});
I want to know how to handle dependencies in using the async library in Node.js, look at the following example:
db.open(function(error, client) {
client.collection(('my-collection', function(error, collection) {
collection.insert({ "email": "test#gmail.com" }, function(error, docs) {
// Do stuff.
});
});
});
Using the async library:
async.parallel([
function(callback) {
db.open(function(error, client) {
callback(error, client);
});
},
function(callback) {
// How do I access the "client" variable at this point?
}
],
function(results){
// Do stuff.
});
You are using async parallell, which runs all functions together and they can finish in any order.
you can use the async waterfall function which passes varibles from one callback to the next function eg.
async.waterfall([
function(callback){
callback(null, 'one', 'two');
},
function(arg1, arg2, callback){
callback(null, 'three');
},
function(arg1, callback){
// arg1 now equals 'three'
callback(null, 'done');
}
], function (err, result) {
// result now equals 'done'
});
or you can use the auto function, which allowes you to specify which other functions must finish first eg.
async.auto({
get_data: function(callback){
// async code to get some data
},
make_folder: function(callback){
// async code to create a directory to store a file in
// this is run at the same time as getting the data
},
write_file: ['get_data', 'make_folder', function(callback){
// once there is some data and the directory exists,
// write the data to a file in the directory
callback(null, filename);
}],
email_link: ['write_file', function(callback, results){
// once the file is written let's email a link to it...
// results.write_file contains the filename returned by write_file.
}]
});
so what you could do in your case is this:
async.auto({
dbReady: function(callback) {
db.open(function(error, client) {
callback(error, client);
});
},
connected: ['dbReady', function(callback, results){
// How do I access the "client" variable at this point?
console.log(results.dbReady);
}
},
function(results){
// Do stuff.
});
Have a look at this to see all the available functions and their uses