We Keep Coding

Calling a callback function from async.each and processing the results

I'm just starting to work with Javascript and Node, and Async and callbacks concepts are not something I have under control right now.
I have to call a function for each element of a documents Array. This function will call to DB and get me an array of the document annotations. I want to get all the annotations and put them on the same array. Something similar to this:
//function in an async waterfall
function(docs,callback){
let annotationsArray = [];
async.each(docs, (doc, callback2) => {
getAnnotationsFromDocument(doc.Id, callback2);
}, function (err,annotations){
if (err){
callback(err);
}
annotationsArray = annotationsArray.concat(annotations);
callback(null, annotationsArray);
});
},
//Next waterfall function
About the getAnnotationsFromDocument function, this is a simplified structure of it:
function getAnnotationsFromDocument(docId,callback){
initDB();
var async = require('async');
async.waterfall([
function authorize(callback){
//checkAuthorization
(...)
},
function getRfpdocAnnotations(auth, metadata, callback){
//call to DB
(...)
},
function processRfpdocAnnotations(rfpDocAnnotations,metadata,callback){
(...)
callback(null,annotationsList);
}
], function (err, result) {
if(err) {
callback(err);
} else {
callback(null, result);
}
});
}
Unfortunately, I'm unable to code it properly. I'm unable to get the results from the function before exiting the async.each. Could somebody explain me how to structurate the code for this?
Debugging I've found that the function getAnnotationsFromDocument gets the data and execute the last callback(null, result); properly, but when I get to function (err,annotations){, annotations is undefined.

Ok, I think I got it:
First problem was that async.each doesn't return the results on the callback like I was expecting. Unlike waterfall, it just returns the errors. I should have payed more attention reading the documentation.
Secondly, I had to create a callback on the getAnnotationsFromDocument call to process the results.
And finally, I was not executing the call to the callback of async.each, so the execution didn't get to the async.each callback and didn't continue to the next async.waterfall function.
To be quite honest, I'm not sure it's a correct answer, but it does what I was trying to achieve.
// function part of an async.waterfall
function(docs,callback){
let annotationsArray = [];
async.each(docs, (doc,callback2) => {
getAnnotationsFromDocument(doc._id, function(err,result){
if (err){
callback2(err);
}else{
annotationsArray = annotationsArray.concat(result);
}
callback2();
})
}, (err) =>{
if( err ) {
callback(err);
} else {
callback(null,annotationsArray); //to the next waterfall function
}
});

What is a simple implementation of async.waterfall?

I'm using some functions from the async library, and want to make sure I understand how they're doing things internally; however, I'm stuck on async.waterfall (implementation here). The actual implementation uses other functions from within the library, and without much experience, I'm finding it difficult to follow.
Could someone, without worrying about optimization, provide a very simple implementation that achieves waterfall's functionality? Probably something comparable to this answer.
From the docs, waterfall's description:
Runs the tasks array of functions in series, each passing their
results to the next in the array. However, if any of the tasks pass an
error to their own callback, the next function is not executed, and
the main callback is immediately called with the error.
An example:
async.waterfall([
function(callback) {
callback(null, 'one', 'two');
},
function(arg1, arg2, callback) {
// arg1 now equals 'one' and arg2 now equals 'two'
callback(null, 'three');
},
function(arg1, callback) {
// arg1 now equals 'three'
callback(null, 'done');
}
], function (err, result) {
// result now equals 'done'
});

Well, here is a simple implementation for chaining functions by queuing them.
First of all, the function:
function waterfall(arr, cb){} // takes an array and a callback on completion
Now, we need to keep track of the array and iterate it:
function waterfall(arr, cb){
var fns = arr.slice(); // make a copy
}
Let's start with handling passed and empty array, by adding an extra parameter so we can pass results around called result:
function waterfall(arr, cb, result){ // result is the initial result
var fns = arr.slice(); // make a copy
if(fns.length === 0){
process.nextTick(function(){ // don't cause race conditions
cb(null, result); // we're done, nothing more to do
});
}
}
That's nice:
waterfall([], function(err, data){
console.log("Done!");
});
Now, let's handle actually having stuff in:
function waterfall(arr, cb, result){ // result is the initial result
var fns = arr.slice(1); // make a copy, apart from the first element
if(!arr[0]){ // if there is nothing in the first position
process.nextTick(function(){ // don't cause race conditions
cb(null, result); // we're done, nothing more to do
});
return;
}
var first = arr[0]; // get the first function
first(function(err, data){ // invoke it
// when it is done
if(err) return cb(err); // early error, terminate entire call
// perform the same call, but without the first function
// and with its result as the result
waterfall(fns, cb, data);
});
}
And that's it! We overcome the fact we can't loop with callbacks by using recursion basically. Here is a fiddle illustrating it.
It's worth mentioning that if we were implementing it with promises we could have used a for loop.

For those who like to keep it short :
function waterfall(fn, done){
fn.length ? fn.shift()(function(err){ err ? done(err) : waterfall(fn, done) }) : done();
}

Node.js: Asynchronous callback confusion

I am trying to figure out how to create asynchronous functions for a web app. I am doing a database query, manipulating the data to a format that is more convenient, and then trying to set my router to pass back that file.
//Module 1
//Module 1 has 2 functions, both are necessary to properly format
function fnA(param1){
db.cypherQuery(query, function(err, result){
if(err){
return err;
}
var reformattedData = {};
//code that begins storing re-formatted data in reformattedData
//the function that handles the rest of the formatting
fnB(param1, param2);
});
});
function fnB(param1, reformattedData){
db.cypherQuery(query, function(err, result){
if(err){
return err;
}
//the rest of the reformatting that uses bits from the second query
return reformattedData;
});
});
exports.fnA = fnA;
Then in my router file:
var db = require('module1');
router.get('/url', function(req,res,next){
db.fnA(param1, function(err, result){
if (err){
return next(err);
}
res.send(result);
});
});
When I tried to test this out by hitting the URL indicated by the router, it just loads indefinitely.
I know what I have above is wrong, since I never wrote my function to require a callback. When I tried figuring out how to rewrite it though, I got really confused - How do I write my function to have a callback when the asynchronous stuff happens inside it?
Can someone help me rewrite my functions to use callbacks correctly, so that when I actually use the function, I can still work with the asynchronous response?

You use db.fa from your router file, and pass the second parameter as a callback function. but the function signature don't have the cb param and doesnt use it.
The main idea - you try to initiate an async operation and cannot know when it ends, so you send it a callback function to get triggered when all operations are done.
Fixed code should be like that:
//Module 1
//Module 1 has 2 functions, both are necessary to properly format
function fnA(param1, cb1){
db.cypherQuery(query, function(err, result){
if(err){
cb1(err); <-- return error to original call
}
var reformattedData = {};
//code that begins storing re-formatted data in reformattedData
//the function that handles the rest of the formatting
fnB(param1, param2, cb1);
});
});
function fnB(param1, reformattedData, cb1){
db.cypherQuery(query, function(err, result){
if(err){
cb1(err); <-- return error to original call
}
//the rest of the reformatting that uses bits from the second query
cb1(false, dataObjectToSendBack); <--- This will call the anonymouse function in your router call
});
});
exports.fnA = fnA;
Router file:
var db = require('module1');
router.get('/url', function(req,res,next){
db.fnA(param1, function(err, result){ <-- This anonymous function get triggered last
if (err){
return next(err);
}
res.send(result);
});
});

Async function works when called once but not multiple occasions

I have used async.waterfall to flatten out a nested function as per the below.
function convertPortfolio(trademarks, fn){
async.waterfall([function(callback){
groupByCountry(trademarks, callback)
},
function(TMsGroupedByCountry, callback){
addEUTrademarks(TMsGroupedByCountry['EU'], TMsGroupedByCountry, callback)
},
function(revisedGroupByCountry, callback){
groupTrademarksByStatus(revisedGroupByCountry, callback)
}], function(err, splitByStatus){
fn(null, splitByStatus);
})
}
Nested alternative
function convertPortfolio(trademarks, fn){
groupByCountry(trademarks, function(err, TMsGroupedByCountry){
addEUTrademarks(TMsGroupedByCountry['EU'], TMsGroupedByCountry, function(err, revisedGroupByCountry){
groupTrademarksByStatus(revisedGroupByCountry, function(err, splitByStatus){
fn(null, splitByStatus)
});
});
});
}
Subsequently when I call this function once as part of another function, it works perfectly. However when I attempt to call the function multiple times using a separate value from a forEach call on an array, it fails to work, when the nested version works fine. I am at a loss to explain why this would be the case and in all honesty, I'm not sure where to start. My understanding is that the forEach call should simply mean that each separate value is processed accordingly and closed over so this shouldn't be the issue.
Async function works correctly and returns value in this instance
exports.convertPortfolioAndAddToGeoJSON = function(geojson, trademarks, fn){
convertPortfolio(trademarks, function(err, revisedTMs){
addToGeoJson(geojson, revisedTMs, function(err, gj){
fn(null, gj)
});
});
}
But in this instance the end target object is not populated:
exports.convertYearPortfolioAndAddToGeoJSON = function(geojson, trademarks, fn){
var target = {};
Object.keys(trademarks).forEach(function(year){
convertPortfolio(trademarks[year], function(err, revisedTMs){
addToGeoJson(geojson, revisedTMs, function(err, revisedGeoJSON){
target[year] = revisedGeoJSON;
});
});
});
fn(null, target);
}
Using console.log at certain points shows that in the nested example, the return values are logged prior to the target object being logged, whereas with the async.waterfall example, the target object is logged prior to the returned data being available (so I suppose it is not surprising that logging the target results in an empty object).
I thought in each case that the presence of callbacks, would mean that the logging of target would only take place once all previous processed had been completed, but although this appears to be the case with the nested alternative, this is not so with the async version at least when it is called multiple times.
Any guidance would be appreciated.
UPDATE Out of interest here's the revised code using async.forEach:
exports.convertYearPortfolioAndAddToGeoJSON = function(geojson, trademarks, fn){
var target = {};
async.forEach(Object.keys(trademarks), function(year, callback){
async.waterfall([
async.apply(convertPortfolio, trademarks[year]),
function(revisedTMs, callback){
addToGeoJson(geojson, revisedTMs, callback)
}],
function(err, revisedGeoJSON){
target[year] = revisedGeoJSON;
callback()
})
}, function(err){
fn(null, target);
});
}

Object.keys(trademarks).forEach is synchronous and doesn't properly track closures/scope. You need to use async.forEach(Object.keys(trademarks), function(year, callback).... there, and adjust the async control flow accordingly.
Also just FYI when you have this pattern with a tiny wrapper function:
function convertPortfolio(trademarks, fn){
async.waterfall([function(callback){
groupByCountry(trademarks, callback)
},
You can use async.apply for that boilerplate:
function convertPortfolio(trademarks, fn){
async.waterfall([async.apply(groupByCountry, trademarks),
...

NodeJS async callback. How to return the list, from a async callback?

So basically I am making a database query, to get all posts with a certain id, then add them to a list, so I can return. But the list is returned, before the callback has finished.
How do I prevent it from being returned before callback has finished?
exports.getBlogEntries = function(opid) {
var list12 =[];
Entry.find({'opid' : opid}, function(err, entries) {
if(!err) {
console.log("adding");
entries.forEach( function(currentEntry){
list12.push(currentEntry);
});
}
else {
console.log("EEEERROOR");
}
//else {console.log("err");}
});
console.log(list12);
return list12;
};

ALL callback is asynchronous, so we don't have any guarantee if they will run exactly in the order we have leave them.
To fix it and make the process "synchronous" and guarantee an order executation you have two solutions:
First: make all process in nested list:
instead of this:
MyModel1.find({}, function(err, docsModel1) {
callback(err, docsModel1);
});
MyModel2.find({}, function(err, docsModel2) {
callback(err, docsModel2);
});
use this:
MyModel1.find({}, function(err, docsModel1) {
MyModel2.find({}, function(err, docsModel2) {
callback(err, docsModel1, docsModel2);
});
});
The last snippet above guarantee us that MyModel2 will be executed AFTER MyModel1 is executed.
Second: Use some framework as Async. This framework is awesome and have several helper functions to execute code in series, parallels, whatever way we want.
Example:
async.series(
{
function1 : function(callback) {
//your first code here
//...
callback(null, 'some result here');
},
function2 : function(callback) {
//your second code here (called only after the first one)
callback(null, 'another result here');
}
},
function(err, results) {
//capture the results from function1 and function2
//if function1 raise some error, function2 will not be called.
results.function1; // 'some result here'
results.function2; // 'another result here'
//do something else...
}
);

You could use sync database calls but that would work around the concept of node.js.
The proper way is to pass a callback to the function that queries the database and then call the provided callback inside the database query callback.

How do I prevent it from being returned before callback has finished?
The callback is asynchronous, and you cannot avoid that. Hence, you must not return a list.
Instead, offer a callback for when it's filled. Or return a Promise for the list. Example:
exports.getBlogEntries = function(opid, callback) {
Entry.find({'opid': opid}, callback); // yes, that's it.
// Everything else was boilerplate code
};

There is an alternate way to handle this scenario. You can use the async module and when the forEach has finished then make the return call. Please find the code snippet below for the same:
var async = requires('async');
exports.getBlogEntries = function(opid) {
var list12 =[];
Entry.find({'opid' : opid}, function(err, entries) {
if(!err) {
console.log("adding");
async.forEachSeries(entries,function(entry,returnFunction){
list12.push(entry);
},function(){
console.log(list12);
return list12;
});
}
else{
console.log("EEEERROOR");
}
});
};