Node.js async module waterfall - dynamically load and execute functions - javascript

I am attempting to dynamically load a series of anonymous functions and execute them, passing the results of the previous to the next.
Here is an example function:
module.exports = function (data) {
// do something with data
return (data);
}
When the functions are loaded (they all sit in separate files) they are returned as an object:
{ bar: [Function], foo: [Function] }
I would like to execute these functions using async.waterfall. This takes an array of functions, not an object of functions, so I convert as follows:
var arr =[];
for( var i in self.plugins ) {
if (self.plugins.hasOwnProperty(i)){
arr.push(self.plugins[i]);
}
}
This gives:
[ [Function], [Function] ]
How can I now execute each function using each async.waterfall passing the result of the previous function to the next?
SOLUTION
Thanks to the comments from #piergiaj I am now using next() in the functions. The final step was to make sure that a predefined function was put first in the array that could pass the incoming data:
var arr =[];
arr.push(function (next) {
next(null, incomingData);
});
for( var i in self.plugins ) {
if (self.plugins.hasOwnProperty(i)){
arr.push(self.plugins[i]);
}
}
async.waterfall(arr,done);

If you want them to pass data on to the next one using async.waterfall, instead of returning at the end of each function, you need to call the next() method. Additionally, you need to have next be the last parameter to each function. Example:
module.exports function(data, next){
next(null, data);
}
The first parameter to next must be null because async.waterfall treats it as an error (if you do encounter an error in one of the methods, pass it there and async.waterfall will stop execute and finish passing the error to the final method).
You can then convert it like you already to (object to array), then call it like so:
async.waterfall(arrayOfFunctions, function (err, result) {
// err is the error pass from the methods (null if no error)
// result is the final value passed from the last method run
});

Related

d3.queue never triggering .await function

I've run into a problem using d3-queue. This is my code:
var dataQueue = d3.queue();
dataQueue.defer(collectData,ISBNs,locations)
.await(processData);
Where collectData is a function that does several API calls (a large number of them to the Google Books API).
Now the problem is that the processData function is never called. I know for a fact that the collectData function runs properly, since I put a print statement just before the return statement, along with several other print statements along the way.
You are not passing your data correctly between the deferred task collectData and the final processData. The documentation has it as follows (emphasis mine):
# queue.defer(task[, arguments…]) <>
Adds the specified asynchronous task callback to the queue, with any optional arguments. The task is a function that will be called when the task should start. It is passed the specified optional arguments and an additional callback as the last argument; the callback must be invoked by the task when it finishes. The task must invoke the callback with two arguments: the error, if any, and the result of the task.
Thus, to pass the result of the deferred task to the function processData, your function collectData() has to be something like this:
function collectData(ISBNs, locations, callback) {
var error = null; // The error, if any
var data = { }; // The actual data to pass on
// ...collect your data...
// Pass the collected data (and the error) by invoking the callback.
callback(error, data);
}

How does JavaScript know to execute the fail function?

I'm working with jasmine and Supertest on an existing project.
let checkResult = require('./check-result');
it('should do something', function(done){
request
.post('/route')
.expect(results => {
expect(results).toBeTruthy();
})
.end(checkResult(done));
});
When I console.log(done) I get the following output: { [Function] fail: [Function] }
Below is our checkResult module.
//check-result
module.exports = function checkResult(done){
return function(seeIfThereIsError){
if(seeIfThereIsError){
done.fail(seeIfThereIsError)
} else {
done()
}
}
};
When an error occurs the if(seeIfThereIsError) block executes.
I have two questions:
When passing done into checkResult how does the returned function in checkResult's seeIfThereIsError argument get populated?
How is the signature { [Function] fail: [Function] } get created?
In short how could I create an arbitrary example from scratch to understand the working parts (syntax) on how this all fits together?
When passing done into checkResult how does the returned function in checkResult's seeIfThereIsError argument get populated?
That function is passed to and called by end. I.e. end will pass a value to the function. In its simplest form it would look like:
request.end = function(callback) {
callback(false);
};
How is the signature { [Function] fail: [Function] } get created?
How console.log represents functions isn't standardized. This output simply tells you that the value is a function object that has a custom property, fail, which is also a function.
If you want to create such a value yourself, this is how it could be done:
function done() {}
done.fail = function() {};
Whether console.log(done) gives you the output you saw depends on the browser and potentially other implementation specific heuristics.

How to pass in a variable to a .then() callback function from an outer scope?

I have an API receiving an request and doing some querys and validation.
I want to use the response reference from the route inside the findUserPlayer function so I can send something to the client. The findUser function will return a data object that will be used on findUserPlayer.
var api = express.Router();
api.post('/signup', function (request, response) {
UserModel.find(req.body)
.then(findUser, createUser)
.then(findUserPlayer, createUserPlayer);
});
var findUserPlayer = function (data, res) {
//...
res.json(object);
};
I've tried calling findUserPlayer(data, res), which gives me 'data is not defined'.
I've tried calling findUserPlayer(res), which will override data.
How can I use the response object inside that callback?
Instead of directly passing findUserPlayer function as a success handler, you can define a success handler function while will then call findUserPlayer.
var api = express.Router();
api.post('/signup', function (request, response) {
UserModel.find(req.body)
.then(findUser, createUser)
.then(function(data) {
return findUserPlayer(data, response);
}, createUserPlayer);
});
var findUserPlayer = function (data, res) {
//...
res.json(object);
};
JavaScript is a very flexible language which means there are many ways you can handle this, one of the easiest (in terms of minimum code change) is to reverse the order of arguments in findUserPlayer and binding your context (.bind(response)).
But that would be a poor design choice. There's no reason for a function that "finds a user" to deal with responses or json. It's out of this functions scope and responsibility. The request handler is the one that should deal with formatting a relevant response, and I'd do that by returning the relevant data in the promise chain and eventually:
.then(function(data) {
response.json(data);
});
I'm a big fan of lodash and sometimes to a fault, but thought I would add the following using lodash's partialRight function.
var _ = require('lodash');
...
UserModel.find(req.body)
.then(findUser, createUser)
.then(_.partialRight(findUserPlayer, response), createUserPlayer);
_.partialRight returns a new function with the partials appended to the new function. The first argument of _.partialRight is the function we want to append the arguments to. The subsequent arguments are the args we want to append to the function.
I'm assuming the following function signature: findUserPlayer(data, response). _.partialRight appends the response object to the new function. The new function signature looks like this now: function (data).
documentation

Node.js calling a callback function inside a callback

I started learning node.js by reading Node JS in Action book recently. This is probably a newbie question, but after reading several posts of callback functions and javascript scope of variables, I still have problem understand the idea behind this code in chapter 5 of the book.
function loadOrInitializeTaskArray(file, cb) {
fs.exists(file, function(exists) {
var tasks = [];
if (exists) {
fs.readFile(file, 'utf8', function(err, data) {
if (err) throw err;
var data = data.toString();
var tasks = JSON.parse(data || '[]');
cb(tasks);
});
} else {
cb([]);
}
});
}
function listTasks(file) {
loadOrInitializeTaskArray(file, function(tasks) {
for(var i in tasks) {
console.log(tasks[i]);
}
});
}
It includes three callback functions divided in two functions. listTasks(..) is called first and it calls loadorInitializeTaskArray(.. ) later. My problem starts here, how is this call handle by node? loadOrInitializeTaskArray takes two arguments and the second one is the callback function which shouldn't accept any parameters according to is signature but it does!!
when does cb(..) get called in loadorInitializeTaskArray and what is that (the same function that calls helper function)?
"tasks" is an array declared inside function loadOrInitializeTaskArray, how do we have access to it in listTasks(..) function?
I know in Javascript, scope of a variable is inside the function it define and all nested functions. But I have a hard time understanding it here. So can someone explain what is going on here?
Thank you
You really are having a hard time in understanding the scope of variable inside the nested functions. So, lets start with it.
Let's consider this code
function foo(){
var a = 3;
function bar(){
console.log(a);
}
return bar;
}
var baz = foo();
baz(); // the value of a i.e 3 will appear on console
If you know langauges like C, C++... Your brain would be interpreting this code like this. If you don't know any of them just ignore thiese points.
When foo() is called, variable a will be declared and assigned value 3.
When foo is returned the stack containing a will be destroyed. And hence a would be destroyed too.
Then how in the world baz() is outputting 3. ???
Well, in javascript when a function is called the things that happen are different than in C. So, first let your all C things go out from your mind before reading further.
In javascript scope resolution is done by travelling down a chain of objects that defines variables that are "in scope" for that code. Let's see how?
When you declare a global JavaScript variable, what you are actually doing is defining a property of the global object.
In top-level JavaScript code (i.e., code not contained within any function definitions), the scope chain consists of a single object, the global object.
In a non-nested function, the scope chain consists of two objects. The first is the object that defines the function’s parameters and local variables, and the second is the global object.
In a nested function, the scope chain has three or more objects. When a function is defined, it stores the scope chain then in effect. When that function is invoked, it creates a new object to store its local variables, and adds that new object to the stored scope chain to create a new, longer, chain that represents the scope for that function invocation.
So, when foo() is executed. It creates a new object to store it's local variables. So, variable a will be stored in that object. And also, bar is defined. When bar is defined it stores the scope chain in effect. So, the scope chain of bar now contains the object that has variable a in it. So, when bar is returned it has a reference to it's scope chain. And hence it knows a.
So, I guess it answers your question how node handles the code.
You wrote:
loadOrInitializeTaskArray takes two arguments and the second one is the callback function which shouldn't accept any parameters according to is signature but it does!!
The callback function is
function(tasks) {
for(var i in tasks) {
console.log(tasks[i]);
}
}
And it accepts an argument tasks. So, you are wrong here.
And When loadOrIntializeTaskArray is called cb refers to this call back function. And cb(arg) basically does this tasks = arg where tasks is the argument in callback function.
I guess you still will be having a lot of questions. Let me know in comments. And I highly recommend you to go through the Core Javascript before diving into node.
First, there is not really any such thing as a function signature in javascript. You can pass as many or as few arguments to a function as you like. The second argument to loadOrInitialiseTaskArray is simply assigned to the local variable cb when the function is called. The line cb(tasks) then invokes this value, so the second argument had better have been a function.
When you call loadOrInitializeTaskArray from listTasks, the second argument is indeed a function, and the first argument to this function is named tasks in its own scope. It isn't reaching into loadOrInitializeTaskArray and using the variable declared in that function's scope. You explicitly passed in that value when you invoked cb.
The code would work the same and perhaps be easier to understand if we renamed the variables:
function loadOrInitializeTaskArray(file, cb) {
fs.exists(file, function(exists) {
var taskArray = [];
if (exists) {
fs.readFile(file, 'utf8', function(err, data) {
if (err) throw err;
var data = data.toString();
var taskArray = JSON.parse(data || '[]');
cb(taskArray);
});
} else {
cb([]);
}
});
}
function listTasks(file) {
loadOrInitializeTaskArray(file, function(listOfTasks) {
for(var i in listOfTasks) {
console.log(listOfTasks[i]);
}
});
}
I'm not sure what you mean by "the second one is the callback function which shouldn't accept any parameters according to is signature". The callback signature (function(tasks)) certainly does expect an argument (tasks).
cb is the callback function that was passed in. In this case it is literally the anonymous function:
function(tasks) {
for(var i in tasks) {
console.log(tasks[i]);
}
}
tasks is the name of two different variables (but they point to the same object because arrays are passed by reference) in different scopes. One is defined in loadOrInitializeTaskArray() as you noted, the other is a parameter for the callback passed to loadOrInitializeTaskArray().
Also, on an unrelated note, typically callbacks follow the "error-first" format. This allows for handling of errors upstream. Modifying the code to follow this convention would result in:
function loadOrInitializeTaskArray(file, cb) {
fs.exists(file, function(exists) {
var tasks = [];
if (exists) {
fs.readFile(file, 'utf8', function(err, data) {
if (err)
return cb(err);
var data = data.toString();
// JSON.parse() throws errors/exceptions on parse errors
try {
var tasks = JSON.parse(data || '[]');
} catch (ex) {
return cb(ex);
}
cb(null, tasks);
});
} else {
cb(null, []);
}
});
}
function listTasks(file) {
loadOrInitializeTaskArray(file, function(err, tasks) {
if (err) throw err;
for (var i in tasks) {
console.log(tasks[i]);
}
});
}

async.js - right method for probing

I have doubts on selecting async.js method for probing several alternatives and stop when the first of them is successful.
For example:
async.probeSeries([
function (callback) {
// try something and call callback
// without arguments - probing fails
callback();
},
function (callback) {
// try something and call callback
// with arguments - probing successful
callback(null, ok);
},
function (callback) {
// will be not executed, because
// the second one is successful
callback();
}
], function (err, result) {
// handle the result returned by the second probe
});
I think that using series and return the result as error way may be a workaround but is there a better way?
Maybe you are looking for detectSeries? It works a littlebit different than your example as it checks values from an array with the same function and then callbacks with one of these values, but maybe you can apply it to your problem.
Btw, this looks like the perfect use case for a promise library like Q, where you'd write
probe1().fail(probe2).fail(probe3).done(resulthandler, errhandler);

Categories