I have a controller which accepts an API call /task/:id/start. My controller method needs to check if Task with at id is valid and if that's valid then I need to create 2 other model instances. I need to create TaskSet and then TaskSetEvents.
TaskSet requires task to be created and TaskSetEvents requires TaskSet to be created. Here is how I'm planning on creating these events. I'm not sure if there is a better way of creating these objects.
TaskSet.create({ task: task}).exec(function(err, taskSet) {
TaskSetEvent.create({ taskSet: taskSet, eventType: 'start'}).exec(function (err, taskSetEvent) {
console.log("Everything created ok");
});
});
This should just work:
TaskSetEvent.create({
eventType: 'start',
taskSet: {
task: myTask
}
})
.then(function (taskSetEvent) {
console.log('should be done here');
});
If you're doing this through a controller endpoint, you shouldn't have to write any code. Just POST your nested object.
Everything is fine with your code. Anyway, when there are more nested functions code becomes hard to read and maintain, something called spaghetti code or callback hell.
In JavaScript common ways of solving callback problem are using promises or using special tools, like async.js.
For your code snippet async.waterfall() is definite way to go. You can rewrite it in such way:
async.waterfall([
function(cb) {
TaskSet.create({ task: task}).exec(cb);
},
function(err, taskSet, cb) {
TaskSetEvent.create({ taskSet: taskSet, eventType: 'start'}).exec(cb);
}
], function(err, taskSetEvent) {
console.log('Everything created ok');
});
waterfall method runs series of functions each passing the results to the next.
Not worth saying that if you want to use async frequently, it is not necessary to require it each time in your modules, you can just install it via npm and save async: true in your globals config.
Related
I started to istanbul as a test coverage tool with mocha and one of the great things is that it shows you the paths (branches) you have tested in your test code logic.
There is a path that can only be taken if a error occurs on the database.
Screenshot of the part I am interested in testing
The [I] indicates the the first if was not tested.
The problem is that it uses a callback function(err, data) and the error is passed to this callback through a the mongoose model method find(), and because of that I don't have the flow control of this part of the code.
In this specific case I using supertest which is a module to test routes in node.js and it make requests to a route that calls a mongoose model method find().
What would be the best option to test this path? Create a stub to simulate the method? Or just remove the if?
EDIT: I noticed that I was using an anonymous function as a callback (err, data) and doing so I can't test it since it's not exposed to the outer scope. One approach I had in mind was to create a function:
handleDbFetchingResponse(res) {
return function(err, data) {
let response = {};
if (err) {
response = {error: true, message: 'Error fetching data'};
} else {
response = {error: false, message: data};
}
res.json(response);
}
}
Now I can expose the function and test it, I create another problem though. Since the other express routes have another logic when fetching data from the database I will have to create a handler function for each one of them. Maybe there is a way to create a handlerBuilder function that returns a new handler passing different arguments to deal with specific cases.
I've looked high and low, and can only find how to write async functions, which I already understand.
What I am trying to do is run an async method in a triggered event [EventEmitter], but such a simple thing seems to be just simply not possible as I can find.
Consider the following...
// Your basic async method..
function doSomething(callback) {
var obj = { title: 'hello' };
// Fire an event for event handlers to alter the object.
// EvenEmitters are called synchronously
eventobj.emit('alter_object', obj);
callback(null, obj);
}
// when this event is fired, I want to manipulate the data
eventobj.on('alter_object', function(obj) {
obj.title += " world!";
// Calling this async function here means that our
// event handler will return before our data is retrieved.
somemodule.asyncFunction(callback(err, data) {
obj.data = data;
});
});
As you can see in the last few lines, the event handler will finish before the object's data property is added.
What I need is something where I can turn the async function into an sync function and get the results there and then. so for example...
obj.data = somemodule.asyncFunction();
I've looked at the wait.for module, the async module, and none of these will not work. I've even looked into the yield method, but it seems not yet fully implemented into the V8 engine.
I've also tried using a while loop too wait for data to populate, but this just brings with it the CPU overload issue.
Has anyone experienced this and found a design pattern to get around this?
You cannot turn an async function into a synchronous one in node.js. It just cannot be done.
If you have an asynchronous result, you cannot return it synchronously or wait for it. You will have to redesign the interface to use an asynchronous interface (which nearly always involves passing in a callback that will be called when the result is ready).
If you're wanting to do something after you .emit() an event that is itself going to do something asynchronously and you want to wait until after the async thing finished, then the event emitter is probably not the right interface. You'd rather have a function call that returns a promise or takes a callback as an argument. You could manipulate an eventEmitter to use this, but you'd have to post back a second event when the async operation finished and have the original caller not do it's second part until it receives the second event (which is really not a good way to go).
Bottom line - you need a different design that works with async responses (e.g. callbacks or promises).
Seems what I wanted to do is just not possible and the two models conflict. To achieve what I wanted in the end, I encapsulated my modules into an array-like object with some event methods that will pass it on to the module's object, which inherited the async-eventemitter class.
So, think of it like so...
My custom app modules may inherit the async-eventemitter module so they have the .on() and .emit(), etc. methods.
I create a customised array item, that will allow me to pass an event on to the module in question that will work asynchronously.
The code I created (and this is by no means complete or perfect)...
// My private indexer for accessing the modules array (below) by name.
var module_dict = {};
// An array of my modules on my (express) app object
app.modules = [];
// Here I extended the array with easier ways to add and find modules.
// Haven't removed some code to trim down this. Let me know if you want the code.
Object.defineProperty(app.modules, 'contains', { enumerable: false, ... });
Object.defineProperty(app.modules, 'find', { enumerable: false, ... });
// Allows us to add a hook/(async)event to a module, if it exists
Object.defineProperty(app.modules, 'on', { enumerable: false, configurable: false, value: function(modulename, action, func) {
if (app.modules.contains(modulename)) {
var modu = app.modules.find(modulename);
if (modu.module && modu.module['on']) {
// This will pass on the event to the module's object that
// will have the async-eventemitter inherited
modu.module.on(action, func);
}
}
} });
Object.defineProperty(app.modules, 'once', { enumerable: false, configurable: false, value: function(modulename, action, func) {
if (app.modules.contains(modulename)) {
var modu = app.modules.find(modulename);
if (modu.on) {
modu.on(action, func);
}
}
} });
This then allows me to bind an event handler to a module by simply calling something like the following... .on(module_name, event_name, callback)
app.modules.on('my_special_module_name', 'loaded', function(err, data, next) {
// ...async stuff, that then calls next to continue to the next event...
if (data.filename.endsWith('.jpg'))
data.dimensions = { width: 100, height: 100 };
next(err, data);
});
And then to execute it I would do something like (express)...
app.get('/foo', function(req, res, next) {
var data = {
filename: 'bar.jpg'
};
// Now have event handlers alter/update our data
// (eg, extend an object about a file with image data if that file is an image file).
my_special_module.emit('loaded', data, function(err, data) {
if (err) next(err);
res.send(data);
next();
});
});
Again, this is just an example of what I did, so i've probably missed something in my copy above, but effectively it's the design I ended up using and it worked like a treat, and I was able to extend data on an object before being pushed out to my HTTP response, without having to replace the main [expressjs] object's standard EventEmitter model.
(eg, I added image data for files that we're loaded that we're image files. If anyone wants the code, let me know, I am more than happy to share what I did)
I am working with Node and I have a "class" that takes a directory as a parameter. It tries to create that directory and if it fails, then it throws an error:
function Config(dir) {
fs.mkdir(dir, function(err) {
if(err) throw new Error('Error', err);
}
}
My question is, is this an approved way of doing this? If I were to use a callback, then the rest of my program would have to reside in that callback, which seems odd to me.
This issue manifested itself when I tried to write a test using mocha which won't work since the exception is thrown in an async call:
it('should throw an error on a bad directory', function() {
var fn = function() {
var badConfig = new Config('/asdf');
};
assert.throws(fn, Error);
});
I've investigated domains as a way to solve the unit test issue, but that didn't seem to solve my problem (or I didn't implement them correctly).
var d = domain.create().on('error', function(err) { throw err; }
d.run(function() {
function Config(dir) {
fs.mkdir(dir, function(err) {
if(err) throw err;
}
}
});
Ultimately, I'm looking for a best practice that allows me to indicate to the application that something bad happened, and allows me to create tests for that solution.
You have three possibilities:
Using a synchronous call. As AsolBerg explained, your case suits exactly why some fs functions have their synchronous equivalent. It's ok because in your case, all your application depends on one Config instance to be loaded. but there are cases
Using a callback as constructor argument.
If constructor callback sounds really too odd for you, put your initialization code into an init() method, that takes a callback. It's a matter of personnal preference, but rather use this technic.
Last option, you can returns a Future in your init() method. There are several future libraries in NodeJS, that are an elegant alternative to callback parameter. But you can't use it in your constructor... as the constructor's return is the created object.
It sounds like in this case you might actually want to make a synchronous call (e.g. the rest of your application depends on this call being finished before proceeding). So although its normally not the way you want to think about building your node apps you could use the synchronous version mkdirSync().
http://nodejs.org/api/fs.html#fs_fs_mkdirsync_path_mode
Then if the call fails you can catch the error and return it and (probably) exit the app.
This question already has answers here:
How to avoid long nesting of asynchronous functions in Node.js
(23 answers)
Closed 9 years ago.
I just started experimenting with node (using Express to build a simple website with a MySql database).
I have basically ran with the application structure Express provides (which doesn't matter for the sake of this question). I have a file routes/index.js which exports the index function that is hit whenever a request is made for my home page. The contents of index.js are:
var db = require('../db');
exports.index = function(req, res){
db.getConnection(function(err, connection) {
connection.query('SELECT * FROM test_table', function (err, rows) {
var templateVariables = {
title: 'Index Page',
response: rows[0].text
};
res.render('index', templateVariables);
});
connection.end();
});
};
This is obviously a very preliminary and lightweight example, however on this particular GET request for the Index page, there is already a 3-deep set of callback functions. Each callbuck must live within the callback of the "parent", for it depends on the result (in a sequentially executed language/environment this would be obvious and trivial).
My question is that when building more complex and potentially very large applications, how does one avoid the issue of having massive nestings of callback functions? This will of course be the case when you have sequential dependency on logic. I know the philosophy of Node is to be asynchronous, but when it comes to waiting for data from the database(s) and say we're running 5 separate queries, what then? Would we just write a single multi-statement query as an atomic unit? Though this question isn't exclusive to databases.
There's a nice general discussion on the issue here:
http://callbackhell.com/
Also, many people use modules like Async to manage the flow control issues.
Since you mention by using Express you can use the next() as an alternative to callbacks.
app.get('/',function first(req,res,next){
res.write('Hello ');
res.locals.user = req.user;
next();
//Execute next handler in queue
});
app.get('/',function second(req,res,next){
res.write('World!');
//Use res.locals.user
res.end();
});
//response shows Hello World!
The route handlers use extra parameter next and are executed in the order they are given, until one of them returns a response. next takes either no parameters at all or an error as a parameter. You can set the variable you want to pass into next function in the res.locals
Use a Promise or Future library, such as Q (available on npm).
Quoting from Q's readme, promises let you turn this:
step1(function (value1) {
step2(value1, function(value2) {
step3(value2, function(value3) {
step4(value3, function(value4) {
// Do something with value4
});
});
});
});
into this:
Q.fcall(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
// Do something with value4
}, function (error) {
// Handle any error from step1 through step4
})
.done();
Every other solution I've seen to callback hell introduces trade-offs that simply seem a step backward. Asynchronous operations do not form the natural logical boundaries between larger operations, so if you factor functions or otherwise modularize along these boundaries you will get microfactored code.
One way that I am fond of doing is like this...
exports.index = function(req, res) {
var connection
db.getConnection(gotConnection)
function gotConnection(err, _c) {
connection = _c
connection.query('SELECT * FROM test_table', gotData)
}
function gotData(err, rows) {
connection.end();
var templateVariables = {
title: 'Index Page',
response: rows[0].text
}
res.render('index', templateVariables);
}
});
You should also always handle errors as well in your code, I'm assuming you left them out to make the code easier to read here.
Callbacks are more and more a requirement in coding, especially when you think about Node.JS non-blocking style of working. But writing a lot of coroutine callbacks quickly becomes difficult to read back.
For example, imagine something like this Pyramid Of Doom:
// This asynchronous coding style is really annoying. Anyone invented a better way yet?
// Count, remove, re-count (verify) and log.
col.count(quertFilter, function(err, countFiltered) {
col.count(queryCached, function(err, countCached) {
col.remove(query, function(err) {
col.count(queryAll, function(err, countTotal) {
util.log(util.format('MongoDB cleanup: %d filtered and %d cached records removed. %d last-minute records left.', countFiltered, countCached, countTotal));
});
});
});
});
is something we see often and can easily become more complex.
When every function is at least a couple of lines longer, it starts to become feasible to separate the functions:
// Imagine something more complex
function mary(data, pictures) {
// Do something drastic
}
// I want to do mary(), but I need to write how before actually starting.
function nana(callback, cbFinal) {
// Get stuff from database or something
callback(nene, cbFinal, data);
}
function nene(callback, cbFinal, data) {
// Do stuff with data
callback(nini, cbFinal, data);
}
function nini(callback, data) {
// Look up pictures of Jeff Atwood
callback(data, pictures);
}
// I start here, so this story doesn't read like a book even if it's quite straightforward.
nana(nene, mary);
But there is a lot of passing vars around happening all the time. With other functions written in between, this becomes hard to read. The functions itself might be too insignificant on their own to justify giving them their own file.
Use an async flow control library like async. It provides a clean way to structure code that requires multiple async calls while maintaining whatever dependency is present between them (if any).
In your example, you'd do something like this:
async.series([
function(callback) { col.count(queryFilter, callback); },
function(callback) { col.count(queryCached, callback); },
function(callback) { col.remove(query, callback); },
function(callback) { col.count(queryAll, callback); }
], function (err, results) {
if (!err) {
util.log(util.format('MongoDB cleanup: %d filtered and %d cached records removed. %d last-minute records left.',
results[0], results[1], results[3]));
}
});
This would execute each of the functions in series; once the first one calls its callback the second one is invoked, and so on. But you can also use parallel or waterfall or whatever flow matches the flow you're looking for. I find it's much cleaner than using promises.
A different approach to callbacks are promises.
Example: jQuery Ajax. this one might look pretty familiar.
$.ajax({
url: '/foo',
success: function() {
alert('bar');
}
});
But $.ajax also returns a promise.
var request = $.ajax({
url: '/foo'
});
request.done(function() {
alert('bar');
});
A benefit is, that you simulate synchronous behavior, because you can use the returned promise instead of providing a callback to $.ajax.success and a callback to the callback and a callback.... Another advantage is, that you can chain / aggregate promises, and have error handlers for one promise-aggregate if you like.
I found this article to be pretty useful.
It describes the pro and cons of callbacks, promises and other techniques.
A popular implementation (used by e.g. AngularJS iirc) is Q.
Combined answers and articles. Please edit this answer and add libraries/examples/doc-urls in a straightforward fasion for everyone's benefit.
Documentation on Promises
Asynchronous Control Flow with Promises
jQuery deferreds
Asynchronous Libraries
async.js
async.waterfall([
function(){ // ... },
function(){ // ... }
], callback);
node fibers
step
Step(
function func1() {
// ...
return value
},
function func2(err, value) {
// ...
return value
},
function funcFinal(err, value) {
if (err) throw err;
// ...
}
);
Q
Q.fcall(func1)
.then(func2)
.then(func3)
.then(funcSucces, funcError)
API reference
Mode examples
More documentation