Consider an example: I have the following express.js app (see code snippet below). I want to have one persistent connection to the DB, and one persistent connection to my own service (which required async call to start) during entire app lifetime. And there are a few entry points, i.e. one can access my app not only via HTTP protocol. Of course, I want to avoid service initialization code duplication and there could be several such async-initializing services.
/* app.js */
var app = require('express')();
// set views, use routes, etc.
var db = require('monk/mongoose/etc')(...); // happily, usually it's a sync operation
var myService = require('./myService'); // however, it's possible to have several such services
myService.init(function(err, result) {
// only here an initialization process is finished!
});
module.exports.app = app;
/* http_server.js (www entry point) */
var app = require('app');
// create an HTTP server with this app and start listening
/* telnet_server.js (other entry point) */
var app = require('app');
// create a Telnet server with this app and start listening
In the code snippet above, by the time http (or telnet, or any other) server is starting, there is no guarantee, that myService already has initialized.
So, I have to somehow reorganize my app creation code. For now I stick with the next solution:
/* app.js */
var app = require('express')();
module.exports.app = app;
module.exports.init = function(callback) {
var myService = require('./myService');
myService.init(callback);
}
/* entry_point.js */
var app = require('app');
app.init(function(err) {
if (!err) {
// create an HTTP/Telnet/etc server and start listening
}
});
So, my question is: what is the common way to initialize services required asynchronous call to start?
I would recommend you to promisify the initialization function of your services(s) and then use them in the following manner:
const app = require('express')();
const util = require('util');
const myService = require('./myService');
const myServiceInit = util.promisify(myService.init);
Promise.all([myServiceInit]).then(() => {
// delayed listening of your app
app.listen(2000);
}).catch(err => {
// handle error here
});
I've used Promise.all in order for you to add initialization of multiple internal services.
The pre-requisite of promisifying your init function is that it should be using an error first callback mechanism. You can read more about it here Node Official Doc
Hope this helps your cause.
I've created a gist here with a sample of the code I normally use for this task. (It uses the Q promise library, but could easily be modified to use any other promises lib).
The basic idea is to describe the app backbone as a sequence of asynchronous initialization steps. Each step calls one or more async functions and binds the result to a name; the startup process only progresses to the next initialization step once all values are resolved for the current step, and subsequent steps can then access all values resolved by previous steps. This allows the dependency order of services and components within the app to be easily described.
For example, a backbone can be defined as follows:
var app = [
{ s1: startService1 },
{ s2: startService2, s3: startService3 },
{ s4: startService4 }
]
(Note that in each step definition, just references to each function are given; the start() function - shown in the gist - will invoke each function in the correct order).
Each of the startXxx vars is a function which takes a single argument, and returns a deferred promise, e.g.:
function startService4( app ) {
var s1 = app.s1;
var s2 = app.s2;
var deferred = Q.defer();
// ... start the service, do async stuff ...
return deferred;
}
The function's app argument represents the configured app backbone, and results from previous initialization steps are available as its named properties.
I've used this pattern in fairly complicated systems, and find it a simple, flexible and effective way to define a system's high level structure.
Related
Performance question - I'm trying to understand if i have a node http server powered by express , is it consider bad practice to create new class instance on every request sent by the user?
The class instance fetch data from another api and expose some functionality to manipulate the fetched data.
example of the code:
//--- Handler.js ---
const _ = require("lodash");
class Handler {
constructor() {
this.fetchData = this.getSiteModel.bind(this);
this.getA = this.getA.bind(this);
this.getB = this.getB.bind(this);
this.getC = this.getC.bind(this);
}
async fetchData(req,res,id){
const result = await fetch(...)
this.data = result;
}
getA(){
...
return this.data.A
}
getB(){
...
return this.data.B
}
getC(){
...
return this.data.C
}
}
//---- controller.js ----
const Handler = require("../Handler/");
exports.getDataById = async function(req ,res) {
const handler = new Handler();
return handler.getA();
}
Would it be better to do this instead
//---- controller.js ----
const fetchData = require("../Handler/");
const getA = require("../Handler/getA");
const getB = require("../Handler/getB");
const getC = require("../Handler/getC");
exports.getDataById = async function(req ,res) {
//no new handler instance created
const data = fetchData(url)
return getA(data);
}
Is it considered bad practice to create new class instance on every request sent by the user?
No, it's not generically considered a bad practice. It's normal to need to create objects during the processing of an incoming request. Look at any database query which is often used during a request handler. It will likely create multiple objects.
Now, whether or not there is a more efficient way to do what you're doing is a different question and we would need to see your actual code in order to offer some advice on that topic.
Would it be better to do this instead?
I don't see a whole lot of reasons for you to put the data into an object before making a single operation on that object.
Something like you proposed (with the addition of await to make it work properly):
exports.getDataById = async function(req ,res) {
//no new handler instance created
const data = await fetchData(url)
return getA(data);
}
Seems perfectly fine to me. When to structure things into a class and put the data into the instance data and why to just use a function to operate on the data is a classic OOP question and it depends upon lots of things which are all dependent upon seeing and understanding your real code, what you're doing with it, how it is most likely to be expanded in the future, how often you are calling multiple functions on the same data in one request, etc...
I currently have a few js files in nodejs which are loaded as module and augment the app object (using express).
So their signatures look like:
module.exports = function(app, callback) {
// ...
callback();
}
So currently as I have about 5 of them my code would look like:
require("./setup/a")(app, function() {
require("./setup/b")(app, function(){
require("./setup/c")(app, function(){
require("./setup/d")(app, function(){
require("./setup/e")(app, function(){
startApp();
})
})
})
})
});
Now that looks unsightly as its the "pyramid of doom", however I am not entirely sure how I need to change this pattern to use Q, as I was assuming I would use Q.fcall(...a).then(...b).etc.done(). However I am unsure how I pass the app into it and if I need to return the callback for it to process as a promise.
Ideally I do not want to start whacking Q all through my code I only want it in the places where I want to remove the pyramid use cases, so in the above example how do I use Q with promises to pass the app into each required module and then start the app at the end?
Assuming your modules don't already use promises you can do something like this:
module.exports = function(app) {
// do some stuff with app
return new Promise(function(resolve,reject){
// when ready to resolve after some actions on app
resolve(); // you can also return a value here as a cb param
});
};
Promise.all(["./setup/a","./setup/b","./setup/c"].map(require.bind(null,app)))
.then(startApp);
You should however use promises at the lowest level possible, which would mean that you can simply return the promise which you used in the process:
module.exports = function(app){
return something(app).then(function(){
return somethingElseAsyncWithApp(app);
});
};
So the promise constructor isn't required. Note that this answer uses native promises but will also work on libraries that use that syntax like Bluebird. For Q change new Promise to new Q.Promise and Promise.all to Q.all.
Alternatively, you can change every require(x) to Q.fcall(require,x) and use Q.all on that directly, but that is both slow (Although Q is slow anyway) and more error prone than promisifying the modules directly. It is best to promisify the lowest level API possible.
Promises are not the silver bullet to callback pyramid of doom. I have seen code where even with promises it looked like a pyramid of doom.
You could get rid of the pyramid and stay with the callback style by doing something like this :
// var app;
// ...etc
var paths = ['a','b','c','d'];
setupNext();
function setupNext() {
var p = paths.pop(); // or shift
var next = paths.length > 0 ? setupNext : startApp
require(p)(app, next);
}
function startApp() {}
I'm new to node and JS and was working thorough the socket.io chat example (http://socket.io/get-started/chat/). I came across this code in the server:
var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
I've looked at other tutorials and never seen double parentheses after require before. What does the (http)part do? Is it a parameter for require, doest it change the type, or something else?
Thanks!
In JavaScript, function is the First-class citizen. This means that it can be returned by another function.
Consider the following simple example to understand this:
var sum = function(a) {
return function(b) {
return a + b;
}
}
sum(3)(2); //5
//...or...
var func = sum(3);
func(2); //5
In your example, require('socket.io') returns another function, which is called immediately with http object as a parameter.
To expand if you had a library http and it has a exported module server.
Lets say we picked apart the line:
var http = require('http').Server(app);
into two lines:
var http = require('http')
Imports the "http" module library as a JSON object into the http variable. This module library has a bunch of modules that you can access now by calling them via the http var.
httpServer = http.Server(app)
This loads the Server module with the express data that you called above (Kind of line a constructor) and puts it into the httpServer var.
The difference above is that instead of the two steps, they are condensing it into one, so that http has the Server module inside it instead of the entire http library. This can be useful if you only want to use that specific part of the http library.
Nodejs allows you to assign an object/function to the exported module using the statement module.exports = something. So each one of those statements are importing a library, and then running the function that was assigned to what was exported.
For example, here is the source code for express where they export the createApplication function.
And here's an article where they go into a bit more detail.
I'd like to use the MongoDB native JS driver with bluebird promises. How can I use Promise.promisifyAll() on this library?
The 2.0 branch documentation contains a better promisification guide https://github.com/petkaantonov/bluebird/blob/master/API.md#promisification
It actually has mongodb example which is much simpler:
var Promise = require("bluebird");
var MongoDB = require("mongodb");
Promise.promisifyAll(MongoDB);
When using Promise.promisifyAll(), it helps to identify a target prototype if your target object must be instantiated. In case of the MongoDB JS driver, the standard pattern is:
Get a Db object, using either MongoClient static method or the Db constructor
Call Db#collection() to get a Collection object.
So, borrowing from https://stackoverflow.com/a/21733446/741970, you can:
var Promise = require('bluebird');
var mongodb = require('mongodb');
var MongoClient = mongodb.MongoClient;
var Collection = mongodb.Collection;
Promise.promisifyAll(Collection.prototype);
Promise.promisifyAll(MongoClient);
Now you can:
var client = MongoClient.connectAsync('mongodb://localhost:27017/test')
.then(function(db) {
return db.collection("myCollection").findOneAsync({ id: 'someId' })
})
.then(function(item) {
// Use `item`
})
.catch(function(err) {
// An error occurred
});
This gets you pretty far, except it'll also help to make sure the Cursor objects returned by Collection#find() are also promisified. In the MongoDB JS driver, the cursor returned by Collection#find() is not built from a prototype. So, you can wrap the method and promisify the cursor each time. This isn't necessary if you don't use cursors, or don't want to incur the overhead. Here's one approach:
Collection.prototype._find = Collection.prototype.find;
Collection.prototype.find = function() {
var cursor = this._find.apply(this, arguments);
cursor.toArrayAsync = Promise.promisify(cursor.toArray, cursor);
cursor.countAsync = Promise.promisify(cursor.count, cursor);
return cursor;
}
I know this has been answered several times, but I wanted to add in a little more information regarding this topic. Per Bluebird's own documentation, you should use the 'using' for cleaning up connections and prevent memory leaks.
Resource Management in Bluebird
I looked all over the place for how to do this correctly and information was scarce so I thought I'd share what I found after much trial and error. The data I used below (restaurants) came from the MongoDB sample data. You can get that here: MongoDB Import Data
// Using dotenv for environment / connection information
require('dotenv').load();
var Promise = require('bluebird'),
mongodb = Promise.promisifyAll(require('mongodb'))
using = Promise.using;
function getConnectionAsync(){
// process.env.MongoDbUrl stored in my .env file using the require above
return mongodb.MongoClient.connectAsync(process.env.MongoDbUrl)
// .disposer is what handles cleaning up the connection
.disposer(function(connection){
connection.close();
});
}
// The two methods below retrieve the same data and output the same data
// but the difference is the first one does as much as it can asynchronously
// while the 2nd one uses the blocking versions of each
// NOTE: using limitAsync seems to go away to never-never land and never come back!
// Everything is done asynchronously here with promises
using(
getConnectionAsync(),
function(connection) {
// Because we used promisifyAll(), most (if not all) of the
// methods in what was promisified now have an Async sibling
// collection : collectionAsync
// find : findAsync
// etc.
return connection.collectionAsync('restaurants')
.then(function(collection){
return collection.findAsync()
})
.then(function(data){
return data.limit(10).toArrayAsync();
});
}
// Before this ".then" is called, the using statement will now call the
// .dispose() that was set up in the getConnectionAsync method
).then(
function(data){
console.log("end data", data);
}
);
// Here, only the connection is asynchronous - the rest are blocking processes
using(
getConnectionAsync(),
function(connection) {
// Here because I'm not using any of the Async functions, these should
// all be blocking requests unlike the promisified versions above
return connection.collection('restaurants').find().limit(10).toArray();
}
).then(
function(data){
console.log("end data", data);
}
);
I hope this helps someone else out who wanted to do things by the Bluebird book.
Version 1.4.9 of mongodb should now be easily promisifiable as such:
Promise.promisifyAll(mongo.Cursor.prototype);
See https://github.com/mongodb/node-mongodb-native/pull/1201 for more details.
We have been using the following driver in production for a while now. Its essentially a promise wrapper over the native node.js driver. It also adds some additional helper functions.
poseidon-mongo - https://github.com/playlyfe/poseidon-mongo
How can I make this work
var asyncToSync = syncFunc();
function syncFunc() {
var sync = true;
var data = null;
query(params, function(result){
data = result;
sync = false;
});
while(sync) {}
return data;
}
I tried to get sync function from async one,
I need it to use FreeTds async query as sync one
Use deasync - a module written in C++ which exposes Node.js event loop to JavaScript. The module also exposes a sleep function that blocks subsequent code but doesn't block entire thread, nor incur busy wait. You can put the sleep function in your while loop:
var asyncToSync = syncFunc();
function syncFunc() {
var sync = true;
var data = null;
query(params, function(result){
data = result;
sync = false;
});
while(sync) {require('deasync').sleep(100);}
return data;
}
Nowadays this generator pattern can be a fantastic solution in many situations:
// nodejs script doing sequential prompts using async readline.question function
var main = (function* () {
// just import and initialize 'readline' in nodejs
var r = require('readline')
var rl = r.createInterface({input: process.stdin, output: process.stdout })
// magic here, the callback is the iterator.next
var answerA = yield rl.question('do you want this? ', res=>main.next(res))
// and again, in a sync fashion
var answerB = yield rl.question('are you sure? ', res=>main.next(res))
// readline boilerplate
rl.close()
console.log(answerA, answerB)
})() // <-- executed: iterator created from generator
main.next() // kick off the iterator,
// runs until the first 'yield', including rightmost code
// and waits until another main.next() happens
You can do it with node-sync lib
var sync = require('sync');
sync(function(){
var result = query.sync(query, params);
// result can be used immediately
})
Notice: your query must use standart callback call (with error first): callback(error, result).
If you can't change query method, just create .async() wrapper (see github link).
I've been using syncrhonize.js with great success. There's even a pending pull request (which works quite well) to support async functions which have multiple parameters. Far better and easier to use than node-sync imho. Added bonus that it has easy-to-understand and thorough documentation, whereas node-sync does not.
The issue you are having is that your tight while loop is blocking. So I don't think your query callback will ever be run. I think you need to use setTimeout or the like to prevent the function from blocking, but should you do so, the function will return before the callback is called. This functionality must be implemented at a lower level.
If you are in the browser, you might check out this article. In node you have to rely on the implementation of whatever you're querying. It may or may not provide synchronous methods.