So I have started with express.js - my first JS web dev framework. I didn't make anything small, but started a bigger project. I'm learning, and building at the same time.
Coming from a Python/Flask background, express seems very complicated.
Like in python, if I want a helper method, I can just put it on top of the file, or in a new module, and import it. Super easy. But in node/express, things are async, and everything is in exports or module.exports (??). Where do helper methods go? How do I call them with callbacks?
In another question I asked, I was doing the same kind of computation multiple times. In Python, I would write a method (with if statements and parameters), and call it multiple times, using a for.. in... loop. The code I have right now is very redundant.
How do I do it in express? What are best practices for writing express code?
It really depends of what your helper is doing. If it operates with data which is passed as a parameter to it then you may save it in an external module and use require to access it.
// helpers/FormatString.js
module.exports = function(str) {
return str.toUpperCase();
}
// app.js
var formatter = require("./helpers/FormatString");
However, if you need to modify the request or the response object then I'll suggest to define it as a middleware. I.e.:
app.use(function(req, res, next) {
// ... do your stuff here
});
#Krasimir gave a correct answer. Regarding your question how to deal with asynchronous helper functions I can give you an example (not the usual foo/bar, but one of my own helper functions):
var cached; // as modules can act like singletons I can share values here
module.exports = function fetchGoogleCerts(options, callback) { // <-- callback
var now = Date.now();
if (ttl > now && cached) {
callback(null, cached); // <-- return with success
return;
}
request({
uri: options.certsUrl || defaultCertsUrl,
strictSSL: true
}, function (err, response, body) {
var error, certs;
if (!err && response.statusCode === 200) {
certs = jsonParse(body); // a local function
if (!certs) {
callback('parse_error', null); // <-- return an error
return;
}
cached = certs;
// ... more code
callback(null, cached); // <-- success case
} else {
error = {
error: err,
statusCode: response.statusCode
};
log.error(error, 'error fetching google certs');
callback(error); // <-- return an error
}
});
}
};
And using the helper:
fetchGoogleCerts = require('./google-certs.js'),
module.exports = function fetchCertsAndDecodeIdToken(token, options, callback) {
fetchGoogleCerts(options, function (err, googleCerts) {
if (err) {
callback({
errorCode: codes.io_error,
errorMsg: 'Unable to fetch Google certificates',
error: err
});
return;
}
decodeAndVerifyGoogleIdToken(googleCerts, token, options, callback);
});
};
As you can see above, the simple solution is to provide a callback function to your asynchronous helper function.
Of course you can also export an Object which extends EventEmitter, then you might not need a callback function, but register for the events. Here is an Example for a helper which emits events.
Related
Background:
I have a gateway that returns the status code 200 via a post request if it is completely booted. If not it returns 500 while booting.
My idea:
I have a NodeJS application which should wait until the gateway returns 200. So I created a while loop which checks the state of the gateway.
My problem:
Unfortunately nothing works, the state is always true. Non of the log statements in the request will be display.
Do you have tips for me how I can fix this?
while (isGatewayUnavailable()) {
log.info('waiting for gateway ...');
sleep(60)
}
function isGatwayUnavailable() {
const url = '...'
let state = true
request.post(url, (err, res, body) => {
log.debug(0)
if (err) {
log.debug("Gateway offline");
log.debug("err: " + err);
}
else if (res.statusCode === 200) {
log.debug("Gateway online");
state = false;
cb(true);
}
else {
log.debug("Status Code: " + res.statusCode);
}
});
log.debug('return state: ' + state);
return state;
}
There is no "waiting" in JS. There's only "running code" and "running code in response to signals" (events, callbacks, promises). In this case, you want to do something based on a process that you do not control the timing of, so you can't use a synchronous function: by the time the function reaches its return keyword, you don't have any information to return yet..
So, instead of making your function return a value and having the caller wait for that value, make your code "do things once the information is in". That is, make your function either generate an event that you have a handler registered for, or pass a callback as argument so that your function can run that callback once it has the information necessary, or have it return a promise whose resolve (or reject) gets called once you have the information necessary.
1. Event-based:
const pubsub = ...;
function checkGatwayAvailability() {
request.post(url, (err, res, body) => {
pubsub.signal("gateway:availability", { available: ..., error: ... });
});
}
with caller code:
const pubsub = ...;
pubsub.register("gateway:availability", data => {...});
...
checkGatewayAvailability();
In this, the code that calls this and the code that handles the result are 100% detached from each other. Also note that pubsub isn't a real thing. Depending on your framework and APIs, there will be different ways to achieve event generation/handling, or you might even need to write your own (which really means "hit up npm and find one that is well documented and used by many folks, then use that").
2. Using a callback:
function checkGatwayAvailability(reportResult) {
request.post(url, (err, res, body) => {
reportResult({ available: ..., error: ... });
});
}
with caller code:
checkGatwayAvailability( result => {
...
});
In this approach, the calling and handling code are coupled in the sense that your call points to the handler, even if your handler is declared somewhere completely different, like:
checkGatwayAvailability(NetworkMonitor.handleGatewayResponse);
3. Using a promise:
function checkGatwayAvailability(reportResult) {
return new Promise((resolve, reject) => {
request.post(url, (err, res, body) => {
if (err) reject(err);
resolve(...);
});
});
}
with caller code:
checkGatwayAvailability().then(result => {...}).catch(err => {...});
Similar to a callback, the calling and handling code are coupled, but with promises you don't "guess" at whether the resultant information is good or bad, you literally have separate code paths for the "good" cases (handled by then), and the "bad" cases (handled by catch).
3b. Using a promise through async/await syntax:
In this case, request.post does not return a Promise, your function would still need to bake its own promise, so using an async declaration doesn't make a lot of sense. We can still use the await keyword in the calling code, though:
try {
const result = await checkGatwayAvailability();
} catch (e) {
...
}
but only if that caller code itself runs inside an async context.
Basically I want to instead of this...
app.get(routes.test, function(req, res, next){
actualRouteHandler(req, res, next) // Always returns a promise or throws.
.catch(function(err) {
next(err);
});
});
Have this:
app.get(routes.test, catchWrap(actualRouteHandler));
Or something similar, I've tried messing around with fn.apply and things but I can't find a way to pass actualRouteHandler the correct parameters (req,res,next) and still have the function. Do I need to return a function or something similar?
Edit: I think there may be libraries that do this but we have no access to the actual express application in this bit of code.
In your specific case, catchWrap would look like this:
function catchWrap(originalFunction) {
return function(req, res, next) {
try {
return originalFunction.call(this, req, res, next);
} catch (e) {
next(e);
}
};
}
That returns a new function that, when called, will call the original with your catch wrapper around it. The key parts are that it creates and returns a function (return function(req, res, next) { ... };) and this line:
return originalFunction.call(this, req, res, next);
Function#call calls the given function saying what to use as this during the call (in the above we pass on the this we recieved) and the arguments to use in the call.
You'd use it as you showed:
app.get(routes.test, catchWrap(actualRouteHandler));
Or if you prefer to define the actual handler as an anonymous function:
app.get(routes.test, catchWrap(function(req, res, next) {
// ...handler code here...
}));
That catchWrap is specific to your situation, because you want to call next(e) with the exception if thrown. The generic form of "wrap this function in another function" is like this:
function catchWrap(originalFunction) {
return function() {
try {
// You can do stuff here before calling the original...
// Now we call the original:
var retVal = originalFunction.apply(this, arguments);
// You can do stuff here after calling the original...
// And we're done
return retVal;
} catch (e) {
// you can do something here if you like, then:
throw e; // Or, of course, handle it
}
};
}
arguments is a pseudo-array provided by JavaScript that includes all of the arguments that the current function was called with. Function#apply is just like Function#call, except you give the arguments to use as an array (or pseudo-array) rather than discretely.
Taking it a step further: You could create a function you passed a router to that returned an updated router that auto-wrapped the handlers when you called get, post, etc. Something along these lines (untested):
const routeCalls = ["all", "get", "post", "put", "delete", /*...*/];
function patchWithCatcher(router) {
for (const call of routeCalls) {
const original = router[call];
// This seemingly-pointless object is to ensure that the new function
// has a *name*; I don't know whether Express relies on the name of
// the function you call (some libs with similar APIs do).
const rep = {
[call]: function (path, ...callbacks) {
return router[call](path, ...callbacks.map(catchWrap));
},
};
router[call] = rep[call];
}
return router; // Just for convenience
}
Using it:
const app = patchWithCatcher(express());
app.get(/*...*/);
app.post(/*...*/);
// ...
or
const router = patchWithCatcher(express.Router(/*...options...*/));
router.get(/*...*/);
router.post(/*...*/);
// ...
That monkey-patches the router you pass in. I considered using the passed-in router as the prototype of a new object in order to avoid modifying the router object passed in, but you'd have to test that really, really thoroughly, there are lots of edge cases around this during calls. Another viable alternative would be to use a Proxy, though at a runtime cost.
The following simple approach can be used to setup a generic error handler -
Step 1: Write the error handler after registering your routes. So, for example, if you are registering your multiple routes like this:
config.getGlobbedFiles('./app/routes/modules/**/*.js').forEach(function (routePath) {
require(path.resolve(routePath))(app);
});
After registering the routes, you can setup a generic error handler -
app.use(function (err, req, res, next) {
// If the error object doesn't exists
if (err != undefined) {
//do error handling
}
else {
//no error found
}
});
Step 2: In the route handler, you need to make sure that you call "next()" if no error is found. In case of error, you need to call "next(err)".
The code is from the IPFS (Inter-planetary file-system) HTTP API JS implementation: https://github.com/ipfs/js-ipfs-api/blob/master/src/api/add.js
'use strict'
const Wreck = require('wreck')
module.exports = (send) => {
return function add(files, opts, cb) {
if (typeof(opts) === 'function' && cb === undefined) {
cb = opts
opts = {}
}
if (typeof files === 'string' && files.startsWith('http')) {
return Wreck.request('GET', files, null, (err, res) => {
if (err) return cb(err)
send('add', null, opts, res, cb)
})
}
return send('add', null, opts, files, cb)
}
}
The function being described is the add() function, used to push data to IPFS.
I'll start by explaining what I do understand: the add() function takes three arguments – if there is no options object (the user omitted it) and it's been replaced by a function: the user is trying to implement a callback function instead – change the callback to opts; cb = opts.
Secondly, if the cited file is a text file && starts with http – it's obviously hosted remotely and we need to fetch it using Wreck.
All of this I understand, but why are we using the (send) => arrow function? Why do we use return function add...? What's the send('add', null, opts, res, cb) and return send('add', null, opts, res, cb) used for? How is the callback (cb) implemented? Help me understand what's happening here
The whole thing being exported is a function, which expects send as an argument; this lets the calling code do dependency injection by passing in the send function to use. It's expecting to be used something like this:
let addBuilder = require("add");
let add = addBuilder(senderFunction);
// This function ----^
// is `send` in the `add.js` file.
// It does the actual work of sending the command
// Then being used:
add(someFiles, someOptions, () => {
// This is the add callback, which is `cb` in the `add.js` file
});
(Frequently, the first two statements above are written as a single statement, e.g. let add = require("add")(senderFunction);)
Basically, the whole thing is one big builder for something that uses a given send function, which is injected into it by calling the builder. That way, it can be tested by injecting a testing version of send, and used for real by injecting a real version of send; and/or various "real" versions of send can be used in different environments, transport mechanisms, etc.
I am trying to create unit tests with Mocha and Chai on Node.JS. Here is a simplified version of the function to test:
router.cheerioParse = function(url, debugMode, db, theme, outCollection, _callback2) {
var nberror = 0;
var localCount = 0;
console.log("\nstarting parsing now : " + theme);
request(url, function(error, response, body) {
//a lot of postprocessing here that returns
//true when everything goes well)
});
}
Here is the test I am trying to write:
describe('test', function(){
it('should find documents', function(){
assert( true ==webscraping.cheerioParse("http://mytest.com, null, null, null ,null,null ));
});
})
How can the request function return true to have it passed to the test? I have tried to use promises but it didn't work either. In this case should I put the return statement in the then callback? What is the best approach?
You should mock request function. You could use e.g. sinon stubs for this (they provide returns function for defining returning value).
In general - the idea of unit tests is to separate particular function (unit of test) and stub every other dependency, as you should do with request :)
To do so, you have to overwrite original request object, e.g. :
before(function() {
var stub = sinon.stub(someObjectThatHasRequestMethod, 'request').returns(true);
});
And after running tests you should unstub this object for future tests like that:
after(function() {
stub.restore();
});
And that's all :) You could use both afterEach/after or beforeEach/before - choose the one that suits you better.
One more note - because your code is asynchronous, it is possible that your solution might need more sophisticated way of testing. You could provide whole request mock function and call done() callback when returning value like this:
it('should find documents', function(done) {
var requestStub = sinon.stub(someObjectThatHasRequestMethod, 'request',
function(url, function (error, response, body) {
done();
return true;
}
assert(true === webscraping.cheerioParse("http://mytest.com, null, null, null ,null,null ));
requestStub.restore();
});
You could find more info here:
Mocha - asynchronous code testing
Actually I am reading some codes in Nodejs, but I cannot get what it means because some common use of function that I don't quite understand. Can someone tell me what the function(done) means? It is a callback or something? in the js file I can't see the definition of done. Thanks
Andes
var getRedisUri = exports.getRedisUri = function (done) {
if (process.env.CLUSTER_CONFIG_URI) {
return done(null, process.env.CLUSTER_CONFIG_URI);
}
//redis uri
Fs.readFile(
'/opt/redis_uri',
{encoding: 'utf8'},
function (err, redis_uri_data) {
if (err) {return done(err);}
var redis_uri = redis_uri_data.toString();
if (process.env.NODE_ENV !== 'production' &&
process.env.VM_DOMAIN &&
(redis_uri.indexOf('localhost') !== -1 || redis_uri.indexOf('127.0.0.1') !== -1)) {
redis_uri = redis_uri.replace('localhost', process.env.VM_DOMAIN);
redis_uri = redis_uri.replace('127.0.0.1', process.env.VM_DOMAIN);
}
done(null, redis_uri);
});
};
That line is just the start of a function definition. function(done) just means that when this function is called, it should be called with one argument, and during the execution of the function, that argument will be referred to as done.
More specifically, I think it has to do with middleware (are you using express.js here?). If you have a bunch of middleware functions in express, the express router will call those functions for you, and pass in a function as an argument which, when called, passes the request to the next middleware function. It seems like that's what's happening here, as done gets called at the end of the function.
Node.js has stuff like this because it's all asynchronous, so you can't use return statements for much.