I've tried to search for a similar problem on here but suprisingly couldn't find one posted already.
I use expressjs v4 framework and I'm constructing my routes like this:
'use strict';
let express = require('express');
let router = express.Router();
let users = require('./modules/users');
router.post('/',users.add);
router.put('/edit/:id',users.edit);
As you can see above, I'm requiring let users = require('./modules/users')
Now the users module looks (let's say) like this:
'use strict';
let usersDbModule = require('...');
let users = {
'add': (req, res, next) => {
let callback = (err, record) => {
//...do something
users.function1(record)
}
usersDbModule.save(req, callback);
},
'function1': (record) => {
users.function2()
},
'function2': () => {
//...do something with next() function
}
}
You can notice, that router from the first code block is using module's add function. add function it's a standard express middleware function but now the things are getting more complicated.
As you can see, add function has next as one of the params, now I'm doing some complex callbacks calls from different functions and let's say that in the end I want to call next in function2.
My question is, what is the best way of passing req, res and next params between different callback functions within the same module.
I come up with 3 different methods of doing it:
Method 1:
Pass req, res or next if necessary around to all the functions in the chain so in this case I would have to pass next to callback than to function1 and than from function1 to function2.
Not the best way in my opinion, difficult to maintain, read and probably test as well.
Method 2:
Wrap function1 and function2 with closures in the add passing all the necessary params. In this particular case I would have to wrap only function2 with closure passing next so it would looks something like this:
'add': (req, res, next) => {
users.function2(next);
//....rest of the code of the function add
}
And than the function2 itself:
'function2': (next) => {4
return () => {
//...now I have access to next here
// without a need to pass it to each and every
// function in the chain
}
}
Method 3:
Append all the necessary functions/variables to res.locals and pass only res object around.
It has exactly the same problem as Method 1 so I would personally be in favour of Method 2 but not sure if it doesn't make the code less readable and maybe there are some other problems with it, haven't tested it in production nor in development environment with the team.
I would really like to hear what are you guys using and how it plays in your projects/teams. Any preferences, best practices, best patterns ? Please share, I really want to know what's the best way.
Maybe there is even better way of doing it ?
All feedback greatly appreciated!
Real life example:
Example usage for function1 & function2 and possibly more...
Let's say we have an adapter that fetches data from an external API, than it needs to save the data into a database, and return a response. Let's also assume that the data returned from the API expires after 5s. If the client hits the route within 5s span, it gets the data from the database, if time between calls was longer, than it repeats the operation calling the API.
This would be of course more complicated than just function1 and function2. It would require a lot of callback functions, both from the adapter and the database, also separate functions for fetching data from the database, adapter, saving data into a database, and eventually deleting data from the database, it gives at least 4 callback functions already.
I think that mix express and app logic is not a good idea.
I use next way in my project
// middlewares/auth.js
// Example middleware
exports.isAdmin = function (req, res, next) {
if (smth-admin-check)
next();
else
next (new Error(403));
}
// routes/index.js
// Include only modules from /routes
let user = require('./user');
let auth = require('../middlewares/auth');
...
app.get('/user/:id(\\d+)', user.get);
app.post('/user', auth.isAdmin, user.post); // only admin can add user
// routes/user.js
// Call model methods and render/send data to browser
// Don't know about db
let User = require('/models/user');
...
exports.get = function(req, res, next) {
let id = req.params.id;
// I cache most data in memory to avoid callback-hell
// But in common case code like below
User.get(id, function(err, u) {
if (!u)
return next(new Error('Bad id'));
... render page or send json ...
});
}
...
exports.post = function(req, res, next) { ... }
// models/user.js
// Encapsulate user logic
// Don't use any express features
let db = require('my-db');
...
class User {
get(id, callback) { ... }
add(data, callback) { ... } // return Error or new user
...
}
Related
I am making a login authentication module using a regular function in my project. However, I see many developers online using middleware to achieve the same purpose. I am so confused about when to use middleware and when to use a normal function. In general, when is it appropriate to use middleware, does it somehow related to some specific scenarios? I want to know if there is any pattern so that I can establish good coding practices.
I tried to make the same authentication module by using middlewares and it behaves exactly the same as expected. I can always pass the (req, res) parameters to a regular function, so why don't I just use a regular function?
Middlewares are useful when you want to apply the function to an entire subset of routes. For example, if you have a dashboard with many routes (/profile, /orders, /prices), and they all require the user to be logged in to view, it would be wise to apply the middleware to the routes, so you won't have to re-write the function in all routes.
I can always pass the (req, res) parameters to a regular function, so why don't I just use a regular function?
You totally can. But middleware is actually less code. I'm thinking in terms of the below example:
// middleware example
const authMiddleware = async (req,res,next) => {
try {
... do some stuff
next()
} catch(err) {
// handle specific errors, 401, 403, etc
next(err)
}
}
const handler = (req,res) => {
... handle the request
}
app.use('path',authMiddleware,handler)
and this:
// function example
const authFn = async (req,res) => {
// ... do some stuff without calling next
}
const handler = async (req,res,next) => {
try {
await authFn(req,res)
} catch(err) {
// handle specific errors, 401, 403, etc
return next(err)
}
... handle the request
}
app.use('path',handler)
In the above example, when using a function you'd have to handle errors in each authenticated route (there's probably more than one) whereas with middleware you can just write it once and use it on all your authenticated routes.
Middleware gets to be an even better separator of concerns when you do stuff like this:
const requirePermissions = (requiredPermissions) => async (req,res,next) => {
try {
// evaluate the user against the required permissions
next()
} catch(err) {
next(err)
}
}
app.use('/somePath', requirePermissions(perm1,perm2), handler)
app.use('/someOtherPath', requirePermissions(perm3,perm4), handler)
As it makes your middleware even more reusable among routes.
Middleware's awesome.
I use express v4.17.1 and would like to make my app.use() middlewares to run in parallel.
So I have found some examples in the internet.
Example:
function runInParallel() {
async.parallel([
getUserProfile,
getRecentActivity,
getSubscriptions,
getNotifications
], function(err, results) {
//This callback runs when all the functions complete
});
}
But what I have in my application is:
const app = express();
const APP_FOLDER = "bdt";
app.use(httpContext.middleware);
app.use(metricsMiddleware);
app.use(rtEndMiddleware);
app.use(trackingContextMiddleware);
app.use(healthRoutes());
app.use("/" + APP_FOLDER + "/api/products", productsRoutes);
app.use("/tcalc/" + APP_FOLDER + "/api/products", productsRoutes);
productRoutes is this:
const jsonParser = bodyParser.json({
limit: "1mb",
});
const accessFilter = accessFilterMiddleware(Registry.list());
const localDevFilter = localDevFilterMiddleware(Registry.list());
const apiRoutes: Router = Router();
apiRoutes.get("/", listProducts);
apiRoutes.get("/healthz", cProductHealth);
apiRoutes.get("/:id", accessFilter, localDevFilter, fetchProductData);
apiRoutes.post(
"/:id",
accessFilter,
localDevFilter,
jsonParser,
fetchProductData,
);
apiRoutes.get(
"/:id/fields/:fieldId/options",
accessFilter,
localDevFilter,
fetchProductOptions,
);
apiRoutes.post(
"/:id/loadmoreoptions",
accessFilter,
localDevFilter,
jsonParser,
loadMoreOptions,
);
apiRoutes.post("/:id/ploy", accessFilter, jsonParser, fetchMultipleProductData);
apiRoutes.post(
"/:id/gxx",
accessFilter,
localDevFilter,
jsonParser,
fetchGssData,
);
apiRoutes.get("/:id/healthz", collectProductHealth);
I think for the first ones it should be easy:
async.parallel([
httpContext.middleware,
metricsMiddleware,
rtEndMiddleware,
trackingContextMiddleware,
healthRoutes()
], function(err, results) {
//This callback runs when all the functions complete
});
But my question: How can I do it with a second parameter (productRoutes) in this case ?
app.use("/" + APP_FOLDER + "/api/products", productsRoutes);
app.use("/tcalc/" + APP_FOLDER + "/api/products", productsRoutes);
You've been going down a rabbit hole and you should change your approach quite a bit, currently you're completely over complicating a simple problem and the solution you're looking for is not possible in javascript.
If you want to create routes in express then don't use app.use for everything, it should only be used for middleware or to register a router, on which you can define routes.
You should be using:
app.get('/', () => ...
To define your routes. Alternatively you can use a router for that as well by doing:
app.use(router)
...
router.get('/', () => ...
More than that if you want to define async or "parallel" routes in javascript then just define async callbacks like normal, remove most of the stuff you've done.
app.get('/', async () => ...
That is now a route that will execute asynchronously.
You should also be careful not to just mess around with express' middleware because you're going to mess up the existing middleware (like error routes).
What's more is that the library you're referring to is just a helper library with neat functionality, it won't fundamentally change how javascript works. When you call an async function it will be added to the event queue and still be executed synchronously one after the other in serial, true multi threading isn't possible except for service workers and browser API calls executing temporarily in a single thread of their own, before being added to the event queue anyways.
What you're looking for is just a simple: router.get('/', async () => ..., that's the best you can do and it will appear that all your routes are executing in parallel.
After you've declared multiple of those then you can invoke all of them with something like Promise.all. My best guess is that this is what something like parallel is doing as well.
Middleware in Concept
As far as I understand it, middleware is akin to a chain, where each middleware item:
performs some initial logic
calls the next link in the chain, then
performs some final logic
where both of the chunks of logic are optional, but calling the next link is not. For example, with your first set of middleware, through to healthRoutes(), it could be visualised like so:
> httpContext.middleware
> metricsMiddleware
> rtEndMiddleware
> trackingContextMiddleware
/healthRoutes
< trackingContextMiddleware
< rtEndMiddleware
< metricsMiddleware
< httpContext.middleware
This chain structure is generally used because each middleware may enhance the common state of the request, sometimes conditionally based on the "output" of previously executed middlewares, and for many middlewares the initial logic needs to be performed before the body of the request, and the final logic after.
Middleware - Parallelisable or Not?
Based on the parallelisation in the blog post, the previously separate middlewares (getUser, getSiteList, getCurrentSite and getSubscription) very likely operate independently, which, in my opinion, makes them quite poor candidates for use as middleware. It's no wonder that they observed a significant performance improvement, as these items should never have been run in series in the first place. If we employ the same parallel() function from the post (note, specifically that function, not async.parallel), for the same middlewares as above, the execution looks more like:
> httpContext.middleware
< httpContext.middleware
> metricsMiddleware
< metricsMiddleware
> rtEndMiddleware
< rtEndMiddleware
> trackingContextMiddleware
< trackingContextMiddleware
/healthRoutes
So, parallelising the middlewares changes the order of execution significantly. Only you can determine whether this would be acceptable for your application, but I can imagine that both metricsMiddleware and trackingContextMiddleware may need to perform some logic both before and after the request being executed.
Express Route Syntax
If you decide that you do want to parallelise some or all of the middleware, I'd suggest taking advantage of native Promises directly, not a separate library, and do something like:
const parallel = (...middlewares) => (req, res, next) => {
Promise.all(middlewares.map(x => new Promise((resolve, reject) => {
x(req, res, resolve);
}))).then(() => next());
}
The important parts of this function are:
returning a function that takes the req, res, and next arguments, satisfying the requirement of Express middleware, that
maps each middleware to a Promise that executes the middleware and is passed the resolve argument of the Promise callback as its next argument
gates all these Promises behind Promise.all, and finally
calls the externally provided next argument when that gated Promise resolves
Then, if you decided that you wanted to have httpContent.middleware and metricsMiddleware run in parallel, but the others in series, you'd use that like:
app.use(parallel(
httpContext.middleware,
metricsMiddleware
));
app.use(rtEndMiddleware);
app.use(trackingContextMiddleware);
app.use(healthRoutes());
the execution of which could be visualised as:
> httpContext.middleware
< httpContext.middleware
> metricsMiddleware
< metricsMiddleware
> rtEndMiddleware
> trackingContextMiddleware
/healthRoutes
< trackingContextMiddleware
< rtEndMiddleware
As for the remaining two app.use() statements, given that each subsequent app.use() will add an item onto the middleware stack, you basically don't have to do anything to parallelise all the middleware up to that point, and if I were going down this path, I would keep the middleware explicitly separate (whether used in parallel or series) from the route implementations themselves, to make it clear where the specific application logic begins.
I have a Koa2 application, which renders templates on different routes. I'd like to introduce a middleware which will modify rendered templates in some way and I need it to be the last in a chain of other middlewares. Are there any way to enforce some middleware to be applied last before firing response using Koa2 and without modification of the already defined routes?
I have tried the code below:
// modification middleware
app.use(async function (ctx, next) {
await next();
ctx.body = ctx.body.toUpperCase();
})
// template rendering
app.use(async function (ctx, next) {
const users = [{ }, { name: 'Sue' }, { name: 'Tom' }];
await ctx.render('content', {
users
});
});
app.listen(7001);
It works as expected, but if any other middleware will be introduced before the modification one, it will be not the last in the chain.
Is it possible to achieve described behavior?
Figured out solution to this question sometime ago. In case someone else will need to do something like in the question, here's the code:
// modification middleware
const mw = async function(ctx, next) {
await next();
ctx.body = ctx.body.toUpperCase();
}
app.middleware.unshift(mw);
Basically middleware member of the application object can be accessed externally. Using standard array method unshift, it can be enforced to add needed middle-ware first in middlewares array, which will be treated by Koa as the last one in chain.
In express I have a handler for a route ex:
router.get(`${api}/path/:params/entrypoint`, routeHandler);
In this example 'routeHandler' function has a lot of logic doing various things. I'd like to break 'routeHandler' into smaller methods to ease readability and testability. So instead of:
routeHandler(req, res) {
//many lines of code
}
We could have:
routeHandler(req, res) {
helperOne(req, res);
helperTwo(req, res);
}
helperOne(req, res) {
//do stuff
}
helper2(req, res) {
//do stuff
}
I am being told not to do this by a coworker who is pretty senior, but I do not understand why. Does anyone know of any issues that can arise by passing the response or request objects into helpers? I can not think of any and google isn't revealing any clear answer.
Thanks!
Does anyone know of any issues that can arise by passing the response or request objects into helpers?
Yes you may run into some problems when passing those parameters, especially res. For example you may res.send multiple times (one in each function) which will raise an exception.
Scenario
A more concrete example is this
routeHandler((req, res) => {
helperOne(req, res);
helperTwo(req, res);
});
Based on some conditions, I want to stop and return an error from helperOne and not go execute any code from helperTwo. My definitions of these functions are like this
helperOne = (req, res) => {
const dataPoint = req.body.dataPoint; // a number for example
if (number > 10) {
return res.send("This is not valid. Stopping here...");
} else {
console.log("All good! Continue..");
}
}
helperTwo = (req, res) => {
res.send("Response from helperTwo");
}
Then let's say I do have req.body.dataPoint = 10, and I'm now expecting my routeHandler to stop after the return res.send in the first block of my if statement in helperOne.
This will not work as expected though, because the return will concern only helperOne which is the returning function. In other terms it won't propagate to routeHandler.
In the end an exception will be raised because routeHandler will call helperTwo and try to send a response again.
Solution
Don't send req or res. Just pass the data you need and handle the reponse in your main handler
An even better alternative is to use Express middlewares. Since you have multiple "sequential" handlers, you can chain multiple middlewares, which is closer to the standard Express.JS way
One reason to avoid doing this is that you're tightly coupling your helper functions to routeHandler, and encouraging complexity in the helpers. If you break up your helper functions so they only have a single responsibility, it's likely you'll only need to pass in a subset of the request.
Why are you passing in res, Are you sending a response from inside the helpers? Without knowing the details of your routeHandler implementation, I would see if you could handle logic in the helpers, but have them each return a value and keep the response-sending in the main routeHandler function. Here's a simple example:
handleRoute('/users/:userID', (req, res) => {
const { userID } = req.params;
const idIsValid = validateUserID(userID);
if (!idIsValid) {
return res.status(400).send('Invalid user ID!');
}
...
});
This question already has answers here:
How to avoid long nesting of asynchronous functions in Node.js
(23 answers)
Closed 9 years ago.
I just started experimenting with node (using Express to build a simple website with a MySql database).
I have basically ran with the application structure Express provides (which doesn't matter for the sake of this question). I have a file routes/index.js which exports the index function that is hit whenever a request is made for my home page. The contents of index.js are:
var db = require('../db');
exports.index = function(req, res){
db.getConnection(function(err, connection) {
connection.query('SELECT * FROM test_table', function (err, rows) {
var templateVariables = {
title: 'Index Page',
response: rows[0].text
};
res.render('index', templateVariables);
});
connection.end();
});
};
This is obviously a very preliminary and lightweight example, however on this particular GET request for the Index page, there is already a 3-deep set of callback functions. Each callbuck must live within the callback of the "parent", for it depends on the result (in a sequentially executed language/environment this would be obvious and trivial).
My question is that when building more complex and potentially very large applications, how does one avoid the issue of having massive nestings of callback functions? This will of course be the case when you have sequential dependency on logic. I know the philosophy of Node is to be asynchronous, but when it comes to waiting for data from the database(s) and say we're running 5 separate queries, what then? Would we just write a single multi-statement query as an atomic unit? Though this question isn't exclusive to databases.
There's a nice general discussion on the issue here:
http://callbackhell.com/
Also, many people use modules like Async to manage the flow control issues.
Since you mention by using Express you can use the next() as an alternative to callbacks.
app.get('/',function first(req,res,next){
res.write('Hello ');
res.locals.user = req.user;
next();
//Execute next handler in queue
});
app.get('/',function second(req,res,next){
res.write('World!');
//Use res.locals.user
res.end();
});
//response shows Hello World!
The route handlers use extra parameter next and are executed in the order they are given, until one of them returns a response. next takes either no parameters at all or an error as a parameter. You can set the variable you want to pass into next function in the res.locals
Use a Promise or Future library, such as Q (available on npm).
Quoting from Q's readme, promises let you turn this:
step1(function (value1) {
step2(value1, function(value2) {
step3(value2, function(value3) {
step4(value3, function(value4) {
// Do something with value4
});
});
});
});
into this:
Q.fcall(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
// Do something with value4
}, function (error) {
// Handle any error from step1 through step4
})
.done();
Every other solution I've seen to callback hell introduces trade-offs that simply seem a step backward. Asynchronous operations do not form the natural logical boundaries between larger operations, so if you factor functions or otherwise modularize along these boundaries you will get microfactored code.
One way that I am fond of doing is like this...
exports.index = function(req, res) {
var connection
db.getConnection(gotConnection)
function gotConnection(err, _c) {
connection = _c
connection.query('SELECT * FROM test_table', gotData)
}
function gotData(err, rows) {
connection.end();
var templateVariables = {
title: 'Index Page',
response: rows[0].text
}
res.render('index', templateVariables);
}
});
You should also always handle errors as well in your code, I'm assuming you left them out to make the code easier to read here.