Hello Stackoverflow users,
Many peoples like me searched for how to pass extra arguments to a callback function. The questions have similar titles but actually they have different challenges and many ways to solve. Plus, it is always a pleasure to share practices to be more experienced.
Recently, I faced a pretty simple challenge in my node js project. One of the APIs I communicate with has an SDK that works synchronically. And I used to pass callback functions every time (which is annoying when you have requests depending on each other and some data needs to transfer within the app layers).
Imagine a plan payment flow that goes like this, a client sends a request to the server including the selected plan and his ID. When the server API layer receives the request data, it passes it to a third-party service function ( .create(...) ). The third-party service function receives a callback with 2 parameters function(err, plan_document). And then, the callback is supposed to apply the selected plan logic on the client by the ID in the request.
** We need to pass the client's and the plan's data to the callback function to apply the logic. The third-party service provides to the callback a plan_document parameter and we still need to somehow pass the client id from the API layer to the service.
The code will look like this.
const create_plan_agreement = (req, res) => {
// some code
var client_id = req.auth.client_id;
third_party.plan_agreement.create({}, update_plan_agreement);
};
const update_plan_agreement = (err, plan_document, client_id) => {
/*
The third-party `third_party.plan_agreement.create` function passes the first
two parameters and somehow we need to add the client_id
*/
console.log('client plan activated');
active_client_plan(plan_document, client_id);
};
------------------ EDIT ------------------
I wonder what if the flow was longer and I need the client id farther than the update function like this.
const create_plan_agreement = (req, res) => {
// some code
var client_id = req.auth.client_id;
third_party.plan_agreement.create({}, update_plan_agreement);
};
const update_plan_agreement = (err, plan_document) => {
console.log('plan activated, send notification to the client');
third_party.plan_agreement.update(plan_document, send_agreement_notification);
};
const send_agreement_notification = (err, plan_document) => {
console.log('client plan activated');
active_client_plan(plan_document, this.client_id);
};
What should I do in this case? Should I keep repeating the.bind({'client_id': client_id}) function until the last step in the flow?
If you want to support older people, you can easily bind using a containing callback, like this:
const create_plan_agreement = (req, res) => {
// some code
var client_id = req.auth.client_id;
third_party.plan_agreement.create({}, function(params, from, create) {
update_plan_agreement(params, from, create, client_id)
});
};
const update_plan_agreement = (err, plan_document, client_id) => {
/*
The third-party `third_party.plan_agreement.create` function passes the first
two parameters and somehow we need to add the client_id
*/
console.log('client plan activated');
active_client_plan(plan_document, client_id);
};
The traditional way is to use a closure. Define the functions inside the parent's scope so that they can access the client_id as an enclosed variable (kind of like global variables):
const create_plan_agreement = (req, res) => {
// some code
var client_id = req.auth.client_id;
const update_plan_agreement = (err, plan_document) => {
console.log('plan activated, send notification to the client');
third_party.plan_agreement.update(plan_document, send_agreement_notification);
};
const send_agreement_notification = (err, plan_document) => {
console.log('client plan activated');
// Note: this function can access client_id
// because it is in scope
active_client_plan(plan_document, client_id);
};
third_party.plan_agreement.create({}, update_plan_agreement);
};
Closures are to scopes what objects are to classes. A closure is an instance of a scope. So unlike regular global variable, each call to create_plan_agreement() will create its own closure with its own copy of client_id.
With modern javascript it is often easier to handle this with Promises. Convert legacy functions to return a Promise and then you can use async/await:
const create_plan_agreement = async (req, res) => {
// some code
var client_id = req.auth.client_id;
try {
var plan_document = await plan_agreement_create({});
var updated_plan_document = await update_plan_agreement(plan_document);
send_agreement_notification(updated_plan_document, client_id);
}
catch (err) {
// handle errors here.
}
};
const plan_agreement_create = (arg) {
return new Promise ((ok, fail) => {
third_party.plan_agreement.create({}, (err, result) => {
if (err) {
return fail(err);
}
ok(result);
});
})
}
const update_plan_agreement = (plan_document) => {
return new Promise ((ok, fail) => {
third_party.plan_agreement.update(plan_document, (err, result) => {
if (err) return fail(err);
ok(result);
});
});
};
const send_agreement_notification = (plan_document, client_id) => {
active_client_plan(plan_document, client_id);
};
Or even without async/await Promises still make callbacks easier to use:
const create_plan_agreement = async (req, res) => {
// some code
var client_id = req.auth.client_id;
plan_agreement_create({})
.then(doc => update_plan_agreement(doc));
.then(doc => {
send_agreement_notification(doc, client_id)
})
.catch(err => {
// handle errors here.
});
};
Related
I am using the code given below to run multiple https GET request for Wikipedia API.
app.get("/data_results", (req, res) => {
const articlesData = names.map(nameObj => {
let name = nameObj.name;
let articleExtract = "";
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
// Getting article content
https.get(contentURL, response => {
response.on("data", async data => {
const wikiArticle = JSON.parse(data);
// Mapping the keys of the JSON data of query to its values.
articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
nameObj.article = articleExtract.substring(0,350);
})
});
return nameObj;
});
res.send(articlesData);});
This is my names array
[
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
]
My aim :-
Run the HTTPS GET request for every value in the names array sequentially.
Add the article extract with its name and save in it in an objects array.
You could also suggest me better ways of doing this.
Try this solution. it creates an object like
{
Mahabharata: response
}
The code
const getValues = async () => {
const name_array = [
{name: 'Indus%20Valley%20Civilisation'},
{name: 'Ramayana'},
{name: 'Mahabharata'},
{name: 'Gautama%20Buddha'},
];
const namePromises = name_array.map(value =>
fetch('baseurl' + value.name).then(res => res.json()),
);
const response = await Promise.all(namePromises);
return response.reduce((obj, responseResults, index) => {
obj[name_array[index].name] = responseResults;
}, {});
};
I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.
something like this
const namePromises = [
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
].map((value) => fetch(baseurl + value.name).then(res=>res.json()));
Promise.all(namePromises).then((response) => {
console.log(response);
});
Problems:
So, you have several things going on here.
http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means you end up calling res.send(articlesData); before any of the http requests are complete.
.map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.
The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.
You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.
You don't show any error handling for errors in your http requests.
You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.
Solutions:
Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.
You can use got() and promises to control the asynchronous flow and error handling like this:
const got = require('got');
app.get("/data_results", (req, res) => {
Promise.all(names.map(nameObj => {
let name = nameObj.name;
// create separate result object
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
return got(contentURL).json().then(wikiArticle => {
// Mapping the keys of the JSON data of query to its values.
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
return result;
});
})).then(articlesData => {
res.send(articlesData);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.
If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:
const got = require('got');
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await got(contentURL).json();
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
If you really want to stick with https.get(), then you can "promisify" it and use it in place of got():
function myGet(url, options = {}) {
return new Promise((resolve, reject) => {
https.get(url, options, (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
} catch (e) {
reject(e);
}
});
}).on('error', reject);
});
}
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await myGet(contentURL);
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
I followed the youtube Node & Express project tutorial and Im confused facing these code:
This is in async js file:
const asyncHWrapper = (fn) => {
return async (req, res, next) => {
try {
await fn(req, res, next);
} catch (error) {
next(error);
}
};
};
module.exports = asyncHWrapper;
And this is the usage:
const Task = require("../models/taskModel");
const asyncWrapper = require("../middleware/async");
const { createCustomError } = require("../errors/customErrors");
const getAllTasks = asyncWrapper(async (req, res) => {
const tasks = await Task.find({});
res.status(200).json({ tasks });
});
Im just confused about these questions:
Does is necessary to return an arrow function in the asyncWrapper? Why doesn't just call the function?
Where do the params (req,res) in the asyncWrapper function come from? In the "fn" function declaration?
Why should I write two pairs of async and await in the wrapper and when I call it?
Thanks a lot!
The youtube tutorial link :Node.js / Express Course - Build 4 Projects
Let me analyze the code first, and then I'll answer your question.
So here is your wrapper
const asyncHWrapper = (fn) => {
return async (req, res, next) => {
try {
await fn(req, res, next);
} catch (error) {
next(error);
}
};
};
and here is how it is used
const getAllTasks = asyncWrapper(async (req, res) => {
const tasks = await Task.find({});
res.status(200).json({ tasks });
});
the asyncWrapper accept an fn param, in this case, it is this function:
async (req, res) => {
const tasks = await Task.find({});
res.status(200).json({ tasks });
}
after asyncHWrapper is called with the above function, it will return another function, in this case, the return function is assigned as the name getAllTasks.
Now for your question:
Does is necessary to return an arrow function in the asyncWrapper? Why doesn't just call the function?
Well basically you can write it like this
const asyncHWrapper = async (fn, req, res, next) => {
try {
await fn(req, res, next);
} catch (error) {
next(error);
}
};
And call it like this
await asyncHWrapper(async (req, res) => {
const tasks = await Task.find({});
res.status(200).json({ tasks });
}, req, res, next)
But in this case, it's just a normal function with callback, it's not a wrapper, and its name shouldn't be asyncHWrapper.
Where do the params (req,res) in the asyncWrapper function come from? In the "fn" function declaration?
No, it comes from getAllTasks, your fn is just consuming two values (req, res), and the next param will be used for error handling. So when you call getAllTask, you must pass in three params like this getAllTasks(req, res, next)
Why should I write two pairs of async and await in the wrapper and when I call it?
I'm not sure what you meant by "two pairs of async and await". I assume you're referring to await when calling getAllTasks and await when calling fn?
That's just because both of them are async functions.
I hope that this answer can help you think about the concept of a "wrapper". Let's say we have a function divide:
const divide = (a,b) => a/b;
You can use this in normal code quite easily:
x = divide(10,5); // sets x to 2.
But you may decide that you care about the possibility of errors. In this case, division by zero is a possibility. You could certainly include some error handling code where you define the divide function. But you could also choose to "wrap" divide with an error handler. This way, we can keep the error handling aspects away from the main division logic. We would like to be able to define a safeDivide function like this:
const safeDivide = catchErrors(divide);
In the same way that divide is a function that takes two arguments, safeDivide also has to be a function that takes two arguments. So the catchErrors wrapper will have to return a function. We will start with something like this:
const catchErrors = (fn) => {
return (p,q) => fn(p,q);
}
If you pass a function fn to catchErrors, it will return a function. That returned function takes two arguments p and q, and returns fn(p,q). So far, this doesn't really achieve anything (except limiting the number of arguments). But now, we can add a try/catch block. I'll do it in a couple of steps, because the notation can be confusing.
The first step is to put an explicit return inside the inner arrow function.
const catchErrors = (fn) => {
return (p,q) => {
return fn(p,q);
}
}
This is technically the same code - it just looks slightly different. Now we add the try/catch.
const catchErrors = (fn) => {
return (p,q) => {
try {
return fn(p,q);
} catch (e) {
console.log("Error occurred. Continuing with null result.");
return null;
}
}
}
So now, the function returned from catchErrors will do the same as the original function, except when an exception is thrown, in which case it will return null. (I'm not saying that this is the best general way to handle exceptions, but it's useful as an example, and it's related to the original question.)
So now look again at where we use this catchErrors wrapper function.
const safeDivide = catchErrors(divide);
When you call the wrapper catchErrors with function divide, it doesn't actually do any dividing - it doesn't yet have any numbers to divide. Instead, it builds up a new function that, whenever it is called, would do the dividing, and catch any exception that arises. I hope that answers your first question.
Your second question is where req and res come from. They are names given to arguments that will be passed to the function. They can be passed to the wrapped function (along with a 3rd argument next), and they will also be passed to the inner (nameless) function which includes calls to Task.find and res.status(200). Those arguments will be provided by the Express (or other) web framework.
I will leave your 3rd question, on the async/await aspects of the wrapper, for another answer.
I have a Vue.js app running on Firebase and as a database I use Firestore. The app has to import data (clientes) form another app (app2), but app2 only exports by sending an XML code through POST to an address. To receive the POST from app2, my app utilizes Firebase Cloud Functions.
const xml2js = require("xml2js");
const functions = require("firebase-functions");
const cors = require("cors");
const express = require("express");
const app = express();
const admin = require("firebase-admin");
const db = admin.initializeApp().firestore();
function parseXml(xml) {
return new Promise((resolve, reject) => {
xml2js.parseString(xml, { explicitArray: false }, (err, result) => {
if (err) {
reject(err);
} else {
resolve(result);
}
});
});
}
app.use(cors({ origen: true }));
app.post("/", async (request, response) => {
let xml = request.body.XMLRecords.toString();
const clientes = db.collection("clientes");
xml = xml.replace(/&(?!(?:apos|quot|[gl]t|amp);|#)/g, "&").trim();
xml = " " + xml;
console.log("Prep XML:");
console.log(xml);
let parsXml = await parseXml(xml);
console.log("parsXML");
console.log(parsXml);
Array.from(parsXml.Records.Clientes.Cliente).forEach(async cli => {
if (cli !== null) {
delete cli.$;
const docRef = clientes.doc(cli.CliCodigo);
console.log("cli " + cli.CliCodigo);
console.log(cli);
const writeResult = await docRef.set(cli);
console.log("Cliente " + cli.CliCodigo + " salvo");
}
});
response.send("Received.");
});
exports.apiClientes = functions.https.onRequest((request, response) => {
if (!request.path) {
request.url = `/${request.url}`; // prepend '/' to keep query params if any
}
return app(request, response);
});
The app does receive the request and is able to process it, but when I try to comunicate with the Firestore database the function stops sending the console.log()staments and won't save the data to the database.
What am I doing wrong?
This is most probably because within your Array.from(parsXml.Records.Clientes.Cliente).forEach() loop you are executing several asynchronous set() operations but you are returning the response (response.send("Received.");) before those asynchronous operations are done.
By doing response.send("Received."); you are indicating to the instance running your Cloud Function that it can terminate it, but in most cases the asynchronous writes to Firestore are not completed.
You need to correctly handle the Promises returned by the set() method calls, as follows (untested):
//....
const promises = [];
Array.from(parsXml.Records.Clientes.Cliente).forEach(cli => {
if (cli !== null) {
delete cli.$;
const docRef = clientes.doc(cli.CliCodigo);
promises.push(docRef.set(cli));
}
});
await Promise.all(promises);
response.send("Received.");
//....
So, we use Promise.all() to execute in parallel all the set() method calls. Promise.all() method returns a single Promise that fulfills when all of the promises passed to as an iterable (i.e. the promises array) have been fulfilled. Therefore you are sure that all the asynchronous work is done when you send back the response, indicating to the Cloud Function platform that it can safely terminate your Cloud Function.
One SaaS provider we use has a webook field, but only allows a single url to be entered. In fact, we need this webhook to be sent to two analytics services, so I need to figure out a way to write a custom endpoint that forwards the entire request to as many other endpoints as we need (currently 2).
What is the easiest way to do this with node and express? If I am not mistaken, a simple redirect would not work for multiple POSTs, right?
I am not sure what the headers or even request content will look like, but it needs to be preserved as much as possible in case auth is in headers etc.
This is what I have so far, but it's nowhere near complete:
app.post('/', (req, res) => {
console.log('Request received: ', req.originalUrl)
const forwardRequests = config.forwardTo.map(url => {
return new Promise((resolve, reject) => {
superagent
.post(url)
.send(req)
.end((endpointError, endpointResponse) => {
if (endpointError) {
console.error(`Received error from forwardTo endpoint (${url}): `, endpointError)
reject(endpointError)
} else {
resolve(endpointResponse)
}
})
})
})
Promise.all(forwardRequests)
.then(() => res.sendStatus(200))
.catch(() => res.sendStatus(500))
})
I get an error, because superagent.send is just for the content... how can I duplicate a request entirely and send it off?
To duplicate a request entirely and send it off to various endpoint, you can use the request module with req.pipe(request(<url>)) and Promise.all.
According to request module's document:
You can also pipe() from http.ServerRequest instances, as well as to http.ServerResponse instances. The HTTP method, headers, and entity-body data will be sent.
Here is an example:
const { Writable } = require('stream');
const forwardToURLs = ['http://...','http://...'];
app.post('/test', function(req, res) {
let forwardPromiseArray = [];
for (let url of forwardToURLs) {
let data = '';
let stream = new Writable({
write: function(chunk, encoding, next) {
data += chunk;
next();
}
});
let promise = new Promise(function(resolve, reject) {
stream.on('finish', function() {
resolve(data);
});
stream.on('error', function(e) {
reject(e);
});
});
forwardPromiseArray.push(promise);
req.pipe(request(url)).pipe(stream);
}
Promise.all(forwardPromiseArray).then(function(result) {
// result from various endpoint, you can process it and return a user-friendly result.
res.json(result);
}).catch(function() {
res.sendStatus(500);
});
});
Please note the above code should be placed before body-parser (if you are using it). Otherwise, the request won't be piped.
I'm trying to do something like this
function retrieveUser(uname) {
var user = User.find({uname: uname}, function(err, users) {
if(err)
console.log(err);
return null;
else
return users[0];
return user;
But this returns a document instead of a user object. The parameter users is an array of user objects matching the query, so how would I store one of the objects into a variable that my function could return?
The function User.find() is an asynchronous function, so you can't use a return value to get a resultant value. Instead, use a callback:
function retrieveUser(uname, callback) {
User.find({uname: uname}, function(err, users) {
if (err) {
callback(err, null);
} else {
callback(null, users[0]);
}
});
};
The function would then be used like this:
retrieveUser(uname, function(err, user) {
if (err) {
console.log(err);
}
// do something with user
});
Updated on 25th Sept. 2019
Promise chaining can also be used for better readability:
Model
.findOne({})
.exec()
.then((result) => {
// ... rest of the code
return Model2.findOne({}).exec();
})
.then((resultOfModel2FindOne) => {
// ... rest of the code
})
.catch((error) => {
// ... error handling
});
I was looking for an answer to the same question.
Hopefully, MongooseJS has released v5.1.4 as of now.
Model.find({property: value}).exec() returns a promise.
it will resolve to an object if you use it in the following manner:
const findObject = (value) => {
return Model.find({property: value}).exec();
}
mainFunction = async => {
const object = await findObject(value);
console.log(object); // or anything else as per your wish
}
Basically, MongoDB and NodeJS have asynchronous functions so we have to make it to synchronous functions then after it will work properly as expected.
router.get('/', async function(req, res, next) {
var users = new mdl_users();
var userData = []; // Created Empty Array
await mdl_users.find({}, function(err, data) {
data.forEach(function(value) {
userData.push(value);
});
});
res.send(userData);
});
In Example, mdl_users is mongoose model and I have a user collection(table) for user's data in MongoDB database and that data storing on "userData" variable to display it.In this find function i have split all documents(rows of table) by function if you want just all record then use direct find() function as following code.
router.get('/', async function(req, res, next) {
var users = new mdl_users();
var userData = await mdl_users.find();
res.send(userData);
});