I am using this async module for asynchronously requesting
web content with the help of another module request, as this is an asynchronous call.
Using async.each method, for requesting data from each link,
the result is also successfully returned by the scrap() function (which I have wrote to scrap returned html data
and return it as array of fuel prices by state).
Now, the problem is that when I try to return prices back to async.each() using cb(null, prices), it shows console.log(prices) as undefined
but logging inside the _check_fuel_prices(), works fine. It seems the callback works with only one argument
(or error only callback, as show as an example in the async.each link above). What if I want to it return prices (I can change it with error like cb(prices), but I also want to log error).
router.get('/someRoute', (req, res, next) => {
const fuels = ['diesel', 'petrol'];
async.each(fuels, _check_fuel_prices, (err, prices) => {
if (!err) {
res.statusCode = 200;
console.log(prices);
return res.json(prices);
}
res.statusCode = 400;
return res.json(err);
});
function _check_fuel_prices(fuel, cb) {
let prices = '';
const url_string = 'http://some.url/';
request(`${url_string}-${fuel}-price/`, (error, response, html) => {
if (error) {
cb(error, null);
return;
}
if (response.statusCode === 404) {
console.log(response.statusCode);
cb('UNABLE TO FIND PAGE', null);
return;
}
prices = scrap(html, fuel);
console.log(prices);
cb(null, prices);
return;
});
}
});
As #generalhenry points out, I was able to get the prices by using async.map which returns error first callback instead of error only apart from that async.series can be used here by slightly changing the code.
Related
So I created a function in my node server which takes in a query string, runs it on my db, and returns the results. I then wanted to use my function asynchronously using async await throughout my routes instead of having nested query within, nested query, within nested query etc.
So here is the code:
const runQuery = queryString => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.log("runQuery: FAILURE");
return error;
}
else {
console.log("runQuery: SUCCESS");
return(results);
}
})
}
register.post("/", async (req, res) => {
console.log(req.body);
const results = await runQuery("select * from test1");
res.send(results);
})
The database should have 3 entries, but unfortunately, it returns nothing. Meaning results is an empty variable when it is sent, meaning JS never properly waits for it to capture the db results. How can I use my function asynchronously, and how is this even feasible?
It seems your function "runQuery" does not return a promise, in fact, it's not returning anything. You are using "return" in the callback of the db.query function, not the function "runQuery" itself.
Since runQuery is performing an asynchronous operation, the result ought to be resolved via a promise (which is what the "await" in your request handler is looking for).
I'm not exactly sure but it seems you are using MySql, and I could not find anything on the npm page of the mysql package regarding the query being promisified, so we'll promisify it ourselves:
const runQuery = (queryString) => new Promise((resolve,reject) => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.error("runQuery: FAILURE");
reject(error);
} else {
console.log("runQuery: SUCCESS");
resolve(results);
}
})
})
register.post("/", async (req, res) => {
console.log(req.body);
try{
const results = await runQuery("select * from test1");
res.send(results);
}catch(e){
console.error(`ERROR THROWN BY runQuery:`,e);
res.status(500).send({
message: e.message || "INTERNAL SERVER ERROR"
})
}
})
Note that if an error occurs, our promisified function will reject the error and it will NOT be stored in the "results" variable in our request handler. Instead, an error will be thrown which needs to be handled. Therefore, it is always a good practice to put any async/await calls inside a try/catch.
Say there is a HTTP GET callback defined as:
router.get('/latestpost', function(req, res, next) {
var data = new FbData();
get_latest_post (data);
get_post_image (data);
res.json(data);
};
Both get_ functions use the fb package to generate a HTTP request and execute a callback when finished. How can the above GET callback be modified in order to wait for the responses from Facebook and only then send a response to the client?
At the time being I solved the problem by executing the get_ functions in series and passing them the res (response) argument, with the last function sending the response:
router.get('/latestpost', function(req, res, next) {
var data = new FbData();
get_latest_post (res, data);
};
function get_latest_post (res, data) {
FB.api(_url, function (res_fb) {
if(!res_fb || res_fb.error) {
console.log(!res_fb ? 'error occurred' : res_fb.error);
return;
}
// Do stuff with data
get_post_image (res, data);
});
}
function get_post_image (res, data) {
FB.api(_url, function (res_fb) {
if(!res_fb || res_fb.error) {
console.log(!res_fb ? 'error occurred' : res_fb.error);
return;
}
// Do stuff with data
/* At the end send the post data to the client */
res.json(data);
});
}
I have found a similar question, but I'm wrapping my head around it, since I can't find a proper way to apply the solution to my problem. I have tried using the patterns described in this manual, but I can't get it to execute using promises, or async/await. Can someone please point me in the right direction?
Your API can easily be modified to return a promise:
function get_post_image (res, data) {
return new Promise((resolve, reject) => {
FB.api(_url, function (res_fb) {
if(!res_fb || res_fb.error) {
reject(res_fb && res_fb.error);
} else resolve(res_fb/*?*/);
});
}
Now that you have a promise, you can await it:
router.get('/latestpost', async function(req, res, next) {
const data = new FbData();
const image = await get_post_image (data);
res.json(data);
});
This is a pseudo code of what I am trying to achieve. First I need to get a list of URLs from the request body then pass those URLs to request function (using request module) which will get the data from each url and then save those data to MongoDB. After all the requests are finished including saving data to the server only then it should send a response.
app.post('/', (req, resp) => {
const { urls } = req.body;
urls.forEach((url, i) => {
request(url, function (err, resp, body) {
if (err) {
console.log('Error: ', err)
} else {
// function to save data to MongoDB server
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
})
});
// Should be called after all data saved from for loop
resp.send('All data saved')
})
I have tried this code and of course the resp.send() function will run without caring if the request has completed. Using this code I get a result on the console like this:
Data saved for URL number - 3
Data saved for URL number - 1
Data saved for URL number - 5
Data saved for URL number - 2
Data saved for URL number - 4
I could write them in nested form but the variable urlscan have any number of urls and that's why it needs to be in the loop at least from my understanding. I want the requests to run sequentially i.e. it should resolve 1st url and then second and so on and when all urls are done only then it should respond. Please help!
app.post('/', async (req, resp) => {
const {
urls
} = req.body;
for (const url of urls) {
try {
const result = await doRequest(url)
console.log(result)
} catch (error) {
// do error processing here
console.log('Error: ', err)
}
}
})
function doRequest(url) {
return new Promise((resolve, reject) => {
request(url, function(err, resp, body) {
err ? reject(err) ? resolve(body)
})
})
}
using async await
You should look at JavaScript Promises
Otherwise, you can do a recursive request like so:
app.post('/', (req, resp) => {
const { urls } = req.body;
sendRequest(urls, 0);
})
function sendRequest(urlArr, i){
request(urlArr[i], function (err, resp, body) {
if (err) {
console.log('Error: ', err)
}
else {
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
i++;
if(i == urlArr.length) resp.send('All data saved') //finish
else sendRequest(urlArr, i); //send another request
})
}
All I had to do is create a separate function I can call over and over again, passing the url array and a base index 0 as arguments. Each success callback increments the index variable which I pass in the same function again. Rinse and repeat until my index hits the length of the url array, I'll stop the recursive loop from there.
You want to wait till all api response you get and stored in db, so you should do async-await and promisify all the response.
You can use Request-Promise module instead of request. So you will get promise on every requested api call instead of callback.
And use promise.all for pushing up all request(module) call inside array.
Using async-await you code execution will wait till all api call get response and stored in db.
const rp = require('request-promise');
app.post('/', async (req, res) => {
try{
const { urls } = req.body;
// completed all will have all the api resonse.
const completedAll = await sendRequest(urls);
// now we have all api response that needs to be saved
// completedAll is array
const saved = await saveAllData(completedAll);
// Should be called after all data saved from for loop
res.status(200).send('All data saved')
}
catch(err) {
res.status(500).send({msg: Internal_server_error})
}
})
function sendRequest(urlArr, i){
const apiCalls = [];
for(let i=0;i < urlArr.length; i++){
apiCalls.push(rp(urlArr[i]));
}
// promise.all will give all api response in order as we pushed api call
return Promise.all(apiCalls);
}
You can refer these links:
https://www.npmjs.com/package/request-promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
Looking at the intention(a crawler) you can use Promise.all because the urls are not dependant upon each other.
app.post('/', (req, resp) => {
const { urls } = req.body;
const promises = urls.map((url, i) => {
return new Promise((resolve, rej)=>{
request(url, function (err, resp, body) {
if (err) {
rej(err);
} else {
resolve(body);
}
})
})
.then((body)=>{
//this should definitely be a promise as you are saving data to mongo
return saveUrlData(body);
})
});
// Should be called after all data saved from for loop
Promise.all(promises).then(()=>resp.send('All data saved'));
})
Note: Need to do error handling as well.
there are multiple ways to solve this.
you can use async/await
Promises
you can also use the async library
app.post('/', (req, res, next) => {
const { urls } = req.body;
async.each(urls, get_n_save, err => {
if (err) return next(err);
res.send('All data saved');
});
function get_n_save (url, callback) {
request(url, (err, resp, body) => {
if (err) {
return callback(err);
}
saveUrlData(body);
callback();
});
}
});
I'm writing a rest api for a node application, and I find myself rewriting something like the following a lot:
function(req, res, next) {
databaseCall()
.then( (results) => {
if (results != null) {
res.status(200).send(results);
} else {
res.sendStatus(404);
}
})
.catch(function(err) {
console.log("Request error: " + err.stack);
res.sendStatus(500);
})
}
I would like to refactor the response portion, so I can do something like
databaseCall()
.then(handleResponse)
where handleResponse would take care of the whole response/catch process.
But I can't quite figure out how to do that. The databaseCall method varies depending on the endpoint - sometimes it takes a parameter, sometimes not. I could make a generic function expression that takes the databaseCall result and stick it in the promise chain, but I don't know how I could access the response object inside that function. I know I could add another function to combine everything, like so:
function(databaseCall, parameter, req, res, next) {
databaseCall(parameter)
.then( (results) => {
if (results != null) {
res.status(200).send(results);
} else {
res.sendStatus(404);
}
})
.catch( (err) => {
console.log("Request error: " + err.stack);
res.sendStatus(500);
})
}
But that seems ugly since databaseCall could have 0-several parameters. I'd think there's a more elegant solution.
You're probably thinking in the right direction, you just need to take it a step further and keep the db call outside the generic handler, and pass it as a promise instead
// generic handler for db promise
// the promise is created outside and passed as arg
function responseFromDb(databaseCallPromise, res) {
databaseCallPromise
.then((results) => {
if (results != null) {
res.status(200).send(results);
} else {
res.sendStatus(404);
}
})
.catch((err) => {
console.log(`Request error: ${err.stack}`);
res.sendStatus(500);
});
}
// handler per request, only needs to create db call with the desired params
// and pass it to the generic handler, which will take care of sending the response
function(req, res, next) {
responseFromDb(databaseCall(param1, param2), res)
}
I am using NodeJS to create an express endpoint that will retrieve the metadata from my images stored on my server. I have the following code for the logic of the endpoint:
/*
* Gallery Controller
*/
var fs = require('fs'),
_ = require('underscore'),
im = require('imagemagick');
/**
* List paths to images stored in selected gallery
*/
exports.list = function(req, res) {
var dir = 'public/images/' + req.params.id;
fs.readdir(dir, function(err, files) {
if (err) return res.send({error: 'No gallery found with provided id'}, 404);
if (files.length > 0) {
var collection = [],
myData = {};
files.forEach(function(file) {
if(file === '.DS_Store') return;
im.readMetadata( dir + '/' + file, function(err, metadata) {
if (err) throw err;
myData = metadata;
console.log(myData); // logs as object with expected data
});
console.log(myData); // logs as empty {}
collection.push(myData);
});
console.log(collection); // logs as [ {}, {} ]
res.json(collection, 200);
} else {
res.json({error: 'Selected gallery is empty'}, 404);
}
});
};
I've listed what the logs appear as in the terminal, why am I getting this scoping issue? I can't seem to wrap my head around it. If I try to return the metadata obj and assign it to the var, I get the following error: TypeError: Converting circular structure to JSON
Use the async module, it'll improve your life in many ways.
The problem you are having is a common one I see, and it is that your loop is asynchronous, but you are treating it as something serial.
Instead of doing files.forEach, you want to loop them asynchronously and then do some more stuff when the looping is done. You can use async.each for that.
async.each(files, function (file, next) {
if (file === '.DS_Store') return next();
im.readMetadata(path.join(dir, file), function (e, data) {
collection.push(data);
next(err);
});
}, function (err) {
if (err) throw err;
console.log(collection);
});
As an alternative, an even more appropriate solution might be to use async.map.
async.map(files, function (file, next) {
if (file === '.DS_Store') return next();
im.readMetadata(path.join(dir, file), next);
}, function (err, collection) {
if (err) throw err;
console.log(collection);
});
You need to restructure your code:
files.forEach(function(file, i) {
if (file === '.DS_Store') return; // see text
im.readMetadata( dir + '/' + file, function(err, metadata) {
if (err) throw err;
collection.push(metadata);
if (i === files.length - 1) {
res.json(collection); // see text
}
});
});
The reason is that the metadata is only available when the callback function to readMetadata is called; that's how asynchronous I/O works in Node.
In that callback, you add the metadata to the collection. If the iteration of the forEach has reached the final element (i is the index of the current element, when its value is one less than the size of the array, it's the last element), the response is sent.
Two issues:
if .DS_Store is the last/only file in the directory, this code will fail because it will never send back a response; I'll leave it to you to deal with that case ;)
res.json will, by default, return a 200 status so you don't have to specify it; if you do want to specify a status, it needs to be res.json(200, collection) (arguments swapped)