I would like a response to differ depending on the finished request I recieve. I am sending a POST request and receive an xml file. The result is either a success or error. I use xml2json to convert the xml into a json object, then depending on the response I want to output json.
The problem is that I can't have a response inside a response. I also can't save the value of the callback for later usage (since its asynchronous).
I have thought about using Promises but I'm not sure. What should I do?
The order of operations should be
1) Send request
2) Get buffer response
3) Join Buffers. Process xml into JSON
4) Depending on the type of JSON entry, output either res.json('success') or res.json('error') if the xml responds with an error.
app.post('/api/submit', (req, res) => {
...
const request = https.request(options, (res) => {
let chunks = [];
res.on("data", function(chunk) {
chunks.push(chunk);
});
res.on("end", function(err) {
if (err) throw err;
let body = Buffer.concat(chunks);
xmlConverter(body, function(err, result) {
console.dir(result);
if (result.entry) {
console.log('✅ Success')
//Respond with json here --> res.json('success')
} else if (result.error) {
console.log('There was an error processing your request');
//or here if there was an error --> res.json('error')
}
});
});
});
request.end()
You can respond inside the callback. The problem is that you have two variable, both named res, so one shadows the other. You just need to change one of the res variable names so your not shadowing it. For example, you can change:
const request = https.request(options, (http_res) // <--change argument name
Then later:
if (result.entry) {
console.log('✅ Success')
http_res.json('success') // <-- use the response object from request
The problem of not being able to save the result for later is a different problem, but easy to solve. The solution though really depends one what you are trying to do. If, for example, you want to further process the data, you can set up a function to call and pass the response data in. Something like:
function process_data(response){
// use the response here
}
Then you can simply call it when you get the data:
if (result.entry) {
console.log('✅ Success')
http_res.json('success') // <-- use the response object from request
process_data(result)
Of course maybe your use case is more complicated but without more details its hard to give a specific answer.
Don't use the same name for both res, because they are different variables. And simply use the out res variable to respond the request with the value you want.
I think it would be something like this:
app.post('/
api/submit', (req, res) => {
...
const request = https.request(options, (resValue) => {
let chunks = [];
resValue.on("data", function(chunk) {
chunks.push(chunk);
});
resValue.on("end", function(err) {
if (err) throw err;
let body = Buffer.concat(chunks);
xmlConverter(body, function(err, result) {
console.dir(result);
if (result.entry) {
console.log('✅ Success')
res.json('success')
} else if (result.error) {
console.log('There was an error processing your request');
res.json('error')
}
});
});
});
request.end()
What exactly is the issue? You are perfectly able to rename the argument of the callback function supplied to https.request(options, callbackFunction) -- it is not important what this variable is named.
app.post('/api/submit', (req, res) => {
const request = https.request(options, (potato) => {
let chunks = [];
potato.on("data", function(chunk) {
chunks.push(chunk);
});
potato.on("end", function(err) {
if (err) throw err; // TODO res.status(500).json({}); ??
let body = Buffer.concat(chunks);
xmlConverter(body, function(err, result) {
console.dir(result);
if (result.entry) {
console.log('✅ Success')
res.status(200).json({});
} else if (result.error) {
console.log('There was an error processing your request');
res.status(500).json({});
}
request.end()
});
});
});
});
Related
INTRODUCTION
I am implementing a function for making any kind of https request to any endpoint (using the https native module). When I make a request to a specific API I get an error response in JSON format. Like this:
{
"error": {
"code": 404,
"message": "ID not found"
}
}
How can I handle this kind of errors? At a first moment, I supposed that they were handled in
request.on("error", (err) => {
reject(err);
});
HTTPs Request function code
I have comment '<---------' in the relevant parts of the code
const https = require("https");
exports.httpsRequest = function (options, body = null) {
/*
This function is useful for making requests over the HTTPs protocol
*/
return new Promise((resolve, reject) => {
const request = https.request(options, (response) => {
// Get the response content type
const contentType =
response.headers["content-type"] &&
response.headers["content-type"].split(";")[0];
// Cumulate data
let chuncks = [];
response.on("data", (chunck) => {
chuncks.push(chunck);
});
response.on("end", () => {
// Concat all received chunks
let response = Buffer.concat(chuncks);
// Some responses might be in JSON format...
if (contentType === "application/json") {
// Jsonify the response
response = JSON.parse(response);
}
// (For the future) TODO - Check and parse more content types if needed.
// Resolve the promise with the HTTPs response
resolve(response); // <--------- The JSON format error responses are resolved too!!
});
});
// Reject on request error
request.on("error", (err) => {
// <------------- At a first moment, I supposed that all error responses were handled in this part of the code
reject(err);
});
// Write the body
if (body) {
request.write(body);
}
// Close HTTPs connection.
request.end();
});
};
Question
Why the error response is not handled in request.on("error", ...) ?
Thank you. I would appreciate any help or suggestion.
You need to create a different code path for when the content type isn't what you were expecting in which you call reject() and you also need to try/catch around JSON parsing errors so you can properly catch them and reject on them too. You can solve those issues with this code:
exports.httpsRequest = function (options, body = null) {
/*
This function is useful for making requests over the HTTPs protocol
*/
return new Promise((resolve, reject) => {
const request = https.request(options, (response) => {
// Get the response content type
const contentType =
response.headers["content-type"] &&
response.headers["content-type"].split(";")[0];
// Cumulate data
let chuncks = [];
response.on("data", (chunck) => {
chuncks.push(chunck);
});
response.on("end", () => {
// Concat all received chunks
let response = Buffer.concat(chuncks);
// Some responses might be in JSON format...
if (contentType === "application/json") {
try {
// Jsonify the response
response = JSON.parse(response);
resolve(response);
return;
} catch(e) {
reject(e);
return;
}
}
reject(new Error("Not JSON content-type"))
});
});
// Reject on request error
request.on("error", (err) => {
reject(err);
});
// Write the body
if (body) {
request.write(body);
}
// Close HTTPs connection.
request.end();
});
};
FYI, libraries such as got() and others listed here, all do this work for you automatically and have a lot of other useful features. You don't really need to build this yourself.
Say there is a HTTP GET callback defined as:
router.get('/latestpost', function(req, res, next) {
var data = new FbData();
get_latest_post (data);
get_post_image (data);
res.json(data);
};
Both get_ functions use the fb package to generate a HTTP request and execute a callback when finished. How can the above GET callback be modified in order to wait for the responses from Facebook and only then send a response to the client?
At the time being I solved the problem by executing the get_ functions in series and passing them the res (response) argument, with the last function sending the response:
router.get('/latestpost', function(req, res, next) {
var data = new FbData();
get_latest_post (res, data);
};
function get_latest_post (res, data) {
FB.api(_url, function (res_fb) {
if(!res_fb || res_fb.error) {
console.log(!res_fb ? 'error occurred' : res_fb.error);
return;
}
// Do stuff with data
get_post_image (res, data);
});
}
function get_post_image (res, data) {
FB.api(_url, function (res_fb) {
if(!res_fb || res_fb.error) {
console.log(!res_fb ? 'error occurred' : res_fb.error);
return;
}
// Do stuff with data
/* At the end send the post data to the client */
res.json(data);
});
}
I have found a similar question, but I'm wrapping my head around it, since I can't find a proper way to apply the solution to my problem. I have tried using the patterns described in this manual, but I can't get it to execute using promises, or async/await. Can someone please point me in the right direction?
Your API can easily be modified to return a promise:
function get_post_image (res, data) {
return new Promise((resolve, reject) => {
FB.api(_url, function (res_fb) {
if(!res_fb || res_fb.error) {
reject(res_fb && res_fb.error);
} else resolve(res_fb/*?*/);
});
}
Now that you have a promise, you can await it:
router.get('/latestpost', async function(req, res, next) {
const data = new FbData();
const image = await get_post_image (data);
res.json(data);
});
This is a pseudo code of what I am trying to achieve. First I need to get a list of URLs from the request body then pass those URLs to request function (using request module) which will get the data from each url and then save those data to MongoDB. After all the requests are finished including saving data to the server only then it should send a response.
app.post('/', (req, resp) => {
const { urls } = req.body;
urls.forEach((url, i) => {
request(url, function (err, resp, body) {
if (err) {
console.log('Error: ', err)
} else {
// function to save data to MongoDB server
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
})
});
// Should be called after all data saved from for loop
resp.send('All data saved')
})
I have tried this code and of course the resp.send() function will run without caring if the request has completed. Using this code I get a result on the console like this:
Data saved for URL number - 3
Data saved for URL number - 1
Data saved for URL number - 5
Data saved for URL number - 2
Data saved for URL number - 4
I could write them in nested form but the variable urlscan have any number of urls and that's why it needs to be in the loop at least from my understanding. I want the requests to run sequentially i.e. it should resolve 1st url and then second and so on and when all urls are done only then it should respond. Please help!
app.post('/', async (req, resp) => {
const {
urls
} = req.body;
for (const url of urls) {
try {
const result = await doRequest(url)
console.log(result)
} catch (error) {
// do error processing here
console.log('Error: ', err)
}
}
})
function doRequest(url) {
return new Promise((resolve, reject) => {
request(url, function(err, resp, body) {
err ? reject(err) ? resolve(body)
})
})
}
using async await
You should look at JavaScript Promises
Otherwise, you can do a recursive request like so:
app.post('/', (req, resp) => {
const { urls } = req.body;
sendRequest(urls, 0);
})
function sendRequest(urlArr, i){
request(urlArr[i], function (err, resp, body) {
if (err) {
console.log('Error: ', err)
}
else {
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
i++;
if(i == urlArr.length) resp.send('All data saved') //finish
else sendRequest(urlArr, i); //send another request
})
}
All I had to do is create a separate function I can call over and over again, passing the url array and a base index 0 as arguments. Each success callback increments the index variable which I pass in the same function again. Rinse and repeat until my index hits the length of the url array, I'll stop the recursive loop from there.
You want to wait till all api response you get and stored in db, so you should do async-await and promisify all the response.
You can use Request-Promise module instead of request. So you will get promise on every requested api call instead of callback.
And use promise.all for pushing up all request(module) call inside array.
Using async-await you code execution will wait till all api call get response and stored in db.
const rp = require('request-promise');
app.post('/', async (req, res) => {
try{
const { urls } = req.body;
// completed all will have all the api resonse.
const completedAll = await sendRequest(urls);
// now we have all api response that needs to be saved
// completedAll is array
const saved = await saveAllData(completedAll);
// Should be called after all data saved from for loop
res.status(200).send('All data saved')
}
catch(err) {
res.status(500).send({msg: Internal_server_error})
}
})
function sendRequest(urlArr, i){
const apiCalls = [];
for(let i=0;i < urlArr.length; i++){
apiCalls.push(rp(urlArr[i]));
}
// promise.all will give all api response in order as we pushed api call
return Promise.all(apiCalls);
}
You can refer these links:
https://www.npmjs.com/package/request-promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
Looking at the intention(a crawler) you can use Promise.all because the urls are not dependant upon each other.
app.post('/', (req, resp) => {
const { urls } = req.body;
const promises = urls.map((url, i) => {
return new Promise((resolve, rej)=>{
request(url, function (err, resp, body) {
if (err) {
rej(err);
} else {
resolve(body);
}
})
})
.then((body)=>{
//this should definitely be a promise as you are saving data to mongo
return saveUrlData(body);
})
});
// Should be called after all data saved from for loop
Promise.all(promises).then(()=>resp.send('All data saved'));
})
Note: Need to do error handling as well.
there are multiple ways to solve this.
you can use async/await
Promises
you can also use the async library
app.post('/', (req, res, next) => {
const { urls } = req.body;
async.each(urls, get_n_save, err => {
if (err) return next(err);
res.send('All data saved');
});
function get_n_save (url, callback) {
request(url, (err, resp, body) => {
if (err) {
return callback(err);
}
saveUrlData(body);
callback();
});
}
});
I am using this async module for asynchronously requesting
web content with the help of another module request, as this is an asynchronous call.
Using async.each method, for requesting data from each link,
the result is also successfully returned by the scrap() function (which I have wrote to scrap returned html data
and return it as array of fuel prices by state).
Now, the problem is that when I try to return prices back to async.each() using cb(null, prices), it shows console.log(prices) as undefined
but logging inside the _check_fuel_prices(), works fine. It seems the callback works with only one argument
(or error only callback, as show as an example in the async.each link above). What if I want to it return prices (I can change it with error like cb(prices), but I also want to log error).
router.get('/someRoute', (req, res, next) => {
const fuels = ['diesel', 'petrol'];
async.each(fuels, _check_fuel_prices, (err, prices) => {
if (!err) {
res.statusCode = 200;
console.log(prices);
return res.json(prices);
}
res.statusCode = 400;
return res.json(err);
});
function _check_fuel_prices(fuel, cb) {
let prices = '';
const url_string = 'http://some.url/';
request(`${url_string}-${fuel}-price/`, (error, response, html) => {
if (error) {
cb(error, null);
return;
}
if (response.statusCode === 404) {
console.log(response.statusCode);
cb('UNABLE TO FIND PAGE', null);
return;
}
prices = scrap(html, fuel);
console.log(prices);
cb(null, prices);
return;
});
}
});
As #generalhenry points out, I was able to get the prices by using async.map which returns error first callback instead of error only apart from that async.series can be used here by slightly changing the code.
I'm trying to establish communication between two node.js scripts.
The first one does a get request and write the response in a file.
The second watch the file after changes, then he read it, and prompt the result.
The first (get then write)
var request = require('request');
var parseString = require('xml2js').parseString;
var fs = require('fs');
//Some needed variables
streamInterval = setInterval(function() {
request.get(addr, function (error, response, body) {
if (!error && response.statusCode == 200) {
parseString(body,{ explicitArray : false, ignoreAttrs : true }, function (err, result) {
var jsonResult = JSON.stringify(result);
var result = JSON.parse(jsonResult);
fs.writeFile(outputDeparts, JSON.stringify(result, null, 4), function(err) {
if(err) {
console.log(err);
}
});
});
}else{
console.log("An error occured : " + response.statusCode);
}
}).auth(LOGIN,PASS,true);
}, 30000);
The second (watch after changes, read and prompt)
var fs = require('fs');
//Some needed variables
fs.watch(outputDeparts, (eventType, filename) => {
console.log(`event type is: ${eventType}`);
if (filename) {
console.log(`filename provided: ${filename}`);
fs.readFile(outputDeparts, 'utf8', function (err, data) {
if (err){
throw err;
}else{
console.log('start parsing');
console.log(data);
var result = JSON.parse(data);
var departs = result["passages"]["train"];
console.log(`next train [${departs[0]["num"]}] at : ${departs[0]["date"]}`);
}
});
} else {
console.log('filename not provided');
}
});
The first time the file is changed everything is ok ! But after 30 second, at the second change I get the following error :
undefined:1
SyntaxError: Unexpected end of input
at Object.parse (native)
at /Users/adobe/Documents/workspace/playWithNode/watchFile.js:17:23
at tryToString (fs.js:414:3)
at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)
I thought it was a problem with reading/writting a file asynchronously but didn't manage to find a fix...
Is everyone able to help me or having clues ? Thanks
..it isn't encouraged to access the file system asynchronously because, while accessing, the target file can be modified by something in the App Life Cycle.
I suggest you to use fs.readFileSync and to wrap the JSON.parse in a try-catch
var fs = require('fs');
function onNextTrain(data) {
console.log("onNextTrain", data);
return data;
}
fs.watch(outputDeparts, (eventType, filename) => {
return new Promise((resolve, reject) => {
if(!filename) {
return reject("FILE NAME NOT PROVIDED");
}
let
data = fs.readFileSync(filename, 'utf8'),
result
;
try {
result = JSON.parse(data);
} catch(error) {
result = data;
}
return resolve(result);
})
.then(onNextTrain)
.catch(err => console.log("error", err))
;
});
Try change code in second file to
var result;
try {
result = JSON.parse(data);
} catch (err) {
console.error(err, data);
}
because SyntaxError: Unexpected end of input it's JSON.parse error.
In first file I see potential problem in
fs.writeFile(outputDeparts, data, function(err) { ...
Callback function must provide ordered write, but your code - only logged error.
Node.js docs
Note that it is unsafe to use fs.writeFile multiple times on the same file without waiting for the callback. For this scenario, fs.createWriteStream is strongly recommended.