I'm working on a small simple Node.js project that I expect to autonomously work with a public-facing API and my mongoDB server. For the most part everything is working smoothly and I don't expect any problem hitting the API once an hour.
However, there are times which instead of returning the body of data the API returns a JSON-formatted error statement (see below). In these cases JSON.parse(data) throws an exception because the format is already in JSON (and not the buffered bytes expected). Since this is a portfolio piece and others will be looking at the code, I want to be as prepared as possible for conditions I can't foresee overtime but haven't found a good way of approaching the parsing problem yet.
At first I had assumed that .on("error", ... ) would handle this error but I guess because it's the JSON parser and not the stream which catches the exception it will have to be handled there. I'm happy to just break out of readStream and get the data next hour but haven't found a configuration of try/catch that will do this. What are some ways to handle this exception so that it doesn't crash the server?
function readStream(stream) {
return new Promise((resolve, reject) => {
let saved = [];
stream
.on('data', function (data) {
console.log('Debug | readStream data: ', data); //See Output Below
const json = JSON.parse(data); // Exception here Line 37
saved.push(json.data);
if (saved.length > 4) {
stream.destroy();
}
})
.on('close', () => {
resolve(saved);
})
.on('error', (error) => reject(error))
.on('timeout', (error) => reject(error));
});
}
// API sometimes returns
{
title: 'ConnectionException',
detail: 'This stream is currently at the maximum allowed connection limit.',
connection_issue: 'TooManyConnections',
type: URL
}
//Exception Text from console
undefined:1
[object Object]
^
SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
at PassThrough.<anonymous> (/Users/?????/Documents/JavaScript-Projects/website-dashbaord1-nodejs/src/API.js:37:23)
Related
I'm occasionally getting a ERR_TIMED_OUT when scraping a site and want to know how I can catch the error so I can start the loop processing it again?
This is the function I'm using the get the response:
const responsePending = page.waitForResponse((response) => {
return response.url().includes('cloudfront.net');
});
console.log('Wait for response...');
var response = await responsePending;
// when there's an error it doesn't print the next line
console.log('Response received: '+response.status());
And this is the error I'm sometimes getting:
This site can’t be XXXXX took too long to respond.
Try:
Checking the connection
Checking the proxy and the firewall
ERR_TIMED_OUT
Puppeteer also records this error in the logs:
crbug/1173575, non-JS module files deprecated.
https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagewaitforresponseurlorpredicate-options
Set the timeout option to 0
const responsePending = page.waitForResponse((response) => {
return response.url().includes('cloudfront.net');
},{timeout:0});
I am new in node.js ecosystem and need some help.
I have controller which triggered when user call URL.
async function get_all_information(req, res, next) {
try {
let start_date = req.query.start_date;
let end_date = req.query.end_date;
const binds = {};
binds.start_date = start_date;
binds.end_date = end_date;
let query = `SOME LONG SQL STATEMENT`;
await pool.query(query, binds, function (error, results) {
if (error) throw error;
console.log(results); // ~ 10 seconds
res.send(JSON.stringify(results)); // ~ 15 seconds
});
} catch (error) {
next(error);
}
}
I tried to use this code but faced the problem. Monthly datapool which return a database has 227011 rows. It seems like stringify method create too huge JSON file. The Postman application just crash when I tried to test. I tried to analyze and notice that daily datapool create ~ 13 MB JSON file. We can say that monthly datapool could create ~ 400 MB JSON file.
Then I tried to streaming query rows like that:
pool.query('HUGE SQL QUERY').stream({ highWaterMark: 5 }).pipe(res);
Unfortunatly such code raise error:
TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be one of type string or Buffer. Received type object
Can someone show me how correctly send huge data from MySQL database in response?
#akram I used fs package as you adviced. I use next code to create JSON file:
await pool.query(sql_statement, binds, function (error, results) {
if (error) throw error;
fs.writeFile("information.json", results, 'utf8', function (error) {
if(error) throw error;
});
});
This code create json file which has ~ 3.5 MB size for monthly datapool. In editor I have next message:
This document contains very long lines. Soft wraps were forcibly enable to improve editor performance.
Also that json file contains:
It seems to me too strange.
Try to use JSONStream https://github.com/dominictarr/JSONStream, i think can resolve your issue, there is a lof of example in doc of JSONStream
I've got some code that looks like this
try {
return inputStream
.pipe(JSONStream.stringify())
.on('error', e => next(`Stringify error: ${e}`))
.pipe(Json2csvTransform)
.on('error', e => next(`json2csv error: ${e}`))
.pipe(res)
.on('finish', () => console.log("Streaming of file is now complete."))
} catch(error) {
return res.status(400).json({ message: msg('fileRetrievalError', -3) })
}
and when I reach .on('error', e => next('json2csv error: ${e}'))
the process does not fall through to the catch, it just keeps on keeping on. I'm guessing this is because it's wrapped in a next
The error i finally was able to extract is this:
Error: Data should not be empty or the "fields" option should be included
I tried to dig through the source in the node modules but it didn't mean much to me.
I guess i have two possible solutions: either I need to understand what the error coming from Json2csv means, or i need to be able to exit and close my stream. i tried to just shove return res.status(400).json({ message: msg('fileRetrievalError', -3) }) into the callback of the .on('error;), but if the process fails one time, it fails every time until the session ends, even when giving it all valid information - if that makes sense..
I don't know a ton about node, and this package doesn't have a lot of support surrounding it.
If you want to stop processing the stream after processing a particular error, you must throw the error:
.on('error', e => throw error.message)
I've tried all sorts to get this to work. I'm trying to request a PDF from an API on node, then send this back to the client who called it to begin with.
For the minute I just want to successfully save and view the PDF on the node server.
The issue is the PDF file is always empty when I open it (Even though it has a size of 30kb).
The basic flow is like this (removed a few bits, but the below code works and returns me the PDF fine)
// We pass through session ID's, request dates etc through in body
app.post("/getPayslipURL", function(client_request, res) {
// create request, which will simply pass on the data to the database (In order to get the NI number we need for the pay API)
const NI_NUMBER_REQUEST = db_api.createRequestTemplate({
body: JSON.stringify(client_request.body)
});
// Create a chain of HTTPS Requests, Starting with our call to the DB
requestPromise(NI_NUMBER_REQUEST)
.then((db_response) => {
const PAY_API_OPTIONS = /*Code to generate options based on furhter DB info (Includes dates etc)*/
return requestPromise(PAY_API_OPTIONS); // Call pay API
})
.then((pay_pdf_data) => {
console.log(typeof pay_pdf_data); // It's a string
// At this point I can log pay_pdf_data, But if I try to save it to file it's always empty
// No matter how I encode it etc
fs.writeFile("./test.pdf", pay_pdf_data, 'binary', function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
})
.catch(err => `Error caught: ${console.log}`) // Catch any errors on our request chain
});
}
I've tried saving with / without the binary flag as suggested in other posts in both the file save aswell as within the requests itself. Also various types of decoding methods have been tried, I always get an empty PDF saved.
My return data looks like this (is much bigger, when saved as test.pdf I get a 30kb file as before mentioned)
%PDF-1.4
%����
1 0 obj
0 obj
<
I've found a post which says about piping the data all the way through, I have a feeling my pdf_data is corrupted when getting converted to a string
Any ideas how would I go about doing this with the current setup?
e/ RequestPromise is a library, could also use the standards request library if it's easier
https://github.com/request/request-promise -
https://github.com/request/request
Thanks!
Your code doesn't work because the underlying request library (used by request-promise) requires the option encoding set to null for binary data - see https://github.com/request/request#requestoptions-callback.
Here's how you download binary data using that module -
app.post("/getPayslipURL", function(client_request, res) {
const NI_NUMBER_REQUEST = db_api.createRequestTemplate({
body: JSON.stringify(client_request.body),
encoding: null
});
requestPromise(NI_NUMBER_REQUEST)
.then((db_response) => {
const PAY_API_OPTIONS = /*Code to generate options based on furhter DB info (Includes dates etc)*/
return requestPromise(PAY_API_OPTIONS); // Call pay API
})
.then((pay_pdf_data) => {
fs.writeFile("./test.pdf", pay_pdf_data, 'binary', (err) => {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
})
.catch(err => `Error caught: ${console.log}`) // Catch any errors on our request chain
});
}
I am having problems trying to report errors from a function where I read a file and pipe it through csv-parse. What I want to do, is if there is a line that does not have three fields, I want to report back an error - to an async.series call. This first implementation fails as
function insertCsvIntoObject(done) {
fs.createReadStream(inputCsvFilepath)
.pipe(csvParser)
.on('data', function (csvRow) {
console.log('csv row:', csvRow);
if (csvRow.length !== 3) {
this.emit('error', "Does this go anywhere?");
}
})
.on('end', function () {
done(null, "Finished reading through the csv data");
})
.on('error', function (err) {
console.log('errors is now:', csvReadErrors);
done(new Error('Error reading csv file'), "CSV Parse error");
});
}
This gives a
Exception: [Error: Callback was already called.] if we have multiple problematic rows in the csv file.
Adding a return before the done
return done(new Error('Error reading csv file'), "CSV Parse error");
is also no help, since we cannot stop and return from the parent function - insertCsvIntoObject.
furthermore, if there are any error events, the .on('end',..) event never gets fired.
What is the correct way to report errors in such a situation?
It looks like csv-parse already emits an error when the number of columns on a line isn't consistent. In other words, you shouldn't have to emit one yourself.
To prevent a callback from being called multiple times, you can use a packaged like once:
function insertCsvIntoObject(done) {
done = once(done);
...
};
Ideally, you should be able to end the processing after the first error, but I haven't been able to stop the parser from parsing additional records (this.end() doesn't seem to work, but perhaps I'm missing something).