reporting errors when using async with streams node.js - javascript

I am having problems trying to report errors from a function where I read a file and pipe it through csv-parse. What I want to do, is if there is a line that does not have three fields, I want to report back an error - to an async.series call. This first implementation fails as
function insertCsvIntoObject(done) {
fs.createReadStream(inputCsvFilepath)
.pipe(csvParser)
.on('data', function (csvRow) {
console.log('csv row:', csvRow);
if (csvRow.length !== 3) {
this.emit('error', "Does this go anywhere?");
}
})
.on('end', function () {
done(null, "Finished reading through the csv data");
})
.on('error', function (err) {
console.log('errors is now:', csvReadErrors);
done(new Error('Error reading csv file'), "CSV Parse error");
});
}
This gives a
Exception: [Error: Callback was already called.] if we have multiple problematic rows in the csv file.
Adding a return before the done
return done(new Error('Error reading csv file'), "CSV Parse error");
is also no help, since we cannot stop and return from the parent function - insertCsvIntoObject.
furthermore, if there are any error events, the .on('end',..) event never gets fired.
What is the correct way to report errors in such a situation?

It looks like csv-parse already emits an error when the number of columns on a line isn't consistent. In other words, you shouldn't have to emit one yourself.
To prevent a callback from being called multiple times, you can use a packaged like once:
function insertCsvIntoObject(done) {
done = once(done);
...
};
Ideally, you should be able to end the processing after the first error, but I haven't been able to stop the parser from parsing additional records (this.end() doesn't seem to work, but perhaps I'm missing something).

Related

Handling JSON Parsing Error in Node.js Stream

I'm working on a small simple Node.js project that I expect to autonomously work with a public-facing API and my mongoDB server. For the most part everything is working smoothly and I don't expect any problem hitting the API once an hour.
However, there are times which instead of returning the body of data the API returns a JSON-formatted error statement (see below). In these cases JSON.parse(data) throws an exception because the format is already in JSON (and not the buffered bytes expected). Since this is a portfolio piece and others will be looking at the code, I want to be as prepared as possible for conditions I can't foresee overtime but haven't found a good way of approaching the parsing problem yet.
At first I had assumed that .on("error", ... ) would handle this error but I guess because it's the JSON parser and not the stream which catches the exception it will have to be handled there. I'm happy to just break out of readStream and get the data next hour but haven't found a configuration of try/catch that will do this. What are some ways to handle this exception so that it doesn't crash the server?
function readStream(stream) {
return new Promise((resolve, reject) => {
let saved = [];
stream
.on('data', function (data) {
console.log('Debug | readStream data: ', data); //See Output Below
const json = JSON.parse(data); // Exception here Line 37
saved.push(json.data);
if (saved.length > 4) {
stream.destroy();
}
})
.on('close', () => {
resolve(saved);
})
.on('error', (error) => reject(error))
.on('timeout', (error) => reject(error));
});
}
// API sometimes returns
{
title: 'ConnectionException',
detail: 'This stream is currently at the maximum allowed connection limit.',
connection_issue: 'TooManyConnections',
type: URL
}
//Exception Text from console
undefined:1
[object Object]
^
SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
at PassThrough.<anonymous> (/Users/?????/Documents/JavaScript-Projects/website-dashbaord1-nodejs/src/API.js:37:23)

Error: Data should not be empty or the "fields" option should be included

I've got some code that looks like this
try {
return inputStream
.pipe(JSONStream.stringify())
.on('error', e => next(`Stringify error: ${e}`))
.pipe(Json2csvTransform)
.on('error', e => next(`json2csv error: ${e}`))
.pipe(res)
.on('finish', () => console.log("Streaming of file is now complete."))
} catch(error) {
return res.status(400).json({ message: msg('fileRetrievalError', -3) })
}
and when I reach .on('error', e => next('json2csv error: ${e}'))
the process does not fall through to the catch, it just keeps on keeping on. I'm guessing this is because it's wrapped in a next
The error i finally was able to extract is this:
Error: Data should not be empty or the "fields" option should be included
I tried to dig through the source in the node modules but it didn't mean much to me.
I guess i have two possible solutions: either I need to understand what the error coming from Json2csv means, or i need to be able to exit and close my stream. i tried to just shove return res.status(400).json({ message: msg('fileRetrievalError', -3) }) into the callback of the .on('error;), but if the process fails one time, it fails every time until the session ends, even when giving it all valid information - if that makes sense..
I don't know a ton about node, and this package doesn't have a lot of support surrounding it.
If you want to stop processing the stream after processing a particular error, you must throw the error:
.on('error', e => throw error.message)

Making my own "database" using Node and FS

So I'm trying to make a database, couple of function snippets that read, write or create X.json files. The way I imagined it to be is a DB folder, then within that folder bunch of username folders, and there, a bunch of files, like account.json, level.json, and so on... So each folder would keep users data, now, this is the code I managed to write so far, and it works.
But the problem is, on the FS docs, it says that using fs.stat to check for the existence of the file before reading/writing to it is a bad idea. I don't understand why tho, as that seems like the only way to do it before I keep asking questions, I'd like to paste my code here:
socket.on('play', (data) => {
fs.stat(`db/${data.username}/account.json`, (error, result) => {
if(!error) {
fs.readFile(`db/${data.username}/account.json`, (error, result) => {
if(error) {
throw error;
} else {
const rawResult = JSON.parse(result);
if(data.password == rawResult.password) {
socket.emit('playResponse', {
success: true,
msg: 'Login Succesfull'
});
} else {
socket.emit('playResponse', {
success: false,
msg: 'Wrong Password!'
});
}
}
});
} else if(error.code == 'ENOENT') {
socket.emit('playResponse', {
success: false,
msg: 'Account not found'
});
}
});
});
I haven't written a general function that does this for me, because I figured that the code above is a mess right now. So, why is it a bad practice to check for the existence of the file (fs.stat) before writing/reading from them? I guess I could do something with the error I get from the readFile function and omit the fs.stat function, but whenever readFile function comes across a folder that doesn't exist, my server just crashes.
I'm not very experienced with Node, so the code above is probably absolute nonsense. That's why I'm here!
How can I make my server not crash if the readFile comes across a non-existent folder, but instead just emit the "Account not Found" through socket.io? If I put that emit code there, my server just crashes anyway.
I would just go with MongoDB or something, but I have a loooot of free time and doing stuff like this is very fun for me. > Is using a DB like mongo like more secure, or do people do it so they don't have to waste time writing their own DB?
Thanks for your help!
But the problem is, on the FS docs, it says that using fs.stat to check for the existence of the file before reading / writing to it is bad idea. I don't understand why tho
The reason is mentioned on the deprecated fs.exists docs:
Using fs.exists() to check for the existence of a file before calling fs.open(), fs.readFile() or fs.writeFile() is not recommended. Doing so introduces a race condition, since other processes may change the file's state between the two calls. Instead, user code should open/read/write the file directly and handle the error raised if the file does not exist.
How can I make my server not crash if the readFile comes across a non existent folder, but instead just emit the "Account not Found" through socket.io?
You don't handle the errors properly. As an example you throw an error in your .readFile callback but your code leaves the error unhandled, that will "crash" your application. You could either wrap your code with a try/catch block or use promises. Promises provide nice APIs to handle errors in your application. Node.js v10.0.0 introduces promise-wrapped APIs for fs module APIs.
const fs = require('fs');
const fsPromises = fs.promises;
fsPromises.readFile(`db/${data.username}/account.json`).then(error => {
// the file exists and readFile could read it successfully!
// you can throw an error and the next `catch` handle catches the error
}).catch(error => {
// there was an error
});
You can also use the APIs with try/catch and await:
try {
const content = await fsPromises.readFile(`db/${data.username}/account.json`);
// the file exists and readFile could read it successfully!
} catch(error) {
// handle the possible error
}
If using node v10.0.0 is not an option you can use a npm package that provides promised-wrapped fs APIs like fs-extra or draxt:
// using draxt
const $ = require('draxt');
const File = $.File;
const file = new File(`db/${data.username}/account.json`);
file.read('utf8').then(contents => {
// the file exists and readFile could read it successfully!
}).catch(error => {
// handle the possible error
});
The problem is that the file could be deleted betweent the time where you did got response the stat call and the readFile call.
The recommend way is to do call the readFile and the check for the error code in the callback of the readFile. For the callback based version this would look that way:
socket.on('play', (data) => {
fs.readFile(`db/${data.username}/account.json`, (error, result) => {
if (error) {
if (error.code === 'ENOENT') {
socket.emit('playResponse', {
success: false,
msg: 'Account not found'
})
} else {
// throw the error if you really want to exit the application for unknown errors
throw error
}
} else {
const rawResult = JSON.parse(result)
if (data.password === rawResult.password) {
socket.emit('playResponse', {
success: true,
msg: 'Login Succesfull'
})
} else {
socket.emit('playResponse', {
success: false,
msg: 'Wrong Password!'
})
}
}
})
})

Node: Wait for python script to run

I have the following code. Where i upload the file first and then i read the file and console the output like console.log(obj). But the response comes first and the python scripts runs behind the scene. How can i make code to wait for the python script to run then proceed?
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log(err);
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj);
});
return res.status(200).send({
message : 'Success',
});
});
I cannot get console.log(obj); output because it runs after the response. How can i make it wait for the python script to run and get console.log(obj) output on console.
To return the result after some async operation, you should call res.send inside the done-callback.
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log('The script work has been finished.'); // (*)
if(err) {
res.status(500).send({
error: err,
});
console.log(err);
return;
}
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj); // (**)
res.status(200).send({
message : 'Success',
});
});
});
Then if you will not see the log (*) in the console, then it would mean that the script does not work or works improperly. The callback is not being called. First of all, you need to be sure that the script (PythonShell.run) works and the callback is being called. The POST handler will wait until you call res.send (with no matter of delay value), so that callback is the main point.
Also readFileSync could fail. In case of readFileSync failure you should see an exception. If it's ok then you'll see the next log (**) and the response will be sent.
I see PythonShell in your code. I have no experience with it, but after some reading I think that the problem could be in how you are using it. It seems the python-shell npm package, so following it's documentation you may try to to instantiate a python shell for your script and then to use listeners:
let pyshell = new PythonShell('calculations.py');
router.post(`${basePath}/file`, (req, res) => {
pyshell.send(settings); // path, args etc
pyshell.end(function (err) {
console.log('The script work has been finished.');
if(err) { res.status(200).send({ error: err }); }
else { res.status(200).send({ message : 'Success' }); }
});
});
This approach could be more appropriate because the pyton shell is kept open between different POST requests. This depends on your needs. But I guess it does not solve the problem of script running. If you are sure that the script itself is fine, then you need just to run it properly in the Node environment. There are some points:
path to script
arguments
other settings
Try to remove all arguments (create some new test script), cleanup settings object (keep only path) and execute it from Node. Handle its result in Node. You should be able to run the simplest script by correct path! Research how to setup correct scriptPath. Then add an argument to your script and run it with an argument. Hanlde the result again. There are not so many options, but each of them could be the cause of improper call.

NodeJS - stat doesn't always run

Here's my code:
console.log("Path 1: " + fullName);
fs.stat(fullName, function(err, stats) {
console.log("Path 2: " + fullName);
}, function(err) { // I don't know if this part actually does something
console.log("An error occurred: " + err); // but I saw it in a different SO answer
});
The code simply doesn't run for some files. As in, my logs will show Path 1 with the file but not path 2 with the file, and also none of the "an error occurred". I was thinking maybe the files have an invalid character, because they all have equal signs in them. They look like this:
...file.device_type=mobile.jsx
Even if that's the case, why no error or anything? And if so, how can I actually stat these files?
You aren't logging or checking for an error. fs.stat() accepts two parameters only, the filename and a callback function. You are passing three parameters, the filename and two separate callbacks. So that second callback is just wrong. Then, in the first callback, you need to check the err variable to see if an error occurred.
This is the correct usage:
fs.stat(fullName, function(err, stats) {
if (err) {
console.log("Error in fs.stat(): ", err);
} else {
console.log("Got stats: ", stats);
}
});
If you're using this proper form and you still don't see either message in the console, then I'd suggest putting an exception handler around it to see if something else is going on:
try {
console.log("about to call fs.stat()");
fs.stat(fullName, function(err, stats) {
if (err) {
console.log("Error in fs.stat(): ", err);
} else {
console.log("Got stats: ", stats);
}
});
} catch(e) {
console.log("fs.stat() exception: ", e);
}
In looking at the source code for fs.stat(), there are several ways it could throw a synchronous exception, particularly if it detects invalid arguments passed to it. As usual, the node.js documentation does not describe that behavior, but you can see it in the code. This is the code for fs.stat():
fs.stat = function(path, callback) {
callback = makeStatsCallback(callback);
if (handleError((path = getPathFromURL(path)), callback))
return;
if (!nullCheck(path, callback)) return;
var req = new FSReqWrap();
req.oncomplete = callback;
binding.stat(pathModule._makeLong(path), req);
};
Both makeStatsCallback() and handleError() can throw an exception (when you look at their implementations in that same file).
I do not know where you got the notion of passing two callbacks to fs.stat(). As documented here, it does not accept two callback functions. It looks remotely like a promisified version of the fs library where every async operation returns a promise and then you can pass two callbacks to fs.statPromise.then(fn1, fn2), but I have no idea if that's where you saw this or not.
https://nodejs.org/api/fs.html#fs_fs_stat_path_callback
As per the documentation it shouldn't have the 3rd param function

Categories