NodeJS - stat doesn't always run - javascript

Here's my code:
console.log("Path 1: " + fullName);
fs.stat(fullName, function(err, stats) {
console.log("Path 2: " + fullName);
}, function(err) { // I don't know if this part actually does something
console.log("An error occurred: " + err); // but I saw it in a different SO answer
});
The code simply doesn't run for some files. As in, my logs will show Path 1 with the file but not path 2 with the file, and also none of the "an error occurred". I was thinking maybe the files have an invalid character, because they all have equal signs in them. They look like this:
...file.device_type=mobile.jsx
Even if that's the case, why no error or anything? And if so, how can I actually stat these files?

You aren't logging or checking for an error. fs.stat() accepts two parameters only, the filename and a callback function. You are passing three parameters, the filename and two separate callbacks. So that second callback is just wrong. Then, in the first callback, you need to check the err variable to see if an error occurred.
This is the correct usage:
fs.stat(fullName, function(err, stats) {
if (err) {
console.log("Error in fs.stat(): ", err);
} else {
console.log("Got stats: ", stats);
}
});
If you're using this proper form and you still don't see either message in the console, then I'd suggest putting an exception handler around it to see if something else is going on:
try {
console.log("about to call fs.stat()");
fs.stat(fullName, function(err, stats) {
if (err) {
console.log("Error in fs.stat(): ", err);
} else {
console.log("Got stats: ", stats);
}
});
} catch(e) {
console.log("fs.stat() exception: ", e);
}
In looking at the source code for fs.stat(), there are several ways it could throw a synchronous exception, particularly if it detects invalid arguments passed to it. As usual, the node.js documentation does not describe that behavior, but you can see it in the code. This is the code for fs.stat():
fs.stat = function(path, callback) {
callback = makeStatsCallback(callback);
if (handleError((path = getPathFromURL(path)), callback))
return;
if (!nullCheck(path, callback)) return;
var req = new FSReqWrap();
req.oncomplete = callback;
binding.stat(pathModule._makeLong(path), req);
};
Both makeStatsCallback() and handleError() can throw an exception (when you look at their implementations in that same file).
I do not know where you got the notion of passing two callbacks to fs.stat(). As documented here, it does not accept two callback functions. It looks remotely like a promisified version of the fs library where every async operation returns a promise and then you can pass two callbacks to fs.statPromise.then(fn1, fn2), but I have no idea if that's where you saw this or not.

https://nodejs.org/api/fs.html#fs_fs_stat_path_callback
As per the documentation it shouldn't have the 3rd param function

Related

Node: Wait for python script to run

I have the following code. Where i upload the file first and then i read the file and console the output like console.log(obj). But the response comes first and the python scripts runs behind the scene. How can i make code to wait for the python script to run then proceed?
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log(err);
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj);
});
return res.status(200).send({
message : 'Success',
});
});
I cannot get console.log(obj); output because it runs after the response. How can i make it wait for the python script to run and get console.log(obj) output on console.
To return the result after some async operation, you should call res.send inside the done-callback.
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log('The script work has been finished.'); // (*)
if(err) {
res.status(500).send({
error: err,
});
console.log(err);
return;
}
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj); // (**)
res.status(200).send({
message : 'Success',
});
});
});
Then if you will not see the log (*) in the console, then it would mean that the script does not work or works improperly. The callback is not being called. First of all, you need to be sure that the script (PythonShell.run) works and the callback is being called. The POST handler will wait until you call res.send (with no matter of delay value), so that callback is the main point.
Also readFileSync could fail. In case of readFileSync failure you should see an exception. If it's ok then you'll see the next log (**) and the response will be sent.
I see PythonShell in your code. I have no experience with it, but after some reading I think that the problem could be in how you are using it. It seems the python-shell npm package, so following it's documentation you may try to to instantiate a python shell for your script and then to use listeners:
let pyshell = new PythonShell('calculations.py');
router.post(`${basePath}/file`, (req, res) => {
pyshell.send(settings); // path, args etc
pyshell.end(function (err) {
console.log('The script work has been finished.');
if(err) { res.status(200).send({ error: err }); }
else { res.status(200).send({ message : 'Success' }); }
});
});
This approach could be more appropriate because the pyton shell is kept open between different POST requests. This depends on your needs. But I guess it does not solve the problem of script running. If you are sure that the script itself is fine, then you need just to run it properly in the Node environment. There are some points:
path to script
arguments
other settings
Try to remove all arguments (create some new test script), cleanup settings object (keep only path) and execute it from Node. Handle its result in Node. You should be able to run the simplest script by correct path! Research how to setup correct scriptPath. Then add an argument to your script and run it with an argument. Hanlde the result again. There are not so many options, but each of them could be the cause of improper call.

Express.js Response Sent Callback

I have the following code in Node/Express that sends a file as a response then deletes the file using a timeout.
res.sendFile(req.params.id, { root: process.env.UPLOADPATH });
setTimeout(function () {
if (fs.existsSync(process.env.UPLOADPATH + req.params.id)) { // check to ensure file still exists on file system
fs.unlink(process.env.UPLOADPATH + req.params.id); // delete file from server file system after 60 seconds
}
}, 60000);
If I didn't use the setTimeout it failed with an error. I'm assuming Express does the sendFile async so it was deleting the file before it actually sent.
Is there a better way to do this tho? Is there a way to check for when the file has been sent so I can safely delete it? Maybe like a sendFile callback or something?
Is there a better way to do this tho? Is there a way to check for when the file has been sent so I can safely delete it? Maybe like a sendFile callback or something?
Yes, you should just remove the file when the res.sendFile() is actually done. You can use the completion callback on res.sendFile() to know when it's done.
Also, it is an anti-pattern to use if (fs.existsSync(...)) and then delete the file because it can be subject to race conditions. If you want the file deleted, just delete it and handle any errors you might get:
let filename = path.join(process.env.UPLOADPATH, req.params.id);
res.sendFile(filename, function (err) {
if (err) {
next(err);
} else {
try {
fs.unlink(filename);
} catch(e) {
console.log("error removing ", filename);
}
}
});
I'm assuming Express does the sendFile async so it was deleting the file before it actually sent.
Yes, that is true.
You could also use the res.on('finish', ...) event to know when the sending of the response is done.
let filename = path.join(process.env.UPLOADPATH, req.params.id);
res.sendFile(filename);
res.on('finish', function() {
try {
fs.unlink(filename);
} catch(e) {
console.log("error removing ", filename);
}
});
The method invokes the callback function fn(err) when the transfer is complete or when an error occurs. If the callback function is specified and an error occurs, the callback function must explicitly handle the response process either by ending the request-response cycle
res.sendFile(fileName, { root: process.env.UPLOADPATH }, function (err) {
if (err) {
next(err);
} else {
// File has been sent
console.log('Sent:', fileName);
if (fs.existsSync(process.env.UPLOADPATH + req.params.id)) {
// check to ensure file still exists on file system
fs.unlink(process.env.UPLOADPATH + req.params.id);
// delete file from server file system after 60 seconds
}
}
});
The main drawback to res.on('finish', ...) is that it isn't called on the response being closed or on an error. Using on-finished will run the callback on closes, finishes, or errors. This is especially helpful in the case of deleting a file where you want to delete the file even on errors or situations like that.

Is this async or blocking code?

Is this blocking code if I have JSON.stringify inside writeFile.Lets say it is big file. Is this why I am getting error conection refused when I push button multiple times in second which triggers this api endpoint?
app.patch('/', function(req, res) {
...some query
fs.writeFile(path.join(__dirname, "../../") + 'Data/deals.json', JSON.stringify(tickets), function (err) {
if (err) {
console.log(err);
return res.status(400).json({error: err});
}
return res.status(200).json({success: true, message: 'Deal Updated successfully'});
});
})
Edit #1:
JSON.stringify parsing object to string synchronously. but you missing that you called JSON.stringify inside the patch function and not inside the writeFIle.
Original:
It's Async. if you want to write a file synchronously you have to use writeFileSync.
For more information visit Node.js fs Docs
Example
// Async callback will called after finish
// Non-Blocking
fs.writeFile(file, data[, options], callback)
// write to file with return the result, may throw exception
// Blocking
fs.writeFileSync(file, data[, options])

reporting errors when using async with streams node.js

I am having problems trying to report errors from a function where I read a file and pipe it through csv-parse. What I want to do, is if there is a line that does not have three fields, I want to report back an error - to an async.series call. This first implementation fails as
function insertCsvIntoObject(done) {
fs.createReadStream(inputCsvFilepath)
.pipe(csvParser)
.on('data', function (csvRow) {
console.log('csv row:', csvRow);
if (csvRow.length !== 3) {
this.emit('error', "Does this go anywhere?");
}
})
.on('end', function () {
done(null, "Finished reading through the csv data");
})
.on('error', function (err) {
console.log('errors is now:', csvReadErrors);
done(new Error('Error reading csv file'), "CSV Parse error");
});
}
This gives a
Exception: [Error: Callback was already called.] if we have multiple problematic rows in the csv file.
Adding a return before the done
return done(new Error('Error reading csv file'), "CSV Parse error");
is also no help, since we cannot stop and return from the parent function - insertCsvIntoObject.
furthermore, if there are any error events, the .on('end',..) event never gets fired.
What is the correct way to report errors in such a situation?
It looks like csv-parse already emits an error when the number of columns on a line isn't consistent. In other words, you shouldn't have to emit one yourself.
To prevent a callback from being called multiple times, you can use a packaged like once:
function insertCsvIntoObject(done) {
done = once(done);
...
};
Ideally, you should be able to end the processing after the first error, but I haven't been able to stop the parser from parsing additional records (this.end() doesn't seem to work, but perhaps I'm missing something).

Return query result state from PostgreSQL query in node.js

I have a node.js server connected to a PostgreSQL database using the pg module. At one point I will insert data into two different database tables for a single HTTP POST. If the first query fails, the second should not be executed, but I have some trouble achieving this.
My generalized query function looks like this:
// General insertion function. If endResponse is true, the response will be ended,
// if it is false it will just write to the response and not end it (unless errors occurs).
function performInsertQuery(request, response, query, endResponse) {
var pg = require('pg');
var client = new pg.Client(request.app.get('postgreConnection'));
client.connect(function(error) {
if (error)
{
message = 'Could not connect to PostgreSQL database: ' + error;
response.end(message);
console.error(message);
client.end();
}
else
{
client.query(query, function (error, result)
{
if (error)
{
var message = 'Error running query ' + query + ': ' + error;
response.writeHead(500, {'content-type':'application/json'});
response.end(message);
console.error(message);
client.end();
}
else
{
var message = 'Query performed: ' + query;
if (endResponse)
{
response.end(message);
}
else
{
response.write(message + '\n');
}
console.log(message);
client.end();
}
});
}
});
}
Later, I have something like the following:
// query = some query
performInsertQuery(request, response, query, false);
// Stop running here if there was any errors running the query.
// Currently, this gets executed even though the first query fails.
// anotherQuery = another query
performInsertQuery(request, response, anotherQuery, true);
I have tried returning true and false from the function performInsertQuery, but since these are inner functions the result is not returned properly from the outer functions. Also, some if it is run asynchronously, which makes things a bit harder as well. I was not able to make it work with try/catch around the call to performInsertQuery either. I guess I could do another query to see if data was inserted, but this seems unnecessary and not very robust.
What would be the best way to return a success or failure state from performInsertQuery?
I know this doesn't exactly handle your question as you intended(dealing with this entirely in node.js), but this sounds like an excellent usecase for a Postgres transaction....Do not commit results unless all insertions/updates are successful. SQL transactions are built for scenarios like yours.
Here are the docs and example code for it with your node module.
https://github.com/brianc/node-postgres/wiki/Transactions
http://www.postgresql.org/docs/9.2/static/tutorial-transactions.html

Categories