Node: Wait for python script to run - javascript

I have the following code. Where i upload the file first and then i read the file and console the output like console.log(obj). But the response comes first and the python scripts runs behind the scene. How can i make code to wait for the python script to run then proceed?
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log(err);
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj);
});
return res.status(200).send({
message : 'Success',
});
});
I cannot get console.log(obj); output because it runs after the response. How can i make it wait for the python script to run and get console.log(obj) output on console.

To return the result after some async operation, you should call res.send inside the done-callback.
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log('The script work has been finished.'); // (*)
if(err) {
res.status(500).send({
error: err,
});
console.log(err);
return;
}
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj); // (**)
res.status(200).send({
message : 'Success',
});
});
});
Then if you will not see the log (*) in the console, then it would mean that the script does not work or works improperly. The callback is not being called. First of all, you need to be sure that the script (PythonShell.run) works and the callback is being called. The POST handler will wait until you call res.send (with no matter of delay value), so that callback is the main point.
Also readFileSync could fail. In case of readFileSync failure you should see an exception. If it's ok then you'll see the next log (**) and the response will be sent.
I see PythonShell in your code. I have no experience with it, but after some reading I think that the problem could be in how you are using it. It seems the python-shell npm package, so following it's documentation you may try to to instantiate a python shell for your script and then to use listeners:
let pyshell = new PythonShell('calculations.py');
router.post(`${basePath}/file`, (req, res) => {
pyshell.send(settings); // path, args etc
pyshell.end(function (err) {
console.log('The script work has been finished.');
if(err) { res.status(200).send({ error: err }); }
else { res.status(200).send({ message : 'Success' }); }
});
});
This approach could be more appropriate because the pyton shell is kept open between different POST requests. This depends on your needs. But I guess it does not solve the problem of script running. If you are sure that the script itself is fine, then you need just to run it properly in the Node environment. There are some points:
path to script
arguments
other settings
Try to remove all arguments (create some new test script), cleanup settings object (keep only path) and execute it from Node. Handle its result in Node. You should be able to run the simplest script by correct path! Research how to setup correct scriptPath. Then add an argument to your script and run it with an argument. Hanlde the result again. There are not so many options, but each of them could be the cause of improper call.

Related

Making my own "database" using Node and FS

So I'm trying to make a database, couple of function snippets that read, write or create X.json files. The way I imagined it to be is a DB folder, then within that folder bunch of username folders, and there, a bunch of files, like account.json, level.json, and so on... So each folder would keep users data, now, this is the code I managed to write so far, and it works.
But the problem is, on the FS docs, it says that using fs.stat to check for the existence of the file before reading/writing to it is a bad idea. I don't understand why tho, as that seems like the only way to do it before I keep asking questions, I'd like to paste my code here:
socket.on('play', (data) => {
fs.stat(`db/${data.username}/account.json`, (error, result) => {
if(!error) {
fs.readFile(`db/${data.username}/account.json`, (error, result) => {
if(error) {
throw error;
} else {
const rawResult = JSON.parse(result);
if(data.password == rawResult.password) {
socket.emit('playResponse', {
success: true,
msg: 'Login Succesfull'
});
} else {
socket.emit('playResponse', {
success: false,
msg: 'Wrong Password!'
});
}
}
});
} else if(error.code == 'ENOENT') {
socket.emit('playResponse', {
success: false,
msg: 'Account not found'
});
}
});
});
I haven't written a general function that does this for me, because I figured that the code above is a mess right now. So, why is it a bad practice to check for the existence of the file (fs.stat) before writing/reading from them? I guess I could do something with the error I get from the readFile function and omit the fs.stat function, but whenever readFile function comes across a folder that doesn't exist, my server just crashes.
I'm not very experienced with Node, so the code above is probably absolute nonsense. That's why I'm here!
How can I make my server not crash if the readFile comes across a non-existent folder, but instead just emit the "Account not Found" through socket.io? If I put that emit code there, my server just crashes anyway.
I would just go with MongoDB or something, but I have a loooot of free time and doing stuff like this is very fun for me. > Is using a DB like mongo like more secure, or do people do it so they don't have to waste time writing their own DB?
Thanks for your help!
But the problem is, on the FS docs, it says that using fs.stat to check for the existence of the file before reading / writing to it is bad idea. I don't understand why tho
The reason is mentioned on the deprecated fs.exists docs:
Using fs.exists() to check for the existence of a file before calling fs.open(), fs.readFile() or fs.writeFile() is not recommended. Doing so introduces a race condition, since other processes may change the file's state between the two calls. Instead, user code should open/read/write the file directly and handle the error raised if the file does not exist.
How can I make my server not crash if the readFile comes across a non existent folder, but instead just emit the "Account not Found" through socket.io?
You don't handle the errors properly. As an example you throw an error in your .readFile callback but your code leaves the error unhandled, that will "crash" your application. You could either wrap your code with a try/catch block or use promises. Promises provide nice APIs to handle errors in your application. Node.js v10.0.0 introduces promise-wrapped APIs for fs module APIs.
const fs = require('fs');
const fsPromises = fs.promises;
fsPromises.readFile(`db/${data.username}/account.json`).then(error => {
// the file exists and readFile could read it successfully!
// you can throw an error and the next `catch` handle catches the error
}).catch(error => {
// there was an error
});
You can also use the APIs with try/catch and await:
try {
const content = await fsPromises.readFile(`db/${data.username}/account.json`);
// the file exists and readFile could read it successfully!
} catch(error) {
// handle the possible error
}
If using node v10.0.0 is not an option you can use a npm package that provides promised-wrapped fs APIs like fs-extra or draxt:
// using draxt
const $ = require('draxt');
const File = $.File;
const file = new File(`db/${data.username}/account.json`);
file.read('utf8').then(contents => {
// the file exists and readFile could read it successfully!
}).catch(error => {
// handle the possible error
});
The problem is that the file could be deleted betweent the time where you did got response the stat call and the readFile call.
The recommend way is to do call the readFile and the check for the error code in the callback of the readFile. For the callback based version this would look that way:
socket.on('play', (data) => {
fs.readFile(`db/${data.username}/account.json`, (error, result) => {
if (error) {
if (error.code === 'ENOENT') {
socket.emit('playResponse', {
success: false,
msg: 'Account not found'
})
} else {
// throw the error if you really want to exit the application for unknown errors
throw error
}
} else {
const rawResult = JSON.parse(result)
if (data.password === rawResult.password) {
socket.emit('playResponse', {
success: true,
msg: 'Login Succesfull'
})
} else {
socket.emit('playResponse', {
success: false,
msg: 'Wrong Password!'
})
}
}
})
})

Pausing Node.js Readable Stream

I'm building a bar code scanning app using the node-serialport. Where I'm stuck is making a AJAX call to trigger a scan and then have Express server respond with the data from the readable stream.
Initialize Device:
// Open device port
var SerialPort = require('serialport');
var port = '/dev/cu.usbmodem1411';
var portObj = new SerialPort(port, (err) => {
if(err) {
console.log('Connection error ' + err);
}
});
//Construct device object
var Scanner = {
// Trigger Scan
scan : () => {
portObj.write(<scan cmd>), (err) => {
if(err) {
console.log('Error on scan' + err);
}
});
}
}
I've tried two approaches and neither produce the 'scan-read-respond' behavior I'm looking for.
First, I tried putting a event listener immediately following a scan, then using a callback in the listener to respond to the AJAX request. With this approach I get a 'Can't set headers after they are sent' error'. From what I understand Node is throwing this error because res.send is being called multiple times.
First Approach -- Response as callback in listener:
app.get('/dashboard', (req, res) => {
Scanner.scan(); //fire scanner
portObj.on('data', (data) => {
res.send(data); //'Can't set headers after they are sent' error'
});
}
In the second approach, I store the scan data into a local variable ('scanned_data') and move the response outside the listener block. The problem with this approach is that res.send executes before the scanned data is captured in the local variable and so comes up as 'undefined'. Also intriguing is the scanned_data that is captured in the listener block seems to multiple with each scan.
Second Approach -- Response outside listener:
app.get('/dashboard', (req, res) => {
var scanned_data; //declare variable outside listener block
Scanner.scan(); //trigger scan
portObj.on('data', (data) => {
scanned_data = data;
console.log(scanned_data); //displays scanned data but data multiplies with each scan. (e.g. 3 triggers logs 'barcode_data barcode_data barcode_data')
});
console.log(scanned_data); //undefined
res.send(scanned_data);
}
I'm a front end developer but have gotten to learn a lot about Node trying to figure this out. Alas, I think I've come to a dead end at this point. I tinkered with the .pipe() command, and have a hunch that's where the solution lies, but wasn't able to zero in on a solution that works.
Any thoughts or suggestions?
You should not make assumptions about what chunk of data you get in a 'data' event. Expect one byte or many bytes. You need to know the underlying protocol being used to know when you have received a full "message" so you can stop listening for data. At that point you should then send a response to the HTTP request.

Passing function on express js Route not working

I'm just really new on Node and Express. Trying to pass a function instead of text on my route but it seems not working. I just looked up at documentation there, They mentioned only text with req.send() method. I'm trying to pass here function's but it's not working. and also the alert() not working like this req.send(alert('Hello world')) it say's alert isn't defined or something similar.
**Update: ** I'm trying to execute this library with express and node https://github.com/przemyslawpluta/node-youtube-dl
I'm trying to do here pass functions like this
function blaBla() {
var youtubedl = require('youtube-dl');
var url = 'http://www.youtube.com/watch?v=WKsjaOqDXgg';
// Optional arguments passed to youtube-dl.
var options = ['--username=user', '--password=hunter2'];
youtubedl.getInfo(url, options, function(err, info) {
if (err) throw err;
console.log('id:', info.id);
console.log('title:', info.title);
console.log('url:', info.url);
console.log('thumbnail:', info.thumbnail);
console.log('description:', info.description);
console.log('filename:', info._filename);
console.log('format id:', info.format_id);
});
}
app.get('/', (req, res) => {
res.send(blaBla());
})
**Instead of **
app.get('/', function (req, res) {
res.send('Hello World!')
})
I hope you guy's understood my question.
res.send() expects a string argument. So, you have to pass a string.
If you want the browser to execute some Javascript, then what you send depends upon what kind of request is coming in from the browser.
If it's a browser page load request, then the browser expects an HTML response and you need to send an HTML page string back. If you want to execute Javascript as part of that HTML page, then you can embed a <script> tag inside the page and then include Javascript text inside that <script> tag and the browser will execute that Javascript when the page is parsed and scripts are run.
If the route is in response to a script tag request, then you can return Javascript text as a string and you need to make sure the MIME type appropriately indicates that it is a script.
If the route is in response to an Ajax call, then it all depends upon what the caller of the Ajax call expects. If they expect a script and are going to execute the text as Javascript, then you can also just send Javascript text as a string. If they expect HTML and are going to process it as HTML, then you probably need to embed the <script> tag inside that HTML in order to get the Javascript executed.
In your example of:
response.send(blaBla());
That will work just fine if blaBla() synchronously returns a string that is formatted properly per the above comments about what the caller is expecting. If you want further help with that, then you need to show or describe for us how the request is initiated in the browser and show us the code for the blaBla() function because the issue is probably in the blaBla() function.
There are lots of issues with things you have in your question:
You show req.send(alert('Hello world')) in the text of your question. The .send() method belongs to the res object, not the req object (the second argument, not the first). So, that would be res.send(), not req.send().
In that same piece of code, there is no alert() function in node.js, but you are trying to execute it immediately and send the result with .send(). That won't work for a bunch of reasons.
Your first code block using blaBla() will work just fine as long as blaBla() returns a string of the right format that matches what the caller expects. If that doesn't work, then there's a problem with what blaBla() is doing so we need to see that code.
Your second code block works because you are send a string which is something the caller is equipped to handle.
Update now that you've shown the code for blaBla().
Your code for blaBla() does not return anything and it's asynchronous so it can't return the result. Thus, you cannot use the structure response.send(blaBla());. There is no way to make that work.
Instead, you will need to do something different like:
blaBla(response);
And, then modify blaBla() to call response.send(someTextValue) when the response string is known.
function blaBla(res) {
var youtubedl = require('youtube-dl');
var url = 'http://www.youtube.com/watch?v=WKsjaOqDXgg';
// Optional arguments passed to youtube-dl.
var options = ['--username=user', '--password=hunter2'];
youtubedl.getInfo(url, options, function(err, info) {
if (err) {
res.status(500).send("Internal Error");
} else {
console.log('id:', info.id);
console.log('title:', info.title);
console.log('url:', info.url);
console.log('thumbnail:', info.thumbnail);
console.log('description:', info.description);
console.log('filename:', info._filename);
console.log('format id:', info.format_id);
// construct your response here as a string
res.json(info);
}
});
}
Note also that the error handling does not use throw because that is really not useful inside an async callback.
No one just could help me with that and after finding things are alone I got to know how to do this. In express there is something called middleware we have to use that thing to get this kind of matter done. Those who are really expert or have working experience with express they know this thing.
to using functions with express you need to use middleware.
like below I'm showing
const express = require('express')
const youtubedl = require('youtube-dl');
const url = 'https://www.youtube.com/watch?v=quQQDGvEP10';
const app = express()
const port = 3000
function blaBla(req, res, next) {
youtubedl.getInfo(url, function(err, info) {
console.log('id:', info.id);
console.log('title:', info.title);
console.log('url:', info.url);
// console.log('thumbnail:', info.thumbnail);
// console.log('description:', info.description);
console.log('filename:', info._filename);
console.log('format id:', info.format_id);
});
next();
}
app.use(blaBla);
app.get('/', (request, response) => {
response.send('Hey Bebs, what is going on here?');
})
app.listen(port, (err) => {
if (err) {
return console.log('something bad happened', err)
}
console.log(`server is listening on ${port}`)
})
And remember that you must need to use app.use(blaBla); on top of getting your route. Otherwise this might not work.

Executing an .HTML in a web-browser - node js

I'm running a program that validates a .html file (index.html) and then runs the program on a web-browser. (localhost:port/index.html).
My program understands that I'm asking for the right file but I can't execute the .html file in web-browser (just trying to display the date).
I've been looking online for references but maybe I'm not looking in the right direction or I'm just overlooking something.
Also the response.end doesn't print onto the web-browser even though other functions with response.end it works perfectly.
function doHTML(http_request, response)
{
// Read the requested file content from file system
fs.readFile('MYHTML/' + http_request, function (err, data) {
console.log('Hey youre in the HTML function');
response.end('Hey you got it working');
});
}
I'm fairly new to NodeJS as well, but I believe the following link may help you out:
https://expressjs.com/en/api.html#res.render
Also, I believe response.end('..text and stuff..') would send the text to your web/desktop client, so that you can process the response there: print it, parse it etc. So it's understandable to me that response.end wouldn't directly print onto the web-browser as console.log would do.
Hope this helps.
Edit: Just realizing that my response assumes you're using the Express framework.
If you are using your function as a request handler for node.js it looks like you should use http_request.url instead of http_request. The following function worked for me (server skeleton from here):
const fs = require('fs')
const http = require('http')
const port = 3000
const requestHandler = (http_request, response) => {
console.log(http_request.url)
fs.readFile('MYHTML/' + http_request.url, function (err, data) {
if (! err){
console.log('Hey youre in the HTML function.');
response.write(data);
response.write('Hey you got it working');
response.end();
} else {
response.end("File not found.");
}
});
}
const server = http.createServer(requestHandler)
server.listen(port, (err) => {
if (err) {
return console.log('something bad happened', err)
}
console.log(`server is listening on ${port}`)
})

Is this async or blocking code?

Is this blocking code if I have JSON.stringify inside writeFile.Lets say it is big file. Is this why I am getting error conection refused when I push button multiple times in second which triggers this api endpoint?
app.patch('/', function(req, res) {
...some query
fs.writeFile(path.join(__dirname, "../../") + 'Data/deals.json', JSON.stringify(tickets), function (err) {
if (err) {
console.log(err);
return res.status(400).json({error: err});
}
return res.status(200).json({success: true, message: 'Deal Updated successfully'});
});
})
Edit #1:
JSON.stringify parsing object to string synchronously. but you missing that you called JSON.stringify inside the patch function and not inside the writeFIle.
Original:
It's Async. if you want to write a file synchronously you have to use writeFileSync.
For more information visit Node.js fs Docs
Example
// Async callback will called after finish
// Non-Blocking
fs.writeFile(file, data[, options], callback)
// write to file with return the result, may throw exception
// Blocking
fs.writeFileSync(file, data[, options])

Categories