We Keep Coding

Node.js Can't set headers after they are sent Error after introducing res.render()

I have this snippet of code:
app.post('/pst', function(req, res) {
var data = req.body.convo;
res.render('waiting.ejs'); //ADDED THIS
myFunc(data).then(result => {
res.render('success.ejs'); //THEN THIS
//---------------------------------
//clever way to send text file to client from the memory of the server
var fileContents = Buffer.from(result, 'ascii');
var readStream = new stream.PassThrough();
readStream.end(fileContents);
res.set('Content-disposition', 'attachment; filename=' + fileName);
res.set('Content-Type', 'text/plain');
readStream.pipe(res);
//--------------------------------------
}).catch( .....
The code i commented as 'clever way to send file from memory of the server' comes from this post:
Node Express.js - Download file from memory - 'filename must be a string'
What this does is is takes a string from the memory and serves it to the client as a .txt file.
This code used to work.
Then i decided to add the res.render('waiting.ejs'); line and i got this error:
Error: Can't set headers after they are sent.
I then experimented with adding another res.render() [in this case res.render('success.ejs');] before and after the code tht sends the .txt file to the client.
The error remained. Also, there is no redirect to success.ejs, in other words the res.render('success.ejs'); never worked, despite whether it is placed before ofr after that piece of code.

app.post('/pst', function(req, res) {
var data = req.body.convo;
myFunc(data).then(result => {
//---------------------------------
//clever way to send text file to client from the memory of the server
var fileContents = Buffer.from(result, 'ascii');
var readStream = new stream.PassThrough();
readStream.end(fileContents);
res.set('Content-disposition', 'attachment; filename=' + fileName);
res.set('Content-Type', 'text/plain');
readStream.pipe(res);
res.redirect(`/success`); //THEN THIS
//--------------------------------------
}).catch( .....
When you add middleware to express (which is built on connect) using the app.use method, you're appending items to Server.prototype.stack in connect.
When the server gets a request, it iterates over the stack, calling the (request, response, next) method.
The problem is, if in one of the middleware items writes to the response body or headers (it looks like it's either/or for some reason), but doesn't call response.end() and you call next() then as the core Server.prototype.handle method completes, it's going to notice that:
there are no more items in the stack, and/or
that response.headerSent is true.
So, it throws an error. But the error it throws is just this basic response (from the connect http.js source code:
res.statusCode = 404;
res.setHeader('Content-Type', 'text/plain');
res.end('Cannot ' + req.method + ' ' + req.url);
The problematic middleware sets the response header without calling response.end() and calls next(), which confuses express's server.
so you set the header through res.render() .Now if you will try to render again it will throw you an error.
app.get('/success',(req,res)=> {
res.render("container/index",{waiting:"waiting",......});
//handle your task then in client side index.ejs with appropriate setTimeout(()=>{},2000) for the waiting div , show waiting div for 2 seconds
});
//then your actual success gets render

You would have to check express.js source code (here):
res.render = function render(view, options, callback) {
var app = this.req.app;
var done = callback;
var opts = options || {};
var req = this.req;
var self = this;
// support callback function as second arg
if (typeof options === 'function') {
done = options;
opts = {};
}
// merge res.locals
opts._locals = self.locals;
// default callback to respond
done = done || function (err, str) {
if (err) return req.next(err);
self.send(str);
};
// render
app.render(view, opts, done);
};
You can see that when You use res.render() method, it will pass the done callback to app.render(...) (source code), it will then pass done to tryInitView etc.
At the end, it will invoke done callback with str in case of success or err in case of failure. It then triggers res.send() inside done callback which simply blocks You from setting headers after that.

res.render() function compiles your template, inserts locals there, and creates html output out of those two things. that's why error comes.
don't use it twice coz it send response.

Node: Wait for python script to run

I have the following code. Where i upload the file first and then i read the file and console the output like console.log(obj). But the response comes first and the python scripts runs behind the scene. How can i make code to wait for the python script to run then proceed?
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log(err);
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj);
});
return res.status(200).send({
message : 'Success',
});
});
I cannot get console.log(obj); output because it runs after the response. How can i make it wait for the python script to run and get console.log(obj) output on console.

To return the result after some async operation, you should call res.send inside the done-callback.
router.post(`${basePath}/file`, (req, res) => {
//Upload file first
PythonShell.run('calculations.py', { scriptPath: '/Path/to/python/script' }, function (err) {
console.log('The script work has been finished.'); // (*)
if(err) {
res.status(500).send({
error: err,
});
console.log(err);
return;
}
let obj = fs.readFileSync('Path/to/file', 'utf8');
console.log(obj); // (**)
res.status(200).send({
message : 'Success',
});
});
});
Then if you will not see the log (*) in the console, then it would mean that the script does not work or works improperly. The callback is not being called. First of all, you need to be sure that the script (PythonShell.run) works and the callback is being called. The POST handler will wait until you call res.send (with no matter of delay value), so that callback is the main point.
Also readFileSync could fail. In case of readFileSync failure you should see an exception. If it's ok then you'll see the next log (**) and the response will be sent.
I see PythonShell in your code. I have no experience with it, but after some reading I think that the problem could be in how you are using it. It seems the python-shell npm package, so following it's documentation you may try to to instantiate a python shell for your script and then to use listeners:
let pyshell = new PythonShell('calculations.py');
router.post(`${basePath}/file`, (req, res) => {
pyshell.send(settings); // path, args etc
pyshell.end(function (err) {
console.log('The script work has been finished.');
if(err) { res.status(200).send({ error: err }); }
else { res.status(200).send({ message : 'Success' }); }
});
});
This approach could be more appropriate because the pyton shell is kept open between different POST requests. This depends on your needs. But I guess it does not solve the problem of script running. If you are sure that the script itself is fine, then you need just to run it properly in the Node environment. There are some points:
path to script
arguments
other settings
Try to remove all arguments (create some new test script), cleanup settings object (keep only path) and execute it from Node. Handle its result in Node. You should be able to run the simplest script by correct path! Research how to setup correct scriptPath. Then add an argument to your script and run it with an argument. Hanlde the result again. There are not so many options, but each of them could be the cause of improper call.

Passing function on express js Route not working

I'm just really new on Node and Express. Trying to pass a function instead of text on my route but it seems not working. I just looked up at documentation there, They mentioned only text with req.send() method. I'm trying to pass here function's but it's not working. and also the alert() not working like this req.send(alert('Hello world')) it say's alert isn't defined or something similar.
**Update: ** I'm trying to execute this library with express and node https://github.com/przemyslawpluta/node-youtube-dl
I'm trying to do here pass functions like this
function blaBla() {
var youtubedl = require('youtube-dl');
var url = 'http://www.youtube.com/watch?v=WKsjaOqDXgg';
// Optional arguments passed to youtube-dl.
var options = ['--username=user', '--password=hunter2'];
youtubedl.getInfo(url, options, function(err, info) {
if (err) throw err;
console.log('id:', info.id);
console.log('title:', info.title);
console.log('url:', info.url);
console.log('thumbnail:', info.thumbnail);
console.log('description:', info.description);
console.log('filename:', info._filename);
console.log('format id:', info.format_id);
});
}
app.get('/', (req, res) => {
res.send(blaBla());
})
**Instead of **
app.get('/', function (req, res) {
res.send('Hello World!')
})
I hope you guy's understood my question.

res.send() expects a string argument. So, you have to pass a string.
If you want the browser to execute some Javascript, then what you send depends upon what kind of request is coming in from the browser.
If it's a browser page load request, then the browser expects an HTML response and you need to send an HTML page string back. If you want to execute Javascript as part of that HTML page, then you can embed a <script> tag inside the page and then include Javascript text inside that <script> tag and the browser will execute that Javascript when the page is parsed and scripts are run.
If the route is in response to a script tag request, then you can return Javascript text as a string and you need to make sure the MIME type appropriately indicates that it is a script.
If the route is in response to an Ajax call, then it all depends upon what the caller of the Ajax call expects. If they expect a script and are going to execute the text as Javascript, then you can also just send Javascript text as a string. If they expect HTML and are going to process it as HTML, then you probably need to embed the <script> tag inside that HTML in order to get the Javascript executed.
In your example of:
response.send(blaBla());
That will work just fine if blaBla() synchronously returns a string that is formatted properly per the above comments about what the caller is expecting. If you want further help with that, then you need to show or describe for us how the request is initiated in the browser and show us the code for the blaBla() function because the issue is probably in the blaBla() function.
There are lots of issues with things you have in your question:
You show req.send(alert('Hello world')) in the text of your question. The .send() method belongs to the res object, not the req object (the second argument, not the first). So, that would be res.send(), not req.send().
In that same piece of code, there is no alert() function in node.js, but you are trying to execute it immediately and send the result with .send(). That won't work for a bunch of reasons.
Your first code block using blaBla() will work just fine as long as blaBla() returns a string of the right format that matches what the caller expects. If that doesn't work, then there's a problem with what blaBla() is doing so we need to see that code.
Your second code block works because you are send a string which is something the caller is equipped to handle.
Update now that you've shown the code for blaBla().
Your code for blaBla() does not return anything and it's asynchronous so it can't return the result. Thus, you cannot use the structure response.send(blaBla());. There is no way to make that work.
Instead, you will need to do something different like:
blaBla(response);
And, then modify blaBla() to call response.send(someTextValue) when the response string is known.
function blaBla(res) {
var youtubedl = require('youtube-dl');
var url = 'http://www.youtube.com/watch?v=WKsjaOqDXgg';
// Optional arguments passed to youtube-dl.
var options = ['--username=user', '--password=hunter2'];
youtubedl.getInfo(url, options, function(err, info) {
if (err) {
res.status(500).send("Internal Error");
} else {
console.log('id:', info.id);
console.log('title:', info.title);
console.log('url:', info.url);
console.log('thumbnail:', info.thumbnail);
console.log('description:', info.description);
console.log('filename:', info._filename);
console.log('format id:', info.format_id);
// construct your response here as a string
res.json(info);
}
});
}
Note also that the error handling does not use throw because that is really not useful inside an async callback.

No one just could help me with that and after finding things are alone I got to know how to do this. In express there is something called middleware we have to use that thing to get this kind of matter done. Those who are really expert or have working experience with express they know this thing.
to using functions with express you need to use middleware.
like below I'm showing
const express = require('express')
const youtubedl = require('youtube-dl');
const url = 'https://www.youtube.com/watch?v=quQQDGvEP10';
const app = express()
const port = 3000
function blaBla(req, res, next) {
youtubedl.getInfo(url, function(err, info) {
console.log('id:', info.id);
console.log('title:', info.title);
console.log('url:', info.url);
// console.log('thumbnail:', info.thumbnail);
// console.log('description:', info.description);
console.log('filename:', info._filename);
console.log('format id:', info.format_id);
});
next();
}
app.use(blaBla);
app.get('/', (request, response) => {
response.send('Hey Bebs, what is going on here?');
})
app.listen(port, (err) => {
if (err) {
return console.log('something bad happened', err)
}
console.log(`server is listening on ${port}`)
})
And remember that you must need to use app.use(blaBla); on top of getting your route. Otherwise this might not work.

Node.js JavaScript Basic Get Request

Just installed node.js, and I'm having trouble sending basic get requests. I used to run things in chrome/firefox's console but wanted to branch out. What I am trying to do (as a test) is send a get request to a webpage, and have it print out some text on it.
In chrome's console, I would do something like this:
$.get("http://stackoverflow.com/questions/1801160/can-i-use-jquery-with-node-js", function(data) {
console.log($(data).find(".question-hyperlink")[0].innerHTML);
});
In node.js, how would I do that? I've tried requiring a few things and gone off a few examples but none of them worked.
Later on, I'll also need to add parameters to get and post requests, so if that involves something different, could you show how to send the request with the parameters {"dog":"bark"}? And say it returned the JSON {"cat":"meow"}, how would I read/get that?

You can install the request module with:
npm install request
And, then do this in your node.js code:
const request = require('request');
request.get("http://stackoverflow.com/questions/1801160/can-i-use-jquery-with-node-js", function(err, response, body) {
if (err) {
// deal with error here
} else {
// you can access the body parameter here to see the HTML
console.log(body);
}
});
The request module supports all sorts of optional parameters you can specify as part of your request for everything from custom headers to authentication to query parameters. You can see how to do all those things in the doc.
If you want to parse and search the HTML with a DOM like interface, you can use the cheerio module.
npm install request
npm install cheerio
And, then use this code:
const request = require('request');
const cheerio = require('cheerio');
request.get("http://stackoverflow.com/questions/1801160/can-i-use-jquery-with-node-js", function(err, response, body) {
if (err) {
// deal with error here
} else {
// you can access the body parameter here to see the HTML
let $ = cheerio.load(body);
console.log($.find(".question-hyperlink").html());
}
});

Getting Data from Node.js file and displaying it in HTML/JS page

I am new to Node.js and this is my first project with it.
I have made a node.js file named test.js. It has an array say a.
Now I want to make a HTML file that calls this test.js on button click event. Then get the data from that file and publish it on a table in the HTML file.
I have already written the node.js file and I can see the results on console.log(a). But I cant understand how to send this array to HTML when it will ask for it.
Meanwhile, I googled and made up some code. The request reaches the server but I always get error response from server. Why so?
Client Side -
function fetch() {
$.ajax({
type: 'POST',
url: "http://127.0.0.1:8888",
data: 'China',
datatype: 'json',
success: function (data) {
alert("hi");
var ret = jQuery.parseJSON(data);
$('#q').html(ret.msg);
},
error: function (xhr, status, error) {
alert("hii");
}
});
Server side :
http.createServer(function(request, response) {
console.log("Request received");
response.writeHeader(200, {"Content-Type": "application/json"});
request.on('data', function (chunk) {
console.log(chunk.toString('utf8'));
consol.log(result);
response.write(JSON.stringify({data : result}));
});
response.end();
}).listen(8888);
I can see China on the console.But I dont get back the result array back to the client. Here result is an array and I get its value on the console. Just that I dont get it back to the client. Any help ?

You should start by setting up a server to serve requests. I use expressjs for this - http://expressjs.com/
This will allow you to run nodejs as a web application.
Setup a route in express JS to serve your data - http://expressjs.com/api.html#express
var express = require('express');
var app = express();
app.get('/data', function(req, res){
res.send('hello world'); //replace with your data here
});
app.listen(3000);
Open up a browser, and type in http://MY_SERVER_ADDR:3000/data and you should see your output there.
Next, you'll need to attach an event handler to your HTML file that will trigger a $.get() request when it is triggered. Add the previous url to your data in your $.get call and do something with it.
$('.my_selector').click(function(){
$.get('http://MY_SERVER_ADDR:3000/data', {}, function(data){
console.log(data)
});
});
That should get you going.

After wrestling with the same question, i found that this is exactly where a template engine comes into the node-picture.
EJS solved it for me, but there are many more available.
This article compares 10 template engines.