I'm having trouble with something very basic. Going through node.js in Action (great book so far!) and I can't get this simple example to work. Perhaps it's because the stream api was updated after the book came out. I'm not sure. Anyway, here's the code:
var http = require('http');
var server = http.createServer(function (req, res) {
req.on('data', function (chunk) {
console.log("Chuck: ", chunk);
});
req.on('end', function () {
console.log("End of Request");
res.end('yay');
});
}).listen(3000);
The console.log('Chunk: ', chunk) never fires. It's almost as if the data events never fire, but according to the documentation the presence of the data handler should switch the readable stream (req) into flowing mode. Am I missing something?
Any help would rock!
The above code is current initally the request body is undefined you have to pass request data inorder to execute this line.
req.on('data', function (chunk) {
console.log("Chuck: ", chunk);
});
Use postman to send data in the request this line will be executed
Related
There is a simple web server that accepts data. Sample code below.
The idea is to track in real time how much data has entered the server and immediately inform the client about this. If you send a small amount of data, then everything works well, but if you send more than X data in size, then the on.data event on the server is triggered with a huge delay. I can see that data is transfering for 5 seconds already but on.data event is not trigerred.
on.data event seems to be triggered only when data is uploaded completely to the server, so that's why it works fine with small data (~2..20Mb), but with big data (50..200Mb) it doesnt work well.
Or maybe it is due to some kind of buffering..?
Do you have any suggestions why on.data triggered with delay and how to fix it?
const app = express();
const port = 3000;
// PUBLIC API
// upload file
app.post('/upload', function (request, response) {
request.on('data', chunk => {
// message appears with delay
console.log('upload on data', chunk.length);
// send message to the client about chunk.length
});
response.send({
message: `Got a POST request ${request.headers['content-length']}`
});
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
TLDR:
The delay that you are experiencing probably is the Queueing from Resource scheduling from the browser.
The Test
I did some tests with express, and then I found that it uses http to handle requests/response, so I used a raw http server listener to test this scenario, which has the same situation.
Backend code
This code, based on sample of Node transaction samples, will create a http server and give log of time on 3 situations:
When a request was received
When the first data event fires
When the end event fires
const http = require('http');
var firstByte = null;
var server = http.createServer((request, response) => {
const { headers, method, url } = request;
let body = [];
request.on('error', (err) => {
}).on('data', (chunk) => {
if (!firstByte) {
firstByte = Date.now();
console.log('received first byte at: ' + Date.now());
}
}).on('end', () => {
console.log('end receive data at: ' + Date.now());
// body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
if (url === '/') {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/html');
response.write('<h1>Hello World</h1>');
}
firstByte = null;
response.end();
});
console.log('received a request at: ' + Date.now());
});
server.listen(8083);
Frontend code (snnipet from devtools)
This code will fire a upload to /upload which some array data, I filled the array before with random bytes, but then I removed and see that it did not have any affect on my timing log, so yes.. the upload content for now is just an array of 0's.
console.log('building data');
var view = new Uint32Array(new Array(5 * 1024 * 1024));
console.log('start sending at: ' + Date.now());
fetch("/upload", {
body: view,
method: "post"
}).then(async response => {
const text = await response.text();
console.log('got response: ' + text);
});
Now running the backend code and then running the frontend code I get some log.
Log capture (screenshots)
The Backend log and frontend log:
The time differences between backend and frontend:
Results
looking at the screenshoots and I get two differences between the logs:
The first, and most important, is the difference between frontend fetch start and backend request recevied, I got 1613ms which is "close" (1430ms) to Resource Scheduling in network timing tab, I think there are more things happening between the frontend fetch call and the node backend event, so I can't direct compare the times:
log.backendReceivedRequest - log.frontEndStart
1613
The second is the difference between receving data on backend, which I got
578ms, close to Request sent (585ms) in network timing tab:
log.backendReceivedAllData - log.backendReceivedFirstData
578
I also changed the frontend code to send different sizes of data and the network timing tab still matches the log
The thing that remains unknown for me is... Why does Google Chrome is queueing my fetch since I'm not running any more requests and not using the bandwidth of the server/host? I readed the conditions for Queueing but not found the reason, maybe is allocating the resources on disk, but not sure: https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
References:
https://nodejs.org/es/docs/guides/anatomy-of-an-http-transaction/
https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
I found a problem. It was in nginx config. Nginx was setup like a reverse proxy. By default proxy request buffering is enabled, so nginx grabs first whole request body and only then forwards it to nodejs, so that's why I saw delay.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering
I am currently buildings proxy using nodejs, which use following syntax for sending and receiving https request and response. However in my project, the response is a liitle bit larger, so typically, req.on('data', callback) will be called 5~7 times before req.on('end', callback) being called.
Here is the simplified code structure:
var http = require("https");
var options = {
hostname: '<WEB SERVICE>',
port: 80,
path: '<WEB PATH>',
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
};
var response = "";
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (body) {
console.log("data");
response += body;
});
res.on('end', function () {
console.log("end");
response = "";
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
// write data to request body
req.write('<SOMETHING>');
req.end();
Ideally, when multiple request comes in, the logging sequence shall be:
data, data, data, data, data, end, data, data, data, data, end
i.e. once one request is done, end will be called once.
However, after doing several tests, when response is big. The sequence becomes:
<response 1 comes>data, data, data ..... data <response 2 comes> data, data, data, ..., data, end
i.e. the end for request 1 is missing.
In short, we need to make sure the callback of 'end' is called exactly once immediate after doing several call back of req.on('data', callback).
I believe there must be some common method for solving this issues (seems a classic bugs in node) and would be appreciated if anyone can indicate how to solve this property.
Thanks for the help!
From the code that you included it is impossible to make two requests. It makes one request with this:
var req = http.request(options, function(res) { ... });
Then binds the error handler here:
req.on('error', function(e) { ... });
And then immediately before even waiting for any response to the request, before even a connection is being made, it calls .write() and .end() methods on the request object:
req.write('<SOMETHING>');
req.end();
Nothing here can possibly cause two requests being made at the same time. But even if the first (and only) request hasn't started yet you already call .write() and .end() methods so maybe there's your problem.
In addition to that you should expect having one request being started before the other one finishes if you are going to do few requests in parallel as you're saying you'd like to.
i have the same issue before, i fixed like this:
res.on('finish', function () {
console.log("res finished");
});
Look here nodejs.org event-finish
I am running this script in node:
var http = require('http');
var server = http.createServer(function (request, response) {
response.writeHead(200, { "Content-Type": "text/plain" });
response.write('Hello World\n');
response.end('Goodbye World', 'utf8', function() {console.log(response.body);
});
server.listen(8000);
console.log('running');
When I load the page (localhost:8000) in Chrome I see:
Hello World
Goodbye World
So far so good, but I'm trying to understand where in the response object the data ('Hello World/nGoodbyeWorld') is. That's why I have 'console.log(response.body)' as the callback in response.end() ( the node http documentation says that the callback will be executed when the response has finished streaming). However the console.log just gives 'undefined'. When I console.log the whole response object it console.logs the response object ok but I can't see any data or body in there even though it has 'hasBody:true'.
So the question is:
a) is there a response.body? I am thinking there has to be one otherwise nothing would show in the browser window.
b) if so how can i access it and why doesn't my way work?
The closest answer i could find was this one: Where is body in a nodejs http.get response? , but I tried adding
response.on('data', function(chunk) {
body += chunk;
});
response.on('end', function() {
console.log(body);
});
, as suggested there and it didn't work. Also people there are just answering HOW you can access the data, not WHY the response.body isn't easily accessible.
Thanks
There is no response body, the data you write to the response stream is just sent to the client as you write it (for the most part). It wouldn't make sense to keep in memory everything ever written to the response.
The same goes for requests. You have to buffer the incoming data yourself if you want that, it is not done behind the scenes, it is merely streamed in.
I really can't figure out what I've done wrong. I've spent about half an hour looking at this code and re-reading code that essentially does the same thing and works. The 'data' event and corresponding callback is never triggered.
var http = require("http");
http.createServer(function(request, response){
response.writeHead(200);
console.log('Executing');
request.on('data', function(chunk){
console.log('data being read');
console.log(chunk.toString());
});
request.on('end', function(){
console.log('done');
response.end();
});
}).listen(8080);
Please help
You probably aren't sending a request body, so the data and end event don't fire. Trying sending a POST or PUT request. If you use a GET request with a query string, you will fire the end event, but not data.
i'm fetching some binary data over http. My code looks like:
var writeStream = fs.createWriteStream(fileName);
request(url, function(err, res) {
res.socket.pipe(writeStream);
});
now the output file is created but the filesize is 0. The url is correct though, i verified that with wget.
Thanks in advance & best regards
The callback for http.request only supplies one argument, which is a reference to the response of the request. Try
http.request(url, function(res) {
res.pipe(writeStream);
});
Also note that the ClientResponse implements ReadableStream, so you should use .pipe rather than .socket.pipe.
I'm assuming that here request is from mikeal's request library rather than being an instance of http.request. In that case you can simply do request(url).pipe(writeStream);
Remember that for debugging purposes, you can always pipe to process.stdout.
var readStream = fs.createReadStream(fileName);
request(url, function(err, res) {
readStream.pipe(res);
readStream.on('end', function() {
//res.end({"status":"Completed"});
});
});