New to Node.js I do understand that createReadStream() function is better for the performance than readFile(), because createReadStream() reads and writes data in chucks while readFile() first reads the whole content. Thus if the file is large, readFile() function might take longer before data can be processed futher. Thus I choose to create server using createReadStream() function as following.
// Create a server with fs.createReadStream(), better performance and less memory usage.
http.createServer( function (request, response) {
// Parse the request containing file name
var pathname = url.parse(request.url).pathname;
// Create a readable stream.
var readerStream = fs.createReadStream(pathname.substr(1));
// Set the encoding to be UTF8.
readerStream.setEncoding('UTF8');
// Handle stream events --> data, end and error
readerStream.on('data', function(chunk) {
// Page found
// HTTP Status: 200 : OK
// Content Type: text/plain
response.writeHead(200, {'Content-type': 'text/html'});
// Write the content of the file to response body.
response.write(chunk);
console.log('Page is being streamed...');
});
readerStream.on('end', function() {
console.log('Page is streamed and emitted successfully.');
});
readerStream.on('error', function(err) {
// HTTP Status: 404 : NOT FOUND
// Content Type: text/plain
response.writeHead(404, {'Content-type': 'text/html'});
console.log('Page streaming error: ' + err);
});
console.log('Code ends!');
}).listen(8081);
// Console will print the message
console.log('Server running at http://127.0.0.1:8081/');
My .html or .txt file contains three short lines of text. After starting my server I visit my web page by going to http://127.0.0.1:8081/index.html. Everything works fine and the content of index.html is echoed on the browser.
But on the tab of the browser, the loader icon keeps turning like it keeps loading for about 1 minute.
Is that normal with Node.js server? Does the icon just keep turning, but costs nothing to the server? Or do I miss something and icon is not supposed to keep turning?
It doesn't look like you are ending your response. The browser probably thinks the request isn't finished and thus continues to "load".
If you look at the Network tab in the developer console you might see the request hasn't finished.
You should be sending response.end()
This method signals to the server that all of the response headers and body have been sent; that server should consider this message complete. The method, response.end(), MUST be called on each response.
I believe you should be calling response.end() in both the readerStream.on('end' and readerStream.on('error' callbacks after you write the head. This will tell the browser the request is finished and it can stop the loading action.
Related
There is a simple web server that accepts data. Sample code below.
The idea is to track in real time how much data has entered the server and immediately inform the client about this. If you send a small amount of data, then everything works well, but if you send more than X data in size, then the on.data event on the server is triggered with a huge delay. I can see that data is transfering for 5 seconds already but on.data event is not trigerred.
on.data event seems to be triggered only when data is uploaded completely to the server, so that's why it works fine with small data (~2..20Mb), but with big data (50..200Mb) it doesnt work well.
Or maybe it is due to some kind of buffering..?
Do you have any suggestions why on.data triggered with delay and how to fix it?
const app = express();
const port = 3000;
// PUBLIC API
// upload file
app.post('/upload', function (request, response) {
request.on('data', chunk => {
// message appears with delay
console.log('upload on data', chunk.length);
// send message to the client about chunk.length
});
response.send({
message: `Got a POST request ${request.headers['content-length']}`
});
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
TLDR:
The delay that you are experiencing probably is the Queueing from Resource scheduling from the browser.
The Test
I did some tests with express, and then I found that it uses http to handle requests/response, so I used a raw http server listener to test this scenario, which has the same situation.
Backend code
This code, based on sample of Node transaction samples, will create a http server and give log of time on 3 situations:
When a request was received
When the first data event fires
When the end event fires
const http = require('http');
var firstByte = null;
var server = http.createServer((request, response) => {
const { headers, method, url } = request;
let body = [];
request.on('error', (err) => {
}).on('data', (chunk) => {
if (!firstByte) {
firstByte = Date.now();
console.log('received first byte at: ' + Date.now());
}
}).on('end', () => {
console.log('end receive data at: ' + Date.now());
// body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
if (url === '/') {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/html');
response.write('<h1>Hello World</h1>');
}
firstByte = null;
response.end();
});
console.log('received a request at: ' + Date.now());
});
server.listen(8083);
Frontend code (snnipet from devtools)
This code will fire a upload to /upload which some array data, I filled the array before with random bytes, but then I removed and see that it did not have any affect on my timing log, so yes.. the upload content for now is just an array of 0's.
console.log('building data');
var view = new Uint32Array(new Array(5 * 1024 * 1024));
console.log('start sending at: ' + Date.now());
fetch("/upload", {
body: view,
method: "post"
}).then(async response => {
const text = await response.text();
console.log('got response: ' + text);
});
Now running the backend code and then running the frontend code I get some log.
Log capture (screenshots)
The Backend log and frontend log:
The time differences between backend and frontend:
Results
looking at the screenshoots and I get two differences between the logs:
The first, and most important, is the difference between frontend fetch start and backend request recevied, I got 1613ms which is "close" (1430ms) to Resource Scheduling in network timing tab, I think there are more things happening between the frontend fetch call and the node backend event, so I can't direct compare the times:
log.backendReceivedRequest - log.frontEndStart
1613
The second is the difference between receving data on backend, which I got
578ms, close to Request sent (585ms) in network timing tab:
log.backendReceivedAllData - log.backendReceivedFirstData
578
I also changed the frontend code to send different sizes of data and the network timing tab still matches the log
The thing that remains unknown for me is... Why does Google Chrome is queueing my fetch since I'm not running any more requests and not using the bandwidth of the server/host? I readed the conditions for Queueing but not found the reason, maybe is allocating the resources on disk, but not sure: https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
References:
https://nodejs.org/es/docs/guides/anatomy-of-an-http-transaction/
https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
I found a problem. It was in nginx config. Nginx was setup like a reverse proxy. By default proxy request buffering is enabled, so nginx grabs first whole request body and only then forwards it to nodejs, so that's why I saw delay.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering
I am running this script in node:
var http = require('http');
var server = http.createServer(function (request, response) {
response.writeHead(200, { "Content-Type": "text/plain" });
response.write('Hello World\n');
response.end('Goodbye World', 'utf8', function() {console.log(response.body);
});
server.listen(8000);
console.log('running');
When I load the page (localhost:8000) in Chrome I see:
Hello World
Goodbye World
So far so good, but I'm trying to understand where in the response object the data ('Hello World/nGoodbyeWorld') is. That's why I have 'console.log(response.body)' as the callback in response.end() ( the node http documentation says that the callback will be executed when the response has finished streaming). However the console.log just gives 'undefined'. When I console.log the whole response object it console.logs the response object ok but I can't see any data or body in there even though it has 'hasBody:true'.
So the question is:
a) is there a response.body? I am thinking there has to be one otherwise nothing would show in the browser window.
b) if so how can i access it and why doesn't my way work?
The closest answer i could find was this one: Where is body in a nodejs http.get response? , but I tried adding
response.on('data', function(chunk) {
body += chunk;
});
response.on('end', function() {
console.log(body);
});
, as suggested there and it didn't work. Also people there are just answering HOW you can access the data, not WHY the response.body isn't easily accessible.
Thanks
There is no response body, the data you write to the response stream is just sent to the client as you write it (for the most part). It wouldn't make sense to keep in memory everything ever written to the response.
The same goes for requests. You have to buffer the incoming data yourself if you want that, it is not done behind the scenes, it is merely streamed in.
In browser javascript is pathetically broken in that the only way to make requests is using script tags and jsonp. To make this useful, I'm trying to make a nodejs server that, given a callback name and address, loads the page at the address and pads it in a call to callback and serves the result. However, I know next to nothing about nodejs. If the server's response is loaded from a script tag it would result in actually loading a web page. Currently, I'm writing the request as localhost:8000/callback/address so a script tag might be <script src="localhost:8000/alert/https://www.google.com" type="text/javascript"></script>. Here is my code for the server:
var http = require("http");
var request = require("request");
var server = http.createServer(function(req, res){
req.on("end", function(){
console.log("alive");
var url = req.url;
var i = url.indexOf("/", 1);
request(url.substring(i + 1), function(err, ret, body){
res.writeHead(200);
res.write(url.substring(1, i) + "(\"" + body + "\");");
res.end();
});
});
});
server.listen(8000);
Why does this stay loading for a very long time but never actually load? By using console.log() it seems as if the req.on("end") callback is never even called.
If you don't care about any request data, you could just add req.resume(); after you add your end event handler.
The reason it's getting "stuck" is that since node v0.10, streams start out in a paused state, so you need to unpause them by reading from them in some way. req.resume(); accomplishes this. Once there is nothing left in the request stream (which there could be nothing), the end event will be emitted.
I'm implementing cache for static serving middleware for Express.js, which works as follows — when request comes, middleware first tries to serve file from filesystem, and if there is none, file is fetched from upstream and stored in file system.
Problem is I don't know how to properly detect “cache hit” event.
staticMiddleware = function(req, res, next) {
// try to read file from fs
filename = urlToFilename(req.url);
stream = fs.createReadStream(filename);
// cache miss - file not found
stream.on('error', function() {
console.log('miss ' + req.url);
// get file from upstream, store it into fs and serve as response
stream = fetchFromUpstream(url);
stream.pipe(fs.createWriteStream(filename));
stream.pipe(res);
});
// cache hit - file is being read
I_DONT_KNOW_WHAT_TO_PUT_HERE(function() {
console.log('hit ' + req.url);
stream.pipe(res);
});
}
So, basically, how can I detect succesful file reading? If I listen to 'data' event, I guess I miss first chunk of data. If I just pipe() it to response, response stream gets closed on error, and I can't serve it with fetched data, and this approach really lacks flexibility. I wonder if there is way to listen for event like fdcreated or opened or similar, or way to push back data I've got in data event, so it will be resent in next data event.
Method createReadStream returns a ReadableStream which also an event open. You can add an event handler for the open event so you will know when the resource is valid before piping:
stream.on('open', function() {
console.log('hit ' + req.url);
stream.pipe(res);
});
I'm creating a reverse HTTP proxy using Node.js for fun. The code is pretty simple at the moment. It listens on 127.0.0.1:8080 for HTTP requests and forwards these to hostname.com, responses from hostname.com are then forwarded back to the client. Nothing fancy is done yet such as rewriting redirect headers, etc. The code is as follows:
var http = require('http');
var server = http.createServer(
function(request, response) {
var proxy = http.createClient(8080, 'hostname.com')
var proxyRequest = proxy.request(request.method, request.url, request.headers);
proxyRequest.on('response', function(proxyResponse) {
proxyResponse.on('data', function(chunk) {
response.write(chunk, 'binary');
});
proxyResponse.on('end', function() {
response.end();
});
response.writeHead(proxyResponse.statusCode, proxyResponse.headers);
});
request.on('data', function(chunk) {
proxyRequest.write(chunk, 'binary');
});
request.on('end', function() {
proxyRequest.end();
});
proxyRequest.on('close', function(err) {
if (err) {
console.log('close error: ' + err + ' for ' + request.url);
}
});
});
server.listen(8080);
server.on('clientError', function(exception) {
console.log('boo a clientError occured :(');
});
All appears to work well until I browse to a page that requires many additional resources (such as images) to be fetched. Naturally the browser will generate a number of GET requests to the reverse proxy to fetch these additional resources.
When I do browse to such a page some of the http.ServerRequests for the additional resources never receive responses. If I restart the page request it almost always results in success as all the resources that were successfully fetched on the first attempt were cached (hence the browser doesn't try GET them again) and so now the browser only needs to grab a few missing ones.
At a guess I would imagine I'm hitting some kind of connection limit although I'm not sure. Any help would be greatly appreciated!
If you set up Wireshark on the proxy, you'll almost certainly see what's happening. (Note that you may need a second machine for this, because some TCP/IP stacks don't provide anything that Wireshark can listen on for loopback traffic - see this)
I'm almost certain that the problem(s) you are running into here are all down to the Connection: header - proxies MUST parse this header and handle it correctly. At a guess, I would say your code is handling the first request in a Connection: keep-alive stream and ignoring the rest. As a proxy, you are supposed to parse and remove/replace this header, and any associated headers (in this case the Keep-Alive: header), before forwarding the request to the server.
If you want to build a HTTP/1.1 proxy, it's very important that you read RFC 2616 and adhere to the many, many rules that it places on their behaviour. The particular problem you are running into here is documented in section 14.10.