We Keep Coding

Why is request.on data firing with a delay on NodeJS?

There is a simple web server that accepts data. Sample code below.
The idea is to track in real time how much data has entered the server and immediately inform the client about this. If you send a small amount of data, then everything works well, but if you send more than X data in size, then the on.data event on the server is triggered with a huge delay. I can see that data is transfering for 5 seconds already but on.data event is not trigerred.
on.data event seems to be triggered only when data is uploaded completely to the server, so that's why it works fine with small data (~2..20Mb), but with big data (50..200Mb) it doesnt work well.
Or maybe it is due to some kind of buffering..?
Do you have any suggestions why on.data triggered with delay and how to fix it?
const app = express();
const port = 3000;
// PUBLIC API
// upload file
app.post('/upload', function (request, response) {
request.on('data', chunk => {
// message appears with delay
console.log('upload on data', chunk.length);
// send message to the client about chunk.length
});
response.send({
message: `Got a POST request ${request.headers['content-length']}`
});
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});

TLDR:
The delay that you are experiencing probably is the Queueing from Resource scheduling from the browser.
The Test
I did some tests with express, and then I found that it uses http to handle requests/response, so I used a raw http server listener to test this scenario, which has the same situation.
Backend code
This code, based on sample of Node transaction samples, will create a http server and give log of time on 3 situations:
When a request was received
When the first data event fires
When the end event fires
const http = require('http');
var firstByte = null;
var server = http.createServer((request, response) => {
const { headers, method, url } = request;
let body = [];
request.on('error', (err) => {
}).on('data', (chunk) => {
if (!firstByte) {
firstByte = Date.now();
console.log('received first byte at: ' + Date.now());
}
}).on('end', () => {
console.log('end receive data at: ' + Date.now());
// body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
if (url === '/') {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/html');
response.write('<h1>Hello World</h1>');
}
firstByte = null;
response.end();
});
console.log('received a request at: ' + Date.now());
});
server.listen(8083);
Frontend code (snnipet from devtools)
This code will fire a upload to /upload which some array data, I filled the array before with random bytes, but then I removed and see that it did not have any affect on my timing log, so yes.. the upload content for now is just an array of 0's.
console.log('building data');
var view = new Uint32Array(new Array(5 * 1024 * 1024));
console.log('start sending at: ' + Date.now());
fetch("/upload", {
body: view,
method: "post"
}).then(async response => {
const text = await response.text();
console.log('got response: ' + text);
});
Now running the backend code and then running the frontend code I get some log.
Log capture (screenshots)
The Backend log and frontend log:
The time differences between backend and frontend:
Results
looking at the screenshoots and I get two differences between the logs:
The first, and most important, is the difference between frontend fetch start and backend request recevied, I got 1613ms which is "close" (1430ms) to Resource Scheduling in network timing tab, I think there are more things happening between the frontend fetch call and the node backend event, so I can't direct compare the times:
log.backendReceivedRequest - log.frontEndStart
1613
The second is the difference between receving data on backend, which I got
578ms, close to Request sent (585ms) in network timing tab:
log.backendReceivedAllData - log.backendReceivedFirstData
578
I also changed the frontend code to send different sizes of data and the network timing tab still matches the log
The thing that remains unknown for me is... Why does Google Chrome is queueing my fetch since I'm not running any more requests and not using the bandwidth of the server/host? I readed the conditions for Queueing but not found the reason, maybe is allocating the resources on disk, but not sure: https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
References:
https://nodejs.org/es/docs/guides/anatomy-of-an-http-transaction/
https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation

I found a problem. It was in nginx config. Nginx was setup like a reverse proxy. By default proxy request buffering is enabled, so nginx grabs first whole request body and only then forwards it to nodejs, so that's why I saw delay.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

Make HTTP requests without opening a new connection

If a browser opens a connection to a remote server, is it possible to access that same connection via Javascript?
I have a small Ethernet module on my network that I program sort of like this (pseudocode):
private var socket
while(true) {
if(socket is disconnected) {
open socket
listen on socket (port 80)
}
if(connection interrupt) {
connect socket
}
if(data receive interrupt) {
serve
}
if(disconnection interrupt) {
disconnect socket
}
}
The point is that it listens on one socket for HTTP requests and serves them.
In my web browser, I can connect to the device, making an HTTP GET request for some HTML/JS that I've written, and it works. A connection is opened on the socket and the files come back as HTTP responses.
Now I want to click a button on the webpage and have the browser send an HTTP POST request over that same connection. In my Javascript, I have (edited and formatted for clarity):
// This function sends an HTTP request
function http(type, url, data, callbacks) {
// make a new HTTP request
var request = new XMLHttpRequest();
// open a connection to the URL
request.open(type, url + (data ? "?" + data : ""));
// add headers
if(type == "POST")
request.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
// register callbacks for the request
Object.keys(callbacks).forEach(function(callback) {
request[callback] = function() {
callbacks[callback](request.responseText);
};
});
// send and return the request
request.send();
return request;
}
// Here is where I call the function
http("POST", // use POST method
"http://192.168.1.99", // IP address of the network device
dataToSend, // the data that needs to be sent
{ // callbacks
onloadend: function(data) {
console.log("success. got: " + data); // print 'success' when the request is done
},
onerror: function(data) {
console.log("There was an error."); // print 'error' when it fails
console.log(data);
}
}
);
The issue here is that this opens a new connection to the device, but I want to use the same socket that the browser is already connected to. Is this possible, and if so, how?

There is no application control inside the browser to decide if a new connection is used for the next request or if an existing connection is used. In fact, it is perfectly normal that the browser will use multiple connections in parallel to the same server and your server has to be able to deal with this.
Since your server architecture seems to be only able to deal with one connection at a time you either would need to change the architecture to handle multiple parallel connections or to make sure that you only need to handle a single connection at a time. The latter could be achieved by not supporting HTTP keep-alive, i.e. by closing the connection immediately after each response. This way a new request will result in a new connection (which is not what you wanted according to your question) but your server will also be able to handle this new connection (which is what you likely ultimately need) since the previous one was closed.

Socket.io, io.use occur before connection is made?

I'm using the Socket.io use function (https://socket.io/docs/server-api/#namespace-use-fn) to authenticate connections, but I'm not sure if I can disconnect a socket from within that function, because I don't know if the connection is made prior to calling 'use'.
Does Socket.io wait until the completion of io.use before making the connection? If the authentication fails, how can I stop the connection from being made?
My testing has shown that io.on('connection') is not fired until next() is called within the 'use' function. But I've also found that when I include socket.disconnect() inside of the 'use' function, the client will in fact disconnect, but the server still thinks the client is connected, as shown by io.clients.
I've read this SO question which cleared up some details about the 'use' function but didn't address my question: Socket.IO middleware, io.use
io.use(function (socket, next) {
let token = socket.handshake.query.token;
if (verify(token) == false) {
socket.disconnect(); // Can I do this here?
}
next();
});
// This isn't fired until the above 'use' function completes, but
// despite disconnecting above, 'on connection' still fires and `io.clients` shows the client is connected
io.on('connection', function (socket) {
setTimeout(function(){
// Get number of connected clients:
io.clients((error, clients) => {
if (error) throw error;
console.log(clients); // => [PZDoMHjiu8PYfRiKAAAF, Anw2LatarvGVVXEIAAAD]
});
}, 2000);
});
EDIT
I have mostly found the answer to this through testing. next() should only be called if the authentication succeeds. If it fails, just don't call next() and the socket will not connect. However, I do not understand why calling socket.disconnect() inside of the 'use' function creates a scenario where the server thinks it's connected to the client, but the client is actually disconnected.

How to detect from the client, a connection closed by server using nodejs request library

I am connecting (stream, long lived connection) to a server that get restarted quite often. I am connecting using http-request (nodejs). When the server gets restarted my clients stops, but not gratefully. Not sure on what's the way to detect that the connection was closed by the server.
I'm using:
request.on('error', function (err) {
console.log("Request on error fired: " + err);
});
but doesn't seem to do the trick. Should I listen to the response object instead? The idea is to catch the error and try to reconnect after a while.

Hope this helps...
req.on("close", function() {
// request closed unexpectedly
});
req.on("end", function() {
// request ended normally
}):

Socket.IO: Reconnect Causes Server Connect Code to Run Twice

Whenever I disconnect using socket.disconnect(); and then reconnect using socket.connect();, the server runs my handshake code twice. The strange thing is, even though the server connection code runs twice, there is only one connection in my array after reconnecting. This happens on an accidental disconnection, intentional, or even if the server restarts. Bit of code:
io.on('connection', OnConnect);
function OnConnect(socket) {
var connection = { socket: socket, realIp: ip, token: GenerateConnToken() };
connections.push(connection);
console.log('Connected');
// Client will respond to this by emitting "client-send-info".
// This is emitted once on initial connect, but twice on reconnect.
socket.emit('acknowledge', connection.token);
socket.on('client-send-info', function() {
console.log('Client Sent Info');
});
socket.on('disconnect', function() {
console.log('Disconnected');
});
}
The above code, when a client connects, disconnects, and then reconnects once, will produce the following log:
Connected
Client Sent Info
Disconnected
Connected
Client Sent Info
Client Sent Info
Why is it that, when reconnecting, the connection code will run twice, but only create one connection object?
EDIT: Upon further inspection, it seems that a different piece of my connection code is being performed twice when the client reconnects. The above code is updated to reflect the relevant information.

Strangely, the solution is completely client side. Instead of the following code:
var socket = io.connect();
socket.on('connect' function() {
socket.on('acknowledge', function() {
});
});
You have to use:
var socket = io.connect();
socket.on('connect' function() {
});
socket.on('acknowledge', function() {
});
Otherwise, the server will appear to be sending multiple emits when it is in reality only sending one, and it's the client that falsely receives multiples. With the second code format, the client successfully connects initially, disconnects, and reconnects without receiving multiple emits.
Simply, don't put any additional socket.on('x') calls inside the on('connection') call. Leave them all outside it.