There is a simple web server that accepts data. Sample code below.
The idea is to track in real time how much data has entered the server and immediately inform the client about this. If you send a small amount of data, then everything works well, but if you send more than X data in size, then the on.data event on the server is triggered with a huge delay. I can see that data is transfering for 5 seconds already but on.data event is not trigerred.
on.data event seems to be triggered only when data is uploaded completely to the server, so that's why it works fine with small data (~2..20Mb), but with big data (50..200Mb) it doesnt work well.
Or maybe it is due to some kind of buffering..?
Do you have any suggestions why on.data triggered with delay and how to fix it?
const app = express();
const port = 3000;
// PUBLIC API
// upload file
app.post('/upload', function (request, response) {
request.on('data', chunk => {
// message appears with delay
console.log('upload on data', chunk.length);
// send message to the client about chunk.length
});
response.send({
message: `Got a POST request ${request.headers['content-length']}`
});
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
TLDR:
The delay that you are experiencing probably is the Queueing from Resource scheduling from the browser.
The Test
I did some tests with express, and then I found that it uses http to handle requests/response, so I used a raw http server listener to test this scenario, which has the same situation.
Backend code
This code, based on sample of Node transaction samples, will create a http server and give log of time on 3 situations:
When a request was received
When the first data event fires
When the end event fires
const http = require('http');
var firstByte = null;
var server = http.createServer((request, response) => {
const { headers, method, url } = request;
let body = [];
request.on('error', (err) => {
}).on('data', (chunk) => {
if (!firstByte) {
firstByte = Date.now();
console.log('received first byte at: ' + Date.now());
}
}).on('end', () => {
console.log('end receive data at: ' + Date.now());
// body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
if (url === '/') {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/html');
response.write('<h1>Hello World</h1>');
}
firstByte = null;
response.end();
});
console.log('received a request at: ' + Date.now());
});
server.listen(8083);
Frontend code (snnipet from devtools)
This code will fire a upload to /upload which some array data, I filled the array before with random bytes, but then I removed and see that it did not have any affect on my timing log, so yes.. the upload content for now is just an array of 0's.
console.log('building data');
var view = new Uint32Array(new Array(5 * 1024 * 1024));
console.log('start sending at: ' + Date.now());
fetch("/upload", {
body: view,
method: "post"
}).then(async response => {
const text = await response.text();
console.log('got response: ' + text);
});
Now running the backend code and then running the frontend code I get some log.
Log capture (screenshots)
The Backend log and frontend log:
The time differences between backend and frontend:
Results
looking at the screenshoots and I get two differences between the logs:
The first, and most important, is the difference between frontend fetch start and backend request recevied, I got 1613ms which is "close" (1430ms) to Resource Scheduling in network timing tab, I think there are more things happening between the frontend fetch call and the node backend event, so I can't direct compare the times:
log.backendReceivedRequest - log.frontEndStart
1613
The second is the difference between receving data on backend, which I got
578ms, close to Request sent (585ms) in network timing tab:
log.backendReceivedAllData - log.backendReceivedFirstData
578
I also changed the frontend code to send different sizes of data and the network timing tab still matches the log
The thing that remains unknown for me is... Why does Google Chrome is queueing my fetch since I'm not running any more requests and not using the bandwidth of the server/host? I readed the conditions for Queueing but not found the reason, maybe is allocating the resources on disk, but not sure: https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
References:
https://nodejs.org/es/docs/guides/anatomy-of-an-http-transaction/
https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
I found a problem. It was in nginx config. Nginx was setup like a reverse proxy. By default proxy request buffering is enabled, so nginx grabs first whole request body and only then forwards it to nodejs, so that's why I saw delay.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering
Related
I have a front end client, which is written in VueJs and a Backend API which is written in Node Js. The Node API communicates with other third party APIs and in turn sent responses back to the client. Now for some of the APIs, it is taking a long time, more than a minute to complete the request and send the response back to the client. As the Node App is proxied over Akamai, it sends a 503 error after a certain time and thus and error will be thrown to the enduser. But the actual process that the third party API do is still in progress and it will send a success response back to the Node App once it is completed. As the client already received the error, it will not receive the success message.
I have this issue with the account creation flow. The client form data is posted to NodeJS backend, which eventually post to another third party API. While waiting for the call to finish, the Akamai proxy will send 503 HTTPS status with Zero Size object response. Client receives this error message and a custom error will be shown. But the account is being created in the backend and eventually it will send success response to the node app, but this never reaches the client and so the user. There is a chance that user will create another account.
The front end call is as follows:
return new Promise((resolve, reject) => {
const config = {}
config.method = 'POST'
config.url = APIaddress
config.data = data
config.params = params
config.withCredentials = true
config.httpsAgent = new https.Agent({ keepAlive: true })
console.log('Config: ', config)
axios(config).then(response => {
console.log('RESPONSE: ', response)
resolve(response)
}).catch(error => {
console.log('ERROR: ', error.response)
reject(error.response.data)
})
})
Here I added the KeepAlive option, but it has no effect and I still get the error.
Now, in the backend also, I use agentkeepalive, and the call is as follows:
const HttpsAgent = agentkeepalive.HttpsAgent
const keepaliveAgent = new HttpsAgent({
timeout:120000,
freeSocketTimeout:60000
});
const options = {
method: 'POST',
url: config.endpoint.url,
headers:
{
'Content-Type': 'application/json;charset=utf-8',
'Accept': 'application/json',
authorization: 'Bearer ' + token
},
data: data,
json: true,
httpsAgent:keepaliveAgent
};
axios(options)
.then(response => response.data)
.then(response => {
resolve(response)
})
.catch(function (error) {
logger.error({
message: `Error while creating account: ${error}`
});
reject(error);
});
Now in order to account for the delays, I am planning to use Server Side Events or WebSockets. I am very new to this and not sure which one to use. I think by using one of these, I can send response back to the client once the account is created. Currently the client is waiting for the account to be created, but I want to make it in such a way that client will send the initial requests and then the server will send notification to the client, once the account is created. This will avoid the unnecessary timeouts and other related issues.
I not sure which solution has to be used here. It will be helpful if someone can shed some light. Thanks for reading.
I switched from SSE and RestAPI to WebSocket on my Node and React app. My setup is as follows:
Create WebSocket server in Node
Create connection from client to server
Then I use "publish-subscribe" pattern.
When client needs something from server, it sends WebSocket message to server with specific sign (In my case it is called "route".) Server filters the message and sends it to proper router (not the Express one, these are routes in my server handling the WebSocket requests.)
As it is processed, server sends WebSocket message back to client, which filters it and processes.
This allows me to have always opened connection to server, what is very swift, and - that's what you are looking for - wait for some message from server without blocking the connection or risking timeout.
Very simple code example:
server:
ws.on('message', m => {
if (m.route === DO_SOMETHING) {
...do something...
ws.send(JSON.stringify({route: DO_SOMETHING_RESPONSE}, message: 'Something was
done'})
}
)
client:
// I want something to be done from server:
ws.send(JSON.stringify({route: DO_SOMETHING, message: 'something should be done'}))
// this is send and you can wait a year for a response, which is catched with:
ws.on('message', m => {
if (m.route === DO_SOMETHING_RESPONSE) {
console.log('Yupeee, something was done!')
}
)
This way you can handle unlimited number of requests. You can make them independent as in this example. Or you can force client to wait for the answger from server.
I have a react.js front end and node.js back end web application. I used mongodb as the database.
I used axios to make HTTP request from front end to back end.
I have tested my app on localhost when making a DELETE request and there is no problem. Below is a screenshot of my browser console of the http traffic.
Below is another screenshot of my browser console when running this web app that is deployed on a 'live environment'(Netlify,Heroku,Mongo Atlas).
I have no clue why the HTTP DELETE request is blocked? Is this because of some sort of setting I have not configured in mongodb atlas?
Below is my javascript code on the front end which I make the axios call
AxiosInstance.get("/statistics")
.then(function (response) {
// handle success
let message = "";
let sortedResultArr = [];
sortedResultArr = response.data.sort(function (a, b) {
return a._id - b._id;
});
for (let i = 0; i < sortedResultArr.length; i++) {
message +=
" Day " +
sortedResultArr[i]._id +
" Profit : " +
sortedResultArr[i].profitPerCup.toFixed(2);
}
AxiosInstance.delete("/statistics").then(function () {
alert(message);
window.location.reload();
});
})
.catch(function (error) {
// handle error
console.log(error);
});
I am wondering if this happened because I nest one axios call inside another ?
EDIT: For my backend, I use the below portion of code to handle CORS
const corsOptions = {
origin: [process.env.FRONTEND_URL, "http://localhost:3001"],
credentials: true
};
app.use(cors(corsOptions));
I also wanted to mention that I also make HTTP DELETE on different routes which worked. For my screenshots above it is DELETE/statistics route that is blocked. However, when the app make HTTP DELETE calls on other routes, the response is correct and the database operation is also correct.
Hello I'm using Ionic framework and express to comunicate between my application, a server API, and a javascript game. The game sends some information to the API using XMLHttpRequest and post and my application retrives this information using Http from angular/http.
The post part works well and I'm able to send information to the server, but I'm struggling with the get part . The get method is called every time that my page is refreshed but at this time the game hasn't sent the data, and when it does the get method isn't called again, so I never get the data from the server.
the game send info like that :
function sendFriendToInvite(friend, callback) {
var req = new XMLHttpRequest();
req.open('POST', 'http://localhost:3001/sendInvite');
req.setRequestHeader('Content-Type', 'application/json');
req.send(JSON.stringify({friend: friend}));
req.addEventListener('load', () => {
console.log(req.responseText);
var results = JSON.parse(req.responseText);
//if(results.error) return console.log(results.error);
if(callback) callback(results);
});
req.addEventListener('error', (e) => {
console.log('an error occured');
console.log(e);
});
}
the server get and post method are :
var friendToInvite;
/**Send from the game**/
api.post('/sendInvite', function(req, res) {
console.log("posts! " + req.body.friend);
friendToInvite = req.body.friend;
res.send({message: 'friend reçu'})
});
/**retrive by the application **/
api.get('/sendInvite', function(req, res) {
console.log("get friend to invite");
//here I need friendToInvite be initialised by the post method before sending the response
res.json({friend: friendToInvite});
});
the application get info like that :
this.http.get('http://localhost:3001/sendInvite').pipe(
map(res => res.json())
).subscribe(response => {
console.log('GET Response:', response);
});
I whant the application retrive data when the game send the data.
I think you should use sockets to communicate between the server and your app.
Another simpler and less efficient option may be to do the GET in an interval function so you can call to the server every X seconds (say 1 minute, for example) to update the data. This implies that your app is constantly calling to the server and I strongly discourage it.
EDIT:
I suggested using sockets because I have used this method occasionally to communicate between server and app but it would also be a good option, if you only need communication from the server to the app and not vice versa, to use push notifications.
I'm new to Node and am having some difficulties with getting the Request library to return an accurate response time.
I have read the thread at nodejs request library, get the response time and can see that the request library should be able to return an "elapsed time" for the request.
I am using it in the following way :
request.get({
url : 'http://example.com',
time : true
},function(err, response){
console.log('Request time in ms', response.elapsedTime);
});
The response.elapsedTime result is in the region of 500-800ms, however I can see the request is actually taking closer to 5000ms.
I am testing this against an uncached nginx page which takes roughly 5 seconds to render the page when profiling via a browser (Chrome).
Here is an example of the timing within Chrome (although the server is under load hence the 10s)
Chrome Profiling example
It looks to me like this isn't actually timing the full start to finish of the request but it "timing" something else. It might be the time taken to download the page once the server starts streaming it.
If this is the case, how can I get the actual start to finish time that this request has taken ? The time I need is from making the request to receiving the entire body and headers.
I am running the request like this with listofURLs being an array of urls to request:
for (var i = 0; i < listofURLs.length; i++) {
collectSingleURL(listofURLs[i].url.toString(),
function (rData) {
console.log(rData['url']+" - "+rData['responseTime']);
});
}
function collectSingleURL(urlToCall, cb) {
var https = require('https');
var http = require('http');
https.globalAgent.maxSockets = 5;
http.globalAgent.maxSockets = 5;
var request = require('request');
var start = Date.now();
// Make the request
request.get({
"url": urlToCall,
"time": true,
headers: {"Connection": "keep-alive"}
}, function (error, response, body) {
//Check for error
if (error) {
var result = {
"errorDetected": "Yes",
"errorMsg": error,
"url": urlToCall,
"timeDate": response.headers['date']
};
//callback(error);
console.log('Error in collectSingleURL:', error);
}
// All Good - pass the relevant data back to the callback
var result = {
"url": urlToCall,
"timeDate": response.headers['date'],
"responseCode": response.statusCode,
"responseMessage": response.statusMessage,
"cacheStatus": response.headers['x-magento-cache-debug'],
"fullHeaders": response.headers,
"bodyHTML": body,
"responseTime" : Date.now() - start
};
cb(result);
//console.log (cb);
});
}
You are missing a key point - it take 5 seconds to render, not to just download the page.
The request module of node is not a full browser, it's a simple HTTP request, so when you for example request www.stackoverflow.com, it will only load the basic HTML returned by the page, it will not load the JS files, CSS file, images etc.
The browser on the otherhand, will load all of that after the basic HTML of the page is loaded (some parts will load before the page has finished loading, together with the page).
Take a look on the network profiling below of stackoverflow - the render finishes at ~1.6 seconds, but the basic HTML page (the upper bar) has finished loading around 0.5 second. So if you use request to fetch a web page, it actually only loading the HTML, meaning - "the upper bar".
Just time it yourself:
var start = Date.now()
request.get({
url : 'http://example.com'
}, function (err, response) {
console.log('Request time in ms', Date.now() - start);
});
Noob question on using callbacks as a control flow pattern with Node and the http class. Based on my understanding of the event loop, all code is blocking, i/o is non-blocking and using callbacks, here's the a simple http server and a pseudo rest function:
// Require
var http = require("http");
// Class
function REST() {};
// Methods
REST.prototype.resolve = function(request,response,callback) {
// Pseudo rest function
function callREST(request, callback) {
if (request.url == '/test/slow') {
setTimeout(function(){callback('time is 30 seconds')},30000);
} else if (request.url == '/test/foo') {
callback('bar');
}
}
// Call pseudo rest
callREST(request, callback);
}
// Class
function HTTPServer() {};
// Methods
HTTPServer.prototype.start = function() {
http.createServer(function (request, response) {
// Listeners
request.resume();
request.on("end", function () {
// Execute only in not a favicon request
var faviconCheck = request.url.indexOf("favicon");
if (faviconCheck < 0) {
//Print
console.log('incoming validated HTTP request: ' + request.url);
//Instantiate and execute on new REST object
var rest = new REST();
rest.resolve(request,response,function(responseMsg) {
var contentType = {'Content-Type': 'text/plain'};
response.writeHead(200, contentType); // Write response header
response.end(responseMsg); // Send response and end
console.log(request.url + ' response sent and ended');
});
} else {
response.end();
}
});
}).listen(8080);
// Print to console
console.log('HTTPServer running on 8080. PID is ' + process.pid);
}
// Process
// Create http server instance
var httpServer = new HTTPServer();
// Start
httpServer.start();
If I open up a browser and hit the server with "/test/slow" in one tab then "/test/foo" in another, I get the following behavior - "foo" responds with "Bar" immediately and then 30 secs late, "slow" responds with "time is 30 seconds". This is what I was expecting.
But if I open up 3 tabs in a browser and hit the server with "/test/slow" successively in each tab, "slow" is being processed and responds serially/synchronously so that the 3 responses appear at 30 second intervals. I was expecting the responses right after each other if they were being processed asynchronously.
What am I doing wrong?
Thank you for your thoughts.
This is actually not the server's fault. Your browser is opening a single connection and re-using it between the requests, but one request can't begin until the previous finishes. You can see this a couple of ways:
Look in the network tab of the Chrome dev tools - the entry for the longest one will show the request in the blocking state until the first two finish.
Try opening the slow page in different browsers (or one each in normal and incognito windows) - this prevents sharing connections.
Thus, this will only happen if the same browser window is making multiple requests to the same server. Also, note that XHR (AJAX) requests will open separate connections so they can be performed in parallel. In the real world, this won't be a problem.