var data = import('./blahblah/data')
app.get('/data', data.getData)
app.listen(3000)
// blahblah/data:
exports.getData = function(req, res) {
setTimeout(function(){
res.send('test')
}, 10000)
};
//code I use to test
for(var i = 0;i<10;i++){
$.ajax({
url: 'http://127.0.0.1:3000/data',
success: function(l){console.log(l);}
});
}
If I send 10 simultaneous requests to this endpoint, then they only receive a reply one at a time 10 seconds apart, with the last request coming 110 seconds after the first request was sent. Aren't nodejs and express supposed to let other requests process when the other requests are running asynchronous code?
Your code is fine, probably something is wrong with the way you're testing it.
Please, make sure that you're sending your requests in parallel.
I tested your code with loadtest utility
loadtest -n 10 -c 10 http://localhost:3000/data
and got expected results
Target URL: http://localhost:3000/data
Max requests: 10
Concurrency level: 10
Agent: none
Completed requests: 10
Total errors: 0
Total time: 10.050681130000001 s
Here is a code snipped I tested:
var app = require('express')();
app.get('/data', function(req, res) {
setTimeout(function(){
res.send('test')
},10000)
});
app.listen(3000)
Update:
Web browsers usually limit the number of parallel requests per host. I just tested it in my Google Chrome and it limited the number of parallel requests by 6:
Here is a detailed timing for one of delayed requests:
Looks like your browser have an even harsher limits.
Related
andI'm trying to work on a caching solution for inflight requests to Koa,
Let's say that i have 100 seperate users hitting the same endpoint concurrently, but the endpoint takes ~4-5 seconds to return a response.
For example:
GET http://mykoa.application.com/getresults
In my router middleware is it possible to cache all of the concurrent inbound requests and then once the response has been generated return the same result to all of them? Potentially something similar to the example below?
const inflight = {};
router.use(async function(ctx, next) {
// Create a cache 'key'
const hash = `${ctx.request.url}-${ctx.state.user?.data?.id}-${JSON.stringify(ctx.request.body)}`;
// Check if there is already a request inflight
if (inflight[hash]) {
// If there is then wait for the promise resolution
return await inflight[hash];
}
// Cache the request resolution for any other identical requests
inflight[hash] = next();
await inflight[hash];
// Clean it up so that the next request will be fresh
inflight[hash].then(function(res) {
delete inflight[hash];
}, function(err) {
delete inflight[hash];
})
})
In my head this should work, and the expectation would be that all 100 concurrent requests would resolve at the same time (after the first one has resolved) however in my tests each request is still being run separately and taking 4-5 seconds each in sequence.
In my express app, I have defined 2 endpoints in my application. One for is-sever-up check and one for simulating a blocking operation.
app.use('/status', (req, res) => {
res.sendStatus(200);
});
app.use('/p', (req, res) => {
const { logger } = req;
logger.info({ message: 'Start' });
let i = 0;
const max = 10 ** 10;
while (i < max) {
i += 1;
}
res.send(`${i}`);
logger.info({ message: 'End' });
});
I am using winston for logging and PM2 for clustering using the following command
$ pm2 start bin/httpServer.js -i 0
It has launched 4 instances.
Now, when I visit the routes /p, /p, /status in order in different tabs with around 1 second delay between (request 1 and request 2) and (request 2 and request 3), I expected to get response for request 1 and request 2 after some time but with around 1 second delay and response for request 3 should come instantly.
Actual: The response for request 3 did come instantly but something weird happened with request 1 and request 2. The request 2 didn't even start until request 1 was completed. Here are logs that I got. You can see the time stamp for the end of request 1 and start of request 2.
{"message":"Start","requestId":"5c1f85bd-94d9-4333-8a87-30f3b3885d9c","level":"info","timestamp":"2020-12-28 07:34:48"}
{"message":"End","requestId":"5c1f85bd-94d9-4333-8a87-30f3b3885d9c","level":"info","timestamp":"2020-12-28 07:35:03"}
{"message":"Start","requestId":"f1f86f68-1ddf-47b1-ae62-f75c7aa7a58d","level":"info","timestamp":"2020-12-28 07:35:03"}
{"message":"End","requestId":"f1f86f68-1ddf-47b1-ae62-f75c7aa7a58d","level":"info","timestamp":"2020-12-28 07:35:17"}
Why did the request 1 and request 2 not start at the same time (with 1 second delay, of course)? And if they are running synchronously, why did request 3 respond instantly and not wait for request 1 and 2 to complete?
That's because connection of the header is keep-alive in the response which your node server respond in default. So, connection will be reused when you use browser (curl also could simulate the reused connection situtation). That means multiple request is served by the same instance within a specified time. Even you have multiple node instances.
Note: You could see specified time in response header like this Keep-Alive: timeout=5
If you use browser, open network tab to see response headers.
If you use curl, add -v options to see response headers
You could try to use multiple separated curl command at the same time in terminal. Separated curl command means connection will not be reused. So, you'll get your expected results. You could add a console.log("status test") in /status router. Then, use pm2 logs to see which instance serve the request like following format (these logs are produced by accessing endpoint with browser).
0|server | status test
0|server | status test
0 means the first instance, you will see this is all the same instance to serve request when you use browser to access endpoint. But, if you use curl, you'll find out the number is always changed which mean every request is served by different node instance.
You could see I sent two request at the same time with curl in terminal. Then, different node instance to serve the request. So, the start and end time of console.log are same. In this example, I have 8 event-loop so I could deal with 8 long-processing (synchronous code) request at the same time.
And, you could use curl to simulate the keep-alive situation. Then, you'll see the request is served by same node instance.
curl http://localhost:8080/status http://localhost:8080/status -v -H "Connection: keep-alive"
You also could use connection close to see the request is served by different node instance.
curl http://localhost:8080/status http://localhost:8080/status -v -H "Connection: close"
You could see the different here.
If you want to close the connection in server side, you could use following code.
res.setHeader("Connection", "close")
This is my test code.
const express = require("express")
const app = express();
const port = 8080;
app.use('/status', (req, res) => {
console.log("status tests");
res.sendStatus(200);
});
app.use('/p', (req, res) => {
console.log(new Date() + " start");
let i = 0;
const max = 10 ** 10;
while (i < max) {
i += 1;
}
res.send(`${i}`);
console.log(new Date() + " end");
});
app.listen(port, () => {
return console.log(`server is listening on ${port}`);
});
I would like to set up an API using express that can either multithread or multiprocess requests.
For instance, below is an api that sleeps 5 seconds before sending a response. If I call it quickly 3 times, the first response will take 5 seconds, the second will take 10, and the third will take 15, indicating the requests were handled sequentially.
How do I architect an application that can handle the requests concurrently.
const express = require('express')
const app = express()
const port = 4000
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
app.get('/', (req, res) => {
sleep(5000).then(()=>{
res.send('Hello World!')
})
})
app.listen(port, () => console.log(`Example app listening on port ${port}!`))
Edit: request -> response
If I call it quickly 3 times, the first response will take 5 seconds, the second will take 10, and the third will take 15, indicating the requests were handled sequentially.
That's only because your browser is serializing the requests, because they're all requesting the same resource. On the Node.js/Express side, those requests are independent of one another. If they were sent from three separate clients one right after another, they'd each get a response roughly five seconds later (not after 5, 10, and 15 seconds).
For instance, I updated your code to output the date/time of the response:
res.send('Hello World! ' + new Date().toISOString())
...and then opened http://localhost:4000 in three separate browsers as quickly as I could (I don't appear to be all that quick :-)). The times on the responses were:
16:15:58.819Z
16:16:00.361Z
16:16:01.164Z
As you can see, they aren't five seconds apart.
But if I do that in three windows in the same browser, they get serialized:
16:17:13.933Z
16:17:18.938Z
16:17:23.942Z
If I further update your code so that it's handling three different endpoints:
function handler(req, res) {
sleep(5000).then(()=>{
res.send('Hello World! ' + new Date().toISOString())
})
}
app.get('/a', handler);
app.get('/b', handler);
app.get('/c', handler);
Then even on the same browser, requests for /a, /b, and /c are not serialized.
I have found strange delay of http.request function. Here is my code
var express = require('express');
var http = require('http');
app.set('port', process.env.PORT || 3000);
var app = express();
app.get('/aaa',function(req,res) {
setTimeout(function(){
res.json({"a":1});
},500);
});
app.get('/bbb',function(req,res){
var options = {
host: '127.0.0.1',
port: 3000,
path: '/aaa',
method: 'GET'
};
var request = http.request(options, function(result) {
result.on("data",function(){
});
res.json({"b":2});
});
request.on('error', function() {
res.json({"b":2});
});
request.end();
});
http.createServer(app).listen(app.get('port'), function(){
});
Client call /bbb, then it's handler call /aaa and within 500ms result returns back to client.
I tried to measure response time in different situations using Apache Bench:
1) 1000 requests with 1 concurrent requests.
Average response time: 500ms
2) 1000 requests with 50 concurrent requests.
Average response time: 5000ms
3) 1000 requests with 100 concurrent requests.
Average response time: 10000ms
Why response time is growing?
It's okay when I call /aaa directly
It's not unusual behaviour. The HTTP Client used in the callback to /bbb (http.request) is limited to 5 concurrent sockets per host. In other words, it can only make 5 HTTP requests in parallel. You can find reference to this here in the documentation
Just to confirm you're hitting the limit, you should run your tests using 5 and 6 concurrent requests. You'll see (as I did) average response time drops significantly at 6 concurrent requests. This is because the 6th concurrent request will be queued until one of the 5 preceeding requests to /aaa is completed.
To answer your question about why response time grows: The more concurrency you add in your benchmark, the more average response time will go up because each request has to wait for more requests in the queue to finish before it can get a socket.
You can increase the number of concurrent sockets your HTTP client can handle by modifying the default agent like this:
var http = require("http");
http.globalAgent.maxSockets = 10;
You can also circumvent pooling altogether by passing agent:false to http.get like so:
http.get({hostname:'localhost', port:80, path:'/', agent:false}, function (res) {
// Do stuff
})
Update (8th Feb 2015)
An important change regarding this answer has come up in Node v 0.12.0.
maxSockets are no longer limited to 5. The default is now set to
Infinity with the developer and the operating system given control
over how many simultaneous connections an application can keep open to
a given host.
I had same issue and it is resolved by keeping it very simple get request as below
var req = http.get(requestUrl)
req.end();
I've created a node.js script, that scans network for available HTTP pages, so there is a lot of connections i want to run in parallel, but it seems that some of the requests wait for previous to complete.
Following is the code fragment:
var reply = { };
reply.started = new Date().getTime();
var req = http.request(options, function(res) {
reply.status = res.statusCode;
reply.rawHeaders = res.headers;
reply.headers = JSON.stringify(res.headers);
reply.body = '';
res.setEncoding('utf8');
res.on('data', function (chunk) {
reply.body += chunk;
});
res.on('end', function () {
reply.finished = new Date().getTime();
reply.time = reply.finished - reply.started;
callback(reply);
});
});
req.on('error', function(e) {
if(e.message == 'socket hang up') {
return;
}
errCallback(e.message);
});
req.end();
This code performs only 10-20 requests per second, but i need 500-1k requests performance. Every queued request is made to a different HTTP server.
I've tried to do something like that, but it didn't help:
http.globalAgent.maxSockets = 500;
Something else must be going on with your code. Node can comfortably handle 1k+ requests per second.
I tested with the following simple code:
var http = require('http');
var results = [];
var j=0;
// Make 1000 parallel requests:
for (i=0;i<1000;i++) {
http.request({
host:'127.0.0.1',
path:'/'
},function(res){
results.push(res.statusCode);
j++;
if (j==i) { // last request
console.log(JSON.stringify(results));
}
}).end();
}
To purely test what node is capable of and not my home broadband connection the code requests from a local Nginx server. I also avoid console.log until all the requests have returned because it is implemented as a synchronous function (to avoid losing debugging messages when a program crash).
Running the code using time I get the following results:
real 0m1.093s
user 0m0.595s
sys 0m0.154s
That's 1.093 seconds for 1000 requests which makes it very close to 1k requests per second.
The simple code above will generate OS errors if you try to make a lot of requests (like 10000 or more) because node will happily try to open all those sockets in the for loop (remember: the requests don't start until the for loop ends, they are only created). You mentioned that your solution also runs into the same errors. To avoid this you should limit the number of parallel requests you make.
The simplest way of limiting number of parallel requests is to use one of the Limit functions form the async.js library:
var http = require('http');
var async = require('async');
var requests = [];
// Build a large list of requests:
for (i=0;i<10000;i++) {
requests.push(function(callback){
http.request({
host:'127.0.0.1',
path:'/'
},function(res){
callback(null,res.statusCode);
}).end()
});
}
// Make the requests, 100 at a time
async.parallelLimit(requests, 100,function(err, results){
console.log(JSON.stringify(results));
});
Running this with time on my machine I get:
real 0m8.882s
user 0m4.036s
sys 0m1.569s
So that's 10k request in around 9 seconds or roughly 1.1k/s.
Look at the functions available from async.js.
I've found solution for me, it is not very good, but works:
childProcess = require('child_process')
I'm using curl:
childProcess.exec('curl --max-time 20 --connect-timeout 10 -iSs "' + options.url + '"', function (error, stdout, stderr) { }
This allows me to run 800-1000 curl processes simultaneously. Of course, this solution has it's weekneses, like requirement for lots of open file decriptors, but works.
I've tried node-curl bindings, but that was very slow too.