I have been reading up on nodejs lately, trying to understand how it handles multiple concurrent requests. I know NodeJs is a single threaded event loop based architecture, and at a given point in time only one statement is going to be executing, i.e. on the main thread and that blocking code/IO calls are handled by the worker threads (default is 4).
Now my question is, what happens when a web server built using NodeJs receives multiple requests? I know that there are lots of similar questions here, but haven't found a concrete answer to my question.
So as an example, let's say we have following code inside a route like /index:
app.use('/index', function(req, res, next) {
console.log("hello index routes was invoked");
readImage("path", function(err, content) {
status = "Success";
if(err) {
console.log("err :", err);
status = "Error"
}
else {
console.log("Image read");
}
return res.send({ status: status });
});
var a = 4, b = 5;
console.log("sum =", a + b);
});
Let's assume that the readImage() function takes around 1 min to read that Image.
If two requests, T1, and T2 come in concurrently, how is NodeJs going to process these request ?
Does it going to take first request T1, process it while queueing the request T2? I assume that if any async/blocking stuff is encountered like readImage, it then sends that to a worker thread (then some point later when async stuff is done that thread notifies the main thread and main thread starts executing the callback?), and continues by executing the next line of code?
When it is done with T1, it then processes the T2 request? Is that correct? Or it can process T2 in between (meaning whilethe code for readImage is running, it can start processing T2)?
Is that right?
Your confusion might be coming from not focusing on the event loop enough. Clearly you have an idea of how this works, but maybe you do not have the full picture yet.
Part 1, Event Loop Basics
When you call the use method, what happens behind the scenes is another thread is created to listen for connections.
However, when a request comes in, because we're in a different thread than the V8 engine (and cannot directly invoke the route function), a serialized call to the function is appended onto the shared event loop, for it to be called later. ('event loop' is a poor name in this context, as it operates more like a queue or stack)
At the end of the JavaScript file, the V8 engine will check if there are any running theads or messages in the event loop. If there are none, it will exit with a code of 0 (this is why server code keeps the process running). So the first Timing nuance to understand is that no request will be processed until the synchronous end of the JavaScript file is reached.
If the event loop was appended to while the process was starting up, each function call on the event loop will be handled one by one, in its entirety, synchronously.
For simplicity, let me break down your example into something more expressive.
function callback() {
setTimeout(function inner() {
console.log('hello inner!');
}, 0); // †
console.log('hello callback!');
}
setTimeout(callback, 0);
setTimeout(callback, 0);
† setTimeout with a time of 0, is a quick and easy way to put something on the event loop without any timer complications, since no matter what, it has always been at least 0ms.
In this example, the output will always be:
hello callback!
hello callback!
hello inner!
hello inner!
Both serialized calls to callback are appended to the event loop before either of them is called. This is guaranteed. That happens because nothing can be invoked from the event loop until after the full synchronous execution of the file.
It can be helpful to think of the execution of your file, as the first thing on the event loop. Because each invocation from the event loop can only happen in series, it becomes a logical consequence, that no other event loop invocation can occur during its execution; Only when the previous invocation is finished, can the next event loop function be invoked.
Part 2, The inner Callback
The same logic applies to the inner callback as well, and can be used to explain why the program will never output:
hello callback!
hello inner!
hello callback!
hello inner!
Like you might expect.
By the end of the execution of the file, two serialized function calls will be on the event loop, both for callback. As the Event loop is a FIFO (first in, first out), the setTimeout that came first, will be be invoked first.
The first thing callback does is perform another setTimeout. As before, this will append a serialized call, this time to the inner function, to the event loop. setTimeout immediately returns, and execution will move on to the first console.log.
At this time, the event loop looks like this:
1 [callback] (executing)
2 [callback] (next in line)
3 [inner] (just added by callback)
The return of callback is the signal for the event loop to remove that invocation from itself. This leaves 2 things in the event loop now: 1 more call to callback, and 1 call to inner.
Now callback is the next function in line, so it will be invoked next. The process repeats itself. A call to inner is appended to the event loop. A console.log prints Hello Callback! and we finish by removing this invocation of callback from the event loop.
This leaves the event loop with 2 more functions:
1 [inner] (next in line)
2 [inner] (added by most recent callback)
Neither of these functions mess with the event loop any further. They execute one after the other, the second one waiting for the first one's return. Then when the second one returns, the event loop is left empty. This fact, combined with the fact that there are no other threads currently running, triggers the end of the process, which exits with a return code of 0.
Part 3, Relating to the Original Example
The first thing that happens in your example, is that a thread is created within the process which will create a server bound to a particular port. Note, this is happening in precompiled C++ code, not JavaScript, and is not a separate process, it's a thread within the same process. see: C++ Thread Tutorial.
So now, whenever a request comes in, the execution of your original code won't be disturbed. Instead, incoming connection requests will be opened, held onto, and appended to the event loop.
The use function, is the gateway into catching the events for incoming requests. Its an abstraction layer, but for the sake of simplicity, it's helpful to think of the use function like you would a setTimeout. Except, instead of waiting a set amount of time, it appends the callback to the event loop upon incoming http requests.
So, let's assume that there are two requests coming in to the server: T1 and T2. In your question you say they come in concurrently, since this is technically impossible, I'm going to assume they are one after the other, with a negligible time in between them.
Whichever request comes in first, will be handled first by the secondary thread from earlier. Once that connection has been opened, it's appended to the event loop, and we move on to the next request, and repeat.
At any point after the first request is added to the event loop, V8 can begin execution of the use callback.
A quick aside about readImage
Since its unclear whether readImage is from a particular library, something you wrote or otherwise, it's impossible to tell exactly what it will do in this case. There are only 2 possibilities though, so here they are:
It's entirely synchronous, never using an alternate thread or the event loop
function readImage (path, callback) {
let image = fs.readFileSync(path);
callback(null, image);
// a definition like this will force the callback to
// fully return before readImage returns. This means
// means readImage will block any subsequent calls.
}
It's entirely asynchronous, and takes advantage of fs' async callback.
function readImage (path, callback) {
fs.readFile(path, (err, data) => {
callback(err, data);
});
// a definition like this will force the readImage
// to immediately return, and allow exectution
// to continue.
}
For the purposes of explanation, I'll be operating under the assumption that readImage will immediately return, as proper asynchronous functions should.
Once the use callback execution is started, the following will happen:
The first console log will print.
readImage will kick off a worker thread and immediately return.
The second console log will print.
During all of this, its important to note, these operations are happening synchronously; No other event loop invocation can start until these are finished. readImage may be asynchronous, but calling it is not, the callback and usage of a worker thread is what makes it asynchronous.
After this use callback returns, the next request has probably already finished parsing and was added to the event loop, while V8 was busy doing our console logs and readImage call.
So the next use callback is invoked, and repeats the same process: log, kick off a readImage thread, log again, return.
After this point, the readImage functions (depending on how long they take) have probably already retrieved what they needed and appended their callback to the event loop. So they will get executed next, in order of whichever one retrieved its data first. Remember, these operations were happening in separate threads, so they happened not only in parallel to the main javascript thread, but also parallel to each other, so here, it doesn't matter which one got called first, it matters which one finished first, and got 'dibs' on the event loop.
Whichever readImage completed first will be the first one to execute. So, assuming no errors occured, we'll print out to the console, then write to the response for the corresponding request, held in lexical scope.
When that send returns, the next readImage callback will begin execution: console log, and writing to the response.
At this point, both readImage threads have died, and the event loop is empty, but the thread that holds the server port binding is keeping the process alive, waiting for something else to add to the event loop, and the cycle to continue.
I hope this helps you understand the mechanics behind the asynchronous nature of the example you provided.
For each incoming request, node will handle it one by one. That means there must be order, just like the queue, first in first serve. When node starts processing request, all synchronous code will execute, and asynchronous will pass to work thread, so node can start to process the next request. When the asynchrous part is done, it will go back to main thread and keep going.
So when your synchronous code takes too long, you block the main thread, node won't be able to handle other request, it's easy to test.
app.use('/index', function(req, res, next) {
// synchronous part
console.log("hello index routes was invoked");
var sum = 0;
// useless heavy task to keep running and block the main thread
for (var i = 0; i < 100000000000000000; i++) {
sum += i;
}
// asynchronous part, pass to work thread
readImage("path", function(err, content) {
// when work thread finishes, add this to the end of the event loop and wait to be processed by main thread
status = "Success";
if(err) {
console.log("err :", err);
status = "Error"
}
else {
console.log("Image read");
}
return res.send({ status: status });
});
// continue synchronous part at the same time.
var a = 4, b = 5;
console.log("sum =", a + b);
});
Node won't start processing the next request until finish all synchronous part. So people said don't block the main thread.
There are a number of articles that explain this such as this one
The long and the short of it is that nodejs is not really a single threaded application, its an illusion. The diagram at the top of the above link explains it reasonably well, however as a summary
NodeJS event-loop runs in a single thread
When it gets a request, it hands that request off to a new thread
So, in your code, your running application will have a PID of 1 for example. When you get request T1 it creates PID 2 that processes that request (taking 1 minute). While thats running you get request T2 which spawns PID 3 also taking 1 minute. Both PID 2 and 3 will end after their task is completed, however PID 1 will continue listening and handing off events as and when they come in.
In summary, NodeJS being 'single threaded' is true, however its just an event-loop listener. When events are heard (requests), it passes them off to a pool of threads that execute asynchronously, meaning its not blocking other requests.
You can simply create child process by shifting readImage() function in a different file using fork().
The parent file, parent.js:
const { fork } = require('child_process');
const forked = fork('child.js');
forked.on('message', (msg) => {
console.log('Message from child', msg);
});
forked.send({ hello: 'world' });
The child file, child.js:
process.on('message', (msg) => {
console.log('Message from parent:', msg);
});
let counter = 0;
setInterval(() => {
process.send({ counter: counter++ });
}, 1000);
Above article might be useful to you .
In the parent file above, we fork child.js (which will execute the file with the node command) and then we listen for the message event. The message event will be emitted whenever the child uses process.send, which we’re doing every second.
To pass down messages from the parent to the child, we can execute the send function on the forked object itself, and then, in the child script, we can listen to the message event on the global process object.
When executing the parent.js file above, it’ll first send down the { hello: 'world' } object to be printed by the forked child process and then the forked child process will send an incremented counter value every second to be printed by the parent process.
The V8 JS interpeter (ie: Node) is basically single threaded. But, the processes it kicks off can be async, example: 'fs.readFile'.
As the express server runs, it will open new processes as it needs to complete the requests. So the 'readImage' function will be kicked off (usually asynchronously) meaning that they will return in any order. However the server will manage which response goes to which request automatically.
So you will NOT have to manage which readImage response goes to which request.
So basically, T1 and T2, will not return concurrently, this is virtually impossible. They are both heavily reliant on the Filesystem to complete the 'read' and they may finish in ANY ORDER (this cannot be predicted). Note that processes are handled by the OS layer and are by nature multithreaded (in a modern computer).
If you are looking for a queue system, it should not be too hard to implement/ensure that images are read/returned in the exact order that they are requested.
Since there's not really more to add to the previous answer from Marcus - here's a graphic that explains the single threaded event-loop mechanism:
assume the following:
http.get('mycoolwebsite.com/somecontent', function(){
console.log('printed');
}
doSomeReallyIntensive100Hourwork();
I understand that we will see the word 'printed' printed on the console after 100 hours.
The questions:
Does javascript delay the request after 100 hours and then execute the callback or does it send out a request immediately and schedule the callback to print later? Basically, do we immediately notice the network request or not?
do we immediately notice the network request or not?
That's up to the browser. JS will immediately tell the browser that it should execute the following request, but it's up to the browser to decide wether it has the capacity to immediately start this request or wether to queue it untill there are ressources available.
Then, eventually the request is sent, the response is there, and the browser will notify JS that the request is done, and pass the recieved data. Then JS, as soon as JS is idle*, it will call the callback-function.
JS is single-threaded. It can't run two pieces of code at once, and it won't interrupt some other code to run your callback-function. It will call it as soon as there is no other pending Javascript to run.
The request is initiated immediately, and the response will possibly also have arrived, but your get callback function will be triggered by an event in the event queue which only gets processed in the next task.
From mdn on XMLHttpRequest.send(), the method that is executed under the hood of the http.get method:
The XMLHttpRequest.send() method sends the request. If the request is asynchronous (which is the default), this method returns as soon as the request is sent.
Note the last part. See also the WHATWG standards, section 4.5.6:
The send(body) method must run these steps:
10.4.1 Wait until either req’s done flag is set or
the timeout attribute value number of milliseconds has passed since these subsubsteps started
while timeout attribute value is not zero.
... so the method does wait to make sure the request has been sent (or a timeout occurred).
Here is a demo with an HTTP call to a service that answers after a 2 second server-sided wait. You can launch it in two ways: with or without the intensive work.
When using the option to not do the work, the response comes back about 2 seconds after the click. If using the other option (with the work), then the work is done after three seconds and the response is processed immediately afterwards (have some patience after clicking):
function work() {
// Be busy for 3 seconds
var until = performance.now() + 3000;
while (performance.now() < until);
console.log('hard work done');
}
function test(work) {
$.get("http://httpstat.us/200?sleep=2000", function() {
console.log('response processed');
});
if (work) work();
}
$('#without').click(test.bind(null, null));
$('#with').click(test.bind(null, work));
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<button id="with">Request with 3 secs of work</button>
<button id="without">Reguest without that work</button>
I initiated a web worker on chrome and it had a simple function that was called repeatedly using setTimeout. Surprisingly the web worker terminated after the function was called around 1000 times. Can anyone explain why? I guess chrome is doing some optimization.
webworker.js
function hi() {
postMessage('1');
setTimeout(hi, 1);
}
hi();
main.js
var blob = new Blob([code]);
var blobURL = window.URL.createObjectURL(blob);
var worker = new Worker(blobURL);
worker.onmessage = function(data) {
console.log(data.data); // gets called around 1000 times and done
};
EDIT:
Reproduced in a fiddle:
http://jsfiddle.net/meovfpv3/1/
It seems to takes arbitrarily long for the onmessage callback to stop firing, as quickly as a few seconds and as long as +5 mins
Here is my best guess at what is happening. By posting a message from the Web Worker every 1ms, you are demanding that the main thread processes each posted message within 1ms.
If the main thread isn't able to process the message within 1ms, you are still sending it a new message even though it isn't finished processing the last message. I would imagine this puts it into a queue of messages waiting to be processed.
Now since you are sending messages from the web worker faster than they can be processed, this queue of unprocessed messages is going to get bigger and bigger. At some point Chrome is going to throw up its hands and say "There are too many messages in the queue", and instead of queueing new messages for processing, it drops them.
This is why if you use a reasonable number in your timeout like 100ms, the message has plenty of time to be processed before the next message is sent, and no problem with unprocessed messages occurs.
I've created a jsFiddle where the worker sends a message to the main thread, and the main thread sends the message back to the worker. If that process doesn't happen before the next message is sent, the counters in both threads will be mismatched and the web worker will terminate.
http://jsfiddle.net/meovfpv3/3/
You can see that with a reasonable setTimeout of 100ms, all messages have adequate time to process before the next message occurs.
When you lower the setTimeout to 1ms, the message chain doesn't have time to finish before the next message is sent and the counters in each thread become eventually desynced, tripping the if clause and terminating the web worker.
One way to fix this problem is instead of blindly posting a message every 1ms whether the last one has been processed or not, only post a new message after you have received a message back from the main thread. This means that you are only posting messages as fast as the main thread can process them.
For completeness here is a copy of the JSFiddle code:
Worker:
var counter2 = 0;
var rcvd = true;
function hi() {
counter2++;
console.log("")
console.log("postMessage", counter2)
postMessage(counter2);
if (!rcvd) {
self.close();
console.log("No message received");
}
rcvd = false;
setTimeout(hi, 1);
}
hi();
onmessage = function(e) {
rcvd = true;
console.log("secondMessage", e.data);
}
Main:
var ww = document.querySelector('script[type="text/ww"]'),
code = ww.textContent,
blob = new Blob([code], {type: 'text/javascript'}),
blobUrl = URL.createObjectURL(blob),
worker = new Worker(blobUrl),
counter = 0;
worker.onmessage = function(e) {
counter++;
console.log("onmessage:", counter);
worker.postMessage(e.data);
}
Firstly, a couple of observations, which I cannot explain but are kind of interesting and might be inspirational for someone:
#Anson - If I put your jsFiddle code into Codepen (still in Chrome) there are no problems there. The onmessage callback just keeps working!
And back in jsFiddle... It fails even changing the setTimeout to a long gap like 10s so it's not the number of times that the worker posts a message, it's how long before the onmessage callback stops firing – which has a lot of variance.
Then I found some ways to keep the onmessage handler alive in this specific example:
Add a button/link in the html and a handler (I used jQuery) that will terminate the worker on click. Just adding this code fixes it. $("#stop").on("click",function(e){e.preventDefault();worker.terminate();});
Just add console.log(worker) after defining onmessage.
Inspired by an answer posted in the related question you can also simply add window.worker = worker after defining onmessage.
Something about mentioning worker again in all cases seems to keep it alive.
Are you trying to postMessage every 1ms? Then you probably meant to use setInterval():
setInterval(function(){
postMessage('1');
}, 1);
Edit: I incorrectly saw recursion which wasn't there, just because I was looking for it. I would still use setInterval over setTimeout though.
What if I want to put a web worker on pause if I cannot proceed processing data, and try a second later? Can I do that in this manner inside a web worker?
var doStuff = function() {
if( databaseBusy() ) {
setTimeout(doStuff, 1000);
} else {
doStuffIndeed();
}
}
I'm getting Uncaught RangeError: Maximum call stack size exceeded and something tells me it's because of the code above.
If by "pause" you mean "further calls to worker.postMessage() will get queued up and not processed by the worker", then no, you cannot use setTimeout() for this. setTimeout() is not a busywait, but rather an in-thread mechanism for delaying work until a later scheduled time. The web worker will still receive a new onmessage event from the main queue as soon as it is posted.
What you can do, however, is queue them up manually and use setTimeout to try to process them later. For example:
worker.js
var workQueue = [];
addEventListener('message',function(evt){
workQueue.push(evt.data);
doStuff();
},false);
function doStuff(){
while (!databaseBusy()) doStuffIndeed(workQueue.shift());
if (workQueue.length) setTimeout(doStuff,1000);
}
Each time a message is received the worker puts it into a queue and calls tryToProcess.
tryToProcess pulls messages from the queue one at a time as long as the database is available.
If the database isn't available and there are messages left, it tries again in 1 second.