Node allows you to spawn child processes and send data between them. You could use it do execute some blocking code for example.
Documentation says "These child Nodes are still whole new instances of V8. Assume at least 30ms startup and 10mb memory for each new Node. That is, you cannot create many thousands of them."
I was wondering if is it efficient, should I worry about some limitations? Here's example code:
//index.js
var childProcess1 = childProcess.fork('./child1.js');
childProcess1.send(largeArray);
childProcess1.once('message', function(formattedData) {
console.log(formattedData);
return false;
});
//child1.js
process.on('message', function(data) {
data = format(data); //do smth with data, then send it back to index.js
try{
process.send(data);
return false;
}
catch(err){
console.log(err);
return false;
}
});
The documentation is telling you that starting new node processes is (relatively) expensive. It is unwise to fork() every time you need to do work.
Instead, you should maintain a pool of long-running worker processes – much like a thread pool. Queue work requests in your main process and dispatch them to the next available worker when it goes idle.
This leaves us with a question about the performance profile of node's IPC mechanism. When you fork(), node automatically sets up a special file descriptor on the child process. It uses this to communicate between processes by reading and writing line-delimited JSON. Basically, when you process.send({ ... }), node JSON.stringifys it and writes the serialized string to the fd. The receiving process reads this data until hitting a line break, then JSON.parses it.
This necessarily means that performance will be highly dependent on the size of the data you send between processes.
I've roughed out some tests to get a better idea of what this performance looks like.
First, I sent a message of N bytes to the worker, which immediately responded with a message of the same length. I tried this with 1 to 8 concurrent workers on my quad-core hyper-threaded i7.
We can see that having at least 2 workers is beneficial for raw throughput, but more than 2 essentially doesn't matter.
Next, I sent an empty message to the worker, which immediately responded with a message of N bytes.
Surprisingly, this made no difference.
Finally, I tried sending a message of N bytes to the worker, which immediately responded with an empty message.
Interesting — performance does not degrade as rapidly with larger messages.
Takeaways
Receiving large messages is slightly more expensive than sending them. For best throughput, your master process should not send messages larger than 1 kB and should not receive messages back larger than 128 bytes.
For small messages, the IPC overhead is about 0.02ms. This is small enough to be inconsequential in the real world.
It is important to realize that the serialization of the message is a synchronous, blocking call; if the overhead is too large, your entire node process will be frozen while the message is sent. This means I/O will be starved and you will be unable to process any other events (like incoming HTTP requests). So what is the maximum amount of data that can be sent over node IPC?
Things get really nasty over 32 kB. (These are per-message; double to get roundtrip overhead.)
The moral of the story is that you should:
If the input is larger than 32 kB, find a way to have your worker fetch the actual dataset. If you're pulling the data from a database or some other network location, do the request in the worker. Don't have the master fetch the data and then try to send it in a message. The message should contain only enough information for the worker to do its job. Think of messages like function parameters.
If the output is larger than 32 kB, find a way to have the worker deliver the result outside of a message. Write to disk or send the socket to the worker so that you can respond directly from the worker process.
This really depends on your server resources and the number of nodes you need to spin up.
As a rule of thumb:
Try reusing running children as much as possible - this will save you 30ms start up time
Do not start unlimited number of children (1 per request for instance) - you will not run out of RAM
The messaging itself it relatively fast i believe. Would be great to see some metrics though.
Also, note that if you have single CPU or running a cluster (using all available cores) it doesn't make much sense. You still have limited CPU capacity and switching context is more expensive than running single process.
Related
I am using Node.js to implement a Websocket client that subscribes to datafeed from multiple Websocket servers.
foo = new WebSocket('ws://foo.host ...')
bar = new WebSocket('ws://barhost ...')
baz = new WebSocket('ws://baz.host ...')
qux = new WebSocket('ws://qux.host ...')
foo.on('data', data => doSomething(data)) // 5 events per second
bar.on('data', data => doSomething(data)) // 1 events per second
baz.on('data', data => doSomething(data)) // 1 events per second
qux.on('data', data => doSomething(data)) // 1 events per second
Question: If we have a multi-core system (eg. 4 cores), is it possible to make use of Node.js Cluster to load balance the processing of the incoming Websocket data, such that each core will approximately receive 2 events per second to be handled?
Or is it better to manually start 8 node.js instances and pass it an argument [foo|bar|baz|qux] for selecting the Websocket server it will connect to?
The nodejs clustering module solves one specific problem. When you have a an http server and you want to load balance incoming connections among multiple processes, that's what the nodejs clustering module does. That is not what you have. You have multiple client-side outgoing webSocket connections and you apparently want to apply multiple processes to processing the incoming data. That's completely different than what the nodejs cluster module does.
First, it's important to understand that receiving the data is not a CPU intensive process for nodejs. The actual socket processing and receiving of incoming data onto the computer is handled by the OS and is outside the nodejs process.
So, if you actually need more than one CPU to work on this, it must be to process the incoming data, not to just receive it.
There are several different ways you could structure that.
You could have one central process that contains all the webSockets and then have and number of worker processes or worker threads that you pass incoming data to for processing. This would apply many CPUs to the processing of the data and would allow the load procesisng to be spread among the CPUs regardless of which socket the data arrived on.
You could create 4 separate child processes and have each child process create one of the four webSocket connections and then have each child process handle just the incoming data for its webSocket. This has the disadvantage that it only applies one process to each webSocket and if most of the data comes on one webSocket, then the other processes will be largely idle.
If one webSocket has a lot more load than the others and for some reason option #1 wouldn't work well, then you could combine #1 and #2. Create a separate process for each webSocket and then have some worker threads for processing the incoming data for each one. Create a work queue that incoming data is inserted into and work can be sent to each worker thread as it finishes its previous chunk of data.
I am currently deeply learning Nodejs platform. As we know, Nodejs is single-threaded, and if it executes blocking operation (for example fs.readFileSync), a thread should wait to finish that operation. I decided to make an experiment: I created a server that responses with the huge amount of data from a file on each request
const { createServer } = require('http');
const fs = require('fs');
const server = createServer();
server.on('request', (req, res) => {
let data;
data =fs.readFileSync('./big.file');
res.end(data);
});
server.listen(8000);
Also, I launched 5 terminals in order to do parallel requests to a server. I waited to see that while one request is being handled, the others should wait for finishing blocking operation from the first request. However, the other 4 requests were responded concurrently. Why does this behavior occur?
What you're likely seeing is either some asynchronous part of the implementation inside of res.end() to actually send your large amount of data or you are seeing all the data get sent very quickly and serially, but the clients can't process it fast enough to actually show it serially and because the clients are each in their own separate process, they "appear" to show it arriving concurrently just because they're too slow reacting to show the actually arrival sequence.
One would have to use a network sniffer to see which of these is actually occurring or run some different tests or put some logging inside the implementation of res.end() or tap into some logging inside the client's TCP stack to determine the actual order of packet arrival among the different requests.
If you have one server and it has one request handler that is doing synchronous I/O, then you will not get multiple requests processes concurrently. If you believe that is happening, then you will have to document exactly how you measured that or concluded that (so we can help you clear up your misunderstanding) because that is not how node.js works when using blocking, synchronous I/O such as fs.readFileSync().
node.js runs your JS as single threaded and when you use blocking, synchronous I/O, it blocks that one single thread of Javascript. That's why you should never use synchronous I/O in a server, except perhaps in startup code that only runs once during startup.
What is clear is that fs.readFileSync('./big.file') is synchronous so your second request will not get started processing until the first fs.readFileSync() is done. And, calling it on the same file over and over again will be very fast (OS disk caching).
But, res.end(data) is non-blocking, asynchronous. res is a stream and you're giving the stream some data to process. It will send out as much as it can over the socket, but if it gets flow controlled by TCP, it will pause until there's more room to send on the socket. How much that happens depends upon all sorts of things about your computer, it's configuration and the network link to the client.
So, what could be happening is this sequence of events:
First request arrives and does fs.readFileSync() and calls res.end(data). That starts sending data to the client, but returns before it is done because of TCP flow control. This sends node.js back to its event loop.
Second request arrives and does fs.readFileSync() and calls res.end(data). That starts sending data to the client, but returns before it is done because of TCP flow control. This sends node.js back to its event loop.
At this point, the event loop might start processing the third or fourth requests or it might service some more events (from inside the implementation of res.end() or the writeStream from the first request to keep sending more data. If it does service those events, it could give the appearance (from the client point of view) of true concurrency of the different requests).
Also, the client could be causing it to appear sequenced. Each client is reading a different buffered socket and if they are all in different terminals, then they are multi-tasked. So, if there is more data on each client's socket than it can read and display immediately (which is probably the case), then each client will read some, display some, read some more, display some more, etc... If the delay between sending each client's response on your server is smaller than the delay in reading and displaying on the client, then the clients (which are each in their own separate processes) are able to run concurrently.
When you are using asynchronous I/O such as fs.readFile(), then properly written node.js Javascript code can have many requests "in flight" at the same time. They don't actually run concurrently at exactly the same time, but one can run, do some work, launch an asynchronous operation, then give way to let another request run. With properly written asynchronous I/O, there can be an appearance from the outside world of concurrent processing, even though it's more akin to sharing of the single thread whenever a request handler is waiting for an asynchronous I/O request to finish. But, the server code you show is not this cooperative, asynchronous I/O.
Maybe is not related directly to your question but i think this is useful,
You can use a stream instead of reading the full file into memory, for example:
const { createServer } = require('http');
const fs = require('fs');
const server = createServer();
server.on('request', (req, res) => {
const readStream = fs.createReadStream('./big.file'); // Here we create the stream.
readStream.pipe(res); // Here we pipe the readable stream to the res writeable stream.
});
server.listen(8000);
The point of doing this is:
Looks nicer.
You don't store the full file in RAM.
This works better because is non blocking, and the res object is already a stream, and this means the data will be transfered in chunks.
Ok so streams = chunked
Why not read chunks from the file and send them in real time instead of reading a really big file and divide that in chunks after?
Also why is really important on a real production server?
Because every time a request is received, your code is going to add that big file into ram, to that add this is concurrent so you are expecting to serve multiple files at the same time, so let's do the most advanced math my poor education allows:
1 request for a 1gb file = 1gb in ram
2 requests for a 1gb file = 2gb in ram
etc
That clearly doesn't scale nicely right?
Streams allows to decouple that data from the current state of the function (inside that scope), so in simple terms its going to be (with the default chunk size of 16kb):
1 request for 1gb file = 16kb in ram
2 requests for 1gb file = 32kb in ram
etc
And also, the OS its already passing a stream to node (fs) so it works with streams end to end.
Hope it helps :D.
PD: Never use sync operations (blocking) inside async operations (non blocking).
I have a program which is using the Websocket TCP: The client is an extension in Chrome and the server is an application written in C++.
When I send small data from the client to the server, it works fine. But when I send large amounts of data (e.g. a source html page), it will be slightly delayed.
For Example:
Client sends: 1,2,3
Server receives: 1,2
Client sends: 4
Server receives: 3
Client sends: 5
Server receives: 4
It's seems like it's a delay.
This is my code client:
var m_cWebsocket = new WebSocket("Servername");
if (m_cWebsocket == null) { return false; }
m_cWebsocket.onopen = onWebsocketOpen(m_cWebsocket); m_cWebsocket.onmessage = onWebsocketMessage;
m_cWebsocket.onerror = onWebsocketError;
m_cWebsocket.onclose = onWebsocketError;
I using m_cWebsocket.send(strMsg) to send data.
Server code
while (true) { recv(sSocket, szBufferTmp, 99990, 0); //recv(sSocket,
szBufferTmp, 99990, MSG_PEEK); //some process }
Since you haven't posted any code to show your implementation of the TCP server or client I can only speculate and try to explain what might be going on here.
That means the potential problems and solutions I outline below may or may not apply to you, but regardless this information should still be helpful to others who might find this question in the future.
TL;DR: (most likely) It's either the server is too slow, the server is not properly waiting for complete 'tcp packets' to be buffered, or the server doesn't know when to properly start and stop and is de-synching while it waits for what it thinks is a 'full packet' as defined by something like a buffer size.
It sounds to me like you are pushing data from the client either faster than the server the server can read, or more likely, the server is buffering a set number of bytes from the current TCP Stream and waiting for the buffer to fill before outputting additional data.
If you are sending this over localhost it's unlikely you are not close to limit of the stream though, and I would expect a server written in C++ would be able to keep up with the javascript client.
So this leads me to believe that the issue is in fact the stream buffer on the C++ side.
Now since the server has no way to know to what data you are sending and or how much of it you are sending, it is common for a TCP stream to utilize a stream buffer that contiguously reads data from the socket until either the buffer has filled to a known size, or until it sees a predefined 'stop character'. This would usually be something like a "line end" or \n character, sometimes \n\r (line feed, carriage feed) depending on your operating system.
Since you haven't specified how you are receiving your data, I'm going to just assume you created either a char or byte buffer of a certain size. I'm a pretty rusty on my C++ socket information so I might be wrong, but I do believe there is a default 'read timeout' on C++ tcp streams as well.
This means you are possibly running into 1 of 2 issues.
Situation 1) You are waiting until that byte/char buffer is filled before outputing it's data. Issue is that will act like a bus that only leaves the station when all seats are filled. If you don't fill all the seats, you server is just sitting and waiting until it gets more data to fill up fully and output your data.
Situation 2) You are running up against the socket read timeout and therefore the function is not getting all the data before outputting the data. This is like a bus that is running by the clock. Every 10 minutes that bus leaves the station, doesn't matter if that bus is full or empty, it's leaving and the next bus will pick up anyone who shows up late. In your case, the TCP stream isn't able to load 1, 2 and 3 onto a bus fast enough, so the bus leaves with just 1, 2 on it because after 20ms of not receiving data, the server is exiting from the function and outputing the data. On the next loop however, there is 3 waiting at the top of the stream buffer ready to get on the next bus out. The Stream will load 3, wait til those 20ms are finished, and then exit before repeating this loop.
I think it's more likely the first situation is occurring though, as I would expect the server to either start catching up, or falling further behind as the 2 servers either begin to sync together, or have internall TPC stream buffer fill up as the server falls further and further behind.
Main point here, you need some way to synchronize the client and the server connections. I would recommend sending a "start byte" and "End byte" to single when a message has begun and finished, so you don't exit the function too early.
Or send a start byte, followed by the packet size in bytes, then filling up the buffer until your buffer has the correct numbers of bytes. Additionally you could include an end byte as well for some basic error checking.
This is a pretty involved topic and hard to really give you a good answer without any code from you, but this should also help anyone in the future who might be having a similar issue.
EDIT I went back and re-read your question and noticed you said it was only with large amounts of data, so I think my original assumption was wrong, and it's more likely situation 2 because the client is sending the data to your server faster than the server can read it, and thus might be bottle necking the connection and the client is only able to send additional data once the server has emptied part of it's TCP stream buffer.
Think of it like a tube of of water. The socket (tube) can only accept (fill up) with so much data (water) before it's full. Once you let some water out the bottom though, you can fill it up a little bit more. The only reason it works for small files is that the file is too small to fill the entire tube.
Additional thoughts: You can see how I was approaching this problem in C# in this question: Continuously reading from serial port asynchronously properly
And another similar question I had previously (again in C#): How to use Task.WhenAny with ReadLineAsync to get data from any TcpClient
It's been awhile since I've played with TCP streams though, so my apologies in that I don't remember all the niche details and caveats of the protocal, but hopefully this information is enough to get you in the ball park for solving your problem.
Full disclaimer, it's been over 2 years since I last touched C++ TCP sockets, and have since worked with sockets/websockets in other languages (such as C# and JavaScript), so I may have some facts wrong about the behavior of C++ TCP sockets specifically, but the core information should still apply. If I got anything wrong, someone in the comments will most likely have the correct information.
Lastly, welcome to stack overflow!
A web client should only expose some features when a backend API is up and running. Therefor, I'm looking for a clean way to monitor the availability of this backend.
As a quick fix, I made a timer-based function that performs a basic GET on the API root. It's not very clean, generates lots of traffic and pollutes the javascript console with errors (in case of server down).
How should one deal with such situation?
You can trigger something in the lines of this when you need it:
function checkServerStatus()
{
setServerStatus("unknown");
var img = document.body.appendChild(document.createElement("img"));
img.onload = function()
{
setServerStatus("online");
};
img.onerror = function()
{
setServerStatus("offline");
};
img.src = "http://myserver.com/ping.gif";
}
Make ping.gif small (1 pixel) to make it as fast as possible.
Ofc you can do it more smoothly by accessing the API that returns true and keeps a really small response time, but that requires you to do some coding in back-end this simply needs you to place a 1-pixel gif image in a correct directory on a server. You can use any picture already present on the server, but expect more traffic and time as image grows larger.
Now put this in some function that calls it with delay, or simply call this when you need to check status, it's up to you.
If you need a server to send to your app a notification when it's down then you need to implement this:
https://en.wikipedia.org/wiki/Push_technology
Ideally, you would have high-reliability server that has fast response rate and is really reliable to be pinging the desired server in some interval to determine whether it up then use the push to get that information to your app. This way that 3rd server would only send you a push if a status of your app server has changed. Ideally, this server's request has a high priority on your app server queue and servers are well connected and close to each other but not on the same network in case that fails.
Recommendation:
First approach should do you good since it's simple to implement and requires the least amount of knowledge.
Consider second if:
You need a really small interval of checking making your application slower and network traffic higher
You have multiple applications that need the same - making load heavier on both each application, network AND the server. The second approach lets you use single ping to determine truth for all apps.
In order to limit number of request, simple solution can be use of server-sent events. This protocol used on top of HTTP allow server to push multiple updates in response of the same client request.
Client side code (javascript) :
var evtSource = new EventSource("backend.php");
evtSource.onmessage = function(e) {
console.log('status:' + e.data);
}
evtSource.onerror = function(e) {
// add some retry then display error to the user
}
Backend code (PHP, also supported by other languages)
header("Content-Type: text/event-stream\n\n");
while (1) {
// Each 30s, send OK status
echo "OK\n";
ob_flush();
flush();
sleep(30);
}
In both case it will limit number of request (only 1 per "session") but you will have 1 socket per client opened, which can be also to heavy for your server.
If you really want to lower the workload, you should delegate it to external monitoring platform which can expose API to publish backend status.
Maybe it already exists if your backend is hosted on cloud platform.
I've been looking around for a definitive answer to this but I seem to keep finding contradictory answers (ex this and this).
Basically, if I
socket.emit('game_update', {n: 1});
from a node.js server and then, 20 ms later,
socket.emit('game_update', {n: 2});
from the same server, is there any way that the n:2 message arrives before the n:1 message? In other words, does the n:1 message "block" the receiving of the n:2 message if the n:1 message somehow got lost on the way?
What if they were volatile emits? My understanding is that the n:1 message wouldn't block the n:2 message -- if the n:1 message got dropped, the n:2 message would still be received whenever it arrived.
Background: I'm building a node.js game server and want to better understand how my game updates are traveling. I'm using volatile emit right now and I would like to increase the server's tick rate, but I want to make sure that independent game updates wouldn't block each other. I would rather the client receive an update every 30 ms with a few dropped updates scattered here and there than have the client receive an update, receive nothing for 200 ms, and then receive 6 more updates all at once.
Disclaimer: I'm not completely familiar with the internals of socket.io.
is there any way that the n:2 message arrives before the n:1 message?
It depends on the transport that you're using. For the polling transport, I think it's fair to say that it's perfectly possible for messages to arrive out-of-order, because each message can arrive over a different connection.
With the websocket transport, which maintains a persistent connection, the message order is reasonably guaranteed.
What if they were volatile emits?
With volatile emits, all bets are off, it's fire-and-forget. I think that in normal situations, the server will wait (and queue up messages) for a client to be ready to receive messages, unless those messages are volatile, in which case the server will just drop them.
From what you're saying, I think that volatile emits are what you want, although once a websocket connection has been established I don't think you'll see the described scenario ("receive an update, receive nothing for 200 ms, and then receive 6 more updates all at once") is likely to happen. Perhaps only when the connection gets lost and is re-established.
The answer is yes it can possibly arrive later, but it is highly unlikely given that sockets are by nature persistent connections and reliability of order is all but guaranteed.
According to the Socket.io documentation messages will be discarded in the case that the client is not connected. This doesn't necessarily fit with your use case, however within the documentation itself it describes Volatile events as an interesting example if you need to send the position of a character.
// server-side
io.on("connection", (socket) => {
console.log("connect");
socket.on("ping", (count) => {
console.log(count);
});
});
// client-side
let count = 0;
setInterval(() => {
socket.volatile.emit("ping", ++count);
}, 1000);
If you restart the server, you will see in the console:
connect
1
2
3
4
# the server is restarted, the client automatically reconnects
connect
9
10
11
Without the volatile flag, you would see:
connect
1
2
3
4
# the server is restarted, the client automatically reconnects and sends its
buffered events
connect
5
6
7
8
9
10
11
Note: The documentation explicitly states that this will happen during a server restart, meaning that your connection to the client likely has to be lost in order for the volatile emits to be dropped.
I would say a good practice would be to write your emits as volatile just in case you do get a dropped client, however this will depend heavily on your game requirements.
As for the goal, I would recommend that you use client side prediction using some sort of dynamic time system or deltatime based on the client and server keeping a sync clock to help alleviate some of the problems you can incur. Here's an example of how you can do that, though I'm not a fan of the creators syntax, it can be easily adapted to your needs.
Hope this helps anyone who hits this topic.
Socket.io - Volatile events
Client Side Prediction