I am creating a small proprietary game server manager in Node.js; currently it runs the game by spawning via child_process:
var server = spawn(cmd, args, { cwd: 'something' });
So long as the manager continues to run I can pipe commands and deal with the child as I would like. However, consider the possibility that my manager crashes or is closed. How would I then reattach to the previously spawned child process (which was still running while the manager was down)? I can store pidfiles to try and reconnect based on pid; but I'm not sure how to get a child_process object with access to the child's stdio objects.
I really would like this to be recoverable; any help is appreciated, thanks!
Please note: The game servers are proprietary, some examples are Minecraft, Source DS, etc. Assume i do not have access to the sever source.
EDIT
After reading some source code from node's child_process it looks like if you specify a property in the options called stdinStream, stdoutStream, or stderrStream it should just open a socket to it. (See lines 428 - 496). So the issue then is, how do I stop spawn from actually doing a spawn and instead just settings its values based on a specified pid and streams I pass. (I would get my stdinStream by doing fs.createWriteStream('/proc/PID/fd/0'); which should work since that fd is created as a pipe.)
To extend what someone said in a comment above, you may be able to use http://nodejs.org/api/net.html where each child process creates a server (net.createServer()) and you keep a list of what children are listening at what ports somewhere, and then when your master restarts, it goes and finds that list of children and connects to each their servers. The Sockets that you get from net.createConnection() replace the child_process objects in your master.
net servers and sockets implement the same Readable and Writable Stream interfaces as stdio, so after setting up and connecting, you should be able to write(...) and pipe() events like you've been doing.
This may not be the best solution but I believe it will work.
After speaking with some of the guys on the node project, it looks like the only way I can accomplish this is by dealing with the raw file descriptors in the proc fs. I will probably mock up something similar to a child_process object and create its streams like:
var child = {};
child.stdin = fs.createWriteStream('/proc/PID/fd/0');
child.stdout = fs.createReadStream('/proc/PID/fd/1');
child.stderr = fs.createReadStream('/proc/PID/fd/2');
Related
I'm currently trying to find the best way to communicate between a master and child processes, in a multi-threaded node.js application. For this I set up some tests, sending a fairly large js object to the child processes and wait for it to respond to the master with the same object.
What i try to achieve with this is to find the fastest possible solution, to sending/receiving and communicating with child_processes or a way to create threads in Node.js (that share the same memory space and possibly scope, but execute on seperate threads) but as far as i'm aware the latter is impossible.
Test sends 1000 messages, child spawning takes on average about 100ms.
Benchmarks:
IPC 399ms
node-shared-cache 350ms
Socket.io 1523ms
fs streams 325ms
Apparently creating 2 additional pipes (child -> parent and parent -> child) to the stdio is the fastest way to communicate, probably because it's an uninterupted connection. I hope the difference becomes even more noticing with bigger datachunks.
Thanks to everyone who replied! Really appreciate it, it pushed me into doing tests with sockets and eventually uninterrupted streams.
I am using jimp (https://www.npmjs.com/package/jimp) in meteor JS to generate an image server side. In other words I am 'calculating' the pixels of the image using a recursive algorithm. The algorithm takes quite some time to complete.
The issue I am having is that this seems to completely block the meteor server. Users trying to visit the webpage while an image is being generated are forced to wait. The website is therefore not rendered at all.
Is there any (meteor) way to run the heavy recursive algorithm in a thread or something so that it does not block the entire website?
Node (and consequently meteor) runs in a single process which blocks on CPU activity. In short, node works really well when you are IO-bound, but as soon as you do anything that's compute-bound you need another approach.
As was suggested in the comments above, you'll need to offload this CPU-intensive activity to another process which could live on the same server (if you have multiple cores) or a different server.
We have a similar problem at Edthena were we need to transcode a subset of our video files. For now I decided to use a meteor-based solution, because it was easy to set up. Here's what we did:
When new transcode jobs need to happen, we insert a "video job" document in to the database.
On a separate server (we max out the full CPU when transcoding), we have an app which calls observe like this:
Meteor.startup(function () {
// Listen for non-failed transcode jobs in creation order. Use a limit of 1 to
// prevent multiple jobs of this type from running concurrently.
var selector = {
type: 'transcode',
state: { $ne: 'failed' },
};
var options = {
sort: { createdAt: 1 }, limit: 1,
};
VideoJobs.find(selector, options).observe({
added: function (videoJob) {
transcode(videoJob);
}, });
});
As the comments indicate this allows only one job to be called at a time, which may or may not be what you want. This has the further limitation that you can only run it on one app instance (multiple instances calling observe would simultaneously complete the job). So it's a pretty simplistic job queue, but it may work for your purposes for a while.
As you scale, you could use a more robust mechanism for dequeuing and processing the tasks like Amazon's sqs service. You can also explore other meteor-based solutions like job-collection.
I believe you're looking for Meteor.defer(yourFunction).
Relevant Kadira article: https://kadira.io/academy/meteor-performance-101/content/make-your-app-faster
Thanks for the comments and answers! It seems to be working now. What I did is what David suggested. I am running a meteor app on the same server. This app deals with the generating of the images. However, this resulted in the app still eating away all the processing power.
As a result of this I set a slightly lower priority on the generating algorithm with the renice command on the PID. (https://www.nixtutor.com/linux/changing-priority-on-linux-processes/) This works! Any time a user logs into the website the other (client) meteor application gains priority over the generating algorithm. Absolutely no delay at all anymore now.
The only issue I am having now is that whenever the server restarts I somehow have to rerun or run the (re)nice command.
Since I am using meteor up for deployment both apps run the same user and the same 'command': node main.js. I am currently trying to figure out how to run the nice command within the startup script of meteor up. (located at /etc/init/.conf)
I wish to suspend a REPL session so that I could shut down the system and then at a later time continue to work on the REPL session as if I'd never closed it, i.e. without having to lose all the environment.
I think that the possible solutions to this could be
Snapshot memory, save to file and load env from file later: I think this would be the neatest solution, like happens when you use the 'hibernate' feature of Windows. I've found this heapdump utility which is intended to take a memory snapshot for analysis of memory leaks, but I don't know if you could resurrect the whole environment from that snapshot and I have found no tools that do so.
Save commands and replay them: A major shortcoming of this method is while it works for simple things like var x = "Hello World";, it wouldn't work for things like var reciptId = bankAccount.makePayment(1000); as it will repeat actions on each replay rather than saving the details of the original function call.
Serialize / Deserialize the whole environment: This would involve making a list of all objects that exist in the environment, and then make a mechanism to write each of them to a file i.e. serialize them, and then make a mechanism that deserializes these and loads them when required. I am yet to see a clean way to serialize and deserialize js variables without limitations. I think that the major limitation of this method is its inability to retain references, so the objects loose their class, things would have to be duplicated upon serialization and lose their equality on deserialization - e.g.
var f = function (x) {...};
var a = {};
a.f = f;
a.f === f? //is true, not true if your serialization mechanism saves a function defn for f and a.f separately and deserializes them separately
and cyclic references would probably not work (x = {}; x.cyclic = x;...). So this method, if it ever works would require a lot of dirty work.
So the question really is, how difficult is it to achieve what I wish to achieve? What could be some other solutions to do this? Is there a major obstruction to achieving this which I'm overlooking?
Also, are there any alternatives to the node repl program (like a console in a browser) that can be suspended like this?
Related :
Swift REPL: how to save/load the REPL state? (a.k.a. suspend/resume, snapshot, clone)
So if you want to be able to "suspend" a REPL session and then pick up where you left off after a shut down doesn't seem to be directly available in Node.js's REPL. The closest thing to this is the Persistent History feature of the REPL which was added (i think) in Node 4.2.1. This will allow you to view the history of the commands in your REPL in plain text but thats the closest thing available out of the box with Node.
Persistent History
By default, the REPL will persist history between node REPL sessions by saving to a .node_repl_history file in the user's home directory. This can be disabled by setting the environment variable NODE_REPL_HISTORY="".
Previously in Node.js/io.js v2.x, REPL history was controlled by using a NODE_REPL_HISTORY_FILE environment variable, and the history was saved in JSON format. This variable has now been deprecated, and your REPL history will automatically be converted to using plain text. The new file will be saved to either your home directory, or a directory defined by the NODE_REPL_HISTORY variable, as documented below.
Full docs for the REPL module are available here.
However, there is a REPL "wrapper" node module that will do what you're asking. What you can do is, save your REPL history out to a file and then load the history file on the next session and gain access to what you saved to the file in your next REPL session.The module is Nesh. It has a lot of additional features including configuring your shell and evaluating different version of JS such as ES6/ES7 (Using Babel) & Coffeescript.
Install nesh:
npm install -g nesh
Launch nesh in the terminal by simply typing nesh. Work as you normally would within any other REPL session and when you want to save you can type the following in nesh to save your REPL history to the given file:
.save <filepath>
In your next REPL session, even after a shutdown, you can relaunch your nesh session and reload your history by typing:
.load <filepath>
This will re-evaluate the entire history file and will makes any variables or functions available in the current REPL/nesh session.
Hope this is helpful and I think it meets your needs.
What I think you are looking for is how to suspend and resume a process. See this answer on how to suspend and resume a process
Is there a way to do a synchronous read of a TCP socket in node.js?
I'm well aware of how to do it asynchronously by adding a callback to the socket's 'data' event:
socket.on('data', function(data) {
// now we have the string data to do whatever with
});
I'm also aware that trying to block with a function call instead of registering callbacks goes against node's design, but we are trying to update an old node module that acts as a client for my university while maintaining backwards compatibility. So we currently have:
var someData = ourModule.getData();
Where getData() previously had a bunch of logic behind it, but now we just want to send to the server "run getData()" and wait for the result. That way all logic is server side, and not duplicated client and server side. This module already maintains a TCP connection to the server so we are just piggybacking on that.
Here are the solutions I've tried:
Find a blocking read function for the socket hidden somewhere similar to python's socket library within node's net module.
string_from_tcp = socket.recv(1024)
The problem here is that it doesn't seem to exist (unsurprisingly because it goes against node's ideology).
This syncnet module adds what I need, but has no Windows support; so I'd have to add that.
Find a function that allow's node to unblock the event loop, then return back, such that this works:
var theData = null;
clientSocket.on('data', function(data) {
theData = data;
});
clientSocket.write("we want some data");
while(theData === null) {
someNodeFunctionThatUnblocksEventLoopThenReturnsHere(); // in this function node can check the tcp socket and call the above 'data' callback, thus changing the value of theData
}
// now theData should be something!
Obvious problem here is that I don't think such a thing exists.
Use ECMAScript 6 generator functions:
var stringFromTcp = yield socketRead(1024);
The problem here is that we'd be forcing students to update their JavaScript clients to this new syntax and understanding ES6 is outside the scopes of the courses that use this.
Use node-gyp and add to our node module an interface to a C++ TCP library that does support synchronous reads such as boost's asio. This would probably work but getting the node module to compile with boost cross platform has been a huge pain. So I've come to Stack Overflow to make sure I'm not over-complicating this problem.
In the simplest terms I'm just trying to create a command line JavaScript program that supports synchronous tcp reads.
So any other ideas? And sorry in advance if this seems blasphemous in context of a node project, and thanks for any input.
I ended up going with option 5. I found a small, fast, and easy to build TCP library in C++ (netLink) and wrote a node module wrapper for it, aptly titled netlinkwrapper.
The module builds on Windows and Linux, but as it is a C++ addon you'll need node-gyp configured to build it.
I hope no one else has to screw with Node.js as I did using this module, but if you must block the event loop with TCP calls this is probably your only bet.
Having worked with NodeJS for some time now, I've been wondering about how node handles file operations internally.
Considering the following pseudo code:
initialize http server
on connection:
modify_some_file:
on success:
print "it worked"
Let's consider two users A & B that try to access the page nearly simultaneously. Let's further assume A is the first one to connect, then the following happens:
A connects
NodeJS initializes the file operation and tells the operating system to be notified once it is done
And here's what I'm wondering about: Let's say, the file operation isn't done yet and B connects, what does node do? How and when does it access the file when it is still in the process of "being modified"?
I hope my question is somewhat clear ;)
Looking forward to your answers!
AFAIK, Node won't care.
At least on Unix, it's perfectly legal to have multiple writers to the same file. Sometimes that's not a problem (say your file consists of fixed-size records, where writer #1 writes to record X and writer #2 writes to record Y, with X !== Y), and sometimes it is (same example: when both writers want to write to record X).
In Node, the problems are mitigated because I/O operations "take turns", but I think there's still potential of two writers getting in each others way. It's up to the programmer to make sure that doesn't happen.
With Node, you could use the *Sync() versions of the fs operations (but those will block your app during the operation), use append mode (which is only atomic up to certain write sizes I think, and it depends on your requirements if appending is actually useful), use some form of locking, or use something like a queue where write operations would be put onto the queue and there's a single queue consumer to handle the writes.