Is there a way to do a synchronous read of a TCP socket in node.js?
I'm well aware of how to do it asynchronously by adding a callback to the socket's 'data' event:
socket.on('data', function(data) {
// now we have the string data to do whatever with
});
I'm also aware that trying to block with a function call instead of registering callbacks goes against node's design, but we are trying to update an old node module that acts as a client for my university while maintaining backwards compatibility. So we currently have:
var someData = ourModule.getData();
Where getData() previously had a bunch of logic behind it, but now we just want to send to the server "run getData()" and wait for the result. That way all logic is server side, and not duplicated client and server side. This module already maintains a TCP connection to the server so we are just piggybacking on that.
Here are the solutions I've tried:
Find a blocking read function for the socket hidden somewhere similar to python's socket library within node's net module.
string_from_tcp = socket.recv(1024)
The problem here is that it doesn't seem to exist (unsurprisingly because it goes against node's ideology).
This syncnet module adds what I need, but has no Windows support; so I'd have to add that.
Find a function that allow's node to unblock the event loop, then return back, such that this works:
var theData = null;
clientSocket.on('data', function(data) {
theData = data;
});
clientSocket.write("we want some data");
while(theData === null) {
someNodeFunctionThatUnblocksEventLoopThenReturnsHere(); // in this function node can check the tcp socket and call the above 'data' callback, thus changing the value of theData
}
// now theData should be something!
Obvious problem here is that I don't think such a thing exists.
Use ECMAScript 6 generator functions:
var stringFromTcp = yield socketRead(1024);
The problem here is that we'd be forcing students to update their JavaScript clients to this new syntax and understanding ES6 is outside the scopes of the courses that use this.
Use node-gyp and add to our node module an interface to a C++ TCP library that does support synchronous reads such as boost's asio. This would probably work but getting the node module to compile with boost cross platform has been a huge pain. So I've come to Stack Overflow to make sure I'm not over-complicating this problem.
In the simplest terms I'm just trying to create a command line JavaScript program that supports synchronous tcp reads.
So any other ideas? And sorry in advance if this seems blasphemous in context of a node project, and thanks for any input.
I ended up going with option 5. I found a small, fast, and easy to build TCP library in C++ (netLink) and wrote a node module wrapper for it, aptly titled netlinkwrapper.
The module builds on Windows and Linux, but as it is a C++ addon you'll need node-gyp configured to build it.
I hope no one else has to screw with Node.js as I did using this module, but if you must block the event loop with TCP calls this is probably your only bet.
Related
I just started getting into child_process and all I know is that it's good for delegating blocking functions (e.g. looping a huge array) to child processes.
I use typeorm to communicate with the mysql database. I was wondering if there's a benefit to move some of the asynchronous database works to child processes. I read it in another thread (Unfortunately I couldn't find it in the browser history) that there's no good reason to delegate async functions to child processes. Is it true?
example code:
child.js
import {createConnection} "./dbConnection";
import {SomeTable} from "./entity/SomeTable";
process.on('message', (m)=> {
createConnection().then(async connection=>{
let repository = connection.getRepository(SomeTable);
let results = await repository
.createQueryBuilder("t")
.orderBy("t.postId", "DESC")
.getMany();
process.send(results);
})
});
main.js
const cp = require('child_process');
const child = cp.fork('./child.js');
child.send('Please fetch some data');
child.on('message', (m)=>{
console.log(m);
});
The big gain about Javascript is its asynchronous nature...
What happens when you call an asynchronous function is that the code continues to execute, not waiting for the answer. And just when the function is done, and an answer is given does it then continue on with that part.
Your database call is already asynchronous. So you would spawn another node process for completely nothing. Since your database takes all the heat, having more nodeJS processes wouldn't help on that part.
Take the same example but with a file write. What could make the write to the disk faster? Nothing much really... But do we care? Nope because our NodeJS is not blocked and keeps answering requests and handling tasks. The only thing that you might want to check is to not send a thousand file writes at the same time, if they are big there would be a negative impact on the file system, but since a write is not CPU intensive, node will run just fine.
child processes really are a great tool, but it is rare to need it. I too wanted to use some when I heard about them, but the thing is that you will certainly not need them at all... The only time I decided to use it was to create a CPU intensive worker. It would make sure it spawns one child process per Core (since node is single threaded) and respawn any faulty ones.
I execute Meteor.call from the client side to a Meteor.methods method and as I console.log things, they are logged in both the command prompt and the browser console.
The issue with this is that it actually seems to be executing on the client side--which does not have access to the proper entities. While I get no errors on the command prompt, here's what is shown on the client side:
Exception while simulating the effect of invoking 'createInvite'
Meteor.makeErrorType.errorClass {error: "not-found", reason:
"evaluator_invite__entity_not_found", details: undefined, message:
"evaluator_invite__entity_not_found [not-found]", errorType:
"Meteor.Error"…} Error: evaluator_invite__entity_not_found [not-found]
at Meteor.methods.createInvite (http://localhost:3000/lib/collections/invites.js?505cdea882e0f829d521280c2057403ec134b075:38:15)
Is it actually running on the client side? Should this error be there?
Meteor methods are expected to run on both environments if defined globally, this allows for a nice feature which is called latency compensation.
The whole latency compensation concept is off topic but basically it means simulating client-side what the server would actually do like inserting documents in the database right-away to design fluid UX. You're essentially predicting server behavior before it's even happening to make your client experience ultra-responsive by eliminating network latency.
An example of this might be inserting a comment in the database immediately after the user submitted it. In this case the Meteor method calls getting executed on both the server and the client will share the same code.
Sometimes it makes absolutely no sense to provide a client-only simulation - or "stub" - when designing a method responsible for sending emails for example.
Some other times it makes sense to share some part of the code but you need to access environment specific objects : for example a method to insert comments might use Email.send on the server to notify an author a comment has been added on his post.
Meteor.methods({
insertComment: function(comment){
check(comment, [...]);
if(! this.isSimulation){
Email.send([...]);
}
return Comments.insert(comment);
}
});
Ultimately, you have to structure your code differently depending on how your method is supposed to behave.
For server-only methods, define them under the server directory, with no client-side counterpart.
For shared methods that can share exactly the same code on both environments, just define them globally.
For shared methods that just slightly differ, you can use Meteor.isClient / Meteor.isServer or method only property isSimulation to check against the current environment and execute specific code accordingly.
For shared methods that share little to no code, I personally use a pattern where I define the Meteor method on both client and server environment and take care of shared behavior like checking arguments validity, then I call a function responsible for actually implementing the correct behavior.
lib/method.js
Meteor.method({
method: function(args){
check(args, [...]);
//
return implementation(args);
}
});
The trick is to define the implementation separately on client and server, depending on the context, the correct function will get called.
client/method.js
implementation = function(args){
// define your client simulation / stub
};
server/method.js
implementation = function(args){
// define your server implementation
};
Read how to structure your app: http://docs.meteor.com/#/full/structuringyourapp
If you want your methods to be run only on the server, put files with methods into server folder.
If you want latency compensation you can use conditionals in your methods:
Meteor.methods({
method1: function() {
if (Meteor.isClient) {
//Do client side stuff
}
if (Meteor.isServer) {
//Do server side stuff
}
}
});
Is there any way i can do some database updation things whenever my node.js server crashes or stopped. Like try{}catch(){}finally(){} in JAVA. I am a bit newbie here.
Is there any events will node emit before it going shutdown. If so i can write my function there.
I have scenario,if i stop the server manually,i need to update some fields in the database.
The same is for Unhandled crashes also.
i here about domain in Node.js. But i have no idea how to monitor a whole server using domain.
An event is emitted when the node process is about to exit:
process.on('exit', function(code) {
console.log('About to exit with code:', code);
});
http://nodejs.org/api/process.html#process_event_exit
You can't query the database here though, since this handler can only perform synchronous operations. Some possible alternatives:
use database transactions so you never need to do "database updation things" when your app crashes
use a tool like Upstart to automatically restart your process, and then do database fixup stuff whenever your process starts
When you are using node JS it's bad practice to use try / catch, because of big number of asynchronous calls. The best practice here to use "promises" review next link, there you can find a good explanation: https://www.promisejs.org/
Having worked with NodeJS for some time now, I've been wondering about how node handles file operations internally.
Considering the following pseudo code:
initialize http server
on connection:
modify_some_file:
on success:
print "it worked"
Let's consider two users A & B that try to access the page nearly simultaneously. Let's further assume A is the first one to connect, then the following happens:
A connects
NodeJS initializes the file operation and tells the operating system to be notified once it is done
And here's what I'm wondering about: Let's say, the file operation isn't done yet and B connects, what does node do? How and when does it access the file when it is still in the process of "being modified"?
I hope my question is somewhat clear ;)
Looking forward to your answers!
AFAIK, Node won't care.
At least on Unix, it's perfectly legal to have multiple writers to the same file. Sometimes that's not a problem (say your file consists of fixed-size records, where writer #1 writes to record X and writer #2 writes to record Y, with X !== Y), and sometimes it is (same example: when both writers want to write to record X).
In Node, the problems are mitigated because I/O operations "take turns", but I think there's still potential of two writers getting in each others way. It's up to the programmer to make sure that doesn't happen.
With Node, you could use the *Sync() versions of the fs operations (but those will block your app during the operation), use append mode (which is only atomic up to certain write sizes I think, and it depends on your requirements if appending is actually useful), use some form of locking, or use something like a queue where write operations would be put onto the queue and there's a single queue consumer to handle the writes.
I am creating a small proprietary game server manager in Node.js; currently it runs the game by spawning via child_process:
var server = spawn(cmd, args, { cwd: 'something' });
So long as the manager continues to run I can pipe commands and deal with the child as I would like. However, consider the possibility that my manager crashes or is closed. How would I then reattach to the previously spawned child process (which was still running while the manager was down)? I can store pidfiles to try and reconnect based on pid; but I'm not sure how to get a child_process object with access to the child's stdio objects.
I really would like this to be recoverable; any help is appreciated, thanks!
Please note: The game servers are proprietary, some examples are Minecraft, Source DS, etc. Assume i do not have access to the sever source.
EDIT
After reading some source code from node's child_process it looks like if you specify a property in the options called stdinStream, stdoutStream, or stderrStream it should just open a socket to it. (See lines 428 - 496). So the issue then is, how do I stop spawn from actually doing a spawn and instead just settings its values based on a specified pid and streams I pass. (I would get my stdinStream by doing fs.createWriteStream('/proc/PID/fd/0'); which should work since that fd is created as a pipe.)
To extend what someone said in a comment above, you may be able to use http://nodejs.org/api/net.html where each child process creates a server (net.createServer()) and you keep a list of what children are listening at what ports somewhere, and then when your master restarts, it goes and finds that list of children and connects to each their servers. The Sockets that you get from net.createConnection() replace the child_process objects in your master.
net servers and sockets implement the same Readable and Writable Stream interfaces as stdio, so after setting up and connecting, you should be able to write(...) and pipe() events like you've been doing.
This may not be the best solution but I believe it will work.
After speaking with some of the guys on the node project, it looks like the only way I can accomplish this is by dealing with the raw file descriptors in the proc fs. I will probably mock up something similar to a child_process object and create its streams like:
var child = {};
child.stdin = fs.createWriteStream('/proc/PID/fd/0');
child.stdout = fs.createReadStream('/proc/PID/fd/1');
child.stderr = fs.createReadStream('/proc/PID/fd/2');