We Keep Coding

NodeJS TCP Client communication

First of all - I am aware of this answer to a kind of similar problem.
Problem
I have a third party protocol, that uses TCP\IP. This protocol defines that the server replies to every message received. On the client side (which I try to implement) I have to wait for the answer from the server.
The problem occurs, when I try to send messages. I need to wait for the answer from the first message before I send the second one (like ping-pong).
I tried to do multiple writes on my NodeJS tcp-client like this, which understandably fails due to async:
client.connect(connectOptions, function () {
client.write(message1);
client.write(message2);
});
Like I said before, I have a third party component, which responses to both messages with a numeric value. So when
client.on('data',function (data) {});
fires an event, I can't distinguish which message, was responsible for the answer. Unlike the linked answer I don't have the ability, to tag the answer on the server side.
I am new to node.js, so I try to figure out the best way to solve this kind of problem, as it´s of the nature: do synchronous things in the async environment.

One way would be to use a common list of handlers to keep track of requests and responses:
var handlers = [];
client.connect(connectOptions, function () {
client.write(message1);
handlers.push(function msg1_handler(data) {});
client.writewrite(message2);
handlers.push(function msg2_handler(data) {});
});
client.on('data',function(data) {
var handler = handlers.shift();
handler(data);
});
All of this should obviously be wrapped in a separate class containing both handlers an client objects. It's just an example of how to do it. The drawback is that if the server fails to respond to some request then you have a complete mess, hard to make it right.
Another idea is to buffer requests:
function BufferedClient(cli) {
this.cli = cli;
this.buffer = [];
this.waiting_for_response = false;
var that = this;
cli.on('data', function(data) {
that.waiting_for_response = false;
var pair = that.buffer.shift();
var handler = pair[0];
process.nextTick(function() {
// we use .nextTick to avoid potential
// exception in handler which would break
// BufferedClient
handler(data);
});
that.flush();
});
};
BufferedClient.prototype = {
request: function(msg, handler) {
this.buffer.push([handler, msg]);
this.flush();
},
flush: function() {
var pair = this.buffer[0];
if (pair && !this.waiting_for_response) {
this.cli.write(pair[1]);
this.waiting_for_response = true;
}
}
};
This time you send requests sequentially (so like synchronous) due to how .request() and .on('data') handler work together with .flush() function. Usage:
client.connect(connectOptions, function () {
var buff_cli = new BufferedClient(client);
buff_cli.request(message1, function(data) { });
buff_cli.request(message2, function(data) { });
});
Now even if the server fails to respond you don't have a mess. However if you issue buff_cli.request parallely and one of them fails then you will have a memory leak (since this.buffer is getting bigger while nothing is draining it because the BufferedClient is waiting for a response). This can be fixed by adding some timeouts on the socket.
Note that both solutions assume that the server never pushes anything to the client without a request.
If I were you I would go with second solution. Note that I haven't tested the code so it might be buggy but the general idea should be ok.
Side note: When you implement a server (and I know that you don't in this case) you should always have a protocol that matches each request with a response in a unique way. One way would be to send a unique ID with each request so that the server would be respond with the same ID. In such scenario matching request with response is very easy and you avoid all that mess.

NodeJS Event Emitter Blocking Issue

I have a node application handling some ZeroMQ events coming from another application utilizing the Node-ZMQ bindings found here: https://github.com/JustinTulloss/zeromq.node
The issue I am running into is one of the operations from an event takes a long time to process and this appears to be blocking any other event from being processed during this time. Although the application is not currently clustered, doing so would only afford a few more threads and doesn't really solve the issue. I am wondering if there is a way of allowing for these async calls to not block other incoming requests while they process, and how I might go about implementing them.
Here is a highly condensed/contrived code example of what I am doing currently:
var zmq = require('zmq');
var zmqResponder = zmq.socket('rep');
var Client = require('node-rest-client').Client;
var client = new Client();
zmqResponder.on('message', function (msg, data) {
var parsed = JSON.parse(msg);
logging.info('ZMQ Request received: ' + parsed.event);
switch (parsed.event) {
case 'create':
//Typically short running process, not an issue
case 'update':
//Long running process this is the issue
serverRequest().then(function(response){
zmqResponder.send(JSON.stringify(response));
});
}
});
function serverRequest(){
var deferred = Q.defer();
client.get(function (data, response) {
if (response.statusCode !== 200) {
deferred.reject(data.data);
} else {
deferred.resolve(data.data);
}
});
return deferred.promise;
}
EDIT** Here's a gist of the code: https://gist.github.com/battlecow/cd0c2233e9f197ec0049

I think, through the comment thread, I've identified your issue. REQ/REP has a strict synchronous message order guarantee... You must receive-send-receive-send-etc. REQ must start with send and REP must start with receive. So, you're only processing one message at a time because the socket types you've chosen enforce that.
If you were using a different, non-event-driven language, you'd likely get an error telling you what you'd done wrong when you tried to send or receive twice in a row, but node lets you do it and just queues the subsequent messages until it's their turn in the message order.
You want to change REQ/REP to DEALER/ROUTER and it'll work the way you expect. You'll have to change your logic slightly for the ROUTER socket to get it to send appropriately, but everything else should work the same.
Rough example code, using the relevant portions of the posted gist:
var zmqResponder = zmq.socket('router');
zmqResponder.on('message', function (msg, data) {
var peer_id = msg[0];
var parsed = JSON.parse(msg[1]);
switch (parsed.event) {
case 'create':
// build parsedResponse, then...
zmqResponder.send([peer_id, JSON.stringify(parsedResponse)]);
break;
}
});
zmqResponder.bind('tcp://*:5668', function (err) {
if (err) {
logging.error(err);
} else {
logging.info("ZMQ awaiting orders on port 5668");
}
});
... you need to grab the peer_id (or whatever you want to call it, in ZMQ nomenclature it's the socket ID of the socket you're sending from, think of it as an "address" of sorts) from the first frame of the message you receive, and then use send it as the first frame of the message you send back.
By the way, I just noticed in your gist you are both connect()-ing and bind()-ing on the same socket (zmq.js lines 52 & 143, respectively). Don't do that. Inferring from other clues, you just want to bind() on this side of the process.

Communicate from server to client side in Google Apps script

I am trying to write a Google Apps script which has a client and server side component. The client side component displays a progress bar. The client calls server side functions (which are called asynchronously), whose progress has to be shown in the client side progress-bar. Now, what I want is to be able to update the client side progress bar based on feedback from the server side functions. Is this possible?
The complexity is created due the the fact that JS makes the server-side calls asynchronously and hence I cannot really have a loop on the client side calling the functions and updating the progress bar.
I could of course split up the execution of the server side function in multiple steps, call one by one from the client side, each time updating the status bar. But I'm wondering if there's a better solution. Is there a way to call a client side function from the server side, and have that update the progress bar based on the argument passed? Or is there a way to access the client side progress-bar object from server side and modify it?

The way I've handled this is to have a middleman (giving a shout out now to Romain Vialard for the idea) handle the progress: Firebase
The HTML/client side can connect to your Firebase account (they're free!) and "watch" for changes.
The client side code can update the database as it progresses through the code - those changes are immediately fed back to the HTML page via Firebase. With that, you can update a progress bar.
Romain has a small example/description here
The code I use:
//Connect to firebase
var fb = new Firebase("https://YOUR_DATABASE.firebaseio.com/");
//Grab the 'child' holding the progress info
var ref = fb.child('Progress');
//When the value changes
ref.on("value", function(data) {
if (data.val()) {
var perc = data.val() * 100;
document.getElementById("load").innerHTML = "<div class='determinate' style='width:" + perc + "%\'></div>";
}
});
On the client side, I use the Firebase library to update the progress:
var fb = FirebaseApp.getDatabaseByUrl("https://YOUR_DATABASE..firebaseio.com/");
var data = { "Progress": .25};
fb.updateData("/",data);

Rather than tying the work requests and progress updating together, I recommend you separate those two concerns.
On the server side, functions that are performing work at the request of the client should update a status store; this could be a ScriptProperty, for example. The work functions don't need to respond to the client until they have completed their work. The server should also have a function that can be called by the client to simply report the current progress.
When the client first calls the server to request work, it should also call the progress reporter. (Presumably, the first call will get a result of 0%.) The onSuccess handler for the status call can update whatever visual you're using to express progress, then call the server's progress reporter again, with itself as the success handler. This should be done with a delay, of course.
When progress reaches 100%, or the work is completed, the client's progress checker can be shut down.

Building on Jens' approach, you can use the CacheService as your data proxy, instead of an external service. The way that I've approached this is to have my "server" application generate an interim cache key which it returns to the "client" application's success callback. The client application then polls this cache key at an interval to see if a result has been returned into the cache by the server application.
The server application returns an interim cache key and contains some helper functions to simplify checking this on the client-side:
function someAsynchronousOperation() {
var interimCacheKey = createInterimCacheKey();
doSomethingComplicated(function(result) {
setCacheKey(interimCacheKey, result);
});
return interimCacheKey;
}
function createInterimCacheKey() {
return Utilities.getUuid();
}
function getCacheKey(cacheKey, returnEmpty) {
var cache = CacheService.getUserCache();
var result = cache.get(cacheKey);
if(result !== null || returnEmpty) {
return result;
}
}
function setCacheKey(cacheKey, value) {
var cache = CacheService.getUserCache();
return cache.put(cacheKey, value);
}
Note that by default getCacheKey doesn't return. This is so that google.script.run's successHandler doesn't get invoked until the cache entry returns non-null.
In the client application (in which I'm using Angular), you call off to the asynchronous operation in the server, and wait for its result:
google.script.run.withSuccessHandler(function(interimCacheKey) {
var interimCacheCheck = $interval(function() {
google.script.run.withSuccessHandler(function(result) {
$interval.cancel(interimCacheCheck);
handleSomeAsynchronousOperation(result);
}).getCacheKey(interimCacheKey, false);
}, 1000, 600); // Check result once per second for 10 minutes
}).someAsynchronousOperation();
Using this approach you could also report progress, and only cancel your check after the progress reaches 100%. You'd want to eliminate the interval expiry in that case.

How does loop a lot of URL request work?

For example:
for(var i = 0; i < 100000000; i++) {
requestify.request('http://www.domain.com/' + i)
.then(function() { // do something; } );
}
So, how does NodeJS handle this type of code? It will try to send 100000000 requests or it will request a few request at the same time and put the rest in the queue so that when active request has finished, it will load more request from queue and do it?
I want to ask this question because I run into similiar problem. I load a database of 1,000,000 records of URL that I will later on make a request on each of that URL. So, I don't want my program to hang because it try to request so much at the same time.

Node will fire these requests as fast as it can, it will not wait one to finish and then fire another one.
Javascript is asynchronous, if you want to control the flow you can use async module.

HTTP calls in loop in a meteor app stops after a few calls

Trying to automate sending mails to a queue of email ids stored in collection email. I am trying meteor for the first time so please pardon my lack of understanding if I have any.
I am using the following code (on isServer ) :
Meteor.methods({
'sendEmails': function () {
this.unblock();
Emails.find({status: "no"}).forEach(function (obj) {
var result = Meteor.http.call("GET", "http://someapidomain/email.php?email=" + obj.email);
console.log(result.content);
});
}
});
This code is called at Meteor.startup.
When this app is run, api is called and i get the results for 13 emails, sometimes for 5 emails, sometimes 2 emails and then nothing happens. Please help.
Let me know if more detail is required.

I would suggest doing
Emails.find({status: "no"}).fetch().forEach(...)
Note the fetch() in the chain, which ensures that all Mongo communication has completed prior to doing HTTP calls. find() alone returns a reactive cursor, which means you're mixing Mongo activity and HTTP activity which might not be playing nicely - just a theory.
Reference: http://docs.meteor.com/#/basic/Mongo-Collection-find