We Keep Coding

Event loop blocking and async programming in Node JS

I am newbie to Node js programming and hence want to understand the core concepts and practises very appropriately. AFAIK node js has non-blocking I/O allowing all disk and other I/O operations running async way while its JS runs in single thread managing the resource and execution paths using Event Loop. As suggested at many places developers are advised to write custom functions/methods using callback pattern e.g.
function processData(inputData, cb){
// do some computation and other stuff
if (err) {
cb(err, null);
}else{
cb(null, result);
}
}
callback = function(err, result){
// check error and handle
// if not error then grab result and do some stuff
}
processData(someData, callback)
// checking whether execution is async or not
console.log('control reached at the end of block');
If I run this code everything runs synchronously printing console message at last. What I believed was that the console message will print first of all followed by execution of processData function code which will in turn invoke the callback function. And if it happens this way, the event loop will be unblocked and would only execute the response to a request when the final response is made ready by 'callback' function.
My question is:- Is the execution sequence alright as per node's nature Or Am I doing something wrong Or missing an important concept?
Any help would be appreciate from you guys !!!

JavaScript (just as pretty much any other language) runs sequentially. If you write
var x = 1;
x *= 2;
x += 1;
you expect the result to be 3, not 4. That would be pretty bad. The same goes with functions. When you write
foo();
bar();
you expect them to run in the very same order. This doesn't change when you nest functions:
var foo = function() {};
var bar = function(clb) {
clb();
foo();
};
var callback = function() {
};
bar(callback);
The expected execution sequence is bar -> callback -> foo.
So the biggest damage people do on the internet is that they put equals sign between asynchronous programming and callback pattern. This is wrong. There are many synchronous functions using callback pattern, e.g. Array.prototype.forEach().
What really happens is that NodeJS has this event loop under the hood. You can tell it to schedule something and run later by calling multiple special functions: setTimeout, setInterval, process.nextTick to name few. All I/O also schedules some things to be done on the event loop. Now if you do
var foo = function() {};
var bar = function(clb) {
procress.nextTick(clb); // <-- async here
foo();
};
var callback = function() {
};
bar(callback);
The expected execution sequence is this bar -> (schedule callback) -> foo and then callback will fire at some point (i.e. when all other queued tasks are processed).
That's pretty much how it works.

You are writing synchronous code all the way down and expecting it to run asynchronously. Just because you are using callbacks does not mean it's an asynchronous operation. You should try using timers, api within your functions to feel node js non blocking asynchronous pattern. I hope The Node.js Event Loop will help you to get started.

Callbacks and returning values

I'm stuck with two external libraries and using them together. One only has an async interface and the other requires me to return a value from the callback.
// this function is called as a callback from an external library
// it is required that the function returns a certain value
function onevent()
{
// the callback for the "thirdparty.foo" function
// with the result that must be returned from onevent
function ondone(result) {
// ???
}
// this is a call to a third party library that only has this asynchronous interface
thirdparty.foo(ondone);
// ??? return result;
}
JavaScript is pretty new for me and I was unable to find any solution for this problem.
I'm looking to something to be able to wait for the completion of ondone before returning from onevent. Comparable from C++ would be .wait or .get from a std::future.

One only has an async interface and the other requires me to return a value from the callback.
That's a problem if you have to use the one providing an async interface to satisfy a callback from the one that doesn't. If you do, you simply cannot use them together. There is no way to make the callback from the one expecting a value returned to it by the callback wait for an asynchronous event to complete.
E.g.:
doSomethingExpectingReturnValueFromCallback(function() {
// You cannot use an asynchronous API here to get the value you return
});
If the synchronous lib understood promises (similar to std::future), then you could return a promise, but it doesn't sound like it does.

AJAX asynch callback working correctly but how do I wait for the returned values

My main EXTJS function is calling three different AJAX asynch functions.The good news is that each AJAX function is successfully returning a boolean callback on whether the AJAX GET call was successful or not.
The problem - I need to perform additional post business logic from my main calling function once the boolean results have been returned from all three AJAX functions.
But as it is async the code will not wait for the async functions to complete and will fly through.
Any ideas on how I can get my code to GRACEFULLY wait until the typeof status is no longer 'undefined' ?
I am not near my code right now but here is a rough approximation.
function main(){
var status1, status2, status3, result1, result2,result3;
thisIsAsynchFunction1(args,function(result1){
status1 = result1;
});
thisIsAsynchFunction2(args,function(result1){
status2 = result1;
});
thisIsAsynchFunction1(args,function(result1){
status3 = result1;
});
// Perform logic based on callback from functions above
}
Thank You.

You cannot make Javascript wait. It simply doesn't work that way. What you can do is to trigger your final action when you discover that all three async results are done. Without redesigning your code to use promises (which would be the modern way to do this), you could do something like this:
function main(callbackFn){
var status1, status2, status3, doneCnt = 0;
thisIsAsynchFunction1(args,function(result1){
status1 = result1;
++doneCnt;
checkDone();
});
thisIsAsynchFunction2(args,function(result1){
status2 = result1;
++doneCnt;
checkDone();
});
thisIsAsynchFunction1(args,function(result1){
status3 = result1;
++doneCnt;
checkDone();
});
// Perform logic based on callback from functions above
function checkDone() {
// if all three results are done, then call the callback
if (doneCnt === 3) {
callbackFn(status1, status2, status3);
}
}
}
// call main and pass it a callback function
main(function(s1, s2, s3) {
// all three async results are available here
});
// Be careful because code here will execute BEFORE the async results are done
// any code that must wait for the async results must be in the above callback
A newer type of approach would be to have each async operation return a promise and when each async operation completes it would resolve its promise with its result. Then, you could use a promise library function (often called .when() to trigger a callback when all three promises were done. Many modern day JS libraries already return promises from ajax calls (such as jQuery) so the promise is already there.
I don't know ExtJS myself, but looked in its documentation and didn't see promise support yet and I found one thread discussing the fact that promises were not going to be in version 5. So, perhaps you just go with the counter design above or maybe ExtJS has some of its own support for monitoring when multiple async operations are all done.

How can I invoke a callback given to a yield in javascript v8 generator code?

I'm attempting to understand javascript generators in node.js v8, without using any third party libraries.
I want to try having a generator that invokes an asynchronous callback (something like a web request) and returns the value that is called back.
My code (using Node v0.11.12) looks like:
var callbackWrapper = function (){
return function(callback) {
callback(null, 5);
};
};
var result = null;
var generator = (function *(){
result = yield callbackWrapper();
})();
var nextIteration = generator.next();
At the end of this code, nextIteration.value is a function, and result is still null.
It was my understanding that if yield is given a method with a callback, it invokes that callback.
My goal is to somehow get the value, 5, that is called back.
What is the way to do this? Do I have to pass some function to nextIteration.value and invoke it?

Let's clarify some things.
It was my understanding that if yield is given a method with a callback, it invokes that callback.
yield does not invoke anything. The purpose of yield is, well, to yield... When you define a generator you define an entity which yields values. It has nothing to do with async code. We can, however, leverage an important property of generators to handle async code.
Generators, by their definition, yield values. Here's the important part - between one yield to another the generator is suspended. In other words, it waits until it is asked to yield the next value. The internal state of the generator is kept in memory (on the stack) until the generator is exhausted (no more values to yield).
Another important property of generators is that we can send values back, thus changing their internal state.
So if we can suspend generators (make them wait) and we can send values back; we can essentially make them wait for some async operation to complete and then send the result back.
What is the way to do this? Do I have to pass some function to nodeIteration.value and invoke it?
Basically you need to wrap you code with a generator. Every time you start an async operation you make the generator wait by using yield. When the async operation completes you resume your generator by sending the result back with next(result).
I wrote an extensive post to explain this issue, which I'm sure will help you understand. Take a look: http://eyalarubas.com/javascript-generators-and-callbacks.html

Generators don't handle node style callbacks on their own. Instead it's returning the function that you wrapped inside of the callbackWrapper thunk. As yield only returns a value and then pauses execution at that point in time.
Generators weren't really designed for control flow but you can build on top of them to create control flow libraries like co, suspend, etc..
Basically what these libraries do (I'm oversimplifying here), is take your generator function and recursively call it until it tells them that it has finished.
Each of these libraries handles the internal yields in different ways, for example co turns everything it can handle into thunks internally. While suspend uses node-style callbacks for everything internally.
At each yield they check to see what was yielded to them a thunk, promise, generator, or whatever constructs that library handles, and abstracts the control out based on when they are completed.
You can build a structure around generators to handle asynchronous thunked functions but it's not for the feint of heart. Basically it would go something like this (Note: don't use this other than for playing around as its missing all the normal checks, error handling, etc..):
function controlFlow(genFunc){
// check to make sure we have a generator function otherwise explode
var generator; // reference for an initialized generator
// return a funcion so that execution time can be postponed
return function(){
runGen(genFunc());
}
function runGen(generator){
var ret = generator.next();
// here the generator is already finished we we'll return the final value
if(ret.done){
return ret.value
}
// here we'll handle the yielded value no matter what type it is
// I'm being naive here for examples sake don't do this
if(typeof ret.value === 'function'){
// oh look we have a regular function (very faulty logic)
// we wouldn't do this either but yeah
ret.value(function(err, result){
console.log(result);
});
}
// oh we got another item like an array or object that means parallelism or serialization depending on what we got back
// turn array, object, whatever into callback mechanisms and then track their completion
// we're just going to fake it here and just handle the next call
runGen(generator);
}
}
function thunked(callback){
return function(){
callback(null, 5);
};
};
function regular(callback){
console.log('regular function call');
callback(null, 'value');
};
controlFlow(function *(){
yield thunked(function(err, result){
console.log(err);
console.log(result);
});
yield regular;
yield thunked(function(err, result){
console.log('Another Thunked');
});
yield regular(function(err, result){
console.log(err);
console.log(result);
});
})();

result won’t get assigned until you send a value back to the generator by calling next again with the value you want to assign to result. In your example that would look like this:
nextIteration.value(function (error, value) {
generator.next(value);
});

Determine whether a method is synchronous or asynchronous

In node.js, is it possible to determine (using a function) whether a method is synchronous or asynchronous?
I'd like to write a function that does the following:
function isSynchonous(methodName) {
//if the method is synchronous, return true. Otherwise, return false.
}

From a language standpoint this is not possible, which I believe llambda's answer proves.
Functions can do things asynchronously but return something synchronously; say, return the number of async tasks that were fired off.
Functions can synchronously return promises... that represent asynchronous information. I would call such a method asynchronous but the language can't really tell that.
In some perverse sense every asynchronous function "returns" something... undefined if nothing else.
From an engineering standpoint:
read the documentation.
If the function accepts a callback, it is likely asynchronous. Look at the function signature.
Read the code.
Use "common sense." If the function does IO and returns a result from IO it must be, in some case, asynchronous. This includes file reading, reading from standard input, saving to a database, and HTTP/AJAX requests. Note streams are often used for this, which represents an asynchronous task, but the callbacks are special.
Furthermore there are functions that mix the two.
function(callback) {
if(ready) {
callback();
}
else {
setTimeout(callback, 5000);
}
}
Arguably this is very evil, and correct practice would be
if(ready) {
process.nextTick(callback);
}
so the function has uniform behavior.
However there is a hacky way to tell if anything asynchronous happened, at least in Node.js. See this discussion.
// untested!! please read the documentation on these functions before using yourself
var work = process._getActiveHandles().length + process._getActiveCallbacks().length;
foo;
var newWork = (process._getActiveHandles().length + process._getActiveCallbacks().length) - work;
if(newWork > 0) {
console.log("asynchronous work took place.");
}
This works because asynchronous work cannot resolve on the same tick, by definition, and because Node.js is single threaded.

You don't necessarily know. A function could even be randomly synchronous or asynchronous.
For example, a function that takes another function could execute that function immediately, or it could schedule to execute it at a later time using setImmediate or nextTick. The function could even randomly choose to call its passed function synchronously or asynchronous, such as:
console.log('Start')
function maybeSynchOrAsync(fun) {
var rand = Math.floor((Math.random() * 2))
if (rand == 0) {
console.log('Executing passed function synchronously')
fun()
console.log('Done.')
} else {
console.log('Executing passed function asynchronously via setImmediate')
setImmediate(fun)
console.log('Done.')
}
}
maybeSynchOrAsync(function () { console.log('The passed function has executed.') });
Further, technically speaking, every function call is synchronous. If you call a function F, and F queues a callback function to be invoked later (using setTimeout or whatever mechanism), the original function F still has a synchronous return value (whether it's undefined, a promise, a thunk, or whatever).

No, that's impossible. The methods aren't just marked synchronous or asynchronous, they either use callbacks or they don't.

Why exactly do you need to know, is the real question.
Having said that; that's possible with the new JS abstractions. Nowadays for the async functions those explicitly defined by a preceding async keyword you can perform a test like;
async function test(){}
var type = Object.getPrototypeOf(test).constructor.name;
console.log(type); // <- 'AsyncFunction'
cool..!
As for the normal functions those happen to return a promise, we must remember that it's just a sync function returning a promise object right away, synchronously. That means you have to check dynamically if a function returns a promise in advance, before invoking the function. I think that takes you to a type level programming adventure which exists in sophisticated languages like Haskell etc.