We Keep Coding

nodejs looping through array with async calls?

I am trying to iterate through an array which pushes a new Thing to a list, inside the Thing it does some async calls of its own. How would I iterate through an array in a synchronous way, as the callback requires the the data from the list to work. Since my for loop is synchronous and does some asynchronous calls, the callback is called before the the list if finished being made.
I do not know how I would iterate through the array and do all the work before doing the callback
load(file, callback) {
fs.readFile(file, (err, fd) => {
var data = JSON.parse(fd);
for(var i of data.array){
this.list.push(new Thing(i.id)); // new Thing is Asynchronous
}
callback(); // Needs a finished list
return;
});
}
Solved it:
By converting my Thing class to synchronous, by removing the asynchronous calls to a function inside the class, and first instantiating all the Things in the loop then calling Promise.all calling the function I solved the issue:
load(file, callback) {
fs.readFile(file, (err, fd) => {
var data = JSON.parse(fd);
for(var i of data.array){
this.list.push(new Thing(i.id));
}
Promise.all(this.list.map(i => i.load()).then(callback);
});
}

You'd have to have some state inside of Thing to track it's doneness for example you could have an instance variable that's a promise. So given this hacked together example of Thing
class Thing {
constructor(id) {
this.id = id;
this.done = new Promise((resolve, reject) => {
asyncThingWithCallback((err, data) {
if (err) {
this.asyncVal = null;
reject(err);
} else {
this.asyncVal = data;
resolve()
}
})
});
}
}
You can use the done property inside of your callback like this:
load(file, callback) {
fs.readFile(file, (err, fd) => {
var data = JSON.parse(fd);
for(var i of data.array){
this.list.push(new Thing(i.id)); // new Thing is Asynchronous
}
Promise.all(this.list.map((thing) => thing.done))
.then(callback)
});
}

First off, it's generally not advisable to have a constructor that needs some asynchronous operation to finish creating a valid object. That just doesn't lead to easily writable or maintainable code because the constructor has to return the object reference and because the operation is asynchronous, it has to return that object reference before you're done creating a valid object. That just leads to messy, partially created objects. You can make it work by requiring a completion callback be passed to the constructor and making sure the calling code does not attempt to use the object until after the completion callback has been called, but this is just not a clean way to do things. It also makes it impossible to have your async operation return a promise (which is the future of async design) because the constructor has to return the object reference so it can't return a promise.
You could embed the promise in the object, but that's messy too because the promise is only really useful during the initial async operation.
What is often done instead is to make the constructor be only synchronous and then have a .init() method that does the async parts. That makes for cleaner code and is compatible with implementations using promises.
Or, you can create a factory function that returns a promise that resolves to the object reference.
Second off, as you already seem to know, your for loop runs synchronously. It doesn't "wait" for any async operations inside it to complete before going onto the next part of the loop. As long as each invocation of the loop is separate and doesn't depend upon the prior iteration, that's all fine. All you need to know is when all the async operations in the loop are done and making your async operations return promises and using Promise.all() is generally the best tool for that.
So, let's supposed you use the .init() method scheme where .init() does the async part of the initialization and the constructor is synchronous and .init() returns a promise. Then, you could do this:
// create all the things objects
let things = data.array.map(i => new Thing(i.id));
// initialize them all asynchronously
Promise.all(things.map(item => {
return item.init();
})).then(function() {
// all things are asynchronously initialized here
});
Or, using the concept of a factory function that returns a promise that resolves to the object:
function newThing(i) {
let o = new Thing(i.id);
return o.init().then(function() {
// resolve to the object itself
return o;
});
}
Promise.all(data.array.map(i => newThing(i))).then(things => {
// all things in the array ready to be used here
});
If you need to sequence your array iteration so the 2nd iteration did not start until the async part of the first iteration was done and 3rd waited until the 2nd iteration was done and so on, then you can't use a for loop because it simply doesn't work that way. There are several different ways to do such a serialized async iteration. You can see several different schemes in these other posts:
How to synchronize a sequence of promises?
JavaScript: Perform a chain of promises synchronously
ES6 Promises - something like async.each?
How can I execute shell commands in sequence?

You can use primise.all to run all the promises after the for loop .Then you can resolve the promise.all .
load(file) {
fs.readFile(file).Then(function (fd){
var data = JSON.parse(fd);
var EachPromise = [ ];
for(var i of data.array){
EachPromise.push(new Thing(i.id)); // new Thing is Asynchronous
}
Promise.all(EachPromise) .then(function (result){
console.log('this is result',result);
}).Catch(function (error){
console.log('this is error', error);
});
}

JavaScript pattern providing optional callback interface but use promise inside

I want to create a function, let's say a readConfig function.
readConfig(path [, callback(err, config)])
The callback here is optional. If there is no callback when called, the function will only use synchronous methods (readFile(path) and configParser(string) which return promises) to read the file and directly return the parsed config object. Exceptions are thrown directly. However when callback is provided, the function will use asynchronous methods and call the callback after finished. Exceptions are directly raised to callback but not thrown.
There are many similar codes to do sync and async work, so I want to use one method for both. How can a promise like function detect an async or sync call according to the caller's callback argument? And how can we ensure that the promise will act in synchronous way? Please show me a pattern for that.

I can't speak to the promise acting in a synchronous way, but in JavaScript you can do arguments.length to get the number of arguments passed to a function. If that equals 2 and typeof arguments[1] === 'function', then you have a second argument that is a function.
As for personal taste, I'd recommend just having 2 different functions, one of them ending in Sync, à la http://nodejs.org/api/fs.html. This module is replete with function pairs where one is async and the other isn't.
Also, remember that this is not async code:
function notAsync(cb) {
cb(null)
}
That callback executes in the same tick. You need to wrap it like so:
function async(cb) {
process.nextTick(function(){ cb(null)})
}
or something like that.
Whatever you decide, happy coding.

In this example, I'm using jQuery's deferred api. I wrap the callback in .when which allows you to attach a .then() to the callback, whether it is synchronous or asynchronous with a promise.
The solution should be to use your promise api's when. http://howtonode.org/promises
Live demo (click).
$('#sync').click(function() {
foo(function() {
});
});
$('#async').click(function() {
foo(function() {
var deferred = new $.Deferred();
setTimeout(function() {
deferred.resolve();
}, 500);
return deferred.promise();
});
});
function foo(callback) {
$.when(callback()).then(function() {
console.log('done!');
});
}

Determine whether a method is synchronous or asynchronous

In node.js, is it possible to determine (using a function) whether a method is synchronous or asynchronous?
I'd like to write a function that does the following:
function isSynchonous(methodName) {
//if the method is synchronous, return true. Otherwise, return false.
}

From a language standpoint this is not possible, which I believe llambda's answer proves.
Functions can do things asynchronously but return something synchronously; say, return the number of async tasks that were fired off.
Functions can synchronously return promises... that represent asynchronous information. I would call such a method asynchronous but the language can't really tell that.
In some perverse sense every asynchronous function "returns" something... undefined if nothing else.
From an engineering standpoint:
read the documentation.
If the function accepts a callback, it is likely asynchronous. Look at the function signature.
Read the code.
Use "common sense." If the function does IO and returns a result from IO it must be, in some case, asynchronous. This includes file reading, reading from standard input, saving to a database, and HTTP/AJAX requests. Note streams are often used for this, which represents an asynchronous task, but the callbacks are special.
Furthermore there are functions that mix the two.
function(callback) {
if(ready) {
callback();
}
else {
setTimeout(callback, 5000);
}
}
Arguably this is very evil, and correct practice would be
if(ready) {
process.nextTick(callback);
}
so the function has uniform behavior.
However there is a hacky way to tell if anything asynchronous happened, at least in Node.js. See this discussion.
// untested!! please read the documentation on these functions before using yourself
var work = process._getActiveHandles().length + process._getActiveCallbacks().length;
foo;
var newWork = (process._getActiveHandles().length + process._getActiveCallbacks().length) - work;
if(newWork > 0) {
console.log("asynchronous work took place.");
}
This works because asynchronous work cannot resolve on the same tick, by definition, and because Node.js is single threaded.

You don't necessarily know. A function could even be randomly synchronous or asynchronous.
For example, a function that takes another function could execute that function immediately, or it could schedule to execute it at a later time using setImmediate or nextTick. The function could even randomly choose to call its passed function synchronously or asynchronous, such as:
console.log('Start')
function maybeSynchOrAsync(fun) {
var rand = Math.floor((Math.random() * 2))
if (rand == 0) {
console.log('Executing passed function synchronously')
fun()
console.log('Done.')
} else {
console.log('Executing passed function asynchronously via setImmediate')
setImmediate(fun)
console.log('Done.')
}
}
maybeSynchOrAsync(function () { console.log('The passed function has executed.') });
Further, technically speaking, every function call is synchronous. If you call a function F, and F queues a callback function to be invoked later (using setTimeout or whatever mechanism), the original function F still has a synchronous return value (whether it's undefined, a promise, a thunk, or whatever).

No, that's impossible. The methods aren't just marked synchronous or asynchronous, they either use callbacks or they don't.

Why exactly do you need to know, is the real question.
Having said that; that's possible with the new JS abstractions. Nowadays for the async functions those explicitly defined by a preceding async keyword you can perform a test like;
async function test(){}
var type = Object.getPrototypeOf(test).constructor.name;
console.log(type); // <- 'AsyncFunction'
cool..!
As for the normal functions those happen to return a promise, we must remember that it's just a sync function returning a promise object right away, synchronously. That means you have to check dynamically if a function returns a promise in advance, before invoking the function. I think that takes you to a type level programming adventure which exists in sophisticated languages like Haskell etc.

Is "if ( somefunction( ) )" blocking in node.js?

test = function(x){
if ( some conditions ) { return true; }
else { return false; }
}
if (test(y)) {document.write("You have done this before!")​}​;​
console.log("Checked!");
The intention is to check if the user performed some action in the past. These are just mock up codes that do not really reflect what I am actually doing though.
Question:
I am relatively new to node.js so please forgive me if this sounds trivial. Suppose test(y) is true. Can I be sure that the console.log will be executed after the document.write? Even if test(y) takes a long time to run?
In other words, I need "if (test(y))..." to be blocking. I understand that passing a function as an argument, e.g. setInterval(test(y),100); can be async and non-blocking. But what about "if(test(y))..."?

NodeJS has both synchronous (blocking) and asynchronous (non-blocking) functions. (More accurately: The functions themselves are always "blocking" but a whole class of them start something that will complete later and then return immediately, not waiting for the thing to finish. Those are what I mean by "non-blocking.")
The default in most cases is asynchronous (and they accept a callback they call when the thing they've started is done); the synchronous ones tend to have names ending in Sync.
So for example, exists is asynchronous (non-blocking), it doesn't have a return value and instead calls a callback when it's done. existsSync is synchronous (blocking); it returns its result rather than having a callback.
If test is your own function, and it only calls synchronous functions, then it's synchronous:
function test(x) { // Obviously a nonsense example, as it just replicates `existsSync`
if (existsSync(x)) {
// The file exists
return true;
}
return false;
}
// This works
if (test("foo.txt")) {
// ...
}
If it calls an asynchronous function, it's asynchronous, and so it can't return a result via a return value that can be tested by if:
// WRONG, always returns `undefined`
function test(x) {
var retVal;
exists(x, function(flag) {
retVal = flag;
});
return retVal;
}
Instead, you have to provide a callback:
function test(x, callback) {
exists(x, function(flag) {
callback(flag);
});
}
// This works
test("foo.txt", function(flag) {
if (flag) {
// ...
}
});

Yes, this code is synchronous executed and "blocking". But console.log will be executed every time the script runs because you only omitting the document.write in the if statement.

The blocking/non-blocking terminology is a bit confusing here, I would say 'all functions in Node.JS are blocking and all IO in node standard library is non-blocking, except when indicated explicitly by Sync suffix in function name'.

How are node.js callbacks asynchronous?

I have recently started with node.js. It quotes that node.js asynchronous behaviour can be used in three ways events, stream, callback.
Events and stream work on emit,pipe event making it truly async, but how is callback async, as it executes before the return of the function unless process.nextTick() is used.
Events:
event.on('data',function(data){
});
Callback:
function(data,cb){
// do something with "data"
cb();
return;
}

Let's take a closer look at your function that has a callback:
function(data, cb) {
// do something with "data"
cb();
return;
}
You have 2 big errors here:
1) You have no reason to use return when you have a callback function. Return is used only when you have synchronous actions that send a response back right away (they do not have to wait for a callback to be triggered sometime in the future).
2) You don't execute the callback immediately, it doesn't make any sense. You either don't use a callback and use a return statement (when you don't really have an async function) or you execute the callback sometime in the future, when you get the result.
Better example:
function (data, cb) {
// you make an Ajax call and
// you don't know when the callback will be triggered
makeAjaxCall(data, cb);
}

You're correct in that callbacks don't necessarily mean asynchronous I/O.
Events aren't necessarily asynchronous either, in the sense that EventEmitter's .emit works immediately, not on the next tick:
var EventEmitter = require('events').EventEmitter;
var emitter = new EventEmitter();
emitter.on('ev', function(x) {
console.log("received event ", x);
});
console.log("Hello");
emitter.emit('ev', "X");
console.log("World");
Result:
Hello
received event X
World
However, callbacks can be made "async" by:
using process.nextTick(). This doesn't make the actual I/O operations inside the callback async. It just defers execution until the next tick. If some operation blocks, the event loop will also block - on the next tick instead of this tick.
function(data, cb) {
process.nextTick(function() { doSomethingWith(data); cb(); })
});
calling the specified callback function from an event that is known to be triggered by an async I/O operation
function(data, cb) {
emitter.startIO();
emitter.on('someIOEvent', function(e) {
doSomethingWith(data,e);
cb();
});
});
using another callback-based function known to be async. Most functions in node core and node's modules are like this.
function(data, cb) {
otherFunction(data, function(moredata) {
doMoreStuffWith(moredata, data); cb();
});
});

In this case, the function with the callback is not being run asynchronously, however there are still cases where such functions can be used asynchronously:
function foo(data, callback) {
// do something with the data
callback();
}
function bar() {
// this will be called as a callback from foo()
}
setInterval(function() { foo(data, bar) }, 1000);
console.log('this is displayed immediately');
You can see that foo() is scheduled to run every second, and that happens asynchronously; however the existence of the callback allows you to set up extra behavior from outside the foo() function, e.g. you might set up several cases where foo() is called with different callbacks, resulting in different application logic.