I'm trying to work through this js/async scenario and i'm trying to know how the rest of the js world handles this.
function doStuff(callback) {
cursor.each(function(err, blahblah) {
...doing stuff here takes some time
});
... Execute this code ONLY after the `cursor.each` loop is finished
callback();
EDIT
Here's a more concrete example updated using most of the suggestions below which still doesn't work.
function doStuff(callback) {
MongoClient.connect(constants.mongoUrl, function(err, db) {
var collection = db.collection('cases2');
var cursor = collection.find();
var promises = []; // array for storing promises
cursor.each(function(err, item) {
console.log('inside each'); // NEVER GETS LOGGED UNLESS I COMMENT OUT THIS LINE: return Q.all(promises).then(callback(null, items));
var def = Q.defer(); // Create deferred object and store
promises.push(def.promise); // Its promise in the array
if(item == null) {
return def.resolve();
}
def.resolve(); // resolve the promise
});
console.log('items'); // ALWAYS GETS CALLED
console.log(items);
// IF I COMMENT THIS LINE OUT COMPLETELY,
// THE LOG STATEMENT INSIDE CURSOR.EACH ACTUALLY GETS LOGGED
return Q.all(promises).then(callback(null, items));
});
}
without using promises or any other dependencies/libraries you can simply
function doStuff(callback) {
add a counter
var cursor = new Array(); // init with some array data
var cursorTasks = cursor.length;
function cursorTaskComplete()
{
cursorTasks--;
if ( cursorTasks <= 0 ) {
// this gets get called after each task reported to be complete
callback();
}
}
for ( var i = 0; i < cursor.length; i++ ) {
...doing stuff here takes some time and does some async stuff
check after each async request
...when async operation is complete call
cursorTaskComplete()
}
}
Without knowing the details of the async calls you're making within the cursor.each loop, I shall assume that you have the ability to invoke a callback each time the functions invoked therein have completed their async task:
function doStuff() {
var promises = []; // array for storing promises
cursor.each(function(err, blahblah) {
var def = Q.defer(); // create deferred object and store
promises.push(def.promise); // its promise in the array
call_async_function(..., def.resolve); // resolve the promise in the async function's callback
});
// pass the array to Q.all, only when all are resolved will "callback" be called
return Q.all(promises);
}
and the usage then becomes:
doStuff().then(callback)
Note how the invocation of the callback now never touches the doStuff function - that function now also returns a promise. You can now register multiple callbacks, failure callbacks, etc, all without modifying doStuff. This is called "separation of concerns".
[NB: all the above based on the Q promises library - https://github.com/kriskowal/q]
EDIT further discussion and experimentation has determined that the .each call is itself async, and gives no indication to the outside when the last row has been seen. I've created a Gist that demonstrates a resolution to this problem.
if you want to do it with the async module, you can make use of the async forEachSeries function
Code snippet:
function doStuff(callback) {
async.forEachSeries(cursor, function(cursorSingleObj,callbackFromForEach){
//...do stuff which takes time
//this callback is to tell when everything gets over execute the next function
callbackFromForEach();
},function(){
//over here the execution of forEach gets over and then the main callback is called
callback();
});
}
In my mind an elegant/ideal solution would be to have something like
cursor.each(........).then( function() { ....your stuff});
But without that you can do this....UPDATED
http://plnkr.co/edit/27l7t5VLszBIW9eFW4Ip?p=preview
The gist of this is as shown below...notice....when
var doStuff = function(callback) {
cursor.forEach(function(cursorStep) {
var deferred = $q.defer();
var promise = deferred.promise;
allMyAsyncPromises.push(promise);
cursorStep.execFn(cursorStep.stepMeta);
promise.resolve;
});
$q.when(allMyAsyncPromises).then(callback);
}
After hitting the start button wait for few seconds...the async tasks have been simulated to finish in 5 seconds so the status will update accordingly.
Not having access to a real cursor object..I had to resort of fake cursor like and array.
Related
Let's say I have multiple functions func1, func2, func3, etc.....
And they all contain an AJAX/async function within them:
function funcX(){
// some ajax request
}
If in a main function I am calling func1, func2, func3 sequentially like so:
$(document).ready(function(){
func1();
func2();
func3();
...
}
Will each ajax/async function's call be certain to execute in the order of their parent functions? At first I thought they might be, but the behavior of my program seems to be suggesting otherwise...
If not, is there a good (hopefully simple?) alternative to having a long chain of callbacks?
Will each ajax/async function's call be certain to execute in the order of their parent functions?
They should execute in order, but their internal callbacks can be called in any order.
If not, is there a good (hopefully simple?) alternative to having a long chain of callbacks?
You could use a promise, and execute the next function when the promise has been resolved.
This example uses jQuery...
var fn1 = function () {
var d = $.Deferred();
setTimeout(function () {
$("body").text("Callback 1 done.") && d.resolve();
}, Math.random() * 1300 + 800);
return d.promise();
};
var fn2 = function () {
var d = $.Deferred();
setTimeout(function () {
$("body").text("Callback 2 done.") && d.resolve();
}, 500);
return d.promise();
};
$.when(fn1(), fn2()).then(function () {
setTimeout(function () {
$("body").text("All done.");
}, 300);
});
jsFiddle.
We use $.when() and pass the invoked functions we want to execute to it. We then use then() to show a final message (I placed a setTimeout() here so you can see the last resolved function's message in the document).
Each of these functions have their own deferred object which return the promise. A setTimeout() mocks an XHR for example's sake. When this callback is executed, we resolve the deferred object.
Once both have been deferred, we reach the callback for then().
To serialize tasks, I've written a helper function, which can also be found in my earlier answer:
function serializeTasks(arr, fn, done)
{
var current = 0;
fn(function iterate() {
if (++current < arr.length) {
fn(iterate, arr[current]);
} else {
done();
}
}, arr[current]);
}
It takes an array of values (in your case those are actually functions), a loop function and a completion handler. Below is the loop function:
function loopFn(nextTask, fn)
{
fn(nextTask);
}
It accepts an intermediate completion function as the first argument and each element of the aforementioned array.
To set everything in motion:
serializeTasks([func1, func2, func3], loopFn, function() {
console.log('all done');
});
Your functions are called with a single argument which should be passed to the AJAX success callback, e.g.
func1(nextTask)
{
$.ajax({
...,
success: nextTask
});
}
The order in which the asynch results are returned is not deterministic, and may wary every time.
func2 might complete before func1 etc
It is important to ensure correct order of execution. One pattern is to call the next function in the success callback of the prior function
Ex:
$.get("/someUrl",function(){
$.get("/nextAjaxCall", function(){
.....
});
});
If the dependency chain is very simple, I don't think it's necessary to introduce a framework to handle this
Or look at async library and it's awesomeness !
async
I've been developing in JavaScript for quite some time but net yet a cowboy developer, as one of the many things that always haunts me is synching JavaScript's callbacks.
I will describe a generic scenario when this concern will be raised: I have a bunch of operations to perform multiple times by a for loop, and each of the operations has a callback. After the for loop, I need to perform another operation but this operation can only execute successfully if all the callbacks from the for loop are done.
Code Example:
for ... in ... {
myFunc1(callback); // callbacks are executed asynchly
}
myFunc2(); // can only execute properly if all the myFunc1 callbacks are done
Suggested Solution:
Initiate a counter at the beginning of the loop holding the length of the loop, and each callback decrements that counter. When the counter hits 0, execute myFunc2. This is essentially to let the callbacks know if it's the last callback in sequence and if it is, call myFunc2 when it's done.
Problems:
A counter is needed for every such sequence in your code, and having meaningless counters everywhere is not a good practice.
If you recall how thread conflicts in classical synchronization problem, when multiple threads are all calling var-- on the same var, undesirable outcomes would occur. Does the same happen in JavaScript?
Ultimate Question:
Is there a better solution?
The good news is that JavaScript is single threaded; this means that solutions will generally work well with "shared" variables, i.e. no mutex locks are required.
If you want to serialize asynch tasks, followed by a completion callback you could use this helper function:
function serializeTasks(arr, fn, done)
{
var current = 0;
fn(function iterate() {
if (++current < arr.length) {
fn(iterate, arr[current]);
} else {
done();
}
}, arr[current]);
}
The first argument is the array of values that needs to be passed in each pass, the second argument is a loop callback (explained below) and the last argument is the completion callback function.
This is the loop callback function:
function loopFn(nextTask, value) {
myFunc1(value, nextTask);
}
The first argument that's passed is a function that will execute the next task, it's meant to be passed to your asynch function. The second argument is the current entry of your array of values.
Let's assume the asynch task looks like this:
function myFunc1(value, callback)
{
console.log(value);
callback();
}
It prints the value and afterwards it invokes the callback; simple.
Then, to set the whole thing in motion:
serializeTasks([1,2, 3], loopFn, function() {
console.log('done');
});
Demo
To parallelize them, you need a different function:
function parallelizeTasks(arr, fn, done)
{
var total = arr.length,
doneTask = function() {
if (--total === 0) {
done();
}
};
arr.forEach(function(value) {
fn(doneTask, value);
});
}
And your loop function will be this (only parameter name changes):
function loopFn(doneTask, value) {
myFunc1(value, doneTask);
}
Demo
The second problem is not really a problem as long as every one of those is in a separate function and the variable is declared correctly (with var); local variables in functions do not interfere with each other.
The first problem is a bit more of a problem. Other people have gotten annoyed, too, and ended up making libraries to wrap that sort of pattern for you. I like async. With it, your code might look like this:
async.each(someArray, myFunc1, myFunc2);
It offers a lot of other asynchronous building blocks, too. I'd recommend taking a look at it if you're doing lots of asynchronous stuff.
You can achieve this by using a jQuery deferred object.
var deferred = $.Deferred();
var success = function () {
// resolve the deferred with your object as the data
deferred.resolve({
result:...;
});
};
With this helper function:
function afterAll(callback,what) {
what.counter = (what.counter || 0) + 1;
return function() {
callback();
if(--what.counter == 0)
what();
};
}
your loop will look like this:
function whenAllDone() { ... }
for (... in ...) {
myFunc1(afterAll(callback,whenAllDone));
}
here afterAll creates proxy function for the callback, it also decrements the counter. And calls whenAllDone function when all callbacks are complete.
single thread is not always guaranteed. do not take it wrong.
Case 1:
For example, if we have 2 functions as follows.
var count=0;
function1(){
alert("this thread will be suspended, count:"+count);
}
function2(){
//anything
count++;
dump(count+"\n");
}
then before function1 returns, function2 will also be called, if 1 thread is guaranteed, then function2 will not be called before function1 returns. You can try this. and you will find out count is going up while you are being alerted.
Case 2: with Firefox, chrome code, before 1 function returns (no alert inside), another function can also be called.
So a mutex lock is indeed needed.
There are many, many ways to achieve this, I hope these suggestions help!
First, I would transform the callback into a promise! Here is one way to do that:
function aPromise(arg) {
return new Promise((resolve, reject) => {
aCallback(arg, (err, result) => {
if(err) reject(err);
else resolve(result);
});
})
}
Next, use reduce to process the elements of an array one by one!
const arrayOfArg = ["one", "two", "three"];
const promise = arrayOfArg.reduce(
(promise, arg) => promise.then(() => aPromise(arg)), // after the previous promise, return the result of the aPromise function as the next promise
Promise.resolve(null) // initial resolved promise
);
promise.then(() => {
// carry on
});
If you want to process all elements of an array at the same time, use map an Promise.all!
const arrayOfArg = ["one", "two", "three"];
const promise = Promise.all(arrayOfArg.map(
arg => aPromise(arg)
));
promise.then(() => {
// carry on
});
If you are able to use async / await then you could just simply do this:
const arrayOfArg = ["one", "two", "three"];
for(let arg of arrayOfArg) {
await aPromise(arg); // wow
}
// carry on
You might even use my very cool synchronize-async library like this:
const arrayOfArg = ["one", "two", "three"];
const context = {}; // can be any kind of object, this is the threadish context
for(let arg of arrayOfArg) {
synchronizeCall(aPromise, arg); // synchronize the calls in the given context
}
join(context).then(() => { // join will resolve when all calls in the context are finshed
// carry on
});
And last but not least, use the fine async library if you really don't want to use promises.
const arrayOfArg = ["one", "two", "three"];
async.each(arrayOfArg, aCallback, err => {
if(err) throw err; // handle the error!
// carry on
});
I have a handler (callback), an object to handle and four functions, which collect the data to object. In my case I wish to asynchronously call four data retrievers and when execution of all four is complete, handle the resulting object (something similar to the following):
var data = {};
function handle (jsObj) {}
// data retrieving
function getColorData () {}
function getSizeData () {}
function getWeightData () {}
function getExtraData () {}
data.color = getColorData();
data.size = getSizeData();
data.weight = getWeightData();
data.extra = getExtraData();
handle( data );
Of course, this code will not work properly. And if I chain data retrieving functions, they will be called one after another, right?
All four functions should be called asynchronously, cause they are being executed for too long to call them one by one.
Updated:
Thanks to everybody for your suggestions! I prefered $.Deferred(), but I found it slightly difficult to make it work the way I need. What I need is to asynchronously make a view, which requires four kinds of data (extraData, colorData, sizeData & weightData) and I have three objects: App, Utils & Tools.
Just a small description: view is created by calling App.getStuff passed App.handleStuff as a callback. Callback in the body of App.getStuff is called only $.when(App.getExtraData(), App.getColorData(), App.getSizeData(), App.getWeightData()). Before that Utils.asyncRequest passed Tools.parseResponse as a callback is called.
So, now the question is should I create four deferred objects inside each App.get*Data() and also return deferred.promise() from each of them?
And should I deferred.resolve() in the last function in my order (Tools.parseResponse for App.getExtraData in my example)?
var view,
App,
Utils = {},
Tools = {};
// Utils
Utils.asyncRequest = function (path, callback) {
var data,
parseResponse = callback;
// do something with 'data'
parseResponse( data );
};
// Tools
Tools.parseResponse = function (data) {
var output = {};
// do something to make 'output' from 'data'
/* So, should the deferred.resolve() be done here? */
deferred.resolve(output);
/// OR deferred.resolve();
/// OR return output;
};
// App
App = {
// Only one method really works in my example
getExtraData : function () {
var deferred = new jQuery.Deferred();
Utils.asyncRequest("/dir/data.txt", Tools.parseResponse);
return deferred.promise();
},
// Others do nothing
getColorData : function () { /* ... */ },
getSizeData : function () { /* ... */ },
getWeightData : function () { /* ... */ }
};
App.getStuff = function (callback) {
$.when(
App.getExtraData(),
App.getColorData(),
App.getSizeData(),
App.getWeightData()
)
.then(function (extraData, colorData, sizeData, weightData) {
var context,
handleStuff = callback;
// do something to make all kinds of data become a single object
handleStuff( context );
});
};
App.handleStuff = function (stuff) { /* ... */ };
/// RUN
view = App.getStuff( App.handleStuff );
I did not expect the code in my example above to work, it is for illustrative purposes.
I've been trying to solve this for quiet a long time and it still gives no result. The documentation for jQuery.Deferred() and discussions around this, unfortunately, did not help me. So, I would be very glad and greatful for any help or advise.
Conceptually, you would use a counter that gets incremented as each asynchronous call completes. The main caller should proceed after the counter has been incremented by all the asynchronous calls.
I think what you're looking for are Promises / Deferreds.
With promises you can write something like:
when(getColorData(), getSizeData(), getWeightData(), getExtraData()).then(
function (colorData, sizeData, weightData, extraData) {
handle(/*..*/);
}
)
The get*Data() functions will return a promise that they fulfill when their assynchronous call is complete.
Ex:
function getData() {
var promise = new Promise();
doAjax("getData", { "foo": "bar" }, function (result) {
promise.resolve(result);
});
return promise;
}
The when simply counts the number arguments, if all it's promises are resolved, it will call then with the results from the promises.
jQuery has an OK implementation: http://api.jquery.com/jQuery.when/
What I could suggest for this scenario would be something like that.
write a function like this
var completed = 0;
checkHandler = function() {
if(completed == 4) {
handle(data);
}
}
where completed is the number of positive callbacks you must receive.
As soon as every function receives a callback you can increment the "completed" counter and invoke the checkHandler function. and you're done!
in example
function getColorData() {
$.get('ajax/test.html', function(data) {
completed++;
checkHandler();
});
}
I am calling a method that has some logic in a deferred, and when that logic finishes, I want to return the value to the callee. See below:
//Callee.js
var myAssistant = new Assistant();
console.log(myAssistant.whatIsTheValue());
//Assistant.js
whatIsTheValue : function(someArg) {
var deferred = someService.getSomething();
deferred.then(lang.hitch(this, this._getTheValue));
//In theory, I want to return whatever this._getTheValue returns, how can I do that?!
}
_getTheValue() {
...
...
return xyz;
}
Deferreds are asynchronous operations. Therefore you can't return a variable in a normal way from them because they will not execute until after the current function context completes.
If you want to do more with that value you will need to do so in terms of another callback (IE chaining the then statements.)
The point of deferreds is to provide sequential operations for callbacks. So you can chain them to achieve the results you want. If you need the results to be available in your current execution context, you're going to have to find a synchronous (not deferreds) method of doing what you want.
So something like this
//Assistant.js
whatIsTheValue : function(someArg) {
var deferred = someService.getSomething();
var next = deferred.then(lang.hitch(this, this._getTheValue));
next.then(/*insert next function here*/);
}
You need to understand that using a deferred lang.hitch isn't going to execute until after whatistheValue is done operating. So instead of returning the value to whatever function called whatisthevalue, you're going to have to put the logic of processing that value into a new function and use that as an additional callback for your deferred. This will probably require some restructuring of your program.
I do not know what your lang.hitch does, but the solution should look like that:
Assistant.prototype.whatIsTheValue = function(someArg) {
var deferred = someService.getSomething();
return deferred.then(lang.hitch(this, this._getTheValue));
// ^^^^^^
};
var myAssistant = new Assistant();
myAssistant.whatIsTheValue().then(console.log); // use console.log.bind(console) in Chrome
// ^^^^ - it is a promise you return
use JQuery's $when instead.
Example
// assuming both getData and getLocation return their respective Promise
var combinedPromise = $.when(getData(), getLocation())
// function will be called when both getData and getLocation resolve
combinePromise.done(function(data,location){
alert("We got data: " + dataResult + " and location: " + location);
});
http://www.html5rocks.com/en/tutorials/async/deferred/
I need to visit each node in a tree, do some asynchronous work, and then find out when all of the asynchronous work has completed. Here are the steps.
Visit a node and modify its children asynchronously.
When async modifications to children are done, visit all children (which might require async work).
When all asynchronous work for all descendants is done, do something else.
Update:
I ended up using a pattern that looks like a monitor/lock (but isn't) for each node to know when to begin step 2. I used events and attributes to keep track of all descendants of a node to know when to begin step 3.
It works, but man is this difficult to read! Is there a cleaner pattern?
function step1(el) { // recursive
var allDone = false;
var monitor = new Monitor();
var lock = monitor.lock(); // obtain a lock
$(el).attr("step1", ""); // step1 in progress for this node
// fires each time a descendant node finishes step 1
$(el).on("step1done", function (event) {
if (allDone) return;
var step1Descendants = $(el).find("[step1]");
if (step1Descendants.length === 0) {
// step 1 done for all descendants (so step 2 is complete)
step3(el); // not async
allDone = true;
}
});
// fires first time all locks are unlocked
monitor.addEventListener("done", function () {
$(el).removeAttr("step1"); // done with step 1
step2(el); // might have async work
$(el).trigger("step1done");
});
doAsyncWork(el, monitor); // pass monitor to lock/unlock
lock.unlock(); // immediately checks if no other locks outstanding
};
function step2(el) { // visit children
$(el).children().each(function (i, child) {
step1(child);
});
};
Here's an updated version that walks the node-tree, processing each child in the initial root node, and then descends recursively into each child's tree and processes its child nodes and so on.
Here's a jsfiddle demo
// Pass the root node, and the callback to invoke
// when the entire tree has been processed
function processTree(rootNode, callback) {
var i, l, pending;
// If there are no child nodes, just invoke the callback
// and return immediately
if( (pending = rootNode.childNodes.length) === 0 ) {
callback();
return;
}
// Create a function to call, when something completes
function done() {
--pending || callback();
}
// For each child node
for( i = 0, l = rootNode.childNodes.length ; i < l ; i++ ) {
// Wrap the following to avoid the good ol'
// index-closure-loop issue. Pass the function
// a child node
(function (node) {
// Process the child node asynchronously.
// I'm assuming the function takes a callback argument
// it'll invoke when it's done.
processChildNodeAsync(node, function () {
// When the processing is done, descend into
// the child's tree (recurse)
processTree(node, done);
});
}(rootNode.childNodes[i]));
}
}
Original Answer
Here's a basic example you might be able to use... though without the specifics of your problem, it's half psuedo-code
function doAsyncTreeStuff(rootNode, callback) {
var pending = 0;
// Callback to handle completed DOM node processes
// When pending is zero, the callback will be invoked
function done() {
--pending || callback();
}
// Recurse down through the tree, processing each node
function doAsyncThingsToNode(node) {
pending++;
// I'm assuming the async function takes some sort of
// callback it'll invoke when it's finished.
// Here, we pass it the `done` function
asyncFunction(node, done);
// Recursively process child nodes
for( var i = 0 ; i < node.children.length ; i++ ) {
doAsyncThingsToNode(node.children[i]);
}
}
// Start the process
doAsyncThingsToNode(rootNode);
}
It seems the right pattern for this problem and for async work in general is Promises. The idea is that any function that will do asynchronous work should return a promise object, to which the caller can attach functions that should be called when the asynchronous work is completed.
jQuery has a great API for implementing this pattern. It's called a jQuery.Deferred object. Here's a simple example:
function asyncWork() {
var deferred = $.Deferred();
setTimeout(function () {
// pass arguments via the resolve method
deferred.resolve("Done.");
}, 1000);
return deferred.promise();
}
asyncWork().then(function (result) {
console.log(result);
});
Very tidy. What's the difference between a Deferred object and its promise object? Good question.
Here's how you might apply this pattern to solve this problem.
function step1(el) { // recursive
var deferred = $.Deferred();
// doAsyncWork needs to return a promise
doAsyncWork(el).then(function () {
step2(el).then(function () {
step3(el); // not async
deferred.resolve();
});
});
return deferred.promise();
};
function step2(el) { // visit children
var deferred = $.Deferred();
var childPromises = [];
$(el).children().each(function (i, child) {
childPromises.push(step1(child));
});
// When all child promises are resolved…
$.when.apply(this, childPromises).then(function () {
deferred.resolve();
});
return deferred.promise();
};
So much cleaner. So much easier to read.
This is something you would probably prefer to do with threads to continue other work, but since you are using JavaScript you need to work around this with some sort of blocking. One way is make an initially empty list of finished tasks, make the asynchronous calls, and have each call register itself on the list when it is finished. While you are waiting for the calls, enter a loop with a timer, and at each iteration check if the finished tasks list is complete; if so, continue with other tasks. You may want to give up if your loop runs too long.