I have a list of items, which I want to run an async task on each.
I want to synchronize so each element will be processed once the preceding element has completed. What I've tried so far is:
function processItems(items) {
var i = 0;
while(i < items.length) {
asyncFunc(items[i], function() { i++ }); // asyncFunc takes a callback parameter
}
}
However this loops forever (I believe i is out of scope in the callback function).
Is there a better way to achieve this?
I think the following accomplishes what you're looking for:
function processItems(items) {
var i = 0,
length = items.length,
fn = function() {
if(i < length) {
asyncFunc(items[i], fn);
i++;
}
};
fn();
}
fn is a function that sets the callback equal to itself as long as the index is less than the length. Then I call fn once to start it off. Here's a fiddle:
http://jsfiddle.net/8A8CG/
Alternatively, if asyncFunc returns a promise, you can use Array#reduce to process the items in a series:
function processItems(items) {
return items.reduce((promise, item) => {
return promise.then(() => asyncFunc(item));
}, Promise.resolve());
}
If you need to execute a set of async function in series, I highly recommend the async module for node.
It offers a series method to execute async tasks in series. It also offers a waterfall method that will pass the results of the previous task to the next task.
Oops. You are looking for eachSeries. Here's how it might work for you:
var async = require('async');
async.eachSeries(items, asyncFunc, function(err){ if(err) return "OH NO"; return "Yay!"})
Make sure asyncFunc invokes a callback with either an error or null.
Related
var promiseReturningFuncs = [];
for(var i = 0; i < 5; i++){
promiseReturningFuncs.push(askQuestion);
}
var programmers = [];
Promise.reduce(promiseReturningFuncs, function(resp, x) {
console.log(typeof resp);
if(typeof resp != "function") {
programmers.push(resp);
}
return x();
})
.then(function(resp) {
programmers.push(resp);
console.log(programmers);
});
My goal: execute the askQuestion function in series and resolve an array of objects created by that function. (this function must execute in series so that it can respond to user input)
So imagine that the askQuestion function returns a promise that resolves a object I want to add to an array.
This is my messy way of doing it.
I am looking to find a cleaner way of doing it, ideally, i wouldn't even need to push to an array, I would just have a final .then, where the response is an array.
Since you appear to be using the Bluebird promise library, you have a number of built-in options for sequencing your promise returning functions. You can use Promise.reduce(), Promise.map() with a concurrency value of 1, Promise.mapSeries or Promise.each(). If the iterator function returns a promise, all of these will wait for the next iteration until that promise resolves. Which to use depends more upon the mechanics of how your data is structured and what result you want (neither of which you actually show or describe).
Let's suppose you have an array of promise returning functions and you want to call them one at a time, waiting for the one to resolve before calling the next one. If you want all the results, then I'd suggest Promise.mapSeries():
let arrayOfPromiseReturningFunctions = [...];
// call all the promise returning functions in the array, one at a time
// wait for one to resolve before calling the next
Promise.mapSeries(arrayOfPromiseReturningFunctions, function(fn) {
return fn();
}).then(function(results) {
// results is an array of resolved results from all the promises
}).catch(function(err) {
// process error here
});
Promise.reduce() could also be used, but it would accumulate a single result, passing it from one to the next and end with one final result (like Array.prototype.reduce() does).
Promise.map() is a more general version of Promise.mapSeries() that lets you control the concurrency number (the number of async operations in flight at the same time).
Promise.each() will also sequence your functions, but does not accumulate a result. It assumes you either don't have a result or you are accumulating the result out-of-band or via side effects. I tend to not like to use Promise.each() because I don't like side effect programming.
You could solve this in pure JS using ES6 (ES2015) features:
function processArray(arr, fn) {
return arr.reduce(
(p, v) => p.then((a) => fn(v).then(r => a.concat([r]))),
Promise.resolve([])
);
}
It applies the function given to the array in series and resolves to an array of the results.
Usage:
const numbers = [0, 4, 20, 100];
const multiplyBy3 = (x) => new Promise(res => res(x * 3));
// Prints [ 0, 12, 60, 300 ]
processArray(numbers, multiplyBy3).then(console.log);
You'll want to double check browser compatibility but this works on reasonably current Chrome (v59), NodeJS (v8.1.2) and probably most others.
You can use recursion so that you can move to the next iteration in a then block.
function promiseToExecuteAllInOrder(promiseReturningFunctions /* array of functions */) {
var resolvedValues = [];
return new Promise(function(resolve, reject) {
function executeNextFunction() {
var nextFunction = promiseReturningFunctions.pop();
if(nextFunction) {
nextFunction().then(function(result) {
resolvedValues.push(result);
executeNextFunction();
});
} else {
resolve(resolvedValues);
}
}
executeNextFunction();
}
}
Executing one after another using a recursive function( in a non promise way):
(function iterate(i,result,callback){
if( i>5 ) callback(result);askQuestion().then(res=>iterate(i+1,result.concat([res]),callback);
})(0,[],console.log);
For shure this can be wrapped in a promise:
function askFive(){
return new Promise(function(callback){
(function iterate(i,result){
if( i>5 ) callback(result);askQuestion().then(res=>iterate(i+1,result.concat([res]),callback);
})(0,[],console.log);
});
}
askFive().then(console.log);
Or:
function afteranother(i,promise){
return new Promise(function(resolve){
if(!i) return resolve([]);
afteranother(i-1,promise).then(val=>promise().then(val2=>resolve(val.concat([val2])));
});
}
afteranother(5,askQuestion).then(console.log);
Supposing we have a script that will execute a certain task for each row in an array.
function execute(err, array){
loop(array, function(err,object){
console.log(object)
//do a certain task when it's finished get into the next object successively.
});
}
function loop(array,callback){
array.forEach(function(object){
callback(null, object);
});
}
function array(callback){
callback(null, [1, 2, 3, 4, 5]);
}
setTimeout(function(){
array(execute);
}, 6000);
Questions:
How to get into the next loop only after finishing the task?
Is my function considered asynchronous ?
You can do something like this to iterate over your array :
var myarray = [1,2,3,4,5];
next(myarray, 0);
function next(array, idx) {
if (idx !== array.length) {
// do something
console.log(array[idx]);
// run other functions with callback
another_fct(array[idx], function() {
next(array, idx + 1); // then the next loop
});
// or run directly the next loop
next(array, idx + 1);
} else {
// the entire array has been proceed
console.log('an array of '+array.length+' number of elements have been proceed');
}
}
function another_fct(array_element, callback) {
// do something with array_element
console.log('value : '+array_element);
callback(); // run the next loop after processing
}
This method will perform your array elements synchronously.
Try this:
function execute(err, array) {
loop(array, function(err, object, next) {
console.log(object);
next(); // this will call recur inside loop function
}, function() {
console.log('All done');
});
}
function loop(array, callback, finish) {
var copy = array.slice();
(function recur() {
var item = copy.shift();
if (item) {
callback(null, item, recur);
} else {
if (typeof finish == 'function') {
finish();
}
}
})();
}
No, you function is not asynchronous but you call it asynchronously using setTimeout.
async provides async.series and other helpers for this purpose.
You can wrap your head around it better if you refactor the code and remove any redundant anonymous functions. The case is, that passing an anonymous function as an argument to another function not yet make the function asynchronous. You read more a more in depth explanation in "Does taking a callback make a function asynchronous?".
From the refactored version,
setTimeout(function() {
[1, 2, 3, 4, 5].forEach(function(object) {
console.log(object)
//do a certain task when it's...
});
}, 6000);
it can be seen that forEach is called for the array. The forEach method executes provided function once per array element and does it synchronously.
So, to answer your question:
yes, it calls the next item only after finishing previous (but if doing anything asynchronous, see below)
this is not considered asynchronous since it doesn't perform any asynchronous operations (not considering the outer setTimeout)
However, if you choose to start an asynchronous operation in the forEach function, then things change considerably. The result is that all operations are in-progress at the same time. This is potentially a hazardous operation resource-wise. There are libraries such as async for handling this use case gracefully.
(Btw, good use of Node.js errbacks, where first function argument reserved for passing potential error.)
function(obj){
for (property in obj) {
if (obj.hasOwnProperty(property)) {
// some code here
if(condition){
obj.children = example.getdata(base, obj.name);
}
// some more code releated to obj
}
}
if (obj.length == 10)
{
//some more function
}
}
Now i want to make example.getdata async but i dont want to execute if statement that is after for loop until all the sync tasks are done.
Its more of like i want all the example.getdata function calls to execute in parallel and after they finish work i execute if (obj.length).
I tried using promises and push all the promises and resolve them but i dont know how to handle the return value for each function call.
You can use Promise s.
Think of something like:
var jobs = get_jobs(data);
when_all(jobs).done(function (jobs_result) {
console.log(jobs_result);
});
Where get_jobs returns a list or Promises and when_all will resolve when all it's input jobs are resolved.
To run async tasks serially in a loop, you can't use a regular for loop because the for loop won't wait for your async operation to complete before executing the next cycle of the loop. Rather, you have to do your own custom iteration a slightly different way. Here's one way:
Assuming your getdata() function is actually asynchronous and has a completion function, you could structure things like this:
function myFunc(obj, callback) {
var keys = Object.keys(obj);
var cntr = 0;
var results = [];
function next() {
if (cntr < keys.length) {
example.getdata(obj[keys[cntr++]], function(result) {
// do something with the result of each async operation here
// and put it in the results array
// kick off the next iteration
next();
});
} else {
// call the final callback because we are done now with all async operations
callback(results);
}
}
// start the first iteration
next();
}
If your getData function returns a promise or can be made to return a promise, then you can use promises for this too:
function myFunc(obj) {
var keys = Object.keys(obj);
keys.reduce(function(p, item) {
return p.then(function(result) {
// do something with the result of each async operation here
// and put it in the results array
return example.getdata(obj[item]);
});
}, Promise.resolve());
}
myFunc.then(function(results) {
// all async operations done here, use the results argument
});
Here is a clean example of a for loop using promises. If you can pick a promise library libraries like Bluebird will make it even simpler:
function(obj){
var props = Object.keys(obj), p = Promise.resolve();
props.forEach(function(prop){
p = p.then(function(){
if(condition){ // can access obj[prop] and obj here
obj.children = example.getData(...); // getData returns a promise
return obj.children; // return the promise.
}
});
});
Promise.resolve(obj.children).then(function(children){
// here the async call to get `.children` is done.
// can wait for `p` too (the loop) if we care about it
if(obj.length === 10) // do stuff
});
I am new to NodeJS and I am not really sure how should the following function be declared. The function contains only a for loop which generates a string. There are no "heavy-weight" calculations done.
Variant 1:
function getRandomString(arg) {
for (var i = 0; i < 100; i++) {
// ...
}
return string;
}
var randomString = getRandomString(arg);
// ... an async code which will use the string
Variant 2: Or should I make it async (async-style)? It would look something like this:
function getRandomString(arg, callback) {
for (var i = 0; i < 100; i++) {
// ...
}
callback(string);
}
getRandomString(arg, function(randomString) {
// Async code...
});
Or should I make it async? So something like this:
function getString(arg, callback) {
for(var i = 0;i<100;i++) {
// ...
}
callback(string);
}
No. That code does still run synchronousls, only with an odd callback style for returning the result. Since JS does no tail call optimisation or has continuation support, it just introduces the pyramid of doom without any benefit.
Do not use this unless you really make it asynchronous (setTimeout, nextTick etc), for example to defer the single loop iterations.
There is no benefit from making it "asynchronous". In fact, the code will run synchronously, with the getString method only exiting after the callback is complete.
The only decision you really have to make is one of coding style (Do you want it to seem asynchronous or not.)
function getString(arg, callback)
{
for(var i = 0;i<100;i++)
{
// ...
}
//Callback will execute synchronously, and you will wait here until it is complete
callback(string);
}
I've been developing in JavaScript for quite some time but net yet a cowboy developer, as one of the many things that always haunts me is synching JavaScript's callbacks.
I will describe a generic scenario when this concern will be raised: I have a bunch of operations to perform multiple times by a for loop, and each of the operations has a callback. After the for loop, I need to perform another operation but this operation can only execute successfully if all the callbacks from the for loop are done.
Code Example:
for ... in ... {
myFunc1(callback); // callbacks are executed asynchly
}
myFunc2(); // can only execute properly if all the myFunc1 callbacks are done
Suggested Solution:
Initiate a counter at the beginning of the loop holding the length of the loop, and each callback decrements that counter. When the counter hits 0, execute myFunc2. This is essentially to let the callbacks know if it's the last callback in sequence and if it is, call myFunc2 when it's done.
Problems:
A counter is needed for every such sequence in your code, and having meaningless counters everywhere is not a good practice.
If you recall how thread conflicts in classical synchronization problem, when multiple threads are all calling var-- on the same var, undesirable outcomes would occur. Does the same happen in JavaScript?
Ultimate Question:
Is there a better solution?
The good news is that JavaScript is single threaded; this means that solutions will generally work well with "shared" variables, i.e. no mutex locks are required.
If you want to serialize asynch tasks, followed by a completion callback you could use this helper function:
function serializeTasks(arr, fn, done)
{
var current = 0;
fn(function iterate() {
if (++current < arr.length) {
fn(iterate, arr[current]);
} else {
done();
}
}, arr[current]);
}
The first argument is the array of values that needs to be passed in each pass, the second argument is a loop callback (explained below) and the last argument is the completion callback function.
This is the loop callback function:
function loopFn(nextTask, value) {
myFunc1(value, nextTask);
}
The first argument that's passed is a function that will execute the next task, it's meant to be passed to your asynch function. The second argument is the current entry of your array of values.
Let's assume the asynch task looks like this:
function myFunc1(value, callback)
{
console.log(value);
callback();
}
It prints the value and afterwards it invokes the callback; simple.
Then, to set the whole thing in motion:
serializeTasks([1,2, 3], loopFn, function() {
console.log('done');
});
Demo
To parallelize them, you need a different function:
function parallelizeTasks(arr, fn, done)
{
var total = arr.length,
doneTask = function() {
if (--total === 0) {
done();
}
};
arr.forEach(function(value) {
fn(doneTask, value);
});
}
And your loop function will be this (only parameter name changes):
function loopFn(doneTask, value) {
myFunc1(value, doneTask);
}
Demo
The second problem is not really a problem as long as every one of those is in a separate function and the variable is declared correctly (with var); local variables in functions do not interfere with each other.
The first problem is a bit more of a problem. Other people have gotten annoyed, too, and ended up making libraries to wrap that sort of pattern for you. I like async. With it, your code might look like this:
async.each(someArray, myFunc1, myFunc2);
It offers a lot of other asynchronous building blocks, too. I'd recommend taking a look at it if you're doing lots of asynchronous stuff.
You can achieve this by using a jQuery deferred object.
var deferred = $.Deferred();
var success = function () {
// resolve the deferred with your object as the data
deferred.resolve({
result:...;
});
};
With this helper function:
function afterAll(callback,what) {
what.counter = (what.counter || 0) + 1;
return function() {
callback();
if(--what.counter == 0)
what();
};
}
your loop will look like this:
function whenAllDone() { ... }
for (... in ...) {
myFunc1(afterAll(callback,whenAllDone));
}
here afterAll creates proxy function for the callback, it also decrements the counter. And calls whenAllDone function when all callbacks are complete.
single thread is not always guaranteed. do not take it wrong.
Case 1:
For example, if we have 2 functions as follows.
var count=0;
function1(){
alert("this thread will be suspended, count:"+count);
}
function2(){
//anything
count++;
dump(count+"\n");
}
then before function1 returns, function2 will also be called, if 1 thread is guaranteed, then function2 will not be called before function1 returns. You can try this. and you will find out count is going up while you are being alerted.
Case 2: with Firefox, chrome code, before 1 function returns (no alert inside), another function can also be called.
So a mutex lock is indeed needed.
There are many, many ways to achieve this, I hope these suggestions help!
First, I would transform the callback into a promise! Here is one way to do that:
function aPromise(arg) {
return new Promise((resolve, reject) => {
aCallback(arg, (err, result) => {
if(err) reject(err);
else resolve(result);
});
})
}
Next, use reduce to process the elements of an array one by one!
const arrayOfArg = ["one", "two", "three"];
const promise = arrayOfArg.reduce(
(promise, arg) => promise.then(() => aPromise(arg)), // after the previous promise, return the result of the aPromise function as the next promise
Promise.resolve(null) // initial resolved promise
);
promise.then(() => {
// carry on
});
If you want to process all elements of an array at the same time, use map an Promise.all!
const arrayOfArg = ["one", "two", "three"];
const promise = Promise.all(arrayOfArg.map(
arg => aPromise(arg)
));
promise.then(() => {
// carry on
});
If you are able to use async / await then you could just simply do this:
const arrayOfArg = ["one", "two", "three"];
for(let arg of arrayOfArg) {
await aPromise(arg); // wow
}
// carry on
You might even use my very cool synchronize-async library like this:
const arrayOfArg = ["one", "two", "three"];
const context = {}; // can be any kind of object, this is the threadish context
for(let arg of arrayOfArg) {
synchronizeCall(aPromise, arg); // synchronize the calls in the given context
}
join(context).then(() => { // join will resolve when all calls in the context are finshed
// carry on
});
And last but not least, use the fine async library if you really don't want to use promises.
const arrayOfArg = ["one", "two", "three"];
async.each(arrayOfArg, aCallback, err => {
if(err) throw err; // handle the error!
// carry on
});