We Keep Coding

How to wait until multiple files are processed before calling finished function js

The following function runs after a drag and drop operation of multiple files.
function getFilesInfo(ev){
for (let i = 0; i < ev.dataTransfer.items.length; i++) {
if (ev.dataTransfer.items[i].kind === 'file') {
let file = ev.dataTransfer.items[i].getAsFile();
//getFileInfo adds string to DOM element
//Note the promise usage ...
file.arrayBuffer().then((data)=>getFileInfo(file.name,data));
}
}
}
I can't figure out how to call a function after all of the promises in this function finish.
Basically I want something like this, sequentially:
getFilesInfo(ev);
//getFileInfo(<file1>);
//getFileInfo(<file2>);
//getFileInfo(<file3>);
// etc.
//run only after all getFileInfo() calls have finished
processResults();
The tricky part is that reading the files generates a promise for each file that gets called when the file has been read into memory (part of the arrayBuffer() call). I can't figure out how to delay processResults because getFilesInfo finishes after all of the read calls have been triggered, not (from what I can tell), after the getFileInfo functions have finished.
It seems like perhaps I could somehow add all arrayBuffer calls to an array and then do some promise chaining (maybe?) but that seems awkward and I'm not even sure how I would do that.

You can use Promise.all to wait for an array of promise to finish:
async function getFilesInfo(ev) {
// create list of jobs
const jobs = [];
for (const item of ev.dataTransfer.items) {
if (item.kind === 'file') {
let file = item.getAsFile();
jobs.push(file.arrayBuffer().then(data => {
getFileInfo(file.name, data);
}));
}
}
// wait for all promise to fullfil
await Promise.all(jobs);
}
https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

You could do it that way:
function getFilesInfo(ev){
return ev.dataTransfer.items.filter(item=>item.kind === 'file').map(item=>{
let file = item.getAsFile();
return file.arrayBuffer().then((data)=>getFileInfo(file.name,data));
});
}
Promise.all(...getFilesInfo(ev)).then(_=>{
processResults();
});
// or with async/await
(async ()=>{
await Promise.all(...getFilesInfo(ev));
processResults();
})()

async function getFilesInfo(ev) {
await Promise.all(ev.dataTransfer.items.map(async (i) => {
const file = i.getAsFile();
const data = await file.arrayBuffer();
return getFileInfo(file.name, data);
}));
}
await getFilesInfo(ev); // will be awaited until all the promises are resolved
processResults();
Let me know if that helps.

The conceptual hurdle I was running into was that I was thinking of the then function as returning the results, not promises. Also, many of the examples I've seen with Promise.all are usually just concatenating explicit calls, not building an array in a loop.
As suggested by Bergi, I simply added the calls to an array, and then passed that array into Promise.all
function getFilesInfo(ev) {
// create list of jobs
let jobs = [];
for (const item of ev.dataTransfer.items) {
if (item.kind === 'file') {
let file = item.getAsFile();
jobs.push(file.arrayBuffer().then(data => {
getFileInfo(file.name, data);
}));
}
}
return jobs;
}
//The call in the parent
let jobs = getFilesInfo(ev);
Promise.all(jobs).then(processResults);

Javascript promise chaining with promise.all not working?

I have the following code. When i run it i get this error " Cannot read property 'then' of undefined" at the first line in my code when calling getQueryToShift. It seems like getQueryToShift is not working like I intended. What is the correct way to use Promise.all so that the original promise I am declaring on the first line waits for all the promises in the promises array declared inside the getQueryToShift function to resolve before executing what is inside the then block ?
promise = getQueryToShift(hourDiff, options, map, baseBuildStart, oldFrom).then(values => { // error is here
// wait for promise before handling data in map passed as parameter
});
function getQueryToShift(hourDiff, options, map, baseBuildStart, oldFrom) {
let promises = [];
datasourceSrv.get(options.targets[0].datasource).then(ds => {
for (let i = 0; i < daysDiff - 1; i++) {
options.range.from._d = dateToMoment(oldFrom, false).add(i, 'h').toDate();
options.range.to._d = dateToMoment(options.range.from._d, false).add(1, 'h').toDate();
ds.query(options).then(result => {
promises.push(createQueryPromise(map, baseBuildStart, result.data));
});
}
return Promise.all(promises);
});
}
function createQueryPromise(map, baseBuildStart, data) {
return new Promise((resolve) => {
data.forEach(datum => {
//parsing data and adding it to map passed in as parameter
})
resolve();
});
}

The question is incomplete because createQueryPromise() function makes no sense by itself. It does not show an asynchronous operation at all, thus without one there is no need for a promise. And, if there is an asynchronous operation inside your .forEach() look, then you will need to coordinate that async operation, but you don't show the code for that so we can't help you fix that and we need to understand what that function is actually doing before we can help with a full and correct implementation of getQueryToShift().
On top of that, here are a couple other things that need fixing:
You aren't returning your promise from getQueryToShift(). Change this:
datasourceSrv.get(options.targets[0].datasource).then(ds => {
to this:
return datasourceSrv.get(options.targets[0].datasource).then(ds => {
And, you are ignoring the promise returned by:
ds.query(options).then(...)
Thus, your code wouldn't wait for that operation to finish. This is probably the promise that you need to collect in the array and use Promise.all() with, not the one you are doing it with.

Your code does the Promise.all on an empty array - you need to return the promise:
function* getDays(daysDiff, oldFrom) {
for (let i = 0; i < daysDiff - 1; i++) {
var options = {};
options.range.from._d = dateToMoment(oldFrom, false).add(i, 'h').toDate();
options.range.to._d = dateToMoment(options.range.from._d, false).add(1, 'h').toDate();
yield options;
}
}
function getQueryToShift(hourDiff, options, map, baseBuildStart, oldFrom) {
// return here
return datasourceSrv.get(options.targets[0].datasource).then(ds => {
return Promise.all(
Array.from(getDays(daysDiff, oldFrom), option => db.query(option))
);
});
}
Although I warmly recommend using an async function instead:
async function getQueryToShift(hourDiff, options, map, baseBuildStart, oldFrom) {
const ds = await datasourceSrv.get(options.targets[0].datasource);
await Promise.all(Array.from(getDays(daysDiff, oldFrom), db.query));
}

Loops and Callback hell

Suppose you have an Array/Object that contains a list of values. Lets say those a mysql commands or urls or filespaths. Now you want to iterate over all of them and execute some code over every entry.
for(let i = 0; i < urls.length; i++){
doSthWith(urls[i]);
}
No Problem so far. But now lets say each function has a callback and needs the result of the last execution. e.g. you request something from one website and you want to use the results of this request for one of your following requests.
for(let i = 0; i < urls.length; i++){
if(resultOfLastIteration.successful){ //or some other result besides the last one
doSthWith(urls[i]);
}
}
Now lets say the length of urls (or sth similar) is over 100. Thats why you normaly use a loop so you dont need to write the same function a 100 times. That also means that Promises wont do the trick either (except Im unaware trick a trick), because you have the same problem:
doSthWith(urls[0]).then(...
doSthWith(urls[1]).then(... //either put them inside each other
).then(...
doSthWith(urls[i]) //or in sequence
...
).catch(err){...}
Either way I dont see a way to use a loop.
A way that I found but isnt really "good" is to use the package "wait.for"(https://www.npmjs.com/package/wait.for). But what makes this package tricky is to launch a fiber each time you want to use wait.for:
//somewhere you use the function in a fiber Context
wait.for(loopedExecutionOfUrls, urls);
//function declaration
function loopedExecutionOfUrls(urls, cb){
//variables:
for(let i = 0; i < urls.length; i++){
if(someTempResultVar[i-1] === true){
someTempResultVar = wait.for(doSthWith,urls[i]);
} else if(...){...}
}
}
But Im not sure if this approach is really good, besides you always have to check if you have wrapped the whole thing in a Fiber so for each function that has loops with functions that have callbacks. Thus you have 3 levels: the lauchFiber level, wait.for(loopedFunction) level and the wait.for the callback function level. (Hope I that was formulated understandable)
So my questions is: Do you guys have a good approach where you can loop throw callback functions and can use results of those whenever you like?
good = easy to use, read, performant, not recursive,...
(Im sorry if this question is stupid, but I really have problems getting along with this asynchronous programming)

If you want to wait for doSthWith to finish before doing the same but with the nex url, you have to chain your promises and you can use array.prototype.reduce to do that:
urls = ["aaa", "bbb", "ccc", "ddd"];
urls.reduce((lastPromise, url) => lastPromise.then((resultOfPreviousPromise) => {
console.log("Result of previous request: ", resultOfPreviousPromise); // <-- Result of the previous request that you can use for the next request
return doSthWith(url);
}), Promise.resolve());
function doSthWith(arg) { // Simulate the doSthWith promise
console.log("do something with: ", arg);
return new Promise(resolve => {
setTimeout(() => resolve("result of " + arg), 2000);
});
}

Use async, specifically async.each:
const async = require('async');
function doSthWith(url, cb) {
console.log('doing something with ' + url);
setTimeout(() => cb(), 2000);
}
const urls = ['https://stackoverflow.com/', 'https://phihag.de/'];
async.each(urls, doSthWith, (err) => {
if (err) {
// In practice, likely a callback or throw here
console.error(err);
} else {
console.log('done!');
}
});
Use async.map if you are interested in the result.

When I need to loop over promises I use my handy dandy ploop function. Here is an example:
// Function that returns a promise
var searchForNumber = function(number) {
return new Promise(function(resolve, reject) {
setTimeout(function() {
var min = 1;
var max = 10;
var val = Math.floor(Math.random()*(max-min+1)+min);
console.log('Value is: ' + val.toString());
return resolve(val);
}, 1000);
});
};
// fn : function that should return a promise.
// args : the arguments that should be passed to fn.
// donefn : function that should check the result of the promise
// and return true to indicate whether ploop should stop or not.
var ploop = function(fn, args, donefn) {
return Promise.resolve(true)
.then(function() {
return(fn.apply(null, args));
})
.then(function(result) {
var finished = donefn(result);
if(finished === true){
return result;
} else {
return ploop(fn, args, donefn);
}
});
};
var searchFor = 4;
var donefn = function(result) {
return result === searchFor;
};
console.log('Searching for: ' + searchFor);
ploop(searchForNumber, [searchFor], donefn)
.then(function(val) {
console.log('Finally found! ' + val.toString());
process.exit(0);
})
.catch(function(err) {
process.exit(1);
});

Node.JS How to set a variable outside the current scope

I have some code that I cant get my head around, I am trying to return an array of object using a callback, I have a function that is returning the values and then pushing them into an array but I cant access this outside of the function, I am doing something stupid here but can't tell what ( I am very new to Node.JS )
for (var index in res.response.result) {
var marketArray = [];
(function () {
var market = res.response.result[index];
createOrUpdateMarket(market, eventObj , function (err, marketObj) {
marketArray.push(marketObj)
console.log('The Array is %s',marketArray.length) //Returns The Array is 1.2.3..etc
});
console.log('The Array is %s',marketArray.length) // Returns The Array is 0
})();
}

You have multiple issues going on here. A core issue is to gain an understanding of how asynchronous responses work and which code executes when. But, in addition to that you also have to learn how to manage multiple async responses in a loop and how to know when all the responses are done and how to get the results in order and what tools can best be used in node.js to do that.
Your core issue is a matter of timing. The createOrUpdateMarket() function is probably asynchronous. That means that it starts its operation when the function is called, then calls its callback sometime in the future. Meanwhile the rest of your code continues to run. Thus, you are trying to access the array BEFORE the callback has been called.
Because you cannot know exactly when that callback will be called, the only place you can reliably use the callback data is inside the callback or in something that is called from within the callback.
You can read more about the details of the async/callback issue here: Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
To know when a whole series of these createOrUpdateMarket() operations are all done, you will have to code especially to know when all of them are done and you cannot rely on a simple for loop. The modern way to do that is to use promises which offer tools for helping you manage the timing of one or more asynchronous operations.
In addition, if you want to accumulate results from your for loop in marketArray, you have to declare and initialize that before your for loop, not inside your for loop. Here are several solutions:
Manually Coded Solution
var len = res.response.result.length;
var marketArray = new Array(len), cntr = 0;
for (var index = 0, index < len; index++) {
(function(i) {
createOrUpdateMarket(res.response.result[i], eventObj , function (err, marketObj) {
++cntr;
if (err) {
// need error handling here
}
marketArray[i] = marketObj;
// if last response has just finished
if (cntr === len) {
// here the marketArray is fully populated and all responses are done
// put your code to process the marketArray here
}
});
})(index);
}
Standard Promises Built Into Node.js
// make a version of createOrUpdateMarket that returns a promise
function createOrUpdateMarketAsync(a, b) {
return new Promise(function(resolve, reject) {
createOrUpdateMarket(a, b, function(err, marketObj) {
if (err) {
reject(err);
return;
}
resolve(marketObj);
});
});
}
var promises = [];
for (var i = 0; i < res.response.result.length; i++) {
promises.push(createorUpdateMarketAsync(res.response.result[i], eventObj));
}
Promise.all(promises).then(function(marketArray) {
// all results done here, results in marketArray
}, function(err) {
// an error occurred
});
Enhanced Promises with the Bluebird Promise library
The bluebird promise library offers Promise.map() which will iterate over your array of data and produce an array of asynchronously obtained results.
// make a version of createOrUpdateMarket that returns a promise
var Promise = require('bluebird');
var createOrUpdateMarketAsync = Promise.promisify(createOrUpdateMarket);
// iterate the res.response.result array and run an operation on each item
Promise.map(res.response.result, function(item) {
return createOrUpdateMarketAsync(item, eventObj);
}).then(function(marketArray) {
// all results done here, results in marketArray
}, function(err) {
// an error occurred
});
Async Library
You can also use the async library to help manage multiple async operations. In this case, you can use async.map() which will create an array of results.
var async = require('async');
async.map(res.response.result, function(item, done) {
createOrUpdateMarker(item, eventObj, function(err, marketObj) {
if (err) {
done(err);
} else {
done(marketObj);
}
});
}, function(err, results) {
if (err) {
// an error occurred
} else {
// results array contains all the async results
}
});

How can I wait for set of asynchronous callback functions?

I have code that looks something like this in javascript:
forloop {
//async call, returns an array to its callback
}
After ALL of those async calls are done, I want to calculate the min over all of the arrays.
How can I wait for all of them?
My only idea right now is to have an array of booleans called done, and set done[i] to true in the ith callback function, then say while(not all are done) {}
edit: I suppose one possible, but ugly solution, would be to edit the done array in each callback, then call a method if all other done are set from each callback, thus the last callback to complete will call the continuing method.

You haven't been very specific with your code, so I'll make up a scenario. Let's say you have 10 ajax calls and you want to accumulate the results from those 10 ajax calls and then when they have all completed you want to do something. You can do it like this by accumulating the data in an array and keeping track of when the last one has finished:
Manual Counter
var ajaxCallsRemaining = 10;
var returnedData = [];
for (var i = 0; i < 10; i++) {
doAjax(whatever, function(response) {
// success handler from the ajax call
// save response
returnedData.push(response);
// see if we're done with the last ajax call
--ajaxCallsRemaining;
if (ajaxCallsRemaining <= 0) {
// all data is here now
// look through the returnedData and do whatever processing
// you want on it right here
}
});
}
Note: error handling is important here (not shown because it's specific to how you're making your ajax calls). You will want to think about how you're going to handle the case when one ajax call never completes, either with an error or gets stuck for a long time or times out after a long time.
jQuery Promises
Adding to my answer in 2014. These days, promises are often used to solve this type of problem since jQuery's $.ajax() already returns a promise and $.when() will let you know when a group of promises are all resolved and will collect the return results for you:
var promises = [];
for (var i = 0; i < 10; i++) {
promises.push($.ajax(...));
}
$.when.apply($, promises).then(function() {
// returned data is in arguments[0][0], arguments[1][0], ... arguments[9][0]
// you can process it here
}, function() {
// error occurred
});
ES6 Standard Promises
As specified in kba's answer: if you have an environment with native promises built-in (modern browser or node.js or using babeljs transpile or using a promise polyfill), then you can use ES6-specified promises. See this table for browser support. Promises are supported in pretty much all current browsers, except IE.
If doAjax() returns a promise, then you can do this:
var promises = [];
for (var i = 0; i < 10; i++) {
promises.push(doAjax(...));
}
Promise.all(promises).then(function() {
// returned data is in arguments[0], arguments[1], ... arguments[n]
// you can process it here
}, function(err) {
// error occurred
});
If you need to make a non-promise async operation into one that returns a promise, you can "promisify" it like this:
function doAjax(...) {
return new Promise(function(resolve, reject) {
someAsyncOperation(..., function(err, result) {
if (err) return reject(err);
resolve(result);
});
});
}
And, then use the pattern above:
var promises = [];
for (var i = 0; i < 10; i++) {
promises.push(doAjax(...));
}
Promise.all(promises).then(function() {
// returned data is in arguments[0], arguments[1], ... arguments[n]
// you can process it here
}, function(err) {
// error occurred
});
Bluebird Promises
If you use a more feature rich library such as the Bluebird promise library, then it has some additional functions built in to make this easier:
var doAjax = Promise.promisify(someAsync);
var someData = [...]
Promise.map(someData, doAjax).then(function(results) {
// all ajax results here
}, function(err) {
// some error here
});

Checking in from 2015: We now have native promises in most recent browser (Edge 12, Firefox 40, Chrome 43, Safari 8, Opera 32 and Android browser 4.4.4 and iOS Safari 8.4, but not Internet Explorer, Opera Mini and older versions of Android).
If we want to perform 10 async actions and get notified when they've all finished, we can use the native Promise.all, without any external libraries:
function asyncAction(i) {
return new Promise(function(resolve, reject) {
var result = calculateResult();
if (result.hasError()) {
return reject(result.error);
}
return resolve(result);
});
}
var promises = [];
for (var i=0; i < 10; i++) {
promises.push(asyncAction(i));
}
Promise.all(promises).then(function AcceptHandler(results) {
handleResults(results),
}, function ErrorHandler(error) {
handleError(error);
});

You can use jQuery's Deferred object along with the when method.
deferredArray = [];
forloop {
deferred = new $.Deferred();
ajaxCall(function() {
deferred.resolve();
}
deferredArray.push(deferred);
}
$.when(deferredArray, function() {
//this code is called after all the ajax calls are done
});

You can emulate it like this:
countDownLatch = {
count: 0,
check: function() {
this.count--;
if (this.count == 0) this.calculate();
},
calculate: function() {...}
};
then each async call does this:
countDownLatch.count++;
while in each asynch call back at the end of the method you add this line:
countDownLatch.check();
In other words, you emulate a count-down-latch functionality.

This is the most neat way in my opinion.
Promise.all
FetchAPI
(for some reason Array.map doesn't work inside .then functions for me. But you can use a .forEach and [].concat() or something similar)
Promise.all([
fetch('/user/4'),
fetch('/user/5'),
fetch('/user/6'),
fetch('/user/7'),
fetch('/user/8')
]).then(responses => {
return responses.map(response => {response.json()})
}).then((values) => {
console.log(values);
})

Use an control flow library like after
after.map(array, function (value, done) {
// do something async
setTimeout(function () {
// do something with the value
done(null, value * 2)
}, 10)
}, function (err, mappedArray) {
// all done, continue here
console.log(mappedArray)
})

In Node.js you can use async/await to control the async flow
async/await is supported in Node.js 7.6
util function to promisify callback is supported in Node.js v8
Sample Code:
const foo = async () => {
try {
const ids = [100, 101, 102];
const fetchFromExternalApi = util.promisify(fetchFromExternalApiCallback);
const promises = ids.map((id) => fetchFromExternalApi(id));
const dataList = await Promise.resolve(promises); // dataList is an array
return dataList;
} catch (err) {
// error handling
}
};

I see several response with Promise.all(), but this function stop if any promise generate an exception...
The best solution in 2022 is Promise.allSettled() (documentation here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled
A quick sample:
const allPromises = [];
for (r in records) {
const promise = update_async(r);
allPromises.push(promise);
};
await Promise.allSettled(allPromises);
At the end, you obtain with allPromises an array with the result of each promise:
when ok --> {status: “fulfilled”, value: xxx }
when error --> {status: "rejected", reason: Error: an error}