We Keep Coding

What does Promise.all actually do under the hood?

I am trying to understand Promise.all here.
What I did here was to covert below code using Promise.all to achieve the same result.
I understand that Promise all combine the data1, data2.
My question here is that how does Promise.All work without resolve method?
Does Promise resolve those data within the method itself?
Please advise.
const readAllUsersChaining = () => {
return new Promise((resolve, reject) => {
let result = [];
getDataFromFilePromise(user1Path)
.then((data) => {
result.push(JSON.parse(data)); // what are you doing? he's gone mad...
return getDataFromFilePromise(user2Path);
})
.then((data) => {
result.push(JSON.parse(data));
result ? resolve(result) : reject(result);
});
});
};
const readAllUsers = () => {
const data1 = getDataFromFilePromise(user1Path);
const data2 = getDataFromFilePromise(user2Path);
console.log(data1, data2);
return Promise.all([data1, data2]).then((data) => {
return data.map((el) => JSON.parse(el));
});
};

My question here is that how does Promise.All work without resolve method?
Not quite sure what you mean. Promise.all simply creates a new promise internally that is resolved when all other promises are resolved.
Here is a simple implementation of Promise.all for the case that arguments are always promises:
function all(promises) {
if (promises.length === 0) {
return Promise.resolve([]);
}
return new Promise((resolve, reject) => {
const results = [];
let resolved = 0;
promises.forEach((promise, i) => {
promise.then(
result => {
results[i] = result;
resolved++;
if (resolved === promised.length) {
resolve(results);
}
},
error => reject(error)
);
});
}

Promise.all allows to wait until all promise passed as arguments to be fullfilled before the then method attach to be execute.
The real use case would be when you have to perform five call to an API, an you want to perform some treatement only when you have get the data from all the API call. you can rely on the Promise.all function which will wait all passed promised to be fullfilled for it to be fullfiled on it turn.
Bellow I provide an example of a Promise.all call. which has two Promises passed as argument. the first one has a timer which fullfilled it after 5 second and the second if fullfiled immediately but. the Promise.all will be fullfilled only when both of the promise passed as argument ar fullfilled
const firstPromise = new Promise((resolve, reject) => {
setTimeout(()=> {
return resolve({
name: "First Promise"
});
}, 5000);
});
const secondPromise = new Promise((resolve, reject) => {
return resolve({
name: "Second Promise"
});
})
const all = Promise.all([firstPromise, secondPromise]).then((response) => {
console.log(response);
});

Iterate over array of queries and append results to object in JavaScript

I want to return results from two database queries in one object.
function route(start, end) {
return new Promise((resolve, reject) => {
const queries = routeQuery(start, end);
var empty_obj = new Array();
for (i=0; i<queries.length; i++) {
query(queries[i], (err, res) => {
if (err) {
reject('query error', err);
console.log(err);
return;
} else {
empty_obj.push(res.rows);
}});
}
console.log(empty_obj);
resolve({coords: empty_obj});
});
}
This is my code right now, the queries are working fine but for some reason, pushing each result into an empty array does not work. When I console log that empty object, it stays empty. The goal is to resolve the promise with the generated object containing the two query results. I'm using node-postgres for the queries.
Output of res is an object:
{
command: 'SELECT',
rowCount: 18,
oid: null,
rows: [
{ ...

I suggest you turn your query function into a Promise so that you can use Promise.all:
// turn the callback-style asynchronous function into a `Promise`
function queryAsPromise(arg) {
return new Promise((resolve, reject) => {
query(arg, (err, res) => {
if (err) {
console.error(err);
reject(err);
return;
}
resolve(res);
});
});
}
Then, you could do the following in your route function:
function route(start, end) {
const queries = routeQuery(start, end);
// use `Promise.all` to resolve with
// an array of results from queries
return Promise.all(
queries.map(query => queryAsPromise(query))
)
// use `Array.reduce` w/ destructing assignment
// to combine results from queries into a single array
.then(results => results.reduce(
(acc, item) => [...acc, ...item.rows],
[]
))
// return an object with the `coords` property
// that contains the final array
.then(coords => {
return { coords };
});
}
route(1, 10)
.then(result => {
// { coords: [...] }
})
.catch(error => {
// handle errors appropriately
console.error(error);
});
References:
Promise.all - MDN
Array.reduce - MDN
Destructing assignment - MDN
Hope this helps.

The issue you currently face is due to the fact that:
resolve({coords: empty_obj});
Is not inside the callback. So the promise resolves before the query callbacks are called and the rows are pushed to empty_obj. You could move this into the query callback in the following manner:
empty_obj.push(res.rows); // already present
if (empty_obj.length == queries.length) resolve({coords: empty_obj});
This would resolve the promises when all rows are pushed, but leaves you with another issue. Callbacks might not be called in order. Meaning that the resulting order might not match the queries order.
The easiest way to solve this issue is to convert each individual callback to a promise. Then use Promise.all to wait until all promises are resolved. The resulting array will have the data in the same order.
function route(start, end)
const toPromise = queryText => new Promise((resolve, reject) => {
query(queryText, (error, response) => error ? reject(error) : resolve(response));
});
return Promise.all(routeQuery(start, end).map(toPromise))
.then(responses => ({coords: responses.map(response => response.rows)}))
.catch(error => {
console.error(error);
throw error;
});
}

Node.js: Unable to return new array from array.map()

I am using a package called Okrabyte to extract words from each image file in a folder. The result should be a new array containing the extracted text that I can use in other functions.
When I run this:
var fs = require("fs");
var okrabyte = require("okrabyte");
fs.readdir("imgs/", function(err, files){
files.map((file)=>{
okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), (err, data)=>{
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
console.log(word);
})
})
})
the console logs each word. To return an array with those words I've tried the following:
async function words() {
await fs.readdir("imgs/", function (err, files) {
return files.map(async (file) => {
await okrabyte.decodeBuffer(fs.readFileSync("imgs/" + file), async (err, data) => {
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
return word
})
})
})
}
var testing = await words();
console.log(testing);
This gives undefined I've tried turning everything into a promise, I've tried async-await, I've tried pushing each word into a new array and returning that array in closure but nothing works - what am I doing wrong??

If your map function is async then it's returning a promise, so your mapped array is in fact an array of promises. But you can then use a Promise.all to get the resolved values of that array.
Additionally, you're trying to await the call to fs.readdir and okrabyte.decodeBuffer, which both accept a callback and do not return a promise. So if you want to use a promise there you'll have to wrap them in a promise constructor manually.
Here's how I would do it:
async function words() {
// Wrap `fs` call into a promise, so we can await it:
const files = await new Promise((resolve, reject) => {
fs.readdir("imgs/", (err, files) => { err ? reject(err) : resolve(files); });
});
// Since map function returns a promise, we wrap into a Promise.all:
const mapped = await Promise.all(files.map((file) => {
// Wrap okrabyte.decodeBuffer into promise, and return it:
return new Promise((resolve, reject) => {
okrabyte.decodeBuffer(fs.readFileSync("imgs/" + file), (err, data) => {
if (err) return reject(err);
const splitWords = data.split(" ");
const word = splitWords[0].substr(1);
resolve(word);
})
})
}))
// Mapped is now an array containing each "word".
return mapped;
}
var testing = await words();
// Should now log your array of words correctly.
console.log(testing);

You should not be using async-await that way. That should be used when you are dealing with promises. The library okrabyte uses the concept of callbacks.
I suggest you follow this approach:
(1) Enclose the okrabyte.decodeBuffer part in a function that returns a promise that resolves in the callback.
(2) Use files.map to generate an array of promises calling the function you defined in (1)
(3) Use Promise.all to wait for all promises to execute and finish before moving on to dealing with all the words.
Walkthrough:
Part 1
const processWord = (file) => {
return new Promise((resolve, reject) => {
okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), (err, data)=>{
if (err) {
reject(err); // <--- reject the promise if there was an error
return;
}
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
resolve(word); // <--- resolve the promise with the word
})
});
}
You make a function that wraps the decoding part into a promise that eventually resolves with the word (or is rejected with an error).
Part 2
const promises = files.map((file)=>{
return processWord(file);
})
The above will generate an array of promises.
Part 3
fs.readdir("imgs/", function(err, files){
const promises = files.map((file)=>{
return processWord(file);
})
Promise.all(promises)
.then(responses => {
// responses holds a list of words
// You will get each word accessing responses[0], responses[1], responses[2], ...
console.log(responses);
})
.catch(error => {
console.log(error); // Deal with the error in some way
});
})
The above uses Promise.all to wait for all promises to resolve before going to the then() block, assuming no errors occurred.
You can further isolate the construct above in a method that will return a promise with a list of all the words, much in the same fashion that was done in the processWord function from Part 1. That way, you can finally use async-await if you wish, instead of handling things in the then() block:
const processEverything = () => {
return new Promise((resolve, reject) => {
fs.readdir("imgs/", function(err, files){
const promises = files.map((file)=>{
return processWord(file);
})
Promise.all(promises)
.then(responses => {
resolve(responses);
})
.catch(error => {
reject(error);
});
})
});
};
const words = await processEverything();
console.log(words);

You're returning word value to the enclosed function but not map() function.
Hope this code help you.
async function words() {
global.words = [];
await fs.readdir("imgs/", function(err, files){
return files.map( async(file)=>{
await okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), async (err, data)=>{
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
global.words.push(word);
})
})
})
}
var testing = await words();
testing = global.words;
console.log(testing);

In JavaScript/Node.js, how to use Async/Await

In JavaScript/Node.js, how to use Async/Await for the below code to return placeName with a,b,c & d.
test.csv is having c,d
var placeName = ["a","b"];
var csvFile = 'test.csv';
fs.readFile(csvFile, 'UTF-8', function(err, csv) {
$.csv.toArrays(csv, {}, function(err, data) {
for(var i=0, len=data.length; i<len; i++) {
console.log(data[i]); //Will print every csv line as a newline
placeName.push(data[i][0].toString());
}
});
console.log(placeName); //Inside the function the array show a,b,c,d
});
// Prints only a and b. Did not add c & d from the csv.
console.log(placeName);

Create a function returning a promise
The await operator waits for a promise to be fulfilled and can only be used within async functions - which return a promise when called. So updating place names will need to be written as a function that returns a promise. Using a single promise for all operations is possible but not recommended:
function addNames(placeName, csvFile) {
return new Promise( (resolve, reject) => {
fs.readFile(csvFile, 'UTF-8', function(err, csv) {
if(err) {
reject(err);
}
else {
$.csv.toArrays(csv, {}, function(err, data) {
if( err) {
reject( err);
}
else {
for(var i=0, len=data.length; i<len; i++) {
console.log(data[i]); //print supplied data elements
placeName.push(data[i][0].toString());
}
}
resolve( placeName);
});
}
});
});
}
Although untested, this is based on the posted code: the input placeName array is modified in place. If fulfilled, the promise value is set to the modified array. However ....
Reducing indented code
Notice the code becomes more indented the more node style callback operations are added? Taken to extreme this can result in a condition known as "call back hell" - difficult to write, compile without error or maintain.
Removing nested callbacks is one of the design aims of promises and can be achieved by promisifying operations separately. The following example separates not just reading and decoding, but transforming the decoded data and adding it to an existing array:
function readCsvFile( csvFile) {
return new Promise( (resolve, reject) => {
fs.readFile(csvFile, 'UTF-8',
(err, csv) => err ? reject(err) : resolve( csv)
);
});
}
function decodeCsv( csv) {
return new Promise( (resolve, reject) => {
$.csv.toArrays(csv, {},
(err, data) => err ? reject( err) : resolve( data)
);
});
}
function extractNames( data) {
return data.map( item => item[0].toString());
}
function addNames( placeName, csvFile) {
return readCsvFile( csvFile)
.then(decodeCsv)
.then(extractNames)
.then( // push new names to placeName and return placeName
names => names.reduce( (a,name) => {a.push(name);return a}, placeName)
);
}
Calling as part of a promise chain
Usage without using async/await requires adding promise handlers for fulfillment and rejection to be called asynchronously after data becomes available or an error occurs:
var placeName = ["a","b"];
var csvFile = 'test.csv';
addNames( placeName, csvFile)
.then( placeName => console.log(placeName)) // log updated places
.catch( err => console.log( err)); // or log the error.
Usage of await
As mentioned, the await operator can only be used inside async functions. It waits for fulfilled promise data so that it can be processed further inside the same function.
async function example() {
var placeName = ["a","b"];
var csvFile = 'test.csv';
await addNames( placeName, csvFile);
// do something with placeName
return somethingElse;
}
Adding fulfillment and rejection handlers to the promise returned by calling example has not been shown.
About return await ...
Note that a promise returned from an async function resolves the promise returned by calling the function. Hence the coding pattern of
return await operand
contains an unnecessary await operator.
Writing async functions instead of ordinary ones (TLDR)
Using async functions in node still requires converting operations which use error/success callbacks into requests that returns a promise ("promisifying" the operation), but syntactically allows writing code processing the request to be written immediately following await operations. Code produced is similar in style to procedural code. For example:
function readCsvFile( csvFile) {
return new Promise( (resolve, reject) => {
fs.readFile(csvFile, 'UTF-8',
(err, csv) => err ? reject(err) : resolve( csv)
);
});
}
function decodeCsv( csv) {
return new Promise( (resolve, reject) => {
$.csv.toArrays(csv, {},
(err, data) => err ? reject( err) : resolve( data)
);
});
}
async function addNames( placeName, csvFile) {
let csv = await readCsvFile( csvFile);
let arrays = await decodeCsv( csv);
let names = arrays.map( item => item[0].toString());
placeName.push.apply( placeName, names);
return placeName;
}

Array undefined after resolving it after for loop

I have the problem of having an undefined array which gets resolved after a for loop where it gets filled. It looks like the following:
function mainFunction() {
getUnreadMails().then(function(mailArray) {
// Do stuff with the mailArray
// Here it is undefined
})
}
function getUnreadMails() {
var mailArray = [];
return new Promise(function(resolve, reject) {
listMessages(oauth2Client).then(
(messageIDs) => {
for(var i = 0; i < messageIDs.length; i++) {
getMessage(oauth2Client, 'me', messageIDs[i]).then(function(r) {
// Array gets filled
mailArray.push(r);
}, function(error) {
reject(error);
})
}
// Array gets resolved
resolve(mailArray);
},
(error) => {
reject(error);
}
)
});
}
Both listMessages() and getMessage() returns a promise, so it is chained here. Any ideas why I am getting an undefined mailArray? My guess is that it is not filled yet when it gets resolved. Secondly I think this flow is not a good practice.

The Array is probably undefined because it is never defined; at least nowhere in your code. And your promise resolves before any iteration in your loop can resolve or better said, throw (trying to push to undefined).
Besides that. you can highly simplyfy your code by using Array#map and Promise.all.
And there's no point in catching an Error just to rethrow the very same error without doing anything else/with that error.
function getUnreadMails() {
//put this on a seperate line for readability
//converts a single messageId into a Promise of the result
const id2message = id => getMessage(oauth2Client, 'me', id);
return listMessages(oauth2Client)
//converts the resolved messageId into an Array of Promises
.then(messageIDs => messageIDs.map( id2message ))
//converts the Array of Promises into an Promise of an Array
//.then(Promise.all.bind(Promise));
.then(promises => Promise.all(promises));
//returns a Promise that resolves to that Array of values
}
or short:
function getUnreadMails() {
return listMessages(oauth2Client)
.then(messageIDs => Promise.all(messageIDs.map( id => getMessage(oauth2Client, 'me', id) )))
}
.then(Promise.all) won't work
I wanted to make the intermediate results more clear by seperating them into distinct steps/functions. But I typed too fast and didn't check it. Fixed the code.
In the short version, where does the mailArray then actually get filled/resolved
Promise.all() takes an an Array of promises and returns a single promise of the resolved values (or of the first rejection).
messageIDs.map(...) returns an Array and the surrounding Promise.all() "converts" that into a single Promise of the resolved values.
And since we return this Promise inside the Promise chain, the returned promise (listMessages(oauth2Client).then(...)) also resolves to this Array of values.

Just picking up on marvel308's answer, I think you need to create a new Promise that resolves when your other ones do. I haven't had a chance to test this, but I think this should work
function getUnreadMails() {
var mailArray = [];
return new Promise(function(resolve, reject) {
listMessages(oauth2Client).then(
(messageIDs) => {
var messages = [];
for(var i = 0; i < messageIDs.length; i++) {
messages.push(
getMessage(oauth2Client, 'me', messageIDs[i]).catch(reject)
);
}
Promise.all(messages).then(resolve);
},
(error) => {
reject(error);
}
)
});
}
This way, the resolve of your first promise gets called when all the messages have resolved

getMessage(oauth2Client, 'me', messageIDs[i]).then(function(r) {
// Array gets filled
mailArray.push(r);
}, function(error) {
reject(error);
})
is an asynchronous call
resolve(mailArray);
won't wait for it to push data and would resolve the array before hand
to resole this you should use Promise.all()
function mainFunction() {
getUnreadMails().then(function(mailArray) {
// Do stuff with the mailArray
// Here it is undefined
})
}
function getUnreadMails() {
var mailArray = [];
return listMessages(oauth2Client).then(
(messageIDs) => {
for(var i = 0; i < messageIDs.length; i++) {
mailArray.push(getMessage(oauth2Client, 'me', messageIDs[i]));
}
// Array gets resolved
return Promise.all(mailArray);
},
(error) => {
reject(error);
}
)
}

Since your getMessage function is async as well you need to wait until all your calls finish.
I would suggest using Promise.all
Here you can find more info: MDN Promise.all()
The code would look something like this:
messageIDs.map(...) returns an array of Promises
use Promise.all() to get an array with all the promises responses
resolve if values are correct reject otherwise
function mainFunction() {
getUnreadMails().then(function(mailArray) {
// Do stuff with the mailArray
// Here it is undefined
})
}
function getUnreadMails() {
return new Promise(function(resolve, reject) {
listMessages(oauth2Client).then(
(messageIDs) => {
return Promise.all(messageIDs.map(id => getMessage(oauth2Client, 'me', id)))
})
.then((messages) => resolve(messages))
.catch(error => reject(error))
});
}
One thing to keep in mind is that Promise.all() rejects if any of your promises failed
Hope this helps!

Explicit construction is an anti-pattern
I believe you can write that piece of code much shorter and, IMHO, cleaner
function mainFunction() {
getUnreadMails().then(function(mailArray) {
// Do stuff with the mailArray
// Here it is undefined
})
}
function getUnreadMails() {
return listMessages(oauth2Client)
.then((messageIDs) => Promise.all(messageIDs.map(id => getMessage(oauth2Client, 'me', id)))
}