We Keep Coding

for loop with axios, how do I know if all data are loaded

this.allMyFacilities = response[1].data
for (let i = 0; i < this.allMyFacilities.length; i++) {
axios
.get(facilityUsed(this.allMyFacilities[i].id))
.then((response) => {
this.facilitiesOnSupplyChain[this.allMyFacilities[i].id] = response.data
})
.catch((err) => {
// An error will also be thrown if you use cancel.
console.error(err)
})
}
I have such a code. I don't know how many facilities do I have. that is I loop it, But each facility has its own Axios. I want to how do I know all data from the loop are completely loaded

You can use Promise.all for this:
this.allMyFacilities = response[1].data
const promises = this.allMyFacilities.map((myFacility) => {
return axios
.get(facilityUsed(myFacility.id))
.then((response) => {
this.facilitiesOnSupplyChain[myFacility.id] = response.data
return response.data;
}).catch((err) => {
// An error will also be thrown if you use cancel.
console.error(err)
});
});
Promise.all(promises).then((allResponsesArray) => /* do sth */)

Assuming that all the asynchronous calls can be made independently, you can leverage upon the Promise.all functionality to achieve this.
this.allMyFacilities = response[1].data
await Promise.all(this.allMyFacilities.map(myFacility) => {
return axios.get(facilityUsed(myFacility.id))
.then(data => this.facilitiesOnSupplyChain[myFacility.id] = data);
});
This will allow you to execute all the axios calls in parallel, and hence decrease the overall execution time of the whole process.
PS:
I have written an await statement hance assuming that you will be wrapping this functionality in an async function.

Defining asynchrosnous function and using it in a forEach loop

I am trying to compile tex files into PFD using data from a firestore database. After completion the script doens't terminate. I found one can use process.exit() to make it quit. However, it terminates the child processes still cimpling the tex files. So, I need an asynchronous function.
The guides I found on how to make them did not particularly help me. I am still very new to javascript and any bloat is still highly confusion to me.
The guides I found on how to make them did not particularly help me. I am still very new to javascript and any bloat is still highly confusion to me.
prepending below mentioned makePDF function with async and the call of the function with await does not work and is, to my understanding, outdated.
I tried implementing a promise, but don't understand how their syntax works. Does simply appending .then() to the function call in the for loop suffice to make the loop wait for the functions completion?
How do I make this specific asynchronous?
Does it matter that it already uses asynchronous functions in its body?
I understand that a promise is used to return what ever the producer has produced to a consumer. Now, my producer doesn't produce anything to be returned. Does this matter at all?
My function called from the for loop:
function makePDF(object){
let input = fs.readFileSync('main.tex', 'utf8');
const outNameTex = object.ID + '.tex';
const outNamePDF = object.ID + '.pdf';
makeTEX(object, input, outNameTex);
const infile = fs.createReadStream(outNameTex);
const outfile = fs.createWriteStream(outNamePDF);
const pdf = latex(infile);
pdf.pipe(outfile);
pdf.on('error', err => console.error(err));
pdf.on('finish', () => {console.log('PDF generated!')});
}
And my function with the loop:
firebase.auth().onAuthStateChanged((user) => {
if (user) {
console.log('user');
db.collection('objects').where('printed', '==', false).get().then((snapshot) => {
snapshot.forEach((doc) => {
console.table(doc.data());
makePDF(doc.data());
})
process.exit();
})
.catch((err) => {
console.log('Error getting documents', err);
});
} else {
console.log('no user');
}
});
It outputs a table for each document, but no PDF generated.

async/await can be tricky to use with for loops, that is because async functions return a promise... if you convert the async/await syntax to native promise syntax you might figure out what the issue is.
What you want to do is use Array.map to map/convert each doc to a promise that resolves once the makePDF is done, then use Promise.all to wait for all the promises to resolve..
The solution should look something like this
function makePDF(object){
return new Promise((resolve, reject) => {
let input = fs.readFileSync('main.tex', 'utf8');
const outNameTex = object.ID + '.tex';
const outNamePDF = object.ID + '.pdf';
makeTEX(object, input, outNameTex);
const infile = fs.createReadStream(outNameTex);
const outfile = fs.createWriteStream(outNamePDF);
const pdf = latex(infile);
pdf.pipe(outfile);
pdf.on('error', reject);
pdf.on('finish', () => {console.log('PDF generated!'); resolve();});
}
firebase.auth().onAuthStateChanged((user) => {
if (user) {
console.log('user');
db.collection('objects').where('printed', '==', false).get().then((snapshot) => {
const promiseArr = snapshot.docs.map((doc) => {
console.table(doc.data());
return makePDF(doc.data());
})
Promise.all(promiseArr)
.then(() => {
process.exit();
})
})
.catch((err) => {
console.log('Error getting documents', err);
});
} else {
console.log('no user');
}
});

Why can't I move "await" to other parts of my code?

Edit2: Solution at the bottom
I am using the chrome-console and I am trying to output fetched data, and I only get the desired output by writing "await" at exactly the right place, even though another solution can do it earlier and I don't know why/how it works.
solution() is the "official" solution from a web-course I am doing. Both functions return the same, currently. In myFunction I tried writing "await" in front of every used function and make every function "async", but I still can't replace the "await" inside log, even though the other solution can.
const urls = ['https://jsonplaceholder.typicode.com/users']
const myFunction = async function() {
// tried await before urls/fetch (+ make it async)
const arrfetched = urls.map( url => fetch(url) );
const [ users ] = arrfetched.map( async fetched => { //tried await in front of arrfetched
return (await fetched).json(); //tried await right after return
});
console.log('users', await users); // but can't get rid of this await
}
const solution = async function() {
const [ users ] = await Promise.all(urls.map(async function(url) {
const response = await fetch(url);
return response.json();
}));
console.log('users', users); // none here, so it can be done
}
solution();
myFunction();
I would think "await" works in a way that makes:
const a = await b;
console.log(a); // this doesn't work
the same as
const a = b;
console.log(await a); // this works
but it doesn't, and I don't understand why not. I feel like Promise.all does something unexpected, as simply writing "await" in the declaration can't do the same, only after the declaration.
Edit1: this does not work
const myFunction = async function() {
const arrfetched = await urls.map( async url => await fetch(url) );
const [ users ] = await arrfetched.map( async fetched => {
return await (await fetched).json();
});
console.log('users', users);
}
Edit2: Thanks for the help everyone, I tried putting ".toString()" on a lot of variables and switching where I put "await" in the code and where not.
As far as I understand it, if I don't use Promise.all then I need to await every time I want to use (as in the actualy data, not just use) a function or variable that has promises. It is insufficient to only have await where the data is being procensed and not further up.
In the Edit1 above users runs bevore any other await is complete, therefore no matter how many awaits i write in, none are being executed. Copying this code in the (in my case chrome-)console demostrates it nicely:
const urls = [
'https://jsonplaceholder.typicode.com/users',
]
const myFunction = async function() {
const arrfetched = urls.map( async url => fetch(url) );
const [ users ] = arrfetched.map( async fetched => {
console.log('fetched', fetched);
console.log('fetched wait', await fetched);
return (await fetched).json();
});
console.log('users', users);
console.log('users wait', await users);
}
myFunction();
// Output in the order below:
// fetched()
// users()
// fetched wait()
// users wait()

TL; DR: Promise.all is important there, but it's nothing magical. It just converts an array of Promises into a Promise that resolves with an array.
Let's break down myFunction:
const arrfetched = urls.map( url => fetch(url) );
This returns an array of Promises, all good so far.
const [ users] = arrfetched.map( async fetched => {
return (await fetched).json();
});
You're destructuring an array to get the first member, so it's the same as this:
const arr = arrfetched.map( async fetched => {
return (await fetched).json();
});
const users = arr[0];
Here we are transforming an array of promises into another array of promises. Notice that calling map with an async function will always result in an array of Promises.
You then move the first member of that array into users, so users now actually contains a single Promise. You then await it before printing it:
console.log('users', await users);
In contrast, the other snippet does something slightly different here:
const [ users ] = await Promise.all(urls.map(async function(url) {
const response = await fetch(url);
return response.json();
}));
Once again, let's separate the destructuring:
const arr = await Promise.all(urls.map(async function(url) {
const response = await fetch(url);
return response.json();
}));
const users = arr[0];
Promise.all transforms the array of Promises into a single Promise that results in an array. This means that, after await Promise.all, everything in arr has been awaited (you can sort of imagine await Promise.all like a loop that awaits everything in the array). This means that arr is just a normal array (not an array of Promises) and thus users is already awaited, or rather, it was never a Promise in the first place, and thus you don't need to await it.

Maybe the easiest way to explain this is to break down what each step achieves:
const urls = ['https://jsonplaceholder.typicode.com/users']
async function myFunction() {
// You can definitely use `map` to `fetch` the urls
// but remember that `fetch` is a method that returns a promise
// so you'll just be left with an array filled with promises that
// are waiting to be resolved.
const arrfetched = urls.map(url => fetch(url));
// `Promise.all` is the most convenient way to wait til everything's resolved
// and it _also_ returns a promise. We can use `await` to wait for that
// to complete.
const responses = await Promise.all(arrfetched);
// We now have an array of resolved promises, and we can, again, use `map`
// to iterate over them to return JSON. `json()` _also_ returns a promise
// so again you'll be left with an array of unresolved promises...
const userData = responses.map(fetched => fetched.json());
//...so we wait for those too, and destructure out the first array element
const [users] = await Promise.all(userData);
//... et voila!
console.log(users);
}
myFunction();

Await can only be used in an async function. Await is a reserved key. You can't wait for something if it isn't async. That's why it works in a console.log but not in the global scope.

Promise.all is not working/waiting JavaScript in combination with mongoose

I got a problem that the await Promise.all is not working in my case. I attached the code and the output I got:
await Promise.all(allIndizes.map(async (index) => {
await axios.get(uri)
.then(async function (response) {
let searchResult = response.data.hits.hits;
console.log('Search Result: ' + searchResult);
await Promise.all(searchResult.map(async (element) => {
await primaryKeyModel.findById(element._id).exec((err, pk) => {
console.log('PK, direct after search: ' + pk);
//DO SOME STUFF HERE BUT DELETED IT TO SHORTEN THE CODE
}
})
console.log('test1');
}));
})
console.log('test2');
}));
The output is the following:
test1
test2
PK, direct after search: { _id: 5bf1c0619674e2052a4f6a64 ... }
I actually would actually expect that the first output is the 'PK, direct after search'. I don't understand why the function is not waiting? Do someone has a hint, whats wrong here? I found a similar issue here and I adopted the logic but its still not working. Thanks for the help.
I tried to shorten the code as much as possible. I only deleted statements which are not affecting the async execution.

Mongoose supports promises for a long time, callback-based API is obsolete, it's a mistake to use it where a promise is expected (await).
then is unwanted inside of async functions, this defies the purpose of using async..await.
It should be:
await Promise.all(allIndizes.map(async (index) => {
const response = await axios.get(uri);
let searchResult = response.data.hits.hits;
await Promise.all(searchResult.map(async (element) => {
const pk = await primaryKeyModel.findById(element._id);
//DO SOME STUFF HERE BUT DELETED IT TO SHORTEN THE CODE
}));
}));

Using Promise.all() to fetch a list of urls with await statements

tl;dr - if you have to filter the promises (say for errored ones) don't use async functions
I'm trying to fetch a list of urls with async and parse them, the problem is that if there's an error with one of the urls when I'm fetching - let's say for some reason the api endpoint doesn't exists - the program crushes on the parsing with the obvious error:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: ext is not iterable
I've tried checking if the res.json() is undefined, but obviously that's not it as it complains about the entire 'ext' array of promises not being iterable.
async function fetchAll() {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
Question 1:
How do I fix the above so it won't crash on an invalid address?
Question 2:
My next step is to write the extracted data to the database.
Assuming the data size of 2-5mgb of content, is my approach of using Promise.all() memory efficient? Or will it be more memory efficient and otherwise to write a for loop which handles each fetch then on the same iteration writes to the database and only then handles the next fetch?

You have several problems with your code on a fundamental basis. We should address those in order and the first is that you're not passing in any URLS!
async function fetchAll(urls) {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
First you have several try catch blocks on DEPENDANT DATA. They should all be in a single try catch block:
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url=>fetch(url)))
let ext = await Promise.all(data.map(res => {
// also fixed the ==! 'undefined'
if (res.json() !== undefined) { return res.json()}
}))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Next is the problem that res.json() returns a promise wrapped around an object if it exists
if (res.json() !== undefined) { return res.json()}
This is not how you should be using the .json() method. It will fail if there is no parsable json. You should be putting a .catch on it
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url => fetch(url).catch(err => err)))
let ext = await Promise.all(data.map(res => res.json ? res.json().catch(err => err) : res))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now when it cannot fetch a URL, or parse a JSON you'll get the error and it will cascade down without throwing. Now your try catch block will ONLY throw if there is a different error that happens.
Of course this means we're putting an error handler on each promise and cascading the error, but that's not exactly a bad thing as it allows ALL of the fetches to happen and for you to distinguish which fetches failed. Which is a lot better than just having a generic handler for all fetches and not knowing which one failed.
But now we have it in a form where we can see that there is some better optimizations that can be performed to the code
async function fetchAll(urls) {
try {
let ext = await Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.catch(error => ({ error, url }))
)
)
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now with a much smaller footprint, better error handling, and readable, maintainable code, we can decide what we eventually want to return. Now the function can live wherever, be reused, and all it takes is a single array of simple GET URLs.
Next step is to do something with them so we probably want to return the array, which will be wrapped in a promise, and realistically we want the error to bubble since we've handled each fetch error, so we should also remove the try catch. At that point making it async no longer helps, and actively harms. Eventually we get a small function that groups all URL resolutions, or errors with their respective URL that we can easily filter over, map over, and chain!
function fetchAll(urls) {
return Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.then(data => ({ data, url }))
.catch(error => ({ error, url }))
)
)
}
Now we get back an array of similar objects, each with the url it fetched, and either data or an error field! This makes chaining and inspecting SUPER easy.

You are getting a TypeError: ext is not iterable - because ext is still undefined when you caught an error and did not assign an array to it. Trying to loop over it will then throw an exception that you do not catch.
I guess you're looking for
async function fetchAll() {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
const ext = await Promise.all(data.map(res => res.json()));
for (let item of ext) {
console.log(item);
}
} catch (err) {
console.log(err);
}
}

Instead of fetch(url) on line 5, make your own function, customFetch, which calls fetch but maybe returns null, or an error object, instead of throwing.
something like
async customFetch(url) {
try {
let result = await fetch(url);
if (result.json) return await result.json();
}
catch(e) {return e}
}

if (res.json()==! 'undefined')
Makes no sense whatsoever and is an asynchronous function. Remove that condition and just return res.json():
try {
ext = await Promise.all(data.map(res => res.json()))
} catch (err) {
console.log(err)
}
Whether or not your approach is "best" or "memory efficient" is up for debate. Ask another question for that.

You can have fetch and json not fail by catching the error and return a special Fail object that you will filter out later:
function Fail(reason){this.reason=reason;};
const isFail = o => (o&&o.constructor)===Fail;
const isNotFail = o => !isFail(o);
const fetchAll = () =>
Promise.all(
urls.map(
url=>
fetch(url)
.then(response=>response.json())
.catch(error=>new Fail([url,error]))
)
);
//how to use:
fetchAll()
.then(
results=>{
const successes = results.filter(isNotFail);
const fails = results.filter(isFail);
fails.forEach(
e=>console.log(`failed url:${e.reason[0]}, error:`,e.reason[1])
)
}
)
As for question 2:
Depending on how many urls you got you may want to throttle your requests and if the urls come from a large file (gigabytes) you can use stream combined with the throttle.

async function fetchAll(url) {
return Promise.all(
url.map(
async (n) => fetch(n).then(r => r.json())
)
);
}
fetchAll([...])
.then(d => console.log(d))
.catch(e => console.error(e));
Will this work for you?

If you don't depend on every resource being a success I would have gone back to basics skipping async/await
I would process each fetch individual so I could catch the error for just the one that fails
function fetchAll() {
const result = []
const que = urls.map(url =>
fetch(url)
.then(res => res.json())
.then(item => {
result.push(item)
})
.catch(err => {
// could't fetch resource or the
// response was not a json response
})
)
return Promise.all(que).then(() => result)
}
Something good #TKoL said:
Promise.all errors whenever one of the internal promises errors, so whatever advice anyone gives you here, it will boil down to -- Make sure that you wrap the promises in an error handler before passing them to Promise.all

Regarding question 1, please refer to this:
Handling errors in Promise.all
Promise.all is all or nothing. It resolves once all promises in the array resolve, or reject as soon as one of them rejects. In other words, it either resolves with an array of all resolved values, or rejects with a single error.