I have this JavaScript pseudocode:
async function main() {
for(let i=0; i<array.length; i++) {
for(let j=0; j<3; j++) {
let value = await promise thisFunctionReturnsAPromise();
If(value) {
// do_something
}
}
}
}
Now I need to iterate 2 promises at time (to speed up the job) and to test each one of them 3 times; so i can't use a simple for anymore.
I need to replace the external for loop.
I was thinking to use async.eachLimit to solve the problem but i can't use await within eachLimit. How can i put an await within async eachLimit to call more then one "test cicle"?
I need something like async.eachLimit because my job needs to go to the next element that needs to be tested after one of the previous two has been tested for a maximum of 3 times.
Sorry for my bad english.
Related
I have below code in javascript in which some asynchronous task is being performed:
async function fetchData(id){
for(let i=1;;++i){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${i}`,{//headers here});
res = await res.json();
if(res.length==0) break;
else{ //do some work here and continue for next iteration}
}
}
async function callApi(){
var arr = [//list of id's here to pass to api one by one, almost 100 id's here];
await Promise.all(arr.map(async(e)=>{
await fetchData(e);
}));
}
callApi();
The above code looks fine to me, except that it doesn't work as expected. Ideally, what should happen is that unless one id's call is not completed( unless break condition not satisfies for one id), the for loop should not proceed to next iteration. Rather, I am getting totally different results. The api calls are happening in random order because the loop is not waiting the iteration to complete. My hard requirement is that unless one iteration is not complete, it should not move to next one.
await seems to have no effect here. Please guide me how can I achieve this. I am running out of ideas.
Thank You!
Your arr.map(...) is not awaiting the different fetchData calls before the next map call, so I'd turn this into a specific for loop to be sure it waits:
async function callApi(){
const arr = [...];
for(let i = 0; i < arr.length; i++){
await fetchData(arr[i]);
}
}
or alternatively use a for of
async function callApi(){
const arr = [...];
for(let a of arr){
await fetchData(a);
}
}
The fetchData function also looks like it could use some improvements with error handling, but since you shortened your code quite a bit, I'm assuming there is something like that going on there, too, and your issue is actually with the callApi() code instead, as the fetch and await looks good to me there.
You should decide either to use promises or async await. Don't mix them.
With promises you can always use funky abstractions but with a simple recursive approach you can do like
function fetchData(hds, id, page = 1, pages = []){
return fetch(`https://some-api/v1/products/${id}/data?page=${page}`,hds)
.then(r => r.ok ? r.json() : Promise.reject({status:r.status,pages})
.then(j => fetchData(hds, id, ++page, pages.push(doSomethingWith(j))))
.catch(e => (console.log(e.status), e.pages));
}
So we use recursion to fetch indefinitelly until the API says enough and r.ok is false.
At the callApi side you can use reduce since we have an ids array.
const ids = [/* ids array */],
hds = { /* headers object */ };
function callApi(ids){
return ids.reduce( (p,id) => p.then(_ => fetchData(hds,id))
.then(pages => process(pages))
, Promise.resolve(null)
)
.catch(e => console.log(e));
}
So now both accesses to the id and page data are working asynchronously but only fired once the previous one finishes. Such as
(id=1,page=1) then (id=1,page=2) then (id=1,page=3) then (process 3 pages of id=1) then
(id=2,page=1) then (id=2,page=2) then (process 2 pages of id=2) etc...
While I love the promises, you can also implement the same functionality with the asyc await abstraction. I believe the idea behind the invention of the async await is to mimic sync imperative code. But keep in mind that it's an abstraction over an abstraction and I urge you to learn promises by heart before even attemting to use async await. The general rule is to never mix both in the same code.
Accordingly the above code could have been written as follows by using async await.
async function fetchData(hds, id){
let page = 1,
pages = [],
res;
while(true){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${page++}`,hds);
if (res.ok) pages.push(await res.json())
else return pages;
}
}
Then the callApi function can be implemented in a similar fashion
const ids = [/* ids array */],
hds = { /* headers object */ };
async function callApi(ids){
let pages;
for(let i = 0; i < ids.length; i++){
try {
pages = await fetchData(hds,ids[i]);
await process(pages); // no need for await if the process function is sync
}
catch(e){
console.log(e);
}
}
}
I'm new to js and async/await stuff and I want to build distance matrix using yandex maps api and then optimize the route using ACO algorithm.
Everything works, but my distance matrix generating so long because of await for every request in loop. I know that i should avoid it but i have no idea how.
My distance matrix should be done before calling ACO algorithm function.
async function buildDistanceMatrix(ymaps) {
const n = routeCoords.length;
let distanceMatrix = [];
console.log(routeCoordsRef.current);
for (let i = 0; i < n; i++) {
let newArr = [];
for (let j = 0; j < n; j++) {
await ymaps.route([routeCoordsRef.current[i], routeCoords[j]]).then((route) => {
newArr.push(route.getLength())
console.log(i, j, routeCoordsRef.current[i], routeCoords[j], route.getLength());
});
}
distanceMatrix.push(newArr);
}
return distanceMatrix;
}
let distanceMatrix = await buildDistanceMatrix(ymaps);
// and here my distance matrix done and i'm calling another function that uses distanceMatrix
I think that you need to take into consideration the following two notes...
Note 1: Typically when dealing with promises, either use "async" / "await" or stick with the ".then()" syntax, but don't mix them. For example...
async function buildDistanceMatrix(ymaps) {...
let route = await ymaps.route( [ routeCoordsRef.current[ i ], routeCoords[ j ] ] );
newArr.push( route.getLength() );
console.log( i, j, routeCoordsRef.current[ i ], routeCoords[ j ], route.getLength() );
...}
...or...
function buildDistanceMatrix(ymaps) {...
ymaps.route( [ routeCoordsRef.current[ i ], routeCoords[ j ] ] ).then( ( route ) => {
newArr.push( route.getLength() );
console.log( i, j, routeCoordsRef.current[ i ], routeCoords[ j ], route.getLength() );
} );
...}
Generally speaking, the "async" / "await" is more readable as it provides a syntax that reads procedurally, and helps avoid nesting hell of the then() functions.
Note 2: It appears that the Yandex router function makes use of a web service...
https://yandex.com/dev/maps/jsapi/
https://yandex.com/dev/maps/jsbox/2.1/router
...and hence the need for the promise, as the javascript code essentially suspends awaiting on the promise to resolve once the Yandex server responds.
Suggest looking into the Promise.all() function...
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
...as this permits your function to initiate a number of promises without waiting for any promises to complete, and only after creating all the promises does your function then await for all the promises to resolve.
The primary benefit is that the remote server is working on all the requests concurrently.
Before running off down this path, check on the Yandex terms of service to determine if there are restrictions on the number of concurrent outstanding calls. Additionally, well designed web services will throttle the number of concurrent requests, so there might be some large gains in performance for a small number of web calls, but after a certain limit, the web server throttle kicks in and the calls are then essentially queued (or ignored!) and handled synchronously again...
EDIT: Use of Promise.all with async / await syntax.
In short, your function should not await when creating the promises, but instead, capture the promises into an array. Then, once all the promises have been created (in this case, initiating a series of web service calls), then await on the Promise.all() function, passing all the unresolved promises. The resulting array will correspond directly with the order that the promises were passed in Promise.all(). Here is generally how your function will look. Note the addition of parallelArray, which is a means of capturing any ancillary data that you need when later making use of arrayOfResults...
async function buildDistanceMatrix(ymaps) {...
let arrayOfPromises = [];
let parallelArray = [];
for ( let i = 0; i < n; i++ ) {
for (let j = 0; j < n; j++ ) {
arrayOfPromises.push( ymaps.route( [ routeCoordsRef.current[ i ], routeCoords[ j ] ] ) );
parallelArray.push( { i: i, j: j, rci: routeCoordsRef.current[ i ], rcj: routeCoords[ j ] );
}
}
let arrayOfResults = await Promise.all( arrayOfPromises );
...}
Hope this helps...
I have a function that uses spawn and calls ffmpeg to trim a file, which returns a Promise. I need to trim an audio file into multiple parts, so I iterate with a for loop which launches the trimming function, I only need the trimming process to complete (which generates the trimmed file) in the for loop (so I don't need to use await) but still need to make sure the trimming function did not return error.
for (let i = 0; i < myArray.length; i++) {
trim().catch(console.error); //<--- Trim asynchronously
linkedList.insertAtEnd(i);
}
My problem is what is a good way to run the for loop trimming the audio asynchronously?
Edit: I don't want the trim() function to complete to iterate the next element in the array.
Edit: The difference between my question and this one is I don't want the for loop to wait for the trim() function in order to go on to the next element in the array.
To know if there is any error, you need to deal with it at some point, and to do that, you need to know when they are all done, even if you do not want to wait on each before the next iteration of the loop.
To do this, either store each promise into an array (or use .map). You can then Promise.all to trigger code when each call of trim has completed.
I've added trim and myArray definitions and commented out the insertAtEnd for a "run-able" example (though having the linked list insert not being dependent on the trim makes little sense, nor does trim not taking any variants from the loop itself).
function trim() {
return new Promise((resolve, reject) => {
setTimeout(() => resolve(), 5)
});
}
let myArray = [1, 2, 3, 4];
let results = [];
for (let i = 0; i < myArray.length; i++) {
results.push(trim().then(() => true).catch(() => false))
console.log(`iteration ${i} finished`);
//linkedList.insertAtEnd(i);
}
Promise.all(results).then((r) => {
console.log(`trim() results: ${r}`);
if (r.every((x) => x)) {
console.log('success');
} else {
console.log('an error occurred');
}
});
I was fetching data asynchronously using async/await and then I am running two for loops one after another. So my question is will the for loops overlap each other for big data sets as js is asynchronous and if yes how to solve this?
And for what condition loops can overlap?
Actually, I am trying to make a dropdown and its working but I had this doubt.
const createDropdown = async language => {
let i = 0;
let listPromise = new Promise((resolve, reject) => {
db.ref("Products/" + language).on('value', snapshot => {
resolve(snapshot.val())
})//Fetching Data
})
let list = await listPromise;
for(i in list)
dropdown.remove(0)
for(i in list)
dropdown.options[dropdown.options.length] = new Option(list[i].name, list[i].name)
}
I am running this code and the for loops are not getting overlapped but is there a condition that it will?
Loops which are put in the code one after the other will never overlap either the code inside the loops are synchronous or asynchronous.
for (var i = 0; i < 10; i++) {
doSomethingSync()
}
for (var j = 0; j < 10; j++) {
createPromise().then((res) => {console.log(res))
}
for (var k = 0; k < 10; k++) {
var res = await createPromise();
console.log(res);
}
Above, the "I" loop completes all its operations and then the "J" loop, and then the "K" loop.
Here is the order and the details of each operation
The "I" loop serializes the 10 synchronous operations.
The "J" loop create 10 different promises. They maybe resolved in a different order.
The "K" loop creates 10 serialized promises. Each iteration waits until the promise is resolved before going for the net one.
1, 2, and 3 always happen one after the other.
I used this solution to work with axios, with about 1000 requests to a server and works fine, maybe this can be your solution too. In this code you will make a promise of your db.ref and wait for the response then you use that to manipulate.
const createDropdown = async (language) => {
const listPromise = await Promise.all(
return await db.ref("Products/" + language).on ('value', snapshot => snapshot.val())
)
for(i in listPromise)
dropdown.remove(0)
for(i in listPromise)
dropdown.options[dropdown.options.length] = new Option(list[i].name, list[i].name)
}
I have a complicated javascript function with promises inside it.
Here is my code :
var chunkProjectList = function(list, project, accounts) {
return new Promise((resolve, reject) => {
// init delta
let delta = 5
if ((accounts.length * delta) > list.length ) {
delta = (list.length / accounts.length)
}
// init chunked list
let chunkedList = []
for (let i = 0; i < accounts.length; i++) {
chunkedList.push([])
}
// loop on all users
for (let i = 0; i < list.length; i++) {
isInBlacklist(list[i], project.id)
.then(() => { // current user is in the blacklist
ProjectModel.deleteElementFromList(project.id, list[i])
if (i === list.length - 1) {
// no screen_names available, cu lan
return resolve(chunkedList)
}
})
.catch(() => { // we continue
let promises = []
for (let j = 0; j < accounts.length; j++) {
promises[j] = FollowedUserModel.getFollowedUserByScreenName(list[i], accounts[j].id)
}
// we checked a screen_name for all accounts
Promise.all(promises)
.then((rows) => {
for (let k = 0; k < rows.length; k++) {
if (rows[k][0].length === 0 && chunkedList[k].length < delta && list[i] !== '') {
console.log('we push')
chunkedList[k].push(list[i])
break
}
console.log('we cut (already followed or filled chunklist)')
if (k === rows.length - 1) {
// determine if is cancer screen_name or just filled chunklists
for (let l = 0; l < chunkedList.length; l++) {
if (chunkedList[l].length < delta) {
console.log('we cut because nobody wants ya')
ProjectModel.deleteElementFromList(project.id, list[i])
ProjectModel.addElementToBlackList(project.id, list[i])
}
if (l === chunkedList.length - 1) {
// you are all filled, cu lan
return resolve(chunkedList)
break
}
}
}
}
if (i === list.length - 1) {
// no screen_names available, cu lan
over = 1
return resolve(chunkedList)
}
})
})
}
})
}
My program is looping on a list of usernames, and tries to share it between the accounts, with a limit called 'delta' for each account
example:
My list contains 1000 usernames, the delta is 100 and I have 4 accounts
The expected result is an array of arrays, containing for each arrays, 100 usernames.
chunkedList (array) => ( array 1: length 100, array 2: length 100 ... )
The problem I have is the following :
When the delta is reached for all the chunkedList arrays, I just wanna exit this function, and stop every work inside, even the running promises. Just when I 'return resolve(chunkedList)'.
But the program continues to run, and it's a real problem for performances.
I know it's difficult to understand the code. Thanks
I am having difficulty understanding what your code is trying to accomplish. However, it looks like you could split your code up into a few separate promise functions. This would help clarity and should allow you stop execution. Without being able to test your code, the main problem seems to be caused by your main loop. In your code you have the following:
.then(() => {
....
return resolve(chunkedList);
}
The "return" statement here is only going to return from the inner .then() call and not from your main Promise. Thus, your main loop will continue to execute all of its code on the whole array. Essentially, it seems that your loop will continue to modify your main Promise's resolved value and never return until the whole loop has completed.
My recommendation is that you split out your main loop contents to be a separate method that takes a list as a parameter. This method will return a Promise. With that method, you can create a Promise.all() using your main loop and then add a .catch() method. The catch() method will occur when you reject one of the method calls (therefore not completing the rest of the promises in the rest of the Promise.all array).
I apologize if my above suggestion does not make any sense. I am trying to write this quickly. In summary, if you can split out the different steps of your method into their own methods that return their own promises, you can use promises very effectively allowing you to chain, execute multiple tasks in parallel, and/or execute tasks sequentially.