I have a complicated javascript function with promises inside it.
Here is my code :
var chunkProjectList = function(list, project, accounts) {
return new Promise((resolve, reject) => {
// init delta
let delta = 5
if ((accounts.length * delta) > list.length ) {
delta = (list.length / accounts.length)
}
// init chunked list
let chunkedList = []
for (let i = 0; i < accounts.length; i++) {
chunkedList.push([])
}
// loop on all users
for (let i = 0; i < list.length; i++) {
isInBlacklist(list[i], project.id)
.then(() => { // current user is in the blacklist
ProjectModel.deleteElementFromList(project.id, list[i])
if (i === list.length - 1) {
// no screen_names available, cu lan
return resolve(chunkedList)
}
})
.catch(() => { // we continue
let promises = []
for (let j = 0; j < accounts.length; j++) {
promises[j] = FollowedUserModel.getFollowedUserByScreenName(list[i], accounts[j].id)
}
// we checked a screen_name for all accounts
Promise.all(promises)
.then((rows) => {
for (let k = 0; k < rows.length; k++) {
if (rows[k][0].length === 0 && chunkedList[k].length < delta && list[i] !== '') {
console.log('we push')
chunkedList[k].push(list[i])
break
}
console.log('we cut (already followed or filled chunklist)')
if (k === rows.length - 1) {
// determine if is cancer screen_name or just filled chunklists
for (let l = 0; l < chunkedList.length; l++) {
if (chunkedList[l].length < delta) {
console.log('we cut because nobody wants ya')
ProjectModel.deleteElementFromList(project.id, list[i])
ProjectModel.addElementToBlackList(project.id, list[i])
}
if (l === chunkedList.length - 1) {
// you are all filled, cu lan
return resolve(chunkedList)
break
}
}
}
}
if (i === list.length - 1) {
// no screen_names available, cu lan
over = 1
return resolve(chunkedList)
}
})
})
}
})
}
My program is looping on a list of usernames, and tries to share it between the accounts, with a limit called 'delta' for each account
example:
My list contains 1000 usernames, the delta is 100 and I have 4 accounts
The expected result is an array of arrays, containing for each arrays, 100 usernames.
chunkedList (array) => ( array 1: length 100, array 2: length 100 ... )
The problem I have is the following :
When the delta is reached for all the chunkedList arrays, I just wanna exit this function, and stop every work inside, even the running promises. Just when I 'return resolve(chunkedList)'.
But the program continues to run, and it's a real problem for performances.
I know it's difficult to understand the code. Thanks
I am having difficulty understanding what your code is trying to accomplish. However, it looks like you could split your code up into a few separate promise functions. This would help clarity and should allow you stop execution. Without being able to test your code, the main problem seems to be caused by your main loop. In your code you have the following:
.then(() => {
....
return resolve(chunkedList);
}
The "return" statement here is only going to return from the inner .then() call and not from your main Promise. Thus, your main loop will continue to execute all of its code on the whole array. Essentially, it seems that your loop will continue to modify your main Promise's resolved value and never return until the whole loop has completed.
My recommendation is that you split out your main loop contents to be a separate method that takes a list as a parameter. This method will return a Promise. With that method, you can create a Promise.all() using your main loop and then add a .catch() method. The catch() method will occur when you reject one of the method calls (therefore not completing the rest of the promises in the rest of the Promise.all array).
I apologize if my above suggestion does not make any sense. I am trying to write this quickly. In summary, if you can split out the different steps of your method into their own methods that return their own promises, you can use promises very effectively allowing you to chain, execute multiple tasks in parallel, and/or execute tasks sequentially.
Related
This question already has answers here:
Limited parallelism with async/await in Typescript/ES7
(3 answers)
Closed 3 years ago.
I have to split a big file into hundreds of uploadParts and upload those uploadParts to the server.
How do I control the number of upload request with 5 requests at most simultaneous?
this.uploadFileChunks.map(uploadPart => doUpload)...
This is similar to throttling. You should be able to find a few libraries in npm by searching for "throttle promise" or "promise limit". These are a few I found:
p-limit
throat
p-queue
Also, here's a simple implementation:
// only allows a function to be run a certain number of times at once. if
// that number is reached, will queue the other function calls and wait
// until a spot opens up.
//
// n (int): number of promises to run concurrently
// f (function): must return a Promise.
const nConcurrent = (n, f) => {
let numRunning = 0
let queue = []
const runOne = ({args, resolve, reject}) => {
numRunning++
return f.apply(null, args)
.then(result => {
numRunning--
if(queue.length) {
runOne(queue.pop())
}
resolve(result)
})
.catch(err => {
if(queue.length) {
runOne(queue.pop())
}
reject(err)
})
}
return (...args) => {
return new Promise((resolve, reject) => {
if(numRunning >= n) {
queue.push({args, resolve, reject})
}
else {
runOne({args, resolve, reject})
}
})
}
}
Your solution would then look like:
const doUploadLimited = nConcurrent(5, doUpload)
this.uploadParts.map(doUploadLimited)
If you want to limit the uploads to 5 simultaneous requests, you could perhaps use async.mapLimit; example:
async.mapLimit(this.uploadParts, 5, (part, onDone) => {
axios.post(/* ... */) // or whatever API you want to use
.then((res) => onDone(undefined, res))
.catch(onDone)
}, (err, results) => {
// if err is undefined, then results is an array of
// responses, or whatever you called your onDone callback
// with.
});
Another approach (pretty reasonable) is to have your this.uploadParts array contain no more than 5 elements, and use async.map without a limit.
Hope this helps.
How about distributing your data among a couple of bags, then start uploading each bag:
let N = 347 // total record count
let M = 5 // number of bags you want
// let me mock your data real quick...
let data = Array(N)
for(var i = 0; i < N; i++){
data[i] = i
}
// create bags...
let bagSize = N / M
let bags = Array(M)
for(var i = 0; i < M; i++) {
let bag = data.slice(i*bagSize, (i+1)*bagSize)
bags[i] = bag
// ...and use immediately
bag.map(uploadPart => doUpload)
}
// ... or later
I'm trying to asynchronously add items to my array, and when the array is completed, it would then continue the code. I've had trouble seeing how I could implement promises because the array doesn't start off fully established.
I was at first thinking that I could perhaps wait for a piece of the table to exist before continuing the function, but then it still has to wait for that one to complete.
async function getListFact() {
for (var i = 0; i < arrayOfPlaces.length; i++) {
placeCell = await resolveList(arrayOfPlaces[i], i)
placeData.push(
placeCell
)
console.log(placeCell)
if (placeData) { // I left this as just placeData, but I need to wait
//for all objects in the array to be completed before it reaches past here.
placeData.sort(function (knit1, pearl2) {
if (Number(knit1.valuePlayers) > Number(pearl2.valuePlayers))
return -1;
if (Number(knit1.valuePlayers) < Number(pearl2.valuePlayers))
return 1;
});
return placeData
}
}
}
I need it to asynchronously push data into the array so it can be done quickly, and then I need it to, after all of the data is in the array, sort the array numerically and then return the data to what called it.
I think you want to use Promise.all. Essentially create a list of tasks you want to do and then wait until all are done.
const arrayOfPlaces = ['US', 'Bills shack', 'Down the road', 'My neighbor', 'Europe'];
async function getListFact() {
const resolvePlace = place => Promise.resolve(place); // Simulate an async task
const promises = arrayOfPlaces.map(resolvePlace);
const placeCells = await Promise.all(promises); // wait until everything is resolved
console.log(placeCells);
}
getListFact();
This should do the trick
async function getListFact() {
for (let i = 0; i < arrayOfPlaces.length; i++) {
placeData.push(resolveList(arrayOfPlaces[i], i));
}
placeData = await Promise.all(placeData);
placeData.sort(function (knit1, pearl2) {
if (Number(knit1.valuePlayers) > Number(pearl2.valuePlayers))
return -1;
if (Number(knit1.valuePlayers) < Number(pearl2.valuePlayers))
return 1;
return 0;
});
return placeData;
}
the issue with your code seems to be mainly } in the wrong places!
This is your code, with an indentation of 4 characters to make it more obvious, and tidied up a bit - but is exactly the code you posted in function
async function getListFact() {
for (var i = 0; i < arrayOfPlaces.length; i++) {
placeCell = await resolveList(arrayOfPlaces[i], i)
placeData.push(placeCell)
console.log(placeCell)
if (placeData) {
placeData.sort(function (knit1, pearl2) {
if (Number(knit1.valuePlayers) > Number(pearl2.valuePlayers)) return -1;
if (Number(knit1.valuePlayers) < Number(pearl2.valuePlayers)) return 1;
});
return placeData
} // 1
} // 2
}
Your code will get the first placeCell, push it in placeData, sort this array (with a single element) and return placeData ... so, only the first arrayOfPlaces element will ever be processed
Now, see those } marked 1 and 2
Watch, all I did was move them
async function getListFact() {
for (var i = 0; i < arrayOfPlaces.length; i++) {
placeCell = await resolveList(arrayOfPlaces[i], i);
placeData.push(placeCell);
console.log(placeCell);
} // 2
if (placeData) {
placeData.sort(function (knit1, pearl2) {
if (Number(knit1.valuePlayers) > Number(pearl2.valuePlayers)) return -1;
if (Number(knit1.valuePlayers) < Number(pearl2.valuePlayers)) return 1;
return 0; //+ should return 0 when equal
});
} // 1
return placeData;
}
Now this code will wait for the loop to finish before sorting and returning placeData - note the additional return 0 in the sort callback
Also note: getListFact returns a PROMISE that resolves to the array placeData - is this what your calling code expects? A promise?
One other thing, since placeData is an array, if (placeData) { will always be true - so instead of
if (placeData) {
I would use
if (placeData.length > 1) {
Also
for (var i = 0; i < arrayOfPlaces.length; i++) {
placeCell = await resolveList(arrayOfPlaces[i], i);
placeData.push(placeCell);
console.log(placeCell);
}
can be rewritten as
placeData.push(... await Promise.all(arrayOfPlaces.map(resolveList)));
Yet another NOTE: Your code as you wrote it uses "external" variables, arrayOfPlaces and placeData. If you call the function getListFact twice without changing placeData, the second call will ADD to the existsing placeData array - not sure this is by design (no comment on this design as it may be what you need), or perhaps you're clearing placeData before calling getListFact (which would be, in my opinion, poor code design) - but as no code outside this function is even described, let alone shown, the code in this answer assumes you know what you want
I was fetching data asynchronously using async/await and then I am running two for loops one after another. So my question is will the for loops overlap each other for big data sets as js is asynchronous and if yes how to solve this?
And for what condition loops can overlap?
Actually, I am trying to make a dropdown and its working but I had this doubt.
const createDropdown = async language => {
let i = 0;
let listPromise = new Promise((resolve, reject) => {
db.ref("Products/" + language).on('value', snapshot => {
resolve(snapshot.val())
})//Fetching Data
})
let list = await listPromise;
for(i in list)
dropdown.remove(0)
for(i in list)
dropdown.options[dropdown.options.length] = new Option(list[i].name, list[i].name)
}
I am running this code and the for loops are not getting overlapped but is there a condition that it will?
Loops which are put in the code one after the other will never overlap either the code inside the loops are synchronous or asynchronous.
for (var i = 0; i < 10; i++) {
doSomethingSync()
}
for (var j = 0; j < 10; j++) {
createPromise().then((res) => {console.log(res))
}
for (var k = 0; k < 10; k++) {
var res = await createPromise();
console.log(res);
}
Above, the "I" loop completes all its operations and then the "J" loop, and then the "K" loop.
Here is the order and the details of each operation
The "I" loop serializes the 10 synchronous operations.
The "J" loop create 10 different promises. They maybe resolved in a different order.
The "K" loop creates 10 serialized promises. Each iteration waits until the promise is resolved before going for the net one.
1, 2, and 3 always happen one after the other.
I used this solution to work with axios, with about 1000 requests to a server and works fine, maybe this can be your solution too. In this code you will make a promise of your db.ref and wait for the response then you use that to manipulate.
const createDropdown = async (language) => {
const listPromise = await Promise.all(
return await db.ref("Products/" + language).on ('value', snapshot => snapshot.val())
)
for(i in listPromise)
dropdown.remove(0)
for(i in listPromise)
dropdown.options[dropdown.options.length] = new Option(list[i].name, list[i].name)
}
I have a loop that cycles around and creates REST Queries, creating and running a different one each time.
In the future I imagine the number of REST Queries that could be made will increase far higher than the browser will take at once (over 20,000).
I want to do something along the lines of counting how many loops have been done, and after every 500 or so, pausing for a few seconds to allow the browser to catch up with the REST responses, and then continuing.
How is this done in react JS?
example code:
for (var i = 0; i < array.length; i++)
{
query += i;
axious.get(query) . then { ...does stuff here... }
//want something along the lines of if multiple of 500, wait(1000)
}
the simplest approach is to make a waitIf function that returns a Promise and accepts a condition.
If the condition is true, it will wait then execute the callback, otherwise, it will execute the callback directly.
a simple implementation would be.
function waitIf(condition, duration, callback) {
if (condition) {
// return a Promise that wait for a `duration` ms using timeout
return Promise((resolve) => {
setTimeout(() => {
resolve(callback());
}, duration);
})
}
return callback();
}
for (let i = 0; i < array.length; i++) {
query += i;
// unify the call here
waitIf(i % 500 === 0, 1000, () => axious.get(query)).then();
}
I've got a method that returns a promise and internally that method makes a call to an API which can only have 20 requests every minute. The problem is that I have a large array of objects (around 300) and I would like to make a call to the API for each one of them.
At the moment I have the following code:
const bigArray = [.....];
Promise.all(bigArray.map(apiFetch)).then((data) => {
...
});
But it doesnt handle the timing constraint. I was hoping I could use something like _.chunk and _.debounce from lodash but I can't wrap my mind around it. Could anyone help me out ?
If you can use the Bluebird promise library, it has a concurrency feature built in that lets you manage a group of async operations to at most N in flight at a time.
var Promise = require('bluebird');
const bigArray = [....];
Promise.map(bigArray, apiFetch, {concurrency: 20}).then(function(data) {
// all done here
});
The nice thing about this interface is that it will keep 20 requests in flight. It will start up 20, then each time one finishes, it will start another. So, this is a potentially more efficient than sending 20, waiting for all to finish, sending 20 more, etc...
This also provides the results in the exact same order as bigArray so you can identify which result goes with which request.
You could, of course, code this yourself with generic promises using a counter, but since it is already built in the the Bluebird library, I thought I'd recommend that way.
The Async library also has a similar concurrency control though it is obviously not promise based.
Here's a hand-coded version using only ES6 promises that maintains result order and keeps 20 requests in flight at all time (until there aren't 20 left) for maximum throughput:
function pMap(array, fn, limit) {
return new Promise(function(resolve, reject) {
var index = 0, cnt = 0, stop = false, results = new Array(array.length);
function run() {
while (!stop && index < array.length && cnt < limit) {
(function(i) {
++cnt;
++index;
fn(array[i]).then(function(data) {
results[i] = data;
--cnt;
// see if we are done or should run more requests
if (cnt === 0 && index === array.length) {
resolve(results);
} else {
run();
}
}, function(err) {
// set stop flag so no more requests will be sent
stop = true;
--cnt;
reject(err);
});
})(index);
}
}
run();
});
}
pMap(bigArray, apiFetch, 20).then(function(data) {
// all done here
}, function(err) {
// error here
});
Working demo here: http://jsfiddle.net/jfriend00/v98735uu/
You could send 1 block of 20 requests every minute or space them out 1 request every 3 seconds (latter probably preferred by the API owners).
function rateLimitedRequests(array, chunkSize) {
var delay = 3000 * chunkSize;
var remaining = array.length;
var promises = [];
var addPromises = function(newPromises) {
Array.prototype.push.apply(promises, newPromises);
if (remaining -= newPromises.length == 0) {
Promise.all(promises).then((data) => {
... // do your thing
});
}
};
(function request() {
addPromises(array.splice(0, chunkSize).map(apiFetch));
if (array.length) {
setTimeout(request, delay);
}
})();
}
To call 1 every 3 seconds:
rateLimitedRequests(bigArray, 1);
Or 20 every minute:
rateLimitedRequests(bigArray, 20);
If you prefer to use _.chunk and _.debounce1 _.throttle:
function rateLimitedRequests(array, chunkSize) {
var delay = 3000 * chunkSize;
var remaining = array.length;
var promises = [];
var addPromises = function(newPromises) {
Array.prototype.push.apply(promises, newPromises);
if (remaining -= newPromises.length == 0) {
Promise.all(promises).then((data) => {
... // do your thing
});
}
};
var chunks = _.chunk(array, chunkSize);
var throttledFn = _.throttle(function() {
addPromises(chunks.pop().map(apiFetch));
}, delay, {leading: true});
for (var i = 0; i < chunks.length; i++) {
throttledFn();
}
}
1You probably want _.throttle since it executes each function call after a delay whereas _.debounce groups multiple calls into one call. See this article linked from the docs
Debounce: Think of it as "grouping multiple events in one". Imagine that you go home, enter in the elevator, doors are closing... and suddenly your neighbor appears in the hall and tries to jump on the elevator. Be polite! and open the doors for him: you are debouncing the elevator departure. Consider that the same situation can happen again with a third person, and so on... probably delaying the departure several minutes.
Throttle: Think of it as a valve, it regulates the flow of the executions. We can determine the maximum number of times a function can be called in certain time. So in the elevator analogy.. you are polite enough to let people in for 10 secs, but once that delay passes, you must go!