Parallel rxJs Observables without waiting for all to complete - javascript

Is there a way to request many similar requests in parallel using rxJs Observables in parallel without waiting for all requests to complete. I want to handle as each one comes. I don't want to use a loop of requests. Use case is one of the request is slow and there are many of them
I tried forkJoin (below) but it waits for all to complete. and mergeMap is not taking an array. Any help? Thanks
/*
// example of many posts
const post1 = getPostHTTP(1);
const post2 = getPostHTTP(2); // A slow post
const post3 = getPostHTTP(3);
...
*/
const promises = [];
for (let i = 0; i < 100; i++) {
promises.push(getPostHTTP(i))
}
forkJoin(promises).pipe(
finalize(() => {
//displayPost
})).subscribe(posts => {
posts.forEach((post: any) => {
displayPost(post)
})
})

I think that a mix of from and mergeMap is what you are looking for. The code could look like this
for (let i = 0; i < 100; i++) {
promises.push(getPostHTTP(i))
}
from(promises).pipe(
mergeMap(post => displayPost(post)),
).subscribe(
next: result => {// do something with the result of each displayPost operation},
error: err => {// handle errors},
complete: () => {// do something when everything is completed}
)

You should convert Promise to Observable via from (https://www.learnrxjs.io/learn-rxjs/operators/creation/from) and merge them into one Observable via merge (https://www.learnrxjs.io/learn-rxjs/operators/combination/merge).
It will be like:
merge(...promises.map(p => from(p))
.subscribe(post => {
// do smth with post
});

You can use Promise API to send multiple requests simultaneously and await them all to resolve.
If you have a condition that you need to run those promises sequentially, then this solution might not work for you.
Excerpt is from MDN.
var p1 = new Promise((resolve, reject) => {
setTimeout(resolve, 1000, 'one');
});
var p2 = new Promise((resolve, reject) => {
setTimeout(resolve, 2000, 'two');
});
var p3 = new Promise((resolve, reject) => {
setTimeout(resolve, 3000, 'three');
});
var p4 = new Promise((resolve, reject) => {
setTimeout(resolve, 4000, 'four');
});
var p5 = new Promise((resolve, reject) => {
reject('reject');
});
Promise.all([p1, p2, p3, p4, p5]).then(values => {
console.log(values);
}, reason => {
console.log(reason)
});
//From console:
//"reject"
//You can also use .catch
Promise.all([p1, p2, p3, p4, p5]).then(values => {
console.log(values);
}).catch(reason => {
console.log(reason)
});

Related

Promise chaining unhandled promise

I'm learning Node.js right now and practicing using EventEmitter along with promises.
The successfulOrder function runs through both promises: verifyStockP and verifyCardP
both promises are super simple.
I noticed that when I purposely make one of the two promises reject, the program works as expected. The catch error code runs as expected.
If I force both promises to fail, I get the following error message:
'(node:2316) UnhandledPromiseRejectionWarning: card invalid;'
How would I make it so we don't get this error if both the verifyStockP and verifyCardP promises reject?
Thank you all in advance!
If you guys have any tips on something else I should fix on the code I'd be super grateful to hear it!
const EventEmitter = require('events');
const emitter = new EventEmitter();
const successfulOrder = (prod) => {
return console.log(`Your ${prod} order has processed!`)
}
const failedOrder = (prod) => {
return console.log(`Your ${prod} order has failed to process.`)
}
emitter.on('order_successful', successfulOrder);
emitter.on('order_failed', failedOrder);
const submitOrder = (prod, prodQuantity, userCardDigits) => {
const currentStock = 10;
const verifyStockP = new Promise((resolve, reject) => {
if(prodQuantity <= currentStock) {
resolve('stock available')
}
else {
reject('out of stock')
}
})
const verifyCardP = new Promise((resolve,reject) => {
let cardDigits = 16;
if (cardDigits === userCardDigits) {
resolve('card valid')
}
else {
reject('card invalid')
}
})
verifyStockP
.then((resolvedVal) => {
console.log(resolvedVal);
return verifyCardP
}).then((resolvedVal) => {
console.log('card valid')
emitter.emit('order_successful', prod)
}).catch((error) => {
console.log(error)
// emitter.emit('order_failed', prod)
})
}
submitOrder('sunglasses', 15, 17)
The issue is that if the first Promise rejects, the second Promise is never part of the Promise chain that has the catch
Two possible solutions ...
One way to handle this is separately add a "dummy" catch to the second promise
const p1 = new Promise((resolve, reject) => setTimeout(reject, 200, 'p1'));
const p2 = new Promise((resolve, reject) => setTimeout(reject, 100, 'p2'));
p1
.then(r => p2)
.catch(e => console.log('error', e));
p2.catch(e => console.log('error2', e));
You can see the result if first promise doesn't reject here
const p1 = new Promise((resolve, reject) => setTimeout(resolve, 200, 'p1'));
const p2 = new Promise((resolve, reject) => setTimeout(reject, 100, 'p2'));
p1
.then(r => p2)
.catch(e => console.log('error', e));
p2.catch(e => console.log('error2', e));
both catch blocks are run
If, however, the second Promise has no reliance at all on the first Promise, and in the code the way you wrote it means that the two Promises do run in parallel (as much as they can since you have no asynchrony in your code) - use Promise.all
const p1 = new Promise((resolve, reject) => setTimeout(reject, 100, 'p1'));
const p2 = new Promise((resolve, reject) => setTimeout(reject, 200, 'p2'));
Promise.all([p1, p2])
.then(([r1, r2]) => console.log(r1, r2))
.catch(e => console.log('error', e));

What does Promise.all actually do under the hood?

I am trying to understand Promise.all here.
What I did here was to covert below code using Promise.all to achieve the same result.
I understand that Promise all combine the data1, data2.
My question here is that how does Promise.All work without resolve method?
Does Promise resolve those data within the method itself?
Please advise.
const readAllUsersChaining = () => {
return new Promise((resolve, reject) => {
let result = [];
getDataFromFilePromise(user1Path)
.then((data) => {
result.push(JSON.parse(data)); // what are you doing? he's gone mad...
return getDataFromFilePromise(user2Path);
})
.then((data) => {
result.push(JSON.parse(data));
result ? resolve(result) : reject(result);
});
});
};
const readAllUsers = () => {
const data1 = getDataFromFilePromise(user1Path);
const data2 = getDataFromFilePromise(user2Path);
console.log(data1, data2);
return Promise.all([data1, data2]).then((data) => {
return data.map((el) => JSON.parse(el));
});
};
My question here is that how does Promise.All work without resolve method?
Not quite sure what you mean. Promise.all simply creates a new promise internally that is resolved when all other promises are resolved.
Here is a simple implementation of Promise.all for the case that arguments are always promises:
function all(promises) {
if (promises.length === 0) {
return Promise.resolve([]);
}
return new Promise((resolve, reject) => {
const results = [];
let resolved = 0;
promises.forEach((promise, i) => {
promise.then(
result => {
results[i] = result;
resolved++;
if (resolved === promised.length) {
resolve(results);
}
},
error => reject(error)
);
});
}
Promise.all allows to wait until all promise passed as arguments to be fullfilled before the then method attach to be execute.
The real use case would be when you have to perform five call to an API, an you want to perform some treatement only when you have get the data from all the API call. you can rely on the Promise.all function which will wait all passed promised to be fullfilled for it to be fullfiled on it turn.
Bellow I provide an example of a Promise.all call. which has two Promises passed as argument. the first one has a timer which fullfilled it after 5 second and the second if fullfiled immediately but. the Promise.all will be fullfilled only when both of the promise passed as argument ar fullfilled
const firstPromise = new Promise((resolve, reject) => {
setTimeout(()=> {
return resolve({
name: "First Promise"
});
}, 5000);
});
const secondPromise = new Promise((resolve, reject) => {
return resolve({
name: "Second Promise"
});
})
const all = Promise.all([firstPromise, secondPromise]).then((response) => {
console.log(response);
});

Promise.all.then is executed before the individual promises are completed

The following sample code represents what am trying to do, and am stumped as to why the console.log inside the Promises.all.then() is completed before even the individual promises are completed. I am thinking that the map statement has something to do with it , may be, but am unsure.
My intention of using a map was to collect all successes and or failures and notify the user later on bulk operations.
var requests = []; var success=[]; var failures = [];
requests.push(new Promise((resolve, reject) => setTimeout(() => resolve("foo success"), 2000)));
requests.push(new Promise((resolve, reject) => setTimeout(() => reject("bar failure"), 1000)));
requests.push(new Promise((resolve, reject) => setTimeout(() => resolve("foo2 success"), 2000)));
requests.push(new Promise((resolve, reject) => setTimeout(() => reject("bar2 failure"), 1000)));
Promise.all(requests.map(p => {
p.then(r => {
console.log(r)
success.push(r);
});
p.catch((e) => { console.log(e); failures.push(e); });
})).then(function () { console.log("ALL promises completed")})
May i know what is wrong with the above code? Am i not implementing the promises as intended.?
It's because your requests.map(...) callback doesn't return anything, so you're mapping the Promises to undefined values. Therefore, you're awaiting an array of undefined values with the Promise.all call. Change to the following:
Promise.all(requests.map(p => {
p.then(r => {
console.log(r)
console.log("POSTed a record successfuly");
success.push(r);
});
p.catch((e) => { console.log(e); failures.push(e); });
return p;
})).then(function () { console.log("POSTED ALL")})
Update:
As #JoeFrambach pointed out, if you want the Promise.all to contain the contents of the original requests Promises, you'll want to return the original Promises. Since that's most likely what you want to do, I've updated my answer.
just return promise from your map callback function
var requests = []; var success=[]; var failures = [];
requests.push(new Promise((resolve, reject) => setTimeout(() => resolve("foo success"), 2000)));
requests.push(new Promise((resolve, reject) => setTimeout(() => reject("bar failure"), 1000)));
requests.push(new Promise((resolve, reject) => setTimeout(() => resolve("foo2 success"), 2000)));
requests.push(new Promise((resolve, reject) => setTimeout(() => reject("bar2 failure"), 1000)));
Promise.all(requests.map(p => {
p.then(r => {
console.log(r)
success.push(r);
});
p.catch((e) => { console.log(e); failures.push(e); });
return p; // ADD THIS LINE
})).then(function () { console.log("ALL promises completed")})
.catch(function() {console.log("FAILED PROMISE ALL")});
You need to return a promise within .map function.
Within .map function make last line as return p.
Promise all expects array of promises. You are returning array of undefined from map function so it immediately gets resolved.

ES6 Promise wait for K out N promises to resolve

var p1 = new Promise((resolve, reject) => {
setTimeout(resolve, 1000, 'one');
});
var p2 = new Promise((resolve, reject) => {
setTimeout(resolve, 2000, 'two');
});
var p3 = new Promise((resolve, reject) => {
setTimeout(resolve, 3000, 'three');
});
Promise.all([p1,p2,p3]).then(values => {
console.log(values);
}, reason => {
console.log(reason)
});
How can i wait for 2 promises to get completed ? Promise.race() wait for one promise to get completed.
Edit
I have n number of promises, what i want to achieve is wait for first k number of promises to get resolved and than trigger some event. assume k < n
Edit - 2
I am sure that k number of promise will be successfully resolved out of n numbers of promise given
(Note: Bluebird has a built-in helper function that serves precisely this purpose and has more or less the same behavior as my waitForN method below, so that option is always available)
I don't believe ES6 promises have an elegant built-in way to do this, but you could define the following relatively short now somewhat long helper function to take care of this.
Edit: Added an additional helper function that includes the indices of the successful promises in the result.
Edit: Updated to reject if the number of rejections reaches the point where successful resolution would be impossible.
Edit: Updated to more gracefully work with promises if it is iterable and check edge cases where the result should immediately resolve (e.g. if n === 0)
function waitForN(n, promises) {
let resolved = [];
let failCount = 0;
let pCount = 0;
return new Promise((resolve, reject) => {
const checkSuccess = () => {
if (resolved.length >= n) { resolve(resolved); }
};
const checkFailure = () => {
if (failCount + n > pCount) {
reject(new Error(`Impossible to resolve successfully. n = ${n}, promise count = ${pCount}, failure count = ${failCount}`));
}
};
for (let p of promises) {
pCount += 1;
Promise.resolve(p).then(r => {
if (resolved.length < n) { resolved.push(r); }
checkSuccess();
}, () => {
failCount += 1;
checkFailure();
});
}
checkFailure();
checkSuccess();
});
}
const waitForNWithIndices = (n, promises) =>
waitForN(n, promises.map((p, index) =>
Promise.resolve(p).then(result => ({ index, result }))
));
var p1 = new Promise((resolve, reject) => {
setTimeout(resolve, 1000, 'one');
});
var p2 = new Promise((resolve, reject) => {
setTimeout(resolve, 2000, 'two');
});
var p3 = new Promise((resolve, reject) => {
setTimeout(resolve, 3000, 'three');
});
var p4 = new Promise((resolve, reject) => {
setTimeout(resolve, 1500, 'four');
});
waitForN(2, [p1,p2,p3,p4]).then(values => {
console.log(values);
}, reason => {
console.log(reason)
});
waitForNWithIndices(2, [p1,p2,p3,p4]).then(values => {
console.log(values);
}, reason => {
console.log(reason)
});
You could use a higher order function to bake in the n then run all promises waiting for n to finish
var p1 = new Promise((resolve, reject) => {
setTimeout(resolve, 1000, 'one');
});
var p2 = new Promise((resolve, reject) => {
setTimeout(resolve, 2000, 'two');
});
var p3 = new Promise((resolve, reject) => {
setTimeout(resolve, 3000, 'three');
});
const raceN = n => (...promises) => {
const resolved = []
return new Promise((res, rej) =>
promises.forEach(promise =>
promise.then(x => {
resolved.push(x)
if (resolved.length === n)
res(resolved)
})
.catch(rej)
)
)
}
const race2 = raceN(2)
race2(p1, p2, p3)
.then(results => console.log(results))
race2(Promise.resolve('resolved'), Promise.reject('rejected'))
.then(results => console.log(results))
.catch(reason => console.log(reason))
Given my many comments on JLRishe's answer, I'll also post my subtly different version of the function:
function any(k, iterable) {
const results = [];
let fullfilled = 0, rejected = 0, n = 0;
return new Promise((resolve, reject) => {
k = Math.max(0, Math.floor(k));
function check() {
if (fulfilled == k)
resolve(results);
else if (rejected + k == n + 1)
reject(new Error(`No ${k} of ${n} got fulfilled`));
}
for (const thenable of iterable) {
const i = n++;
Promise.resolve(thenable).then(res => {
if (fulfilled++ < k) results[i] = res;
check();
}, () => {
rejected++;
check();
});
}
check();
});
}
A bit late but maybe this will do it for you:
const first = howMany => (promises,resolved=[],rejected=[]) => {
if(promises.length-rejected.length<howMany){
return Promise.reject([resolved,rejected]);
}
if(resolved.length===howMany){
return Promise.resolve(resolved);
}
if(resolved.length===0&&rejected.length===0){
promises=promises.map(
(p,index)=>Promise.resolve(p)
.then(resolve=>[resolve,index])
.catch(err=>Promise.reject([err,index]))
);
}
return Promise.race(promises)
.then(
([resolve,index])=>
first(howMany)(
promises.filter(
(p,i)=>i!==index
),
resolved.concat([resolve]),
rejected
)
)
.catch(
([err,index])=>
first(howMany)(
promises.filter(
(p,i)=>i!==index
),
resolved,
rejected.concat([err])
)
);
};
const first2 = first(2);
first2([p1,p2,p3])
.then(
result=>//you will have an array of 2 with resolve value(s)
)
.catch(
err=>//you will have 2 arrays, resolved ones and errors
)
EDIT: Updated to match question edits.
Pretty easy to roll your own when the abstraction limits your expressiveness.
This code is short and can easily be reworked to run in legacy ES3 environments.
You can also easily expand its usefulness by adding code to cancel unresolved timeouts. This is what you get when you don't hand everything over to a library; greater simplicity and more capability.
function raceN(n, fns, resolve, reject) {
const res = [], rej = [];
let halt = false;
for (const [idx, fn] of fns.entries()) {
fn(data => update(res, data, idx), data => update(rej, data, idx));
}
function update(arr, data, idx) {
if (halt) return;
arr.push({idx, data});
if ((halt=res.length >= n)) resolve(res);
else if ((halt=rej.length > fns.length - n)) reject(rej);
}
}
DEMO:
function raceN(n, fns, resolve, reject) {
const res = [], rej = [];
let halt = false;
for (const [idx, fn] of fns.entries()) {
fn(data => update(res, data, idx), data => update(rej, data, idx));
}
function update(arr, data, idx) {
if (halt) return;
arr.push({idx, data});
if ((halt=res.length >= n)) resolve(res);
else if ((halt=rej.length > fns.length - n)) reject(rej);
}
}
function rand() { return Math.ceil(Math.random() * 5000) }
var fns = [(resolve, reject) => {
setTimeout(resolve, rand(), 'one');
},
(resolve, reject) => {
setTimeout(resolve, rand(), 'two');
},
(resolve, reject) => {
setTimeout(reject, rand(), 'three');
}];
raceN(2, fns, values => {
console.log("SUCCESS:", values);
}, reason => {
console.log("REJECT:", reason)
});
If you're guaranteed that at least n elements will succeed, it then becomes even shorter and simpler.
function raceN(n, fns, resolve, reject) {
const res = [];
for (const [idx, fn] of fns.entries()) {
fn(data => res.length < n && res.push({idx, data}) == n) && resolve(res),
data => reject({idx, data}));
}
}
DEMO:
function raceN(n, fns, resolve, reject) {
const res = [];
for (const [idx, fn] of fns.entries()) {
fn(data => res.length < n && res.push({idx, data}) == n) && resolve(res),
data => reject({idx, data}));
}
}
function rand() { return Math.ceil(Math.random() * 4000) }
var fns = [(resolve, reject) => {
setTimeout(resolve, rand(), 'one');
},
(resolve, reject) => {
setTimeout(resolve, rand(), 'two');
},
(resolve, reject) => {
setTimeout(reject, rand(), 'three');
}];
raceN(2, fns, values => {
console.log("SUCCESS:", values);
}, reason => {
console.log("REJECTING:", reason)
});
One other idea could be using Promise.reject() to cut it short when the condition is met while catching possible errors thrown, could be as follows;
var p1 = new Promise((v, x) => setTimeout(v, 1000, 'one')),
p2 = new Promise((v, x) => setTimeout(v, 5000, 'two')),
p3 = new Promise((v, x) => setTimeout(v, 1500, 'three')),
pn = (n, ps, k = 0, r = []) => Promise.all(ps.map(p => p.then(v => (k === n - 1 ? Promise.reject(r.concat(v))
: (++k, r.push(v))))))
.catch(r => Array.isArray(r) ? r : Promise.reject(r));
pn(2,[p1,p2,p3]).then(rs => console.log(rs))
.catch(e => console.log(e));
A more modern approach, in 2022, using iter-ops library:
import {pipeAsync, take, waitRace} from 'iter-ops';
// easy-to-read way to define a job function:
const job = (delay, value) => new Promise(resolve => {
setTimeout(() => resolve(value), delay);
});
// list of jobs:
const input = [
job(1000, 'one'),
job(2000, 'two'),
job(3000, 'three')
];
const i = pipeAsync(
input, // our input
waitRace(10), // race-resolve with cache up to 10 items
take(2) // take only the first 2 items
).catch(err => {
console.log(err); // report any iteration-time error
});
// iterating through our list of jobs:
(async function () {
for await(const a of i) {
console.log(a);
}
})();
This will output one and two. You can twist the delays to see the result change accordingly.
CREDITS
I am the author of the library, while operator waitRace was implemented by #Bergi.

How to structure nested Promises

I have a situation where I think the only choice for me is to nest some Promises within each other. I have a Promise that needs to be performed and a method that does something until that Promise is complete. Something like this:
let promise = new Promise((resolve, reject) => {
// Do some stuff
});
doSomethingUntilPromiseisDone(promise);
However, within my Promise, I need to execute another method that returns another Promise:
let promise = new Promise((resolve, reject) => {
fetchValue(url)
.then((value) => {
// Do something here
}).catch((err) => {
console.error(err);
});
});
doSomethingUntilPromiseisDone(promise);
But now, in the fetchValue method's then statement, I have another method I need to execute that, guess what, returns another Promise:
let promise = new Promise((resolve, reject) => {
fetchValue(url)
.then((value) => {
saveToCache(value)
.then((success) => {
console.log('success!!');
resolve('success');
});
}).catch((err) => {
console.error(err);
});
});
doSomethingUntilPromiseisDone(promise);
So in the end, I have a Promise, within a Promise, within a Promise. Is there someway I can structure this better so that it is more straightforward? It seems like nesting them within each other is counter to Promise's intended chaining approach.
Use .then()
let doStuff = (resolve, reject) => {/* resolve() or reject() */};
let promise = new Promise(doStuff);
doSomethingUntilPromiseisDone(
promise
.then(value => fetchValue(url))
.then(value => value.blob())
.then(saveToCache)
)
.then(success => console.log("success!!"))
.catch(err => console.error(err))
you can use generator to flatten your nested promises (Bluebird.couroutine or Generators)
//Bluebird.couroutine
const generator = Promise.coroutine(function*() {
try {
const value = yield fetchValue(url);
const success = yield saveToCache(value);
console.log('success:', success);
} catch(e) {
console.error(err);
}
}));
generator();
Each function will call the next one with the result of the method before.
var promises = [1,2,3].map((guid)=>{
return (param)=> {
console.log("param", param);
var id = guid;
return new Promise(resolve => {
// resolve in a random amount of time
setTimeout(function () {
resolve(id);
}, (Math.random() * 1.5 | 0) * 1000);
});
}
}).reduce(function (acc, curr, index) {
return acc.then(function (res) {
return curr(res[index-1]).then(function (result) {
console.log("result", result);
res.push(result);
return res;
});
});
}, Promise.resolve([]));
promises.then(console.log);

Categories