I'm using Firebase Storage and I'm trying to load all assets via a function call. The only way to get an assets url is to call getDownloadURL which returns a promise. I need to call this for every asset but I can't make it wait for all promises to be done before continuing for some reason.
I thought returning a promise from mergeMap would make it wait for all of them but that doesn't seem to be the case.
I've look at a number of questions regarding promises and RXJS but I can't seem to figure out what's wrong with the code.
getAssets() {
return this.authService.user$.pipe(
first(),
switchMap(user => defer(() => from(this.afs.storage.ref(`${user.uid}/assets`).listAll()))),
switchMap(assets => from(assets.items).pipe(
mergeMap(async (asset) => {
return new Promise((res, rej) => {
asset.getDownloadURL().then(url => {
const _asset = {
name: asset.name,
url,
};
this.assets.push(_asset);
res(_asset);
})
.catch((e) => rej(e));
});
}),
)),
map(() => this.assets),
);
}
...
this.getAssets().subscribe(assets => console.log(assets)); // this runs before all asset's url has been resolved
Overview
mergeMap doesn't wait for all internal observables. It spins up n internal observable pipes that run in parallel, and spits all the values out the same coupling at the bottom of the pipe (your subscribe statement in this case) as individual emissions. Hence why this.getAssets().subscribe(assets => console.log(assets)) runs before all your parallel internal mergeMap pipes complete their individual computations, because mergeMap doesn't wait for all of them before emitting (it will emit one by one as they finish). If you want to wait for n observable pipes to finish, then you need to use forkJoin.
Fork Join
forkJoin is best used when you have a group of observables and only care about the final emitted value of each. One common use case for this is if you wish to issue multiple requests on page load (or some other event) and only want to take action when a response has been received for all. In this way it is similar to how you might use Promise.all.
Solution
getAssets(): Observable<Asset[]> {
return this.authService.user$.pipe(
// first() will deliver an EmptyError to the observer's error callback if the
// observable completes before any next notification was sent. If you don't
// want this behavior, use take(1) instead.
first(),
// Switch to users firebase asset stream.
switchMap(user => {
// You might have to tweak this part. I'm not exactly sure what
// listAll() returns. I guessed that it returns a promise with
// firebase asset metadata.
return from(this.afs.storage.ref(`${user.uid}/assets`).listAll());
}),
// Map to objects that contain method to get image url.
map(firebaseAssetMetadata => firebaseAssetMetadata?.items ?? []),
// Switch to parallel getDownloadUrl streams.
switchMap(assets => {
// Not an rxjs map, a regular list map. Returns a list of getAssetUrlPipes.
const parallelGetAssetUrlPipes = assets.map(asset => {
return from(asset.getDownloadUrl()).pipe(
map(url => { name: asset.name, url })
);
});
// 1) Listen to all parallel pipes.
// 2) Wait until they've all completed.
// 3) Merge all parallel data into a list.
// 4) Then move list down the pipe.
return forkJoin(parallelGetAssetUrlPipes);
}),
// Outputs all parallel pipe data as a single emission in list form.
// Set local variable to users asset data.
tap(assetObjects => this.assets = assetObjects)
);
}
// Outputs the list of user asset data.
this.getAssets().subscribe(console.log);
Good luck out there, and enjoy your Swedish meatballs!
const { from } = rxjs
const { mergeMap } = rxjs.operators
const assets = [1,2,3,4,5]
function getUrl (index) {
return new Promise((res) => {
setTimeout(() => res(`http://example.com/${index}`), Math.random() * 3 + 1000)
})
}
// add param2 1 for mergeMap === concatMap
from(assets).pipe(
mergeMap(asset => {
return getUrl(asset)
}, 1)
).subscribe(console.log)
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
use concatMap to run one by one.
Related
I'm currently fetching data from an API and I need to do multiple GET requests (using axios). After all those GET requests are completed, I return a resolved promise.
However, I need to do these GET requests automatically based on an array list:
function do_api_get_requests() {
return promise = new Promise(function(resolve, reject) {
API_IDs = [0, 1, 2];
axios.get('https://my.api.com/' + API_IDs[0])
.then(data => {
// Do something with data
axios.get('https://my.api.com/' + API_IDs[1])
.then(data => {
// Do something with data
axios.get('https://my.api.com/' + API_IDs[2])
.then(data => {
// Do something with data
// Finished, resolve
resolve("success");
}
}
}
}
}
This works but the problem is API_IDs isn't always going to be the same array, it will change. So I'm not sure how to chain these requests automatically.
Since you said it may be a variable length array and you show sequencing the requests, you can just loop through the array using async/await:
async function do_api_get_requests(API_IDS) {
for (let id of API_IDS) {
const data = await axios.get(`https://my.api.com/${id}`);
// do something with data here
}
return "success";
}
And, since you said the list of API ids would be variable, I made it a parameter that you can pass into the function.
If you wanted to run all the API requests in parallel (which might be OK for a small array, but might be trouble for a large array) and you don't need to run them in a specific order, you can do this:
function do_api_get_requests(API_IDS) {
return Promise.all(API_IDS.map(async (id) => {
const data = await axios.get(`https://my.api.com/${id}`);
// do something with data here for this request
})).then(() => {
// make resolved value be "success"
return "success";
});
}
Depending upon your circumstances, you could also use Promise.allSettled(). Since you don't show getting results back, it's not clear whether that would be useful or not.
You can use Promise.all() method to do all API requests at the same time, and resolve when all of them resolves.
function do_api_get_requests() {
const API_IDs = [0, 1, 2];
let promises = [];
for (const id of API_IDS) {
promises.push(axios.get(`https://my.api.com/${id}`));
}
return Promise.all(promises);
}
If you use Bluebird.js (a better promise library, and faster than the in-built Promise), you can use Promise.each(), Promise.mapSeries(), or Promisme.reduce() to do what you want.
http://bluebirdjs.com
I am currently making a request that returns data for some files and a user.
For each file I want to return an object with data in it but one of the keys is dependent on another request.
How can I make this request so that getData function waits until the inner observable resolves to a value?
function getData() {
return forkJoin([
filesApiRequest),
userApiRquest
])
.pipe(map(([files, userInfo]) => {
return files.getFilesList()
.map((file) => {
const name = file.getName();
const importantInfo = importantInfoCall(userInfo.name, name); // returns an observable.
// How can I get the value from this before returning? I need this piece of data for each
// file in the files list.
return {
data1,
data2,
name,
...,
hasImportantInfo: importantInfo
};
})
}));
}
You need to use a "Higher order mapping operator" (switchMap) which will subscribe to your inner observable source and emit the data you need. In your case, you need to make many calls (one for each file), so you can map your file list to an array of observables that emit the desired data. You can use forkJoin again to wrap all these calls into a single observable:
function getData() {
return forkJoin([filesApiRequest, userApiRquest]).pipe(
switchMap(([files, userInfo]) => forkJoin(
// return array of observables
files.getFilesList().map(f => toDataWithImportantInfo$(f, userInfo))
))
);
}
function toDataWithImportantInfo$(file, userInfo) {
const name = file.getname();
return importantInfoCall(userInfo.name, name).pipe(
map(importantInfo => ({
data1,
data2,
name,
...,
hasImportantInfo: importantInfo
})
);
}
forkJoin isn't the best solution when you have a large number of requests, because it will execute them all at the same time (maybe you don't want to kick off 500 http requests at once).
In order to limit the number of requests, we can use merge instead, since it provides a concurrency parameter:
function getData() {
return forkJoin([filesApiRequest, userApiRquest]).pipe(
switchMap(([files, userInfo]) => merge(
files.getFilesList().map(f => toDataWithImportantInfo$(f, userInfo))
, 5)), // MAX 5 CONCURRENT
toArray()
);
}
forkJoin emits an array of all results when all sources complete. merge emits each result individually, so toArray is needed if you want to emit a single array once all the inner sources complete (rather than emitting individually).
I am new to rxjs, but need to use this asynchronous regime to fulfill such task:
I have many calculation requests, say 10K, and I want to execute them in batches: 1K per batch and the batch is pre-determined. Only when the current batch is done, I will move on to the next batch. The function will be:
calcBatch(dataset)
The input dateset will come from an array of datasets: datasets = [...]
the synchronous loop would look like:
datasets.foreach(dataset=> {
while (!calculation_is_done) {
wait();
}
calcBatch(dataset);
});
calcBatch(dataset) {
calculation_is_done = false;
/* calculation */
calculation_is_done = true;
}
Now switching to an asynchronous regime, how should I construct the flow? What I am thinking is in the calcBatch, when the work is done, a promise or an observable will be returned. Then in the loop, a subscriber will listen on this promise or observable, once it is caught, calcBatch will be called for the next batch.
The calculation needs to process in batches because the backend (HTTP) cannot handle a full set of calculation.
I think bufferCount and concatMap might be what you need:
function calcBatch(dataset) {
console.log('Processing batch:', dataset)
// Fake async request
return new Promise(res => {
setTimeout(() => res(dataset), 2000)
})
}
from(datasets)
.pipe(
bufferCount(1000),
concatMap(dataset => calcBatch(dataset))
)
.subscribe(dataset => {
console.log('Batch done:', dataset)
})
Note: calcBatch needs to return a promise or an observable.
In my last post, I was trying to buffer pending http requests using RxJS. I thought bufferCount was what I needed, but I found it my items were under the buffer size, it would just wait, which is not what I was after.
I now have a new scheme, using take. It seems to do what I am after, except when my resulting observable has no items (left), the complete is never called.
Eg I have something like the following..
const pendingRequests = this.store$.select(mySelects.getPendingRequests).pipe(
// FlatMap turns the observable of a single Requests[] to observable of Requests
flatMap(x => x),
// Only get requests unprocessed
filter(x => x.processedState === ProcessedState.unprocessed),
// Batches of batchSize in each emit
take(3),
);
let requestsSent = false;
pendingRequests.subscribe(nextRequest => {
requestsSent = true;
this.sendRequest(nextEvent);
},
error => {
this.logger.error(`${this.moduleName}.sendRequest: Error ${error}`);
},
() => {
// **** This is not called if pendingRequests is empty ****
if (requestsSent ) {
this.store$.dispatch(myActions.continuePolling());
} else {
this.store$.dispatch(myActions.stopPolling());
}
}
);
So the take(3) will get the next 3 pending requests and send them ()where I also dispatch an action to set the processed state to not ProcessedState.pending so we don't get them in the next poll)
This all works fine, but when pendingRequests eventually returns nothing (is empty), the completed block, marked with the ****. is not called. I would have thought this would just be called straight away.
I am not sure if this matters, as since I don't then dispatch the action to continue polling, the polling does stop.
But my biggest concern is if pendingRequests is not completed, do I need to unsubscribe from it to prevent any leaks? I assume if the complete is called I do not need to unsubscribe?
Update
To get the pendingRegueststo always complete, I have taken a slightly different approach. Rather than using the rx operators to "filter", I Just get the whole list every time, and just take(1) on it. I will always get the list, even if it is empty, so the pendingReguests will complete every time.
ie
const pendingRequests = this.store$.select(mySelects.getPendingRequests).pipe(take(1))
And then I can just filter and batch inside the observable..
pendingRequests.subscribe(nextRequest => {
let requestToSend = nextRequest.filter(x => x.processedState === ProcessedState.unprocessed);
const totalPendingCount = requestToSend.length;
requestToSend = requestToSend slice(0, this.batchSize);
for (const nextRequest of requestToSend) {
this.sendRequest(nextRequest);
}
if (totalPendingCount > this.batchSize) {
this.store$.dispatch(myActions.continuePolling());
}
In my testing so far, I have now always got the complete to fire.
Also, by having 2 actions (a startPolling, and a continuePolling) I can put the delay just in the continuePolling, so the first time we start the polling (eg the app has just come back online after being out of network range), we submit straight away, and only delay if we have more than the batch size
Maybe this is not 100% the "rxy" way of doing it, but seems to work so far. Is there any problem here?
I would substitute take with toArray and a bit of buffering logic afterwards.
This is how your code could look like. I have added the delay logic, which I think was suggested by your previous post, and provided comments to describe each line added
// implementation of the chunk function used below
// https://www.w3resource.com/javascript-exercises/fundamental/javascript-fundamental-exercise-265.php
const chunk = (arr, size) =>
Array.from({ length: Math.ceil(arr.length / size) }, (v, i) =>
arr.slice(i * size, i * size + size)
);
const pendingRequests = this.store$.select(mySelects.getPendingRequests).pipe(
// FlatMap turns the observable of a single Requests[] to observable of Requests
flatMap(x => x),
// Only get requests unprocessed
filter(x => x.processedState === ProcessedState.unprocessed),
// Read all the requests and store them in an array
toArray(),
// Split the array in chunks of the specified size, in this case 3
map(arr => chunk(arr, 3)), // the implementation of chunk is provided above
// Create a stream of chunks
concatMap((chunks) => from(chunks)),
// make sure each chunk is emitted after a certain delay, e.g. 2 sec
concatMap((chunk) => of(chunk).pipe(delay(2000))),
// mergeMap to turn an array into a stream
mergeMap((val) => val)
);
let requestsSent = false;
pendingRequests.subscribe(nextRequest => {
requestsSent = true;
this.sendRequest(nextEvent);
},
error => {
this.logger.error(`${this.moduleName}.sendRequest: Error ${error}`);
},
() => {
// **** THIS NOW SHOULD BE CALLED ****
if (requestsSent ) {
this.store$.dispatch(myActions.continuePolling());
} else {
this.store$.dispatch(myActions.stopPolling());
}
}
);
I doubt that pendingRequests will ever complete by itself. The Store, at least in ngrx, is a BehaviorSubject. So, whenever you do store.select() or store.pipe(select()), you're just adding another subscriber to the internal list of subscribers maintained by the BehaviorSubject.
The BehaviorSubject extends Subject, and here is what happens when the Subject is being subscribed to:
this.observers.push(subscriber);
In your case, you're using take(3). After 3 values, the take will emit a complete notification, so your complete callback should be called. And because the entire chain is actually a BehaviorSubject's subscriber, it will remove itself from the subscribers list on complete notifications.
I assume if the complete is called I do not need to unsubscribe
Here is what happens when a subscriber(e.g TakeSubscriber) completes:
protected _complete(): void {
this.destination.complete();
this.unsubscribe();
}
So, there is no need to unsubscribe if a complete/error notification already occurred.
I am creating a script in node.js (V8.1.3) which looks at similar JSON data from multiple API's and compares the values. To be more exact I am looking at different market prices of different stocks (actually cryptocurrencies).
Currently, I am using promise.all to wait for all responses from the respective APIs.
let fetchedJSON =
await Promise.all([getJSON(settings1), getJSON(settings2), getJSON(settings3) ... ]);
However, Promise.all throws an error if even just one promise rejects with an error. In the bluebird docos there is a function called Promise.some which is almost what I want. As I understand it takes an array of promises and resolves the two fastest promises to resolve, or otherwise (if less than 2 promises resolve) throws an error.
The problem with this is that firstly, I don't want the fastest two promises resolved to be what it returns, I want any successful promises to be returned, as long as there is more than 2. This seems to be what Promise.any does except with a min count of 1. (I require a minimum count of 2)
Secondly, I don't know how many Promises I will be awaiting on (In other words, I don't know how many API's I will be requesting data from). It may only be 2 or it may be 30. This depends on user input.
Currently writing this it seems to me there is probably just a way to have a promise.any with a count of 2 and that would be the easiest solution. Is this possible?
Btw, not sure if the title really summarizes the question. Please suggest an edit for the title :)
EDIT: Another way I may be writing the script is that the first two APIs to get loaded in start getting computed and pushed to the browser and then every next JSON that gets loaded and computed after it. This way I am not waiting for all Promises to be fulfilled before I start computing the data and passing results to the front end. Would this be possible with a function which also works for the other circumstances?
What I mean kind of looks like this:
Requesting JSON in parallel...
|-----JSON1------|
|---JSON-FAILS---| > catch error > do something with error. Doesn't effect next results.
|-------JSON2-------| > Meets minimum of 2 results > computes JSON > to browser.
|-------JSON3---------| > computes JSON > to browser.
How about thening all the promises so none fail, pass that to Promise.all, and filter the successful results in a final .then.
Something like this:
function some( promises, count = 1 ){
const wrapped = promises.map( promise => promise.then(value => ({ success: true, value }), () => ({ success: false })) );
return Promise.all( wrapped ).then(function(results){
const successful = results.filter(result => result.success);
if( successful.length < count )
throw new Error("Only " + successful.length + " resolved.")
return successful.map(result => result.value);
});
}
This might be somewhat clunky, considering you're asking to implement an anti-pattern, but you can force each promise to resolve:
async function fetchAllJSON(settingsArray) {
let fetchedJSON = await Promise.all(
settingsArray.map((settings) => {
// force rejected ajax to always resolve
return getJSON(settings).then((data) => {
// initial processing
return { success: true, data }
}).catch((error) => {
// error handling
return { success, false, error }
})
})
).then((unfilteredArray) => {
// only keep successful promises
return dataArray.filter(({ success }) => success)
})
// do the rest of your processing here
// with fetchedJSON containing array of data
}
You can use Promise.allSettled([]). the difference is that allSettled will return an array of objects after all the promises are settled regardless if successful or failed. then just find the successful o whatever you need.
let resArr = await Promise.allSettled(userNamesArr.map(user=>this.authenticateUserPassword(user,password)));
return resArr.find(e=>e.status!="rejected");
OR return resArr.find(e=>e.status=="fulfilled").
The other answers have the downside of having to wait for all the promises to resolve, whereas ideally .some would return as soon as any (N) promise(s) passes the predicate.
let anyNPromises = (promises, predicate = a => a, n = 1) => new Promise(async resolve => {
promises.forEach(async p => predicate(await p) && !--n && resolve(true));
await Promise.all(promises);
resolve(false);
});
let atLeast2NumbersGreaterThan5 = promises => anyNPromises(promises, a => a > 5, 2);
atLeast2NumbersGreaterThan5([
Promise.resolve(5),
Promise.resolve(3),
Promise.resolve(10),
Promise.resolve(11)]
).then(a => console.log('5, 3, 10, 11', a)); // true
atLeast2NumbersGreaterThan5([
Promise.resolve(5),
Promise.resolve(3),
Promise.resolve(10),
Promise.resolve(-43)]
).then(a => console.log('5, 3, 10, -43', a)); // false
atLeast2NumbersGreaterThan5([
Promise.resolve(5),
Promise.resolve(3),
new Promise(() => 'never resolved'),
Promise.resolve(10),
Promise.resolve(11)]
).then(a => console.log('5, 3, unresolved, 10, 11', a)); // true