I need to fetch a large number of data points from our API.
These can't however all be fetched at once, as the response time would be too long, so I want to break it into multiple requests. The response looks something like this:
{
href: www.website.com/data?skip=0&limit=150,
nextHref: www.website.com/data?skip=150&limit=150,
maxCount: 704,
skip: 0,
count: 150,
limit:150,
results: [...]
}
So, ultimately I need to continually call the nextHref until we actually reach the last one.
After each request, I want to take the results and concatenate them into a list of data, which will be updated on the UI.
I am relatively new to the world of Obervables but would like to create a solution with RxJS. Does anyone have an idea of how to implement this?
The part that gets me the most is that I don't know how many requests I will have to do in advance. It just needs to keep looping until it's done.
It looks like you can determine the number of calls to make after the first response is received. So, we can make the first call, and build an observable that returns the results of the "first call" along with the results of all subsequent calls.
We can use scan to accumulate the results into a single array.
const results$ = makeApiCall(0, 150).pipe(
switchMap(firstResponse => {
const pageCount = Math.ceil(firstResponse.maxCount / firstResponse.limit);
const pageOffsets = Array(pageCount - 1).fill(0).map((_, i) => (i + 1) * firstResponse.limit);
return concat(
of(firstResponse),
from(pageOffsets).pipe(
mergeMap(offset => makeApiCall(offset, firstResponse.limit), MAX_CONCURRENT_CALLS)
)
);
}),
scan((acc, cur) => acc.concat(cur.results), [])
);
Here's a breakdown of what this does:
we first call makeApiCall() so we can determine how many other calls need made
from creates an observable that emits our array of offsets one at a time
mergeMap will execute our subsequent calls to makeApiCall() with the passed in offsets and emit the results. Notice you can provide a "concurrency" limit, to control how many calls are made at a time.
concat is used to return an observable that emits the first response, followed by the results of the subsequent calls
switchMap subscribes to this inner observable and emits the results
scan is used to accumulate the results into a single array
Here's a working StackBlitz demo.
Related
So imagine you have an array of URLs:
urls: string[]
You make a collection of requests (in this example I am using Angular's HTTPClient.get which returns an Observable)
const requests = urls.map((url, index) => this.http.get<Film>(url)
Now I want to execute this requests concurrently but not wait for all response to see everything. In other words, if I have something like films$: Observable<Film[]>, I want films$ to update gradually every time a response arrives.
Now to simulate this, you can update the requests above into something like this
const requests = urls.map((url, index) => this.http.get<Film>(url).pipe(delay((index + 1)* 1000))
With the above array of Observables you should get data from each request one by one since they aren't requested at the same time. Note that this is just faking the different times of arrival of data from the individual requests. The requests itself should be done concurrently.
The goal is to update the elements in films$ every time value emitted by any of the requests.
So before I had something like this when I misunderstood how combineLatest works
let films$: Observable<Film[]> = of([]);
const requests = urls.map(url => this.http.get<Film>(url)
.pipe(
take(1),
// Without this handling, the respective observable does not emit a value and you need ALL of the Observables to emit a value before combineLatest gives you results.
// rxjs EMPTY short circuits the function as documented. handle the null elements on the template with *ngIf.
catchError(()=> of(null))
));
// Expect a value like [{...film1}, null, {...film2}] for when one of the URL's are invalid for example.
films$ = combineLatest(requests);
I was expecting the above code to update films$ gradually, overlooking a part of the documentation
To ensure the output array always has the same length, combineLatest will actually wait for all input Observables to emit at least once, before it starts emitting results.
Which is not what I was looking for.
If there is an rxjs operator or function that can achieve what I am looking for, I can have a cleaner template with simply utilizing the async pipe and not having to handle null values and failed requests.
I have also tried
this.films$ = from(urls).pipe(mergeMap(url => this.http.get<Film>(url)));
and
this.films$ = from(requests).pipe(mergeAll());
which isn't right because the returned value type is Observable<Film> instead of Observable<Film[]> that I can use on the template with *ngFor="let film of films$ | async". Instead, if I subscribe to it, it's as if I'm listening to a socket for one record, getting updates realtime (the individual responses coming in). I can manually subscribe to any of the two and make an Array.push to a separate property films: Film[] for example, but that defeats the purpose (use Observable on template with async pipe).
The scan operator will work nicely for you here:
const makeRequest = url => this.http.get<Film>(url).pipe(
catchError(() => EMPTY))
);
films$: Observable<Film[]> = from(urls).pipe(
mergeMap(url => makeRequest(url)),
scan((films, film) => films.concat(film), [])
);
Flow:
from emits urls one at a time
mergeMap subscribes to "makeRequest" and emits result into stream
scan accumulates results into array and emits each time a new emission is received
To preserve order, I would probably use combineLatest since it emits an array in the same order as the input observables. We can start each observable with startWith(undefined), then filter out the undefined items:
const requests = urls.map(url => this.http.get<Film>(url).pipe(startWith(undefined));
films$: Observable<Film[]> = combineLatest(requests).pipe(
map(films => films.filter(f => !!f))
);
Suppose I have a list of items that is queried from an API depending on a parameter that can be changed in the UI. When changing the value of this parameter, I dispatch an action:
this.store$.dispatch(new ChangeParameterAction(newParameterValue));
Now, on the receiving end, I want to trigger a new API call on every parameter change. I do this by subscribing to the store and then switching to the API observable. Then, I dispatch the result back into the store:
/** Statement 1 **/
this.store$.select(selectParameter).pipe(
switchMap(parameter => this.doApiCall$(parameter))
).subscribe(apiResult => {
this.store$.dispatch(new ResultGotAction(apiResult))
});
My UI is receiving the items by subscribing to
/** Statement 2 **/
this.store$.select(selectResults);
Now my question is: How can I join these two statements together so that we only have the subscription for Statement 1 for as long as the UI showing the results is active (and not destroyed)? I will always subscribe to the result of Statement 2, so Statement 1 will never be unsubscribed.
I've tried merging both observables and ignoring the elements for Statement 1, then subscribing tothe merged observables. But this looks like a very unreadable way for doing such a basic task. I think there must be a better way, but I can't find one. Hope you can help!
I can't tell exactly if this would run correctly, but I would go with moving the dispatch of ResultGotAction to a tap operator and then switching to this.store$.select(selectResults)
For example:
this.store$.select(selectParameter).pipe(
switchMap(parameter => this.doApiCall$(parameter)),
tap(apiResult => this.store$.dispatch(new ResultGotAction(apiResult))),
switchMapTo(this.store$.select(selectResults))
);
I'm facing a problem, and I've been trying to find a solution using RxJs, but can't seem to find one that fits it...
I have 3 different REST requests, that will be called sequentially, and each of them needs the response of the previous one as an argument
I want to implement a progress bar, which increments as the requests succeed
Here is what I thought :
I am going to use pipes and concatMap() to avoid nested subscriptions and subscribe to each request when the previous one is done.
Consider this very simplified version. Assume that each of represents a whole REST successful request (will handle errors later), and that I will do unshown work with the n parameter...
const request1 = of('success 1').pipe(
delay(500),
tap(n => console.log('received ' + n)),
);
const request2 = (n) => of('success 2').pipe(
delay(1000),
tap(n => console.log('received ' + n))
);
const request3 = (n) => of('success 3').pipe(
delay(400),
tap(n => console.log('received ' + n))
);
request1.pipe(
concatMap(n => request2(n).pipe(
concatMap(n => request3(n))
))
)
However, when I subscribe to the last piece of code, I will only get the response of the last request, which is expected as the pipe resolves to that.
So with concatMap(), I can chain my dependent REST calls correctly, but can't follow the progress.
Though I could follow the progress quite easily with nested subscriptions, but I am trying hard to avoid this and use the best practice way.
How can I chain my dependent REST calls, but still be able to do stuff each time a call succeeds ?
This is a generalized solution, though not as simple. But it does make progress observable while still avoiding the share operator, which can introduce unexpected statefulness if used incorrectly.
const chainRequests = (firstRequestFn, ...otherRequestFns) => (
initialParams
) => {
return otherRequestFns.reduce(
(chain, nextRequestFn) =>
chain.pipe(op.concatMap((response) => nextRequestFn(response))),
firstRequestFn(initialParams)
);
};
chainRequests takes a variable number of functions and returns a function that accepts initial parameters and returns an observable that concatMaps the functions together as shown manually in the question. It does this by reducing each function into an accumulation value that happens to be an observable.
Remember, RxJS leads us out of callback hell if we know the path.
const chainRequestsWithProgress = (...requestFns) => (initialParams) => {
const progress$ = new Rx.BehaviorSubject(0);
const wrappedFns = requestFns.map((fn, i) => (...args) =>
fn(...args).pipe(op.tap(() => progress$.next((i + 1) / requestFns.length)))
);
const chain$ = Rx.defer(() => {
progress$.next(0);
return chainRequests(...wrappedFns)(initialParams);
});
return [chain$, progress$];
};
chainRequestsWithProgress returns two observables - the one that eventually emits the last response, and one that emits progress values when the first observable is subscribed to. We do this by creating a BehaviorSubject to serve as our stream of progress values, and wrapping each of our request functions to return the same observable they normally would, but we also pipe it to tap so it can push a new progress value to the BehaviorSubject.
The progress is zeroed out upon each subscription to the first observable.
If you wanted to return a single observable that produced the progress state as well as the eventual result value, you could have chainRequestsWithProgress instead return:
chain$.pipe(
op.startWith(null),
op.combineLatest(progress$, (result, progress) => ({ result, progress }))
)
and you'll have an observable that emits an object representing the progress toward the eventual result, then that result itself. Food for thought - does progress$ have to emit just numbers?
Caveat
This assumes request observables emit exactly one value.
The simplest solution would be to have a progress counter variable that is updated from a tap when each response comes back.
let progressCounter = 0;
request1.pipe(
tap(_ => progressCounter = 0.33),
concatMap(n => request2(n).pipe(
tap(_ => progressCounter = 0.66),
concatMap(n => request3(n)
.pipe(tap(_ => progressCounter = 1)))
))
);
If you want the progress itself to be observable then you want to share the request observables as to not make duplicate requests) and then combine them to get the progress.
An example of how you may want to approach that can be found at: https://www.learnrxjs.io/recipes/progressbar.html
I want to reach an API that returns data in pages of 50 items, but I dont know how many items (and therefore pages) there are.
My idea is to send 20 requests in parallel, each one will request the ith page and then the ith+20 page, and so on, until a page returns blank, in which case I end.
With this approach I would do at most 20 unnecessary requests.
The thing is, I don't know how to structure this loop using observables.
I imagined something like this:
return Observable.from(_.range(0, 20))
.map((pageNo) => fetchPage(pageNo))
.while((page) => isValid(page));
but this while method or similars dont exist/work
I found this similar question but he uses interval, which seems inefficient RxJs Observable interval until reached desired value
From my understanding, I can't use takeWhilebecause it validates the condition already met, and not a response of the request still to be made.
This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)
You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60
I'd like to call multiple times the same API but with different keys to get results faster.
The thing is I need to not wait to receive the result from the first call to start the second call, etc...
The steps are :
1) I have an array with all the different keys.
2) This gets data from the API ("APIKeys" is the array that contains all the keys) :
_.map(APIKeys,function(value, index){
var newCount = count+(25*index);
parseResult(Meteor.http.get("http://my.api.com/content/search/scidir?query=a&count=25&start="+newCount+"&apiKey="+value+""));
});
3) I call a function (named "parseResult") that will format and filter the result I get from the API and save it into the database.
I want to call the function (step 3) without having to wait that I get the data from the API and continue with the other keys while the request is being made.
Do you know how I could do that with meteor ?
Thanks
Do something like this to use HTTP.get() in an async manner:
HTTP.get("http://my.api.com/content/search/scidir?query=a&count=25&start="+newCount+"&apiKey="+value+"", function (error, result) {
// parse the result here
});
And see the docs here:
http://docs.meteor.com/#/full/http_get