RxJs Observable make requests in parallel until it fails - javascript

I want to reach an API that returns data in pages of 50 items, but I dont know how many items (and therefore pages) there are.
My idea is to send 20 requests in parallel, each one will request the ith page and then the ith+20 page, and so on, until a page returns blank, in which case I end.
With this approach I would do at most 20 unnecessary requests.
The thing is, I don't know how to structure this loop using observables.
I imagined something like this:
return Observable.from(_.range(0, 20))
.map((pageNo) => fetchPage(pageNo))
.while((page) => isValid(page));
but this while method or similars dont exist/work
I found this similar question but he uses interval, which seems inefficient RxJs Observable interval until reached desired value
From my understanding, I can't use takeWhilebecause it validates the condition already met, and not a response of the request still to be made.

This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)

You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60

Related

How to run an array of requests with rxjs like forkJoin and combineLatest but without having to wait for ALL to complete before seeing the results?

So imagine you have an array of URLs:
urls: string[]
You make a collection of requests (in this example I am using Angular's HTTPClient.get which returns an Observable)
const requests = urls.map((url, index) => this.http.get<Film>(url)
Now I want to execute this requests concurrently but not wait for all response to see everything. In other words, if I have something like films$: Observable<Film[]>, I want films$ to update gradually every time a response arrives.
Now to simulate this, you can update the requests above into something like this
const requests = urls.map((url, index) => this.http.get<Film>(url).pipe(delay((index + 1)* 1000))
With the above array of Observables you should get data from each request one by one since they aren't requested at the same time. Note that this is just faking the different times of arrival of data from the individual requests. The requests itself should be done concurrently.
The goal is to update the elements in films$ every time value emitted by any of the requests.
So before I had something like this when I misunderstood how combineLatest works
let films$: Observable<Film[]> = of([]);
const requests = urls.map(url => this.http.get<Film>(url)
.pipe(
take(1),
// Without this handling, the respective observable does not emit a value and you need ALL of the Observables to emit a value before combineLatest gives you results.
// rxjs EMPTY short circuits the function as documented. handle the null elements on the template with *ngIf.
catchError(()=> of(null))
));
// Expect a value like [{...film1}, null, {...film2}] for when one of the URL's are invalid for example.
films$ = combineLatest(requests);
I was expecting the above code to update films$ gradually, overlooking a part of the documentation
To ensure the output array always has the same length, combineLatest will actually wait for all input Observables to emit at least once, before it starts emitting results.
Which is not what I was looking for.
If there is an rxjs operator or function that can achieve what I am looking for, I can have a cleaner template with simply utilizing the async pipe and not having to handle null values and failed requests.
I have also tried
this.films$ = from(urls).pipe(mergeMap(url => this.http.get<Film>(url)));
and
this.films$ = from(requests).pipe(mergeAll());
which isn't right because the returned value type is Observable<Film> instead of Observable<Film[]> that I can use on the template with *ngFor="let film of films$ | async". Instead, if I subscribe to it, it's as if I'm listening to a socket for one record, getting updates realtime (the individual responses coming in). I can manually subscribe to any of the two and make an Array.push to a separate property films: Film[] for example, but that defeats the purpose (use Observable on template with async pipe).
The scan operator will work nicely for you here:
const makeRequest = url => this.http.get<Film>(url).pipe(
catchError(() => EMPTY))
);
films$: Observable<Film[]> = from(urls).pipe(
mergeMap(url => makeRequest(url)),
scan((films, film) => films.concat(film), [])
);
Flow:
from emits urls one at a time
mergeMap subscribes to "makeRequest" and emits result into stream
scan accumulates results into array and emits each time a new emission is received
To preserve order, I would probably use combineLatest since it emits an array in the same order as the input observables. We can start each observable with startWith(undefined), then filter out the undefined items:
const requests = urls.map(url => this.http.get<Film>(url).pipe(startWith(undefined));
films$: Observable<Film[]> = combineLatest(requests).pipe(
map(films => films.filter(f => !!f))
);

Using RxJS for unknown number of consequtive HTTP Requests

I need to fetch a large number of data points from our API.
These can't however all be fetched at once, as the response time would be too long, so I want to break it into multiple requests. The response looks something like this:
{
href: www.website.com/data?skip=0&limit=150,
nextHref: www.website.com/data?skip=150&limit=150,
maxCount: 704,
skip: 0,
count: 150,
limit:150,
results: [...]
}
So, ultimately I need to continually call the nextHref until we actually reach the last one.
After each request, I want to take the results and concatenate them into a list of data, which will be updated on the UI.
I am relatively new to the world of Obervables but would like to create a solution with RxJS. Does anyone have an idea of how to implement this?
The part that gets me the most is that I don't know how many requests I will have to do in advance. It just needs to keep looping until it's done.
It looks like you can determine the number of calls to make after the first response is received. So, we can make the first call, and build an observable that returns the results of the "first call" along with the results of all subsequent calls.
We can use scan to accumulate the results into a single array.
const results$ = makeApiCall(0, 150).pipe(
switchMap(firstResponse => {
const pageCount = Math.ceil(firstResponse.maxCount / firstResponse.limit);
const pageOffsets = Array(pageCount - 1).fill(0).map((_, i) => (i + 1) * firstResponse.limit);
return concat(
of(firstResponse),
from(pageOffsets).pipe(
mergeMap(offset => makeApiCall(offset, firstResponse.limit), MAX_CONCURRENT_CALLS)
)
);
}),
scan((acc, cur) => acc.concat(cur.results), [])
);
Here's a breakdown of what this does:
we first call makeApiCall() so we can determine how many other calls need made
from creates an observable that emits our array of offsets one at a time
mergeMap will execute our subsequent calls to makeApiCall() with the passed in offsets and emit the results. Notice you can provide a "concurrency" limit, to control how many calls are made at a time.
concat is used to return an observable that emits the first response, followed by the results of the subsequent calls
switchMap subscribes to this inner observable and emits the results
scan is used to accumulate the results into a single array
Here's a working StackBlitz demo.

RXJS Chain dependent observables sequentially, but getting each emission for a progress bar

I'm facing a problem, and I've been trying to find a solution using RxJs, but can't seem to find one that fits it...
I have 3 different REST requests, that will be called sequentially, and each of them needs the response of the previous one as an argument
I want to implement a progress bar, which increments as the requests succeed
Here is what I thought :
I am going to use pipes and concatMap() to avoid nested subscriptions and subscribe to each request when the previous one is done.
Consider this very simplified version. Assume that each of represents a whole REST successful request (will handle errors later), and that I will do unshown work with the n parameter...
const request1 = of('success 1').pipe(
delay(500),
tap(n => console.log('received ' + n)),
);
const request2 = (n) => of('success 2').pipe(
delay(1000),
tap(n => console.log('received ' + n))
);
const request3 = (n) => of('success 3').pipe(
delay(400),
tap(n => console.log('received ' + n))
);
request1.pipe(
concatMap(n => request2(n).pipe(
concatMap(n => request3(n))
))
)
However, when I subscribe to the last piece of code, I will only get the response of the last request, which is expected as the pipe resolves to that.
So with concatMap(), I can chain my dependent REST calls correctly, but can't follow the progress.
Though I could follow the progress quite easily with nested subscriptions, but I am trying hard to avoid this and use the best practice way.
How can I chain my dependent REST calls, but still be able to do stuff each time a call succeeds ?
This is a generalized solution, though not as simple. But it does make progress observable while still avoiding the share operator, which can introduce unexpected statefulness if used incorrectly.
const chainRequests = (firstRequestFn, ...otherRequestFns) => (
initialParams
) => {
return otherRequestFns.reduce(
(chain, nextRequestFn) =>
chain.pipe(op.concatMap((response) => nextRequestFn(response))),
firstRequestFn(initialParams)
);
};
chainRequests takes a variable number of functions and returns a function that accepts initial parameters and returns an observable that concatMaps the functions together as shown manually in the question. It does this by reducing each function into an accumulation value that happens to be an observable.
Remember, RxJS leads us out of callback hell if we know the path.
const chainRequestsWithProgress = (...requestFns) => (initialParams) => {
const progress$ = new Rx.BehaviorSubject(0);
const wrappedFns = requestFns.map((fn, i) => (...args) =>
fn(...args).pipe(op.tap(() => progress$.next((i + 1) / requestFns.length)))
);
const chain$ = Rx.defer(() => {
progress$.next(0);
return chainRequests(...wrappedFns)(initialParams);
});
return [chain$, progress$];
};
chainRequestsWithProgress returns two observables - the one that eventually emits the last response, and one that emits progress values when the first observable is subscribed to. We do this by creating a BehaviorSubject to serve as our stream of progress values, and wrapping each of our request functions to return the same observable they normally would, but we also pipe it to tap so it can push a new progress value to the BehaviorSubject.
The progress is zeroed out upon each subscription to the first observable.
If you wanted to return a single observable that produced the progress state as well as the eventual result value, you could have chainRequestsWithProgress instead return:
chain$.pipe(
op.startWith(null),
op.combineLatest(progress$, (result, progress) => ({ result, progress }))
)
and you'll have an observable that emits an object representing the progress toward the eventual result, then that result itself. Food for thought - does progress$ have to emit just numbers?
Caveat
This assumes request observables emit exactly one value.
The simplest solution would be to have a progress counter variable that is updated from a tap when each response comes back.
let progressCounter = 0;
request1.pipe(
tap(_ => progressCounter = 0.33),
concatMap(n => request2(n).pipe(
tap(_ => progressCounter = 0.66),
concatMap(n => request3(n)
.pipe(tap(_ => progressCounter = 1)))
))
);
If you want the progress itself to be observable then you want to share the request observables as to not make duplicate requests) and then combine them to get the progress.
An example of how you may want to approach that can be found at: https://www.learnrxjs.io/recipes/progressbar.html

Better way to use fork join subsequent to another observable

I have a login process that has fairly complicated login variations and has to be scalable to easily add more in the future. So initially the user is authenticated in the typical manner and a user object is returned. Then I must make additional http calls to get information that will determine the various requirements before the user is granted access to the app. This is done using some of the values returned in the user object. I want to write the code in a way that I can easily add http calls without changing current code so I thought using fork join for the subsequent calls would be good since they can be done in parallel. Below is my working code.
I can easily add new requests to the fork join call and while it doesn't look too bad to me I have been told nested subscriptions is a code smell and typically bad practice. Any ideas on how to do this better would be great.
Thanks.
this.authenticate.login(this.model)
.subscribe(
_data => {
this.subscription = Observable.forkJoin(
this.devicesHttp.getDevicesByMacAddress(this.macAddress),
this.teamsService.getTeamsByUserId(_data['userId'])
);
this.subscription.subscribe(
_data => {
// Check login type and other stuff...
}
);
}
);
For example like this using the concatMap() operator:
this.authenticate.login(this.model)
.concatMap(_data => Observable.forkJoin(
this.devicesHttp.getDevicesByMacAddress(this.macAddress),
this.teamsService.getTeamsByUserId(_data['userId'])
))
.subscribe(_data => {
// Check login type and other stuff...
});
The Observables in forkJoin will run in parallel and forkJoin will wait until they both finish.
Also concatMap() waits until the inner Observable completes and then pushes the result further.
In 2021 this must be written with pipe, stand-alone operators, array in forkJoin and Observer argument in subscribe:
import { concatMap, forkJoin } from 'rxjs';
this.getFirst().pipe(
concatMap(data =>
forkJoin([
this.getSecond(data),
this.getThird(data)
])
)
).subscribe({
next: result => ...,
error: e => ...
});
How about this:
this.authenticate.login(this.model)
.switchMap(data => Observable.forkJoin(
this.devicesHttp.getDevicesByMacAddress(this.macAddress),
this.teamsService.getTeamsByUserId(data['userId'])
))
.subscribe(...,...)

Why complete is not invoked after performing a concatenation?

I need to query a device multiple times. Every query needs to be asynchronous and the device doesn't support simultaneous queries at a time.
Moreover, once it is queried, it can not be queried again immediately after. It needs at least a 1 second pause to work properly.
My two queries, performed by saveClock() and saveConfig(), return a Promise and both resolve by returning undefined as expected.
In the following code why removing take() prevents toArray() from being called?
What's happening here, is there a better way to achieve the same behavior?
export const saveEpic = (action$, store) =>
action$.ofType(SAVE)
.map(action => {
// access store and create object data
// ...
return data;
})
.mergeMap(data =>
Rx.Observable.from([
Rx.Observable.of(data).mergeMap(data => saveClock(data.id, data.clock)),
Rx.Observable.timer(1000),
Rx.Observable.of(data).mergeMap(data => saveConfig(data.id, data.config)),
Rx.Observable.of(data.id)
])
)
.concatAll()
.take(4)
.toArray()
// [undefined, 0, undefined, "id"]
.map(x => { type: COMPLETED, id: x[3] });
There are a couple things I see:
Your final .map() is missing parenthesis, which in its current form is a syntax error but a subtle change could make it accidentally a labeled statement instead of returning an object. Because in its current form it's a syntax error, I imagine this is just a bug in this post, not in your code (which wouldn't even run), but double check!
// before
.map(x => { type: COMPLETED, id: x[3] });
// after
.map(x => ({ type: COMPLETED, id: x[3] }));
With that fixed, the example does run with a simple redux-observable test case: http://jsbin.com/hunale/edit?js,output So if there's nothing notable I did differently than you, problem appears to be in code not provided. Feel free to add additional insight or even better, reproduce it in a JSBin/git repo for us.
One thing you didn't mention but is very very noteworthy is that in redux-observable, your epics will typically be long-lived "process managers". This epic will actually only process one of these saves, then complete(), which is probably not what you actually want? Can the user only save something one time per application boot? Seems unlikely.
Instead, you'll want to keep the top-level stream your epic returns alive and listening for future actions by encapsulating this logic inside the mergeMap. The take(4) and passing the data.id then become extraneous:
const saveEpic = (action$, store) =>
action$.ofType(SAVE)
.mergeMap(data =>
Rx.Observable.from([
Rx.Observable.of(data).mergeMap(data => saveClock(data.id, data.clock)),
Rx.Observable.timer(1000),
Rx.Observable.of(data).mergeMap(data => saveConfig(data.id, data.config))
])
.concatAll()
.toArray()
.map(() => ({ type: COMPLETED, id: data.id }))
);
This separation of streams is described by Ben Lesh in his recent AngularConnect talks, in the context of errors but it's still applicable: https://youtu.be/3LKMwkuK0ZE?t=20m (don't worry, this isn't Angular specific!)
Next, I wanted to share some unsolicited refactoring advice that may make your life easier, but certainly this is opinionated so feel free to ignore:
I would refactor to more accurately reflect the order of events visually, and reduce the complexity:
const saveEpic = (action$, store) =>
action$.ofType(SAVE)
.mergeMap(data =>
Rx.Observable.from(saveClock(data.id, data.clock))
.delay(1000)
.mergeMap(() => saveConfig(data.id, data.config))
.map(() => ({ type: COMPLETED, id: data.id }))
);
Here we're consuming the Promise returned by saveClock, delaying it's output for 1000ms, the mergeMapping the result to a call to saveConfig() which also returns a Promise that will be consumed. Then finally mapping the result of that to our COMPLETE action.
Finally, keep in mind that if your Epic does stay alive and is long lived, there's nothing in this epic as-is to stop it from receiving multiple SAVE requests while other ones are still in-flight or have not yet exhausted the required 1000ms delay between requests. i.e. if that 1000ms space between any request is indeed required, your epic itself does not entirely prevent your UI code from breaking that. In that case, you may want to consider adding a more complex buffered backpressure mechanism, for example using the .zip() operator with a BehaviorSubject.
http://jsbin.com/waqipol/edit?js,output
const saveEpic = (action$, store) => {
// used to control how many we want to take,
// the rest will be buffered by .zip()
const requestCount$ = new Rx.BehaviorSubject(1)
.mergeMap(count => new Array(count));
return action$.ofType(SAVE)
.zip(requestCount$, action => action)
.mergeMap(data =>
Rx.Observable.from(saveClock(data.id, data.clock))
.delay(1000)
.mergeMap(() => saveConfig(data.id, data.config))
.map(() => ({ type: COMPLETED, id: data.id }))
// we're ready to take the next one, when available
.do(() => requestCount$.next(1))
);
};
This makes it so that requests to save that come in while we're still processing an existing one is buffered, and we only take one of them at a time. Keep in mind though that this is an unbounded buffer--meaning that the queue of pending actions can potentially grow infinitely quicker than the buffer is flushed. This is unavoidable unless you adopted a strategy for lossy backpressure, like dropping requests that overlap, etc.
If you have other epics which have overlapping requirements to not sending requests more than once a second, you would need to create some sort of single supervisor that makes this guarantee for all the epics.
This may all seem very complex, but perhaps ironically this is much easier to do in RxJS than with traditional imperative code. The hardest part is actually knowing the patterns.

Categories