I need to build a stream that makes api request immediately after the call and that each hour (1:00 PM, 2:00 PM) if page wasn't refreshed. I build this with setTimeout() but I want to implement it with RxJs. Can you help me pls.
isUpdated() {
return new Observable<Object>(function update(obs) {
this.http.get(`/update`).subscribe(data => {
obs.next(data);
});
const timer = (60 - new Date().getMinutes()) * 60 * 1000;
setTimeout(() => update.call(this, obs), timer);
}.bind(this));
}
//call
isUpdated().subscribe(data => console.log(data));
I think that to solve this problem, you need to break it down into smaller pieces.
First of all, we know that at some point, based on the current time, we'll want to know when to trigger the next call. If we get a timestamp which gives us the current time in ms and we want to get the number of ms before the next hour, here's how we can do it:
const timeToNextHourInMs = (currentTimestampMs) => {
const timestampSeconds = currentTimestampMs / 1000;
const numberOfSecondsIntoTheCurrentHour = timestampSeconds % 3600;
const numberOfSecondsToTheNextHour = 3600 - numberOfSecondsIntoTheCurrentHour;
return numberOfSecondsToTheNextHour * 1000;
};
I hope that the variable names are explicit enough that I do not need to comment but let me know otherwise.
Next, we want to tackle the stream issue:
We want to trigger an HTTP call straight away
Get the emitted value straight away
Do all the above again every time a new hour start (1:00, 2:00, 3:00, etc..)
Here's how you can do this:
this.http.get(`/update`).pipe(
timestamp(),
switchMap(({ timestamp, value }) =>
concat(
of(value),
EMPTY.pipe(delay(timeToNextHourInMs(timestamp)))
)
),
repeat()
);
Let's go through the above logic:
First, we make the HTTP call straight away
Once the HTTP call is done, we get the current timestamp (to later on based on that find out when we want to do the next call)
We do a switchMap but as our HTTP call is only ever going to return 1 value it doesn't really matter in this very specific case. We could use flatMap or concatMap too
Inside the switchMap, we use concat to first of all send the value that we just got from the HTTP call but also keep that observable alive until the end of the current our (by using the function we created earlier)
At the end of the current hour, the stream will therefore complete. BUT, as we've got a retry, as soon as the stream completes, we'll subscribe to it again (and as a reminder, the stream will only complete at the very beginning of a new hour!)
One thing I'd suggest to add here but which isn't a requirement of the initial issue would be to have some error handling so that if something goes wrong when you make that call, it automatically retries it a few seconds after. Otherwise imagine when the polling kicks in if your network doesn't work for 5s at that exact time your stream will error straight away.
For this, you can refer to this brilliant answer and do that in a reusable custom operator:
const RETRY_DELAY = 2000;
const MAX_RETRY_FOR_ONE_HTTP_CALL = 3;
const automaticRetry = () => (obs$) =>
obs$.pipe(
retryWhen((error$) =>
error$.pipe(
concatMap((error, index) =>
iif(
() => index >= MAX_RETRY_FOR_ONE_HTTP_CALL,
throwError(error),
of(error).pipe(delay(RETRY_DELAY))
)
)
)
)
);
This will retry the observable 3 times with a delay between each retry. After 3 times, the stream will error by throwing the last emitted error.
Now, we can just add this custom operator to our stream:
this.http.get(`/update`).pipe(
automaticRetry(),
timestamp(),
switchMap(({ timestamp, value }) =>
concat(
of(value),
EMPTY.pipe(delay(timeToNextHourInMs(timestamp)))
)
),
repeat()
);
I haven't actually tested the code above so please do that on your side and let me know how it goes. But if my logic is correct here's how things should go:
Imagine you start your app at 2:40
An HTTP call is made straight away
You get the response pretty much instantly
The stream is set to be kept open for 20mn
At 3:00, the stream is completed and the retry kicks in: We do another HTTP call
This time, the server got re-deployed and was not available for a few seconds
Internally, the stream errors but thanks to our custom operator automaticRetry it waits for 3 seconds then retries the HTTP call once, still nothing. It waits another 3 seconds and this time it's fine, the result is passed downstream
Repeat this indefinitely :)
Let me know how it goes
I think it is very simple.
Timer accepts two parameters, the first is the delay for the first call and then the interval in which the call should be made. You got the next full hour offset right, I'm using your code for that.
Use startWith(null) to start immediately.
const startDelayMs = (60 - new Date().getMinutes()) * 60 * 1000;
const hourMs = 60 * 60 * 1000;
timer(startDelayMs, hourMs).pipe(
startWith(null),
switchMap(()) // do your call here
)
Maybe you could do it like this (I didn't test this code):
import { merge, concat, of, timer, interval } from 'rxjs';
import { switchMap } from 'rxjs/operators';
...
isUpdated() {
return concat(
merge(
of(null), // make call inmediately
timer((60 - new Date().getMinutes()) * 60 * 1000), // the next full hour eg. 2:00
),
interval(60 * 60 * 1000), // every hour
).pipe(
switchMap(() => this.http.get(...)),
)
})
concat will subscribe to interval only after the first merge completes which is at the next full hour eg.: 2:00 and then emit every hour.
#MrkSef had a good idea that this could be simplified to something like this:
isUpdated() {
return timer(
(60 - new Date().getMinutes()) * 60 * 1000, // the next full hour eg. 2:00
60 * 60 * 1000 // every hour
).pipe(
startWith(null), // initial request
switchMap(() => this.http.get(...)),
)
})
You can simply use rxjs interval
import { interval } from 'rxjs';
const seconds = 3600;
const source = interval(seconds * 1000);
source.subscribe(()=> {
// make api request here
});
Ok, so you want to:
create two streams: first one will always tick every hour (60 000 ms), second - will control the allowance of emitting these hourly ticks
const every60Minutes$ = interval(60 * 1000);
const isAllowedToProceedSubject = new Subject<boolean>(); // subject is also an Observable stream
combine those two streams and only allow to emit the values if the second stream has "true" as the latest value
const every60MinsIfAllowed$ = combineLatest([
isAllowedToProceedSubject,
every60Minutes$
]).pipe(
filter(([isAllowed, intervalTick]) => {
return isAllowed;
}),
flatMap(() => {
return of(new Date()); // replace with HTTP call
})
)
be able to control the second stream
setTimeout(() => {
isAllowedToProceedSubject.next(true); // this allows the values to be emitted
}, 2500);
// or add some click handler
Working example (but with emitting every 2000ms for better clarity)
Related
so I have a piece of asynchronous(setInterval) code in the firebase function.
export const auto_play = functions
.runWith({ memory: "512MB", timeoutSeconds: 540 })
.pubsub.schedule("*/15 * * * *")
.onRun(async (context) => {
const nums = polledDoc.data()?.nums as number[];
setInterval(() => {
const polledNum = nums.shift();
// function suppose to run for atleast 10-15 mins coz nums.length can be any number from 60 to 90.
// a function which save data to realtime database
autoPollAlgo({ gameId: scheduledGame.docs[0].id, number: polledNum as number });
}, 10 * 1000);
})
now this code works fine 3/5 times but it sometimes exits from intervals before the num array completes. sometimes it just got stop after a min and sometimes after 5mins.
I know the fact functions have a max timeout of 9mins, but how does this async code work even after 9mins.
after some digging found out I'm not returning any promise so this code can terminate any time. now to make things perfect I added a promise code to the end of the block.
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve(true);
//to make sure it exit after 15mins when interval code is done.
}, 900000);
});
now what is happening, functions is got consistent now its end exactly after 9mins(in the middle of setinterval exec.). it won't wait for the promise to resolve.
how can I keep a function to run an async task for 15mins with consistency?
After struggling with cloud functions I came with a solution.
I divided the logic into two functions.
First, I divided the nums array in half.
I executed setInterval for half of the nums then I use Cloud Pub/Sub to execute another function with the other half nums.
// somewhere in first function
pubSubClient.topic("play_half").publish(Buffer.from(JSON.stringify(restNum), "utf-8"), { gameId: scheduledGame.docs[0].id });
export const auto_play_2 = functions
.runWith({ memory: "512MB", timeoutSeconds: 540 })
.pubsub.topic("play_half")
.onPublish((message) => {
const non_parse_arr = Buffer.from(message.data, "base64").toString();
//other half nums
const nums = JSON.parse(non_parse_arr);
//...
})
I'm making a music composer and to play the notes i'm waiting for the time (in ms) to pass before playing the next notes, i've seen that this seems farily inaccurate with sometimes up to 10ms of inaccuracy, is there a way i can make a more accurate timeout or delay function to get down to 0/1 ms of discrepancy?
My code currently is:
function delayMs(ms) {
return new Promise(resolve => {
setTimeout(resolve, ms)
})
}
with the "tick" function:
while (this.state.isPlaying) {
const { song, settings } = this.state
let msPerBPM = Math.floor(60000 / settings.bpm.value)
await delayMs(msPerBPM)
this.handleTick()
}
i'm able to use service workers as i've noticed that i get this issue with delays from re renders in react.
It is easy enough to demonstrate that using setTimeout (and by extension setInterval) you will never get millisecond accuracy.
The below demonstrates that even telling javascript to execute in 1 millisecond it can be anywhere from a few ms to tens of ms later the call is actually made.
function recall(count){
console.log(new Date())
if(++count<10)
setTimeout(recall,1,count);
}
recall(1)
You might have more luck with requestAnimationFrame but again you'll need to calculate how long has elapsed since last step and use that to "synchronise" whatever you want to do:
let start;
function step(timestamp) {
if (start === undefined)
start = timestamp;
const elapsed = timestamp - start;
console.log(elapsed);
if (elapsed < 500) { // Stop the animation after 0.5 seconds
window.requestAnimationFrame(step);
}
}
window.requestAnimationFrame(step);
In my last post, I was trying to buffer pending http requests using RxJS. I thought bufferCount was what I needed, but I found it my items were under the buffer size, it would just wait, which is not what I was after.
I now have a new scheme, using take. It seems to do what I am after, except when my resulting observable has no items (left), the complete is never called.
Eg I have something like the following..
const pendingRequests = this.store$.select(mySelects.getPendingRequests).pipe(
// FlatMap turns the observable of a single Requests[] to observable of Requests
flatMap(x => x),
// Only get requests unprocessed
filter(x => x.processedState === ProcessedState.unprocessed),
// Batches of batchSize in each emit
take(3),
);
let requestsSent = false;
pendingRequests.subscribe(nextRequest => {
requestsSent = true;
this.sendRequest(nextEvent);
},
error => {
this.logger.error(`${this.moduleName}.sendRequest: Error ${error}`);
},
() => {
// **** This is not called if pendingRequests is empty ****
if (requestsSent ) {
this.store$.dispatch(myActions.continuePolling());
} else {
this.store$.dispatch(myActions.stopPolling());
}
}
);
So the take(3) will get the next 3 pending requests and send them ()where I also dispatch an action to set the processed state to not ProcessedState.pending so we don't get them in the next poll)
This all works fine, but when pendingRequests eventually returns nothing (is empty), the completed block, marked with the ****. is not called. I would have thought this would just be called straight away.
I am not sure if this matters, as since I don't then dispatch the action to continue polling, the polling does stop.
But my biggest concern is if pendingRequests is not completed, do I need to unsubscribe from it to prevent any leaks? I assume if the complete is called I do not need to unsubscribe?
Update
To get the pendingRegueststo always complete, I have taken a slightly different approach. Rather than using the rx operators to "filter", I Just get the whole list every time, and just take(1) on it. I will always get the list, even if it is empty, so the pendingReguests will complete every time.
ie
const pendingRequests = this.store$.select(mySelects.getPendingRequests).pipe(take(1))
And then I can just filter and batch inside the observable..
pendingRequests.subscribe(nextRequest => {
let requestToSend = nextRequest.filter(x => x.processedState === ProcessedState.unprocessed);
const totalPendingCount = requestToSend.length;
requestToSend = requestToSend slice(0, this.batchSize);
for (const nextRequest of requestToSend) {
this.sendRequest(nextRequest);
}
if (totalPendingCount > this.batchSize) {
this.store$.dispatch(myActions.continuePolling());
}
In my testing so far, I have now always got the complete to fire.
Also, by having 2 actions (a startPolling, and a continuePolling) I can put the delay just in the continuePolling, so the first time we start the polling (eg the app has just come back online after being out of network range), we submit straight away, and only delay if we have more than the batch size
Maybe this is not 100% the "rxy" way of doing it, but seems to work so far. Is there any problem here?
I would substitute take with toArray and a bit of buffering logic afterwards.
This is how your code could look like. I have added the delay logic, which I think was suggested by your previous post, and provided comments to describe each line added
// implementation of the chunk function used below
// https://www.w3resource.com/javascript-exercises/fundamental/javascript-fundamental-exercise-265.php
const chunk = (arr, size) =>
Array.from({ length: Math.ceil(arr.length / size) }, (v, i) =>
arr.slice(i * size, i * size + size)
);
const pendingRequests = this.store$.select(mySelects.getPendingRequests).pipe(
// FlatMap turns the observable of a single Requests[] to observable of Requests
flatMap(x => x),
// Only get requests unprocessed
filter(x => x.processedState === ProcessedState.unprocessed),
// Read all the requests and store them in an array
toArray(),
// Split the array in chunks of the specified size, in this case 3
map(arr => chunk(arr, 3)), // the implementation of chunk is provided above
// Create a stream of chunks
concatMap((chunks) => from(chunks)),
// make sure each chunk is emitted after a certain delay, e.g. 2 sec
concatMap((chunk) => of(chunk).pipe(delay(2000))),
// mergeMap to turn an array into a stream
mergeMap((val) => val)
);
let requestsSent = false;
pendingRequests.subscribe(nextRequest => {
requestsSent = true;
this.sendRequest(nextEvent);
},
error => {
this.logger.error(`${this.moduleName}.sendRequest: Error ${error}`);
},
() => {
// **** THIS NOW SHOULD BE CALLED ****
if (requestsSent ) {
this.store$.dispatch(myActions.continuePolling());
} else {
this.store$.dispatch(myActions.stopPolling());
}
}
);
I doubt that pendingRequests will ever complete by itself. The Store, at least in ngrx, is a BehaviorSubject. So, whenever you do store.select() or store.pipe(select()), you're just adding another subscriber to the internal list of subscribers maintained by the BehaviorSubject.
The BehaviorSubject extends Subject, and here is what happens when the Subject is being subscribed to:
this.observers.push(subscriber);
In your case, you're using take(3). After 3 values, the take will emit a complete notification, so your complete callback should be called. And because the entire chain is actually a BehaviorSubject's subscriber, it will remove itself from the subscribers list on complete notifications.
I assume if the complete is called I do not need to unsubscribe
Here is what happens when a subscriber(e.g TakeSubscriber) completes:
protected _complete(): void {
this.destination.complete();
this.unsubscribe();
}
So, there is no need to unsubscribe if a complete/error notification already occurred.
I have an Angular 7 application and I am using RxJS.
I want to fetch a list of stuff from an endpoint every 30 seconds, but sometimes the request can take a while and I want to account for that time.
Example: if the request takes 10 seconds, then I want to call it again after 40 seconds (30 + 10), not 20 (30 - 10).
I am trying with:
fetchList() {
this.service.fetchListFromHttp()
.pipe(
finalize(() =>
setTimeout(() => {
this.fetchList();
}, 30000)
)
)
.subscribe(
result => this.list = result,
err => this.logError(err)
);
}
I would imagine that when the http requests, it would trigger another call to fetchInfo after 30 seconds, but what happens is that I get weird intervals. For instance, the funcion gets called every 10 seconds, or 20 seconds, and weird intervals like that.
I would assume that the interval would always be bigger then 30 seconds.
Are you connecting to your own backend? If you're the one who has coded the API to fetch information form, socket.io is probably what you're looking for to accomplish this.
Have you looked into using RxJS? it comes with Angular, I'm sure what you could accomplish something that fetches then pauses
something similar to this might work
this.service.fetchInfoFromHttp().pipe(delay(30 * 60 * 10000), () => ...), repeat(Infinity))
Here is the link to the documentation on RxJS
RxJS delay
RxJS repeat
Socket.io
EDIT:
How Can I make a Timer using an observable in Angular 9
EDIT AGAIN:
Thanks for your comment #goga-koreli, maybe debounce will work
https://www.learnrxjs.io/learn-rxjs/operators/filtering/debounce
Improving upon SVNTY's answer, you can do it like so:
fetchInfo() {
const passAndWait = (data) => interval(30000).pipe(
take(1),
ignoreElements(), // this is needed to discard interval numbers
startWith(data),
)
// get data from network pass it to subscriber and wait for specified time
// repeat will start everything again when interval completes with take(1)
this.service.fetchInfoFromHttp().pipe(
concatMap((data) => passAndWait(data)),
repeat(Infinity),
).subscribe();
}
use observable from rxjs https://stackblitz.com/edit/angular-q714uq
import { interval, Subject, PartialObserver, Observable } from 'rxjs';
import { takeUntil } from 'rxjs/operators';
after import libraries set interval to 30 then start after finish call your service when service get result again start interval
ispause = new Subject();
private time = 30;
timer: Observable<number>;
timerObserver: PartialObserver<number>;
ngOnInit(){
this.timer = interval(1000)
.pipe(
takeUntil(this.ispause)
);
this.timerObserver = {
next: (_: number) => {
if(this.time==0){
this.ispause.next;
this.callservice();
}
this.time -= 1;
}
};
this.timer.subscribe(this.timerObserver);
}
callservice(){
//after u get data
this.service.fetchListFromHttp()
.pipe(
finalize(() =>
this.time=30;
this.timer.subscribe(this.timerObserver);
)
)
.subscribe(
result => this.list = result,
err => this.logError(err)
);
}
I have the following code:
const rl = require('readline').createInterface({
input: require('fs').createReadStream(__dirname + '/../resources/profiles.txt'),
terminal: true
});
for await (const line of rl) {
scrape_profile(line)
}
scrape_profile is a function that makes some request to the web and perform some processing. now the issue is that i wanted to limit so that 5 scrape_profile is executed per 30 seconds.. as of now if i have a text file with 1000 lines, it would go ahead and execute 1000 concurrent requests at one time.. how do i limit this ?
I'm not entirely sure why you're using a readlineInterface if you're asynchronously reading the entire file into memory at once, so for my answer, I've replaced it with a call to fs.readFileSync as it's much easier to deal with finite values than a stream and the question didn't explicitly state the file IO needed to be streamed.
You could try using Bluebird Promise.reduce:
const fs = require('fs');
const lines = fs.readFileSync('./test.txt').toString().split('\r\n');
const Promise = require('bluebird');
const BATCHES = 5;
const scrape_profile = file => new Promise((resolve, reject) => {
setTimeout(() => {
console.log("Done with", file);
resolve(Math.random());
}, Math.random() * 1000);
});
const runBatch = batchNo => {
const batchSize = Math.round(lines.length / BATCHES);
const start = batchSize * batchNo;
const end = batchSize * (batchNo + 1);
const index = start;
return Promise.reduce(lines.slice(start, end), (aggregate, line) => {
console.log({ aggregate });
return scrape_profile(line)
.then(result => {
aggregate.push(result);
return aggregate;
});
}, []);
}
runBatch(0).then(/* batch 1 done*/)
runBatch(1).then(/* batch 2 done*/)
runBatch(2).then(/* batch 3 done*/)
runBatch(3).then(/* batch 4 done*/)
runBatch(4).then(/* batch 5 done*/)
// ... preferably use a for loop to do this
This is a full example; you should be able to run this locally (with a file called 'test.txt' that has any contents), for each line it will spend a random amount of time generating a random number; it runs 5 separate batches. You need to change the value of BATCHES to reflect the number of batches you need
you can use setinterval for 30 seconds to perform a loop of scrape_profile for 5 times, your loop is using the number of lines which is like you specified 1000 lines without stopping, then make a loop for 5 times and put it in a function that you call with setinterval and of course keep the index of the current line as a variable too to continue from where you left off