I am trying to make a network retry-er using RxJS repeatWhen operator. The idea is that when a new request receives to the scheduler, then I try the request directly and if it results in a network failure result, I add it to a pool to be retried on sometime later. So the entering point of my scheduler is the queue function which does the job like this:
queue({ operation, body }) {
const behaviourSubject = new BehaviorSubject();
const task = {
operation,
body,
behaviourSubject,
};
this.doTask(task).subscribe({
error: _ => this.notifier.next({ tasks: [task], remove: false }),
complete: () => console.log('Task successfully done on first try: ', task),
});
return behaviourSubject;
}
and this.notifier is a Subject which is used as the notifier of the worker. So the worker itself is like this:
const rawWorker = new Observable(subscriber => {
const doneTasks = [];
const jobs = [];
for (const task of this.tasks) {
jobs.push(
this.doTask(task).pipe(
tap(_ => {
doneTasks.push(task);
}),
catchError(error => { task.behaviourSubject.next(error); return of(error); }),
)
);
}
if (jobs.length > 0) {
forkJoin(...jobs).subscribe(_ => {
this.notifier.next({ tasks: doneTasks, remove: true });
subscriber.next(`One cycle of worker done. #${doneTasks.length} task(s) done and #${this.tasks.length} remaining.`);
// subscriber.complete();
if (this.tasks.length > 0) {
this.notifier.next();
}
})
} else {
subscriber.complete();
}
});
const scheduledWorker = rawWorker.pipe( // TODO: delay should be added to retry and repeat routines
retry(),
repeatWhen(_ => this.notifier.pipe(
filter(_ => this.tasks.length > 0),
)),
);
and also the notifier keeps track of all undone requests in an array like this:
this.notifierSubscription = this.notifier
.pipe(
filter(data => data && data.tasks)
)
.subscribe({
next: ({ tasks = [], remove = false }) => {
if (remove) {
console.log('removing tasks: ', tasks);
this.tasks = this.tasks.filter(task => !tasks.some(tsk => task === tsk));
} else {
console.log('inserting: ', tasks);
this.tasks.push.apply(
this.tasks,
tasks,
);
}
console.log('new tasks array: ', this.tasks);
}
});
As I know if a cycle of the worker is not completed then repeatWhen has nothing to do. For example if I remove the part:
else {
subscriber.complete();
}
from the worker, on the first try of the worker (an empty cycle) the Observable does not complete and repeatWhen won't do anything. But on the other hand as you see I have commented // subscriber.complete(); when there exist jobs but the repeat is occurring. And the worst part of the problem is that many different instances of the worker run in parallel which makes many duplicate requests.
I have spent a lot of time on this problem but do not have any clue to trace.
Related
Is there a clean way to do something on first emit from multiple subscriptions ?
e.g.:
this.subscription1 = this.service.getData1().subscribe(data => {
this.data1 = data;
console.log('1');
});
this.subscription2 = this.service.getData2().subscribe(data => {
this.data2 = data2;
console.log('2');
});
// Do something after first emit from subscription1 AND subscription2
doSomething();
...
doSomething() {
console.log('Hello world !');
}
Output goal:
1
2
Hello world !
1
2
1
1
2
1
2
2
...
There've multiple times where I also needed such a isFirst operator that'll run some predicate only for the first emission. I've slapped together a quick custom operator that uses a single state variable first to decide if the emission is indeed first and run some predicate using the tap operator.
Since it uses tap internally it does not modify the source emission in any way. It only runs the passed predicate when the emission is indeed first.
Try the following
isFirst() operator
export const isFirst = (predicate: any) => {
let first = true;
return <T>(source: Observable<T>) => {
return source.pipe(
tap({
next: _ => {
if (first) {
predicate();
first = false;
}
}
})
);
};
};
For combining multiple streams that will be triggered when any of the source emits, you could use RxJS combineLatest function.
Example
import { Component } from "#angular/core";
import { timer, Observable, Subject, combineLatest } from "rxjs";
import { tap, takeUntil } from "rxjs/operators";
#Component({
selector: "my-app",
template: `<button (mouseup)="stop$.next()">Stop</button>`
})
export class AppComponent {
stop$ = new Subject<any>();
constructor() {
combineLatest(timer(2000, 1000), timer(3000, 500))
.pipe(
isFirst(_ => {
console.log("first");
}),
takeUntil(this.stop$)
)
.subscribe({
next: r => console.log("inside subscription:", r)
});
}
}
Working example: Stackblitz
In your case it might look something like
this.subscription = combineLatest(
this.service.getData1().pipe(
tap({
next: data => {
this.data1 = data;
console.log('1');
}
})
),
this.service.getData2().pipe(
tap({
next: data => {
this.data2 = data;
console.log('2');
}
})
)
).pipe(
isFirst(_ => {
console.log("first");
})
).subscribe({
next: r => console.log("inside subscription:", r)
});
The easiest strategy is to have a 3rd Observable that will perform this action.
See below example
const Observable1$ = timer(1000, 2000).pipe(
map(() => 1),
tap(console.log)
);
const Observable2$ = timer(1700, 1700).pipe(
map(() => 2),
tap(console.log)
);
const Observable3$ = combineLatest([Observable1$, Observable2$]).pipe(
take(1),
map(() => "Hello World!"),
tap(console.log)
);
Observable1$.subscribe();
Observable2$.subscribe();
Observable3$.subscribe();
The console output is as per below, since there are two subscribers to Observable1$ (i.e Observable1$ and Observable3$same as two subscribers toObservable2$(i.eObservable2$ and Observable3$ we see console logs 1 1 2 2 'hello world ...'
Here is the link to the stackblitz
In the above we notice that we get 2 subscriptions hence 2 console logs for each. To solve this we can use Subjects to generate new Observables and combine these instead
const track1Subject$ = new Subject();
const track1$ = track1Subject$.asObservable();
const track2Subject$ = new Subject();
const track2$ = track2Subject$.asObservable();
const Observable1$ = timer(1000, 2000).pipe(
map(() => 1),
tap(console.log),
tap(() => track1Subject$.next()),
take(5)
);
const Observable2$ = timer(1700, 1700).pipe(
map(() => 2),
tap(console.log),
tap(() => track2Subject$.next()),
take(5)
);
const Observable3$ = combineLatest([track1$, track2$]).pipe(
take(1),
map(() => "Hello World!"),
tap(console.log)
);
Observable1$.subscribe();
Observable2$.subscribe();
Observable3$.subscribe();
See Link to final solution
With some further restrictions, this problem becomes easier. Unfortunately, operators like combineLatest, and zip add extra structure to your data. I'll provide a solution with zip below, but it doesn't extend at all (if you want to add more logic downstream of your zip, you're out of luck in many cases).
General solution.
Assuming, however, that getData1 and getData2 are completely orthogonal (How they emit and how they are consumed by your app are not related in any predictable way), then a solution to this will require multiple subscriptions or a custom operator tasked with keeping track of emissions.
It's almost certainly the case that you can do something more elegant than this, but this is the most general solution I could think of that meets your very general criteria.
Here, I merge the service calls, tag each call, and pass through emissions until each call has emitted at least once.
merge(
this.service.getData1().pipe(
tap(_ => console.log('1')),
map(payload => ({fromData: 1, payload}))
),
this.service.getData2().pipe(
tap(_ => console.log('2')),
map(payload => ({fromData: 2, payload}))
)
).pipe(
// Custom Operator
s => defer(() => {
let fromData1 = false;
let fromData2 = false;
let done = false;
return s.pipe(
tap(({fromData}) => {
if(done) return;
if(fromData === 1) fromData1 = true;
if(fromData === 2) fromData2 = true;
if(fromData1 && fromData2){
done = true;
doSomething();
}
})
);
})
).subscribe(({fromData, payload}) => {
if(fromData === 1) this.data1 = payload;
if(fromData === 2) this.data2 = payload;
});
In the subscription, we have to separate out the two calls again. Since you're setting a global variable, you could throw that logic as a side effect in the tap operator for each call. This should have similar results.
merge(
this.service.getData1().pipe(
tap(datum => {
console.log('1')
this.data1 = datum;
),
map(payload => ({fromData: 1, payload}))
),
...
The zip Solution
This solution is much shorter to write but does come with some drawbacks.
zip(
this.service.getData1().pipe(
tap(datum => {
console.log('1')
this.data1 = datum;
)
),
this.service.getData2().pipe(
tap(datum => {
console.log('2')
this.data2 = datum;
)
)
).pipe(
map((payload, index) => {
if(index === 0) doSomething();
return payload;
})
).subscribe();
What is passed into your subscription is the service calls paired off. Here, you absolutely must set a global variable as a side effect of the original service call. The option of doing so in the subscription is lost (unless you want them set as pairs).
I get an Observable<Group[]> from my Firebase collection.
In this Group class is an id which I wanna use to retrieve another dataset array from Firebase, which would be messages for each unique group Observable<Message[]>.(each group has its own chat: Message[])
And it want to return an observable which hold an array of a new Type:
return { ...group, messages: Message[] } as GroupWithMessages
the final goal should be Observable<GroupWithMessages[]>
getGroupWithChat(): Observable<GroupWithMessages[]> {
const groupColl = this.getGroups(); // Observable<Group[]>
const messages = groupColl.pipe(
map(groups => {
return groups.map(meet => {
const messages = this.getMessagesFor(group.uid);
return { messages:messages, ...group} as GroupWithMessages
});
})
);
return messages;
}
}
and here the Message function
getMessagesFor(id: string): Observable<Message[]> {
return this.afs.collection<Message>(`meets/${id} /messages`).valueChanges();
}
sadly that doesnt work because when i create the new Obj I cannot bind messages:messages because messages ist vom typ Observable<Message[]>
I hope that cleares things
UPDATE:
my main problem now comes down to this:
getGroupsWithMessages() {
this.getJoinedGroups()
.pipe(
mergeMap(groups =>
from(groups).pipe(
mergeMap(group => {
return this.getMessagesFor(group.uid).pipe(
map(messages => {
return { ...group, messages } as GroupIdMess;
})
);
}),
tap(x => console.log('reaching here: ', x)),
toArray(),
tap(x => console.log('not reaching here = completed: ', x))
)
),
tap(x => console.log('not reaching here: ', x))
)
.subscribe(x => console.log('not reaching here: ', x));
}
when i call that function my console.log is as follows:
Not sure if I follow what you're doing here but the logic look like you'd want:
getGroupWithChat() {
return this.getGroups.pipe(map(groups=> {
return groups.map(group => this.getMessagesFor(group.uid));
})).subscribe(); // trigger "hot" observable
}
Let me know if I can help further after you clarify.
UPDATE:
So it looks like you need to get the UID of the group before making the call to get the GroupMessages[]?
get Group: Observable
call getMessagesFor(Group.uid)
this example gets groups result$ then
concatMap uses groups result$ to make the messages query
this.getGroups().pipe(
concatMap((group: Group) => this.getMessagesFor(group.uid))
).subscribe((messages: GroupWithMessages[]) => {
console.log(messages);
});
You may still want to map them together but it seems like you know how to do that. concatMap waits for the first to finish, then makes the second call which you need.
Is this closer?
Use forkJoin to wait for messages to be received for all groups. Then map the result of forkJoin to an array of GroupWithMessages like this -
getGroupWithChat(): Observable<GroupWithMessages[]> {
return this.getGroups()
.pipe(
switchMap(groups => {
const messagesForAllGroups$ = groups.map(group => this.getMessagesFor(group.uid));
return forkJoin(messagesForAllGroups$)
.pipe(
map(joined => {
//joined has response like -
//[messagesArrayForGroup0, messagesArrayForGroup1, messagesArrayForGroup2....];
const messagesByGroup = Array<GroupWithMessages>();
groups.forEach((group, index) => {
//assuming that GroupWithMessages has group and messages properties.
const gm = new GroupWithMessages();
gm.group = group;
gm.messages = joined[index];
messagesByGroup.push(gm);
});
return messagesByGroup;
})
)
})
)
}
I usually do that by splitting Observable<any[]> to Observable<any> and then mergeMap the results to inner Observable.
Something like this should work:
getMessagesFor(id: string): Observable<number> {
return of(1);
}
getGroups(): Observable<string[]> {
return of(["1", "2"]);
}
getGroupWithChat() {
this.getGroups().pipe(
mergeMap(groups => from(groups)), // Split the stream into individual group elements instead of an array
mergeMap(group => {
return this.getMessagesFor(group).pipe(
map(messages => {
return Object.assign(group, messages);
})
);
})
);
}
Edit:
Consider BehaviorSubject. It doesn't complete at all:
const behSub: BehaviorSubject<number[]> = new BehaviorSubject([1, 2, 3]);
setTimeout(() => {
behSub.next([4, 5, 6]);
}, 5000);
behSub
.pipe(
mergeMap(arr =>
from(arr).pipe(
tap(), // Do something with individual items, like mergeMap to messages
toArray() // Go back to array
)
)
)
.subscribe(console.log, null, () => {
console.log('Complete');
});
I have a firebase subscription in my angular app which fires multiple times.
How can ich achieve that the tasks are processed as a queue so that I can run each task synchronously once?
this.tasks.subscribe(async tasks => {
for (const x of tasks)
await dolongtask(x); // has to be sync
await removetask(x);
});
The problem is that the subribe event fires when the longtask is still processing.
IMHO, I would try and leverage the power of rxjs since we're using it here already anyway and avoid implementing a custom queuing concept as suggested by another answer (though you certainly can do that).
If we simplify the given case a bit, we just have some observable and want to perform a long-running procedure for each emission – in sequence. rxjs allows doing this by means of the concatMap operator essentially out of the box:
$data.pipe(concatMap(item => processItem(item))).subscribe();
This only assumes that processItem returns an observable. Since you used await, I assume your function(s) currently return Promises. These can be trivially converted into observables using from.
The only detail left to look at from the OP is that the observable actually emits an array of items and we want to perform the operation on each item of each emission. To do that, we just flatten the observable using mergeMap.
Let's put it all together. Note that if you take away preparing some stub data and logging, the actual implementation of this is only two lines of code (using mergeMap + concatMap).
const { from, interval } = rxjs;
const { mergeMap, concatMap, take, bufferCount, tap } = rxjs.operators;
// Stub for the long-running operation
function processTask(task) {
console.log("Processing task: ", task);
return new Promise(resolve => {
setTimeout(() => {
console.log("Finished task: ", task);
resolve(task);
}, 500 * Math.random() + 300);
});
}
// Turn processTask into a function returning an observable
const processTask$ = item => from(processTask(item));
// Some stubbed data stream
const tasks$ = interval(250).pipe(
take(9),
bufferCount(3),
);
tasks$.pipe(
tap(task => console.log("Received task: ", task)),
// Flatten the tasks array since we want to work in sequence anyway
mergeMap(tasks => tasks),
// Process each task, but do so consecutively
concatMap(task => processTask$(task)),
).subscribe(null, null, () => console.log("Done"));
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.3.2/rxjs.umd.js"></script>
I am making a couple of assumptions from the code you gave,
other applications add tasks to the firebase db (asynchronously), and this code is implementing the task processor.
your firebase query returns all unprocessed tasks (in a collection) and it emits the full list every time a new task is added.
the query will drop a task only after removeTask() has been run
If this is so, you need a deduping mechanism before the processor.
For the purpose of illustration, I've simulated the firebase query with a subject (renamed it to tasksQuery$) and a sequence of firebase events are simulated at the bottom of the script.
I hope it's not too confusing!
console.clear()
const { mergeMap, filter } = rxjs.operators;
// Simulate tasks query
const tasksQuery$ = new rxjs.Subject();
// Simulate dolongtask and removetask (assume both return promises that can be awaited)
const dolongtask = (task) => {
console.log( `Processing: ${task.id}`);
return new Promise(resolve => {
setTimeout(() => {
console.log( `Processed: ${task.id}`);
resolve('done')
}, 1000);
});
}
const removeTask = (task) => {
console.log( `Removing: ${task.id}`);
return new Promise(resolve => {
setTimeout(() => {
console.log( `Removed: ${task.id}`);
resolve('done')
}, 200);
});
}
// Set up queue (this block could be a class in Typescript)
let tasks = [];
const queue$ = new rxjs.Subject();
const addToQueue = (task) => {
tasks = [...tasks, task];
queue$.next(task);
}
const removeFromQueue = () => tasks = tasks.slice(1);
const queueContains = (task) => tasks.map(t => t.id).includes(task.id)
// Dedupe and enqueue
tasksQuery$.pipe(
mergeMap(tasks => tasks), // flatten the incoming task array
filter(task => task && !queueContains(task)) // check not in queue
).subscribe(task => addToQueue(task) );
//Process the queue
queue$.subscribe(async task => {
await dolongtask(task);
await removeTask(task); // Assume this sends 'delete' to firebase
removeFromQueue();
});
// Run simulation
tasksQuery$.next([{id:1},{id:2}]);
// Add after delay to show repeated items in firebase
setTimeout(() => {
tasksQuery$.next([{id:1},{id:2},{id:3}]);
}, 500);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.3.2/rxjs.umd.js"></script>
Leaving aside your title 'Rxjs subscription queue', you can actually fix your async/await code.
The problem is that async/await does not play nicely with for loops, see this question Using async/await with a forEach loop.
For example, you can replace the for loop as per #Bergi's answer,
with Promise.all()
console.clear();
const { interval } = rxjs;
const { take, bufferCount } = rxjs.operators;
function processTask(task) {
console.log(`Processing task ${task}`);
return new Promise(resolve => {
setTimeout(() => {
resolve(task);
}, 500 * Math.random() + 300);
});
}
function removeTask(task) {
console.log(`Removing task ${task}`);
return new Promise(resolve => {
setTimeout(() => {
resolve(task);
}, 50);
});
}
const tasks$ = interval(250).pipe(
take(10),
bufferCount(3),
);
tasks$.subscribe(async tasks => {
await Promise.all(
tasks.map(async task => {
await processTask(task); // has to be sync
await removeTask(task);
console.log(`Finished task ${task}`);
})
);
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.3.2/rxjs.umd.js"></script>
Better yet, you can shape the query to avoid using a for loop,
with mergeMap()
console.clear();
const { interval } = rxjs;
const { mergeMap, take, bufferCount } = rxjs.operators;
function processTask(task) {
console.log(`Processing task ${task}`);
return new Promise(resolve => {
setTimeout(() => {
resolve(task);
}, 500 * Math.random() + 300);
});
}
function removeTask(task) {
console.log(`Removing task ${task}`);
return new Promise(resolve => {
setTimeout(() => {
resolve(task);
}, 50);
});
}
const tasks$ = interval(250).pipe(
take(10),
bufferCount(3),
);
tasks$
.pipe(mergeMap(tasks => tasks))
.subscribe(
async task => {
await processTask(task); // has to be sync
await removeTask(task);
console.log(`Finished task ${task}`);
}
);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.3.2/rxjs.umd.js"></script>
I'm trying to extend some existing code with additional promises, but they are a new topic for me at the moment and i'm obviously missing something. This is running as part of a build scrip for npm.
All i am currently trying to make happen is for the final then to be called after the pack operation has happened for each architecture. I have tried wrapping it in a
return new Promise
But at the moment i am not returning anything from that function so i'm not sure what i should include in the resolve call at the end. If i just call the resolve with a true nothing happens, and wrapping it in a promise seems to cause the function to not actually run, and no errors are caught anywhere?
I'm guessing i am going about this completely wrong, all i want to achieve is to run another function once the previous one has completed?
Here's the code as it stands with the additional .then that i can't get to be called.
function build(cfg) {
return new Promise((resolve, reject) => {
webpack(cfg, (err, stats) => {
if (err) return reject(err);
resolve(stats);
});
});
}
function startPack() {
console.log('start pack...');
build(electronCfg)
.then(() => build(cfg))
.then(() => del('release'))
.then(paths => {
if (shouldBuildAll) {
// build for all platforms
const archs = ['ia32', 'x64'];
const platforms = ['linux', 'win32', 'darwin'];
platforms.forEach(plat => {
archs.forEach(arch => {
pack(plat, arch, log(plat, arch));
});
});
} else {
// build for current platform only
pack(os.platform(), os.arch(), log(os.platform(), os.arch()));
}
})
.then(() => {
console.log('then!');
})
.catch(err => {
console.error(err);
});
}
function pack(plat, arch, cb) {
// there is no darwin ia32 electron
if (plat === 'darwin' && arch === 'ia32') return;
const iconObj = {
icon: DEFAULT_OPTS.icon + (() => {
let extension = '.png';
if (plat === 'darwin') {
extension = '.icns';
} else if (plat === 'win32') {
extension = '.ico';
}
return extension;
})()
};
const opts = Object.assign({}, DEFAULT_OPTS, iconObj, {
platform: plat,
arch,
prune: true,
'app-version': pkg.version || DEFAULT_OPTS.version,
out: `release/${plat}-${arch}`,
'osx-sign': true
});
packager(opts, cb);
}
You didn't say what log is, but if it's a plain logging function, then it looks like you're passing in undefined (the result from calling log(...)) as the cb argument to pack. Perhaps you meant:
pack(plat, arch, () => log(plat, arch));
In any case, this won't do anything to wait for packing to finish. I don't know why you're not seeing any console output, but if you're looking for this output to happen after all the packing has finished, then you need to wrap packager in a promise. Something like:
var pack = (plat, arch) => new Promise(resolve => {
// ...
packager(opts, resolve);
});
And then use Promise.all instead of forEach to do all the packaging (in parallel if that's OK):
.then(paths => {
if (!shouldBuildAll) {
return pack(os.platform(), os.arch());
}
return Promise.all(['linux', 'win32', 'darwin'].map(plat =>
Promise.all(['ia32', 'x64'].map(arch => pack(plat, arch))));
})
.then(() => console.log('then!'))
.catch(err => console.error(err));
I have the following recursive function that recursively calls an iterator
The conversions.next() call is a request-promise call to the next page of an api endpoint
The conversions.done just checks if there are no more pages
This loop is kicked off by passing a call to the api endpoint as the first parameter.
The problem is that my node process never exits using this, I seem to have 10 + open TCP socket connections when the process._getActiveHandles() is run at the end of the code, but node has no further code to execute, but does not exit
const loadConversions = function loadConversions(httpCallPromise, allConversions) {
allConversions = typeof allConversions !== 'undefined' ? allConversions : [];
return httpCallPromise.then((conversions) => {
allConversions = allConversions.concat(conversions.value);
if (conversions.done === false) {
return loadConversions(conversions.next(), allConversions);
}
return Promise.resolve(allConversions);
});
};
// this is the entry point for the code
if (args[0] === 'conversions') {
loadConversions(queries.conversions())
.then((allConversions) => {
console.log(allConversions.length);
return Promise.all(allConversions.map(db.insertConversion));
})
.then(() => {
console.log('done db insertions');
console.log(process._getActiveHandles().length);
})
.catch((err) => {
console.log(err);
});
}
the output of process.getActiveHandles().length is 13 node processes of type tcp socket
Here is the iterator
const cakeRequest = function(options) {
// make the request
return rp(options)
.then((response) => {
const processed = processResponse(response);
return {
next: function next() {
const nextOptions = Object.assign({}, options, {
qs: Object.assign({}, options.qs, {
start_at_row: parseInt(options.qs.start_at_row, 10) + parseInt(options.qs.row_limit, 10),
}),
});
return cakeRequest(nextOptions);
},
value: processed.value,
done: processed.done,
row_count: processed.row_count,
};
})
.catch(handleError);
};
My Issue seemed to be with the knex library used for the DB insert. If the pool is never closed, the process never exits. This seems like strange behavior to me, but calling knex.destroy at the end let the process exit