I want to retrieve a list of products in relation to the user's position, for this I use Geofirestore and update my Flatlist
When I have my first 10 closest collections, I loop to have each of the sub-collections.
I manage to update my state well, but every time my collection is modified somewhere else, instead of updating my list, it duplicates me the object that has been modified and adds it (updated) at the end of my list and keep the old object in that list too.
For example:
const listListeningEvents = {
A: {Albert, Ducon}
B: {Mickael}
}
Another user modified 'A' and delete 'Ducon', I will get:
const listListeningEvents = {
A: {Albert, Ducon},
B: {Mickael},
A: {Albert}
}
And not:
const listListeningEvents = {
A: {Albert},
B: {Mickael},
}
That's my useEffect:
useEffect(() => {
let geoSubscriber;
let productsSubscriber;
// 1. getting user's location
getUserLocation()
// 2. then calling geoSubscriber to get the 10 nearest collections
.then((location) => geoSubscriber(location.coords))
.catch((e) => {
throw new Error(e.message);
});
//Here
geoSubscriber = async (coords) => {
let nearbyGeocollections = await geocollection
.limit(10)
.near({
center: new firestore.GeoPoint(coords.latitude, coords.longitude),
radius: 50,
})
.get();
// Empty array for loop
let nearbyUsers = [];
// 3. Getting Subcollections by looping onto the 10 collections queried by Geofirestore
productsSubscriber = await nearbyGeocollections.forEach((geo) => {
if (geo.id !== user.uid) {
firestore()
.collection("PRODUCTS")
.doc(geo.id)
.collection("USER_PRODUCTS")
.orderBy("createdDate", "desc")
.onSnapshot((product) => {
// 4. Pushing each result (and I guess the issue is here!)
nearbyUsers.push({
id: product.docs[0].id.toString(),
products: product.docs,
});
});
}
});
setLoading(false);
// 4. Setting my state which will be used within my Flatlist
setListOfProducts(nearbyUsers);
};
return () => {
if (geoSubscriber && productsSubscriber) {
geoSubscriber.remove();
productsSubscriber.remove();
}
};
}, []);
I've been struggling since ages to make this works properly and I'm going crazy.
So I'm dreaming about 2 things :
Be able to update my state without duplicating modified objects.
(Bonus) Find a way to get the 10 next nearest points when I scroll down onto my Flatlist.
In my opinion the problem is with type of nearbyUsers. It is initialized as Array =[] and when you push other object to it just add new item to at the end (array reference).
In this situation Array is not very convenient as to achieve the goal there is a need to check every existing item in the Array and find if you find one with proper id update it.
I think in this situation most convenient will be Map (Map reference). The Map indexes by the key so it is possible to just get particular value without searching it.
I will try to adjust it to presented code (not all lines, just changes):
Change type of object used to map where key is id and value is products:
let nearbyUsersMap = new Map();
Use set method instead of push to update products with particular key:
nearbyUsersMap.set(product.docs[0].id.toString(), product.docs);
Finally covert Map to Array to achieve the same object to use in further code (taken from here):
let nearbyUsers = Array.from(nearbyUsersMap, ([id, products]) => ({ id, products }));
setListOfProducts(nearbyUsers);
This should work, but I do not have any playground to test it. If you get any errors just try to resolve them. I am not very familiar with the geofirestore so I cannot help you more. For sure there are tones of other ways to achieve the goal, however this should work in the presented code and there are just few changes.
I have a case where I may or may not need to add observables to a list. I then want to forkJoin the observables I do have so the page can load once all of the data is available.
let observables: Observable<any>[] = [];
observables.push(this.taskService.getStep(this.housingTransactionId, this.task.stageReferenceId, this.task.stepReferenceId));
if (this.task.associatedChatThreadId) {
observables.push(this.messageHubService.getChatThread(this.housingTransactionId, this.task.associatedChatThreadId));
}
if (this.task.associatedDocuments && this.task.associatedDocuments.length > 0) {
this.task.associatedDocuments.forEach(documentId => {
observables.push(this.documentHubService.getDocumentProperties(this.housingTransactionId, documentId));
});
}
Observable.forkJoin(observables)
.subscribe(([step, chatThread, ...documents]) => {
this.step = step;
this.chatThread = chatThread;
this.documents = documents;
this.isPageLoading = false;
}, error => {
this.isPageLoading = false;
console.log(error);
});
The problem I'm getting is that if I don't have a this.task.associatedChatThreadId, then the observable is not added to the list and when the forkJoin is executed, the ...documents are in the position of the chatThread property in the subscribe method (well, the first document!).
Is there a way to ensure the positioning of the responses from a forkJoin? Or should I/can I use a different approach?
Most easily you can add a dumb Observable.of(null) with null value if the condition is not met in order to keep the same order of responses:
if (this.task.associatedChatThreadId) {
observables.push(this.messageHubService....);
} else {
observables.push(Observable.of(null))
}
Then in the subscription you can check if chatThread === null becauese it'll always be present at the same position.
Alternatively, you could wrap each Observable in observables with some extra object that would make it uniquely identifiable in the subscriber but that would be unnecessarily complicated so I'd personally stick to the first option.
Another approach would be not to use folkJoin but subscribe separately. At the same time, make isPageLoading a BehaviorSubject which counts how many async requests you currently have. Each time when you make a request, you can have isPageLoading.next(1), and isPageLoading.next(-1) when you finish a request.
You could make a helper function that accepts an object which has string keys and observable values and returns an observable that will emit an object with the same keys, but having the resulting values instead of the observables as values.
I would not really say that this is a cleaner version than using of(null) like suggested by martin, but it might be an alternative.
function namedForkJoin(map: {[key: string]: Observable<any>}): Observable<{[key: string]: any}> {
// Get object keys
const keys = Object.keys(map);
// If our observable map is empty, we want to return an empty object
if (keys.length === 0) {
return of({});
}
// Create a fork join operation out of the available observables
const forkJoin$ = Observable.forkJoin(...keys.map(key => map[key]))
return forkJoin$
.map(array => {
const result = {};
for (let index = 0; index < keys.length; index++) {
result[keys[index]] = array[index];
}
}));
}
Please keep in mind, I did not have angular or rxjs running here at the moment, so I could not verify the function really works. But the idea is:
1. Get the keys from the input map.
2. Use the keys to get an array of observables and pass that to fork join.
3. Add a mapping function that converts the resulting array back into an object.
I'm building an Electron application and I'm trying to keep an array of images that are on each page so if the page is deleted I can easily delete all the associated images from the filesystem.
What I have now:
const initialState = [{
uuid: '65ec81f5-a783-4abd-bd0d-1451adda58c6',
imageUUIDs: []
}];
const actionsMap = {
[ActionTypes.ADD_IMAGE_TO_PAGE](state, action) {
const pageUUID = action.pageUUID;
const imgUUID = action.imgUUID;
return state.map(page =>
(page.uuid === pageUUID ?
Object.assign({}, page, {
imageUUIDs: page.imageUUIDs.splice(0, 0, imgUUID),
}) : page)
);
},
};
This initially seems to work when viewed at runtime, but I end up with an empty array after completion. What is the right way to maintain a list like this?
Use Array.prototype.concat instead of Array.prototype.splice. Array.prototype.concat returns a new array whereas Array.prototype.splice modifies the original array but does not return a new array, which is not what you are expecting.
I would like to use make a series of requests to a server, but the server has a hard rate limit of 10 request per second. If I try to make the requests in a loop, it will hit the rate limit since all the requests will happen at the same time.
for(let i = 0; i < 20; i++) {
sendRequest();
}
ReactiveX has lots of tools for modifying observable streams, but I can't seem to find the tools to implement rate limiting. I tried adding a standard delay, but the requests still fire at the same time, just 100ms later than they did previously.
const queueRequest$ = new Rx.Subject<number>();
queueRequest$
.delay(100)
.subscribe(queueData => {
console.log(queueData);
});
const queueRequest = (id) => queueRequest$.next(id);
function fire20Requests() {
for (let i=0; i<20; i++) {
queueRequest(i);
}
}
fire20Requests();
setTimeout(fire20Requests, 1000);
setTimeout(fire20Requests, 5000);
The debounceTime and throttleTime operators are similar to what I'm looking for as well, but that is lossy instead of lossless. I want to preserve every request that I make, instead of discarding the earlier ones.
...
queueRequest$
.debounceTime(100)
.subscribe(queueData => {
sendRequest();
});
...
How do I make these requests to the server without exceeding the rate limit using ReactiveX and Observables?
The implementation in the OP's self answer (and in the linked blog) always imposes a delay which is less than ideal.
If the rate-limited service allows for 10 requests per second, it should be possible to make 10 requests in, say, 10 milliseconds, as long as the next request is not made for another 990 milliseconds.
The implementation below applies a variable delay to ensure the limit is enforced and the delay is only applied to requests that would see the limit exceeded.
function rateLimit(source, count, period) {
return source
.scan((records, value) => {
const now = Date.now();
const since = now - period;
// Keep a record of all values received within the last period.
records = records.filter((record) => record.until > since);
if (records.length >= count) {
// until is the time until which the value should be delayed.
const firstRecord = records[0];
const lastRecord = records[records.length - 1];
const until = firstRecord.until + (period * Math.floor(records.length / count));
// concatMap is used below to guarantee the values are emitted
// in the same order in which they are received, so the delays
// are cumulative. That means the actual delay is the difference
// between the until times.
records.push({
delay: (lastRecord.until < now) ?
(until - now) :
(until - lastRecord.until),
until,
value
});
} else {
records.push({
delay: 0,
until: now,
value
});
}
return records;
}, [])
.concatMap((records) => {
const lastRecord = records[records.length - 1];
const observable = Rx.Observable.of(lastRecord.value);
return lastRecord.delay ? observable.delay(lastRecord.delay) : observable;
});
}
const start = Date.now();
rateLimit(
Rx.Observable.range(1, 30),
10,
1000
).subscribe((value) => console.log(`${value} at T+${Date.now() - start}`));
<script src="https://unpkg.com/rxjs#5/bundles/Rx.min.js"></script>
This blog post does a great job of explaining that RxJS is great at discarding events, and how they came to the answer, but ultimately, the code you're looking for is:
queueRequest$
.concatMap(queueData => Rx.Observable.of(queueData).delay(100))
.subscribe(() => {
sendRequest();
});
concatMap adds concatenates the newly created observable to the back of the observable stream. Additionally, using delay pushes back the event by 100ms, allowing 10 request to happen per second. You can view the full JSBin here, which logs to the console instead of firing requests.
Actually, there's an easier way to do this with the bufferTime() operator and its three arguments:
bufferTime(bufferTimeSpan, bufferCreationInterval, maxBufferSize)
This means we can use bufferTime(1000, null, 10) which means we'll emit a buffer of max 10 items or after max 1s. The null means we want to open a new buffer immediately after the current buffer is emitted.
function mockRequest(val) {
return Observable
.of(val)
.delay(100)
.map(val => 'R' + val);
}
Observable
.range(0, 55)
.concatMap(val => Observable.of(val)
.delay(25) // async source of values
// .delay(175)
)
.bufferTime(1000, null, 10) // collect all items for 1s
.concatMap(buffer => Observable
.from(buffer) // make requests
.delay(1000) // delay this batch by 1s (rate-limit)
.mergeMap(value => mockRequest(value)) // collect results regardless their initial order
.toArray()
)
// .timestamp()
.subscribe(val => console.log(val));
See live demo: https://jsbin.com/mijepam/19/edit?js,console
You can experiment with different initial delay. With only 25ms the request will be sent in batches by 10:
[ 'R0', 'R1', 'R2', 'R3', 'R4', 'R5', 'R6', 'R7', 'R8', 'R9' ]
[ 'R10', 'R11', 'R12', 'R13', 'R14', 'R15', 'R16', 'R17', 'R18', 'R19' ]
[ 'R20', 'R21', 'R22', 'R23', 'R24', 'R25', 'R26', 'R27', 'R28', 'R29' ]
[ 'R30', 'R31', 'R32', 'R33', 'R34', 'R35', 'R36', 'R37', 'R38', 'R39' ]
[ 'R40', 'R41', 'R42', 'R43', 'R44', 'R45', 'R46', 'R47', 'R48', 'R49' ]
[ 'R50', 'R51', 'R52', 'R53', 'R54' ]
But with .delay(175) we'll emit batches of less than 10 items because we're limited by the 1s delay.
[ 'R0', 'R1', 'R2', 'R3', 'R4' ]
[ 'R5', 'R6', 'R7', 'R8', 'R9', 'R10' ]
[ 'R11', 'R12', 'R13', 'R14', 'R15' ]
[ 'R16', 'R17', 'R18', 'R19', 'R20', 'R21' ]
[ 'R22', 'R23', 'R24', 'R25', 'R26', 'R27' ]
[ 'R28', 'R29', 'R30', 'R31', 'R32' ]
[ 'R33', 'R34', 'R35', 'R36', 'R37', 'R38' ]
[ 'R39', 'R40', 'R41', 'R42', 'R43' ]
[ 'R44', 'R45', 'R46', 'R47', 'R48', 'R49' ]
[ 'R50', 'R51', 'R52', 'R53', 'R54' ]
There's however one difference to what you might need. This solution starts initially starts emitting values after 2s delay because of the .bufferTime(1000, ...) and delay(1000). All other emissions happen after 1s.
You could eventually use:
.bufferTime(1000, null, 10)
.mergeAll()
.bufferCount(10)
This will always collect 10 items and only after that it'll perform the request. This would be probably more efficient.
I wrote a library to do this, you set up the maximum number of requests per interval and it rate limits observables by delaying subscriptions. It's tested and with examples: https://github.com/ohjames/rxjs-ratelimiter
Go with Adam’s answer. However, bear in mind the traditional of().delay() will actually add a delay before every element. In particular, this will delay the first element of your observable, as well as any element that wasn’t actually rate limited.
Solution
You can work around this by having your concatMap return a stream of observables that immediately emit a value, but only complete after a given delay:
new Observable(sub => {
sub.next(v);
setTimeout(() => sub.complete(), delay);
})
This is kind of a mouthful, so I’d create a function for it. That said, since there’s no use for this outside of actual rate limiting, you’d probably be better served just writing a rateLimit operator:
function rateLimit<T>(
delay: number,
scheduler: SchedulerLike = asyncScheduler): MonoTypeOperatorFunction<T> {
return concatMap(v => new Observable(sub => {
sub.next(v);
scheduler.schedule(() => sub.complete(), delay);
}));
}
Then:
queueRequest$.pipe(
rateLimit(100),
).subscribe(...);
Limitation
This will now create a delay after every element. This means that if your source observable emits its last value then completes, your resulting rate-limited observable will have a little delay between itself between its last value, and completing.
Updated cartant's answer as pipe-able operator for newer rxjs versions:
function rateLimit(count: number, period: number) {
return <ValueType>(source: Observable<ValueType>) => {
return source.pipe
( scan((records, value) => {
let now = Date.now();
let since = now - period;
// Keep a record of all values received within the last period.
records = records.filter((record) => record.until > since);
if (records.length >= count) {
// until is the time until which the value should be delayed.
let firstRecord = records[0];
let lastRecord = records[records.length - 1];
let until = firstRecord.until + (period * Math.floor(records.length / count));
// concatMap is used below to guarantee the values are emitted
// in the same order in which they are received, so the delays
// are cumulative. That means the actual delay is the difference
// between the until times.
records.push(
{ delay: (lastRecord.until < now) ?
(until - now) :
(until - lastRecord.until)
, until
, value });
} else {
records.push(
{ delay: 0
, until: now
, value });
}
return records;
}, [] as RateLimitRecord<ValueType>[])
, concatMap((records) => {
let lastRecord = records[records.length - 1];
let observable = of(lastRecord.value);
return lastRecord.delay ? observable.pipe(delay(lastRecord.delay)) : observable;
}) );
};
}
interface RateLimitRecord<ValueType> {
delay: number;
until: number;
value: ValueType;
}
How to group an Observable, and from each GroupedObservable keep in memory only the last emitted item?
So that each group would behave just like BehaviorSubject.
Something like this:
{user: 1, msg: "Anyone here?"}
{user: 2, msg: "Hi"}
{user: 2, msg: "How are you?"}
{user: 1, msg: "Hello"}
{user: 1, msg: "Good"}
So in memory we'd have only have the last item for each user:
{user: 2, msg: "How are you?"}
{user: 1, msg: "Good"}
And when a subscriber subscribes, these two items were issued right away (each in it's own emission). Like we had BehaviorSubject for each user.
onCompleted() is never expected to fire, as people may chat forever.
I don't know in advance what user values there can be.
I assume your chatlog observable is hot. The groupObservables emitted by #groupBy will consequently also be hot and won't keep anything in memory by themselves.
To get the behavior you want (discard everything but the last value from before subscription and continue from there) you could use a ReplaySubject(1).
Please correct me if I'm wrong
see jsbin
var groups = chatlog
.groupBy(message => message.user)
.map(groupObservable => {
var subject = new Rx.ReplaySubject(1);
groupObservable.subscribe(value => subject.onNext(value));
return subject;
});
You can write the reducing function that turns out the latest emitted items of grouped observables, pass that to a scan observable, and use shareReplay to recall the last values emitted for new subscribers. It would be something like this :
var fn_scan = function ( aMessages, message ) {
// aMessages is the latest array of messages
// this function will update aMessages to reflect the arrival of the new message
var aUsers = aMessages.map(function ( x ) {return x.user;});
var index = aUsers.indexOf(message.user);
if (index > -1) {
// remove previous message from that user...
aMessages.splice(index, 1);
}
// ...and push the latest message
aMessages.push(message);
return aMessages;
};
var groupedLatestMessages$ = messages$
.scan(fn_scan, [])
.shareReplay(1);
So what you get anytime you subscribe is an array whose size at any moment will be the number of users who emitted messages, and whose content will be the messages emitted by the users ordered by time of emission.
Anytime there is a subscription the latest array is immediately passed on to the subscriber. That's an array though, I can't think of a way how to pass the values one by one, at the same time fulfilling your specifications. Hope that is enough for your use case.
UPDATE : jsbin here http://jsfiddle.net/zs7ydw6b/2