How to use javascript rxjs to run batches of calculations - javascript

I am new to rxjs, but need to use this asynchronous regime to fulfill such task:
I have many calculation requests, say 10K, and I want to execute them in batches: 1K per batch and the batch is pre-determined. Only when the current batch is done, I will move on to the next batch. The function will be:
calcBatch(dataset)
The input dateset will come from an array of datasets: datasets = [...]
the synchronous loop would look like:
datasets.foreach(dataset=> {
while (!calculation_is_done) {
wait();
}
calcBatch(dataset);
});
calcBatch(dataset) {
calculation_is_done = false;
/* calculation */
calculation_is_done = true;
}
Now switching to an asynchronous regime, how should I construct the flow? What I am thinking is in the calcBatch, when the work is done, a promise or an observable will be returned. Then in the loop, a subscriber will listen on this promise or observable, once it is caught, calcBatch will be called for the next batch.
The calculation needs to process in batches because the backend (HTTP) cannot handle a full set of calculation.

I think bufferCount and concatMap might be what you need:
function calcBatch(dataset) {
console.log('Processing batch:', dataset)
// Fake async request
return new Promise(res => {
setTimeout(() => res(dataset), 2000)
})
}
from(datasets)
.pipe(
bufferCount(1000),
concatMap(dataset => calcBatch(dataset))
)
.subscribe(dataset => {
console.log('Batch done:', dataset)
})
Note: calcBatch needs to return a promise or an observable.

Related

Aggregate multiple calls then separate result with Promise

Currently I have many concurrent identical calls to my backend, differing only on an ID field:
getData(1).then(...) // Each from a React component in a UI framework, so difficult to aggregate here
getData(2).then(...)
getData(3).then(...)
// creates n HTTP requests... inefficient
function getData(id: number): Promise<Data> {
return backend.getData(id);
}
This is wasteful as I make more calls. I'd like to keep my getData() calls, but then aggregate them into a single getDatas() call to my backend, then return all the results to the callers. I have more control over my backend than the UI framework, so I can easily add a getDatas() call on it. The question is how to "mux" the JS calls into one backend call, the "demux" the result into the caller's promises.
const cache = Map<number, Promise<Data>>()
let requestedIds = []
let timeout = null;
// creates just 1 http request (per 100ms)... efficient!
function getData(id: number): Promise<Data> {
if (cache.has(id)) {
return cache;
}
requestedIds.push(id)
if (timeout == null) {
timeout = setTimeout(() => {
backend.getDatas(requestedIds).then((datas: Data[]) => {
// TODO: somehow populate many different promises in cache??? but how?
requestedIds = []
timeout = null
}
}, 100)
}
return ???
}
In Java I would create a Map<int, CompletableFuture> and upon finishing my backend request, I would look up the CompletableFuture and call complete(data) on it. But I think in JS Promises can't be created without an explicit result being passed in.
Can I do this in JS with Promises?
A little unclear on what your end goal looks like. I imagine you could loop through your calls as needed; Perhaps something like:
for (let x in cache){
if (x.has(id))
return x;
}
//OR
for (let x=0; x<id.length;x++){
getData(id[x])
}
Might work. You may be able to add a timing method into the mix if needed.
Not sure what your backend consists of, but I do know GraphQL is a good system for making multiple calls.
It may be ultimately better to handle them all in one request, rather than multiple calls.
The cache can be a regular object mapping ids to promise resolution functions and the promise to which they belong.
// cache maps ids to { resolve, reject, promise, requested }
// resolve and reject belong to the promise, requested is a bool for bookkeeping
const cache = {};
You might need to fire only once, but here I suggest setInterval to regularly check the cache for unresolved requests:
// keep the return value, and stop polling with clearInterval()
// if you really only need one batch, change setInterval to setTimeout
function startGetBatch() {
return setInterval(getBatch, 100);
}
The business logic calls only getData() which just hands out (and caches) promises, like this:
function getData(id) {
if (cache[id]) return cache[id].promise;
cache[id] = {};
const promise = new Promise((resolve, reject) => {
Object.assign(cache[id], { resolve, reject });
});
cache[id].promise = promise;
cache[id].requested = false;
return cache[id].promise;
}
By saving the promise along with the resolver and rejecter, we're also implementing the cache, since the resolved promise will provide the thing it resolved to via its then() method.
getBatch() asks the server in a batch for the not-yet-requested getData() ids, and invokes the corresponding resolve/reject functions:
function getBatch() {
// for any
const ids = [];
Object.keys(cache).forEach(id => {
if (!cache[id].requested) {
cache[id].requested = true;
ids.push(id);
}
});
return backend.getDatas(ids).then(datas => {
Object.keys(datas).forEach(id => {
cache[id].resolve(datas[id]);
})
}).catch(error => {
Object.keys(datas).forEach(id => {
cache[id].reject(error);
delete cache[id]; // so we can retry
})
})
}
The caller side looks like this:
// start polling
const interval = startGetBatch();
// in the business logic
getData(5).then(result => console.log('the result of 5 is:', result));
getData(6).then(result => console.log('the result of 6 is:', result));
// sometime later...
getData(5).then(result => {
// if the promise for an id has resolved, then-ing it still works, resolving again to the -- now cached -- result
console.log('the result of 5 is:', result)
});
// later, whenever we're done
// (no need for this if you change setInterval to setTimeout)
clearInterval(interval);
I think I've found a solution:
interface PromiseContainer {
resolve;
reject;
}
const requests: Map<number, PromiseContainer<Data>> = new Map();
let timeout: number | null = null;
function getData(id: number) {
const promise = new Promise<Data>((resolve, reject) => requests.set(id, { resolve, reject }))
if (timeout == null) {
timeout = setTimeout(() => {
backend.getDatas([...requests.keys()]).then(datas => {
for (let [id, data] of Object.entries(datas)) {
requests.get(Number(id)).resolve(data)
requests.delete(Number(id))
}
}).catch(e => {
Object.values(requests).map(promise => promise.reject(e))
})
timeout = null
}, 100)
}
return promise;
}
The key was figuring out I could extract the (resolve, reject) from a promise, store them, then retrieve and call them later.

Resolving multiple promises inside an observable not working

I'm using Firebase Storage and I'm trying to load all assets via a function call. The only way to get an assets url is to call getDownloadURL which returns a promise. I need to call this for every asset but I can't make it wait for all promises to be done before continuing for some reason.
I thought returning a promise from mergeMap would make it wait for all of them but that doesn't seem to be the case.
I've look at a number of questions regarding promises and RXJS but I can't seem to figure out what's wrong with the code.
getAssets() {
return this.authService.user$.pipe(
first(),
switchMap(user => defer(() => from(this.afs.storage.ref(`${user.uid}/assets`).listAll()))),
switchMap(assets => from(assets.items).pipe(
mergeMap(async (asset) => {
return new Promise((res, rej) => {
asset.getDownloadURL().then(url => {
const _asset = {
name: asset.name,
url,
};
this.assets.push(_asset);
res(_asset);
})
.catch((e) => rej(e));
});
}),
)),
map(() => this.assets),
);
}
...
this.getAssets().subscribe(assets => console.log(assets)); // this runs before all asset's url has been resolved
Overview
mergeMap doesn't wait for all internal observables. It spins up n internal observable pipes that run in parallel, and spits all the values out the same coupling at the bottom of the pipe (your subscribe statement in this case) as individual emissions. Hence why this.getAssets().subscribe(assets => console.log(assets)) runs before all your parallel internal mergeMap pipes complete their individual computations, because mergeMap doesn't wait for all of them before emitting (it will emit one by one as they finish). If you want to wait for n observable pipes to finish, then you need to use forkJoin.
Fork Join
forkJoin is best used when you have a group of observables and only care about the final emitted value of each. One common use case for this is if you wish to issue multiple requests on page load (or some other event) and only want to take action when a response has been received for all. In this way it is similar to how you might use Promise.all.
Solution
getAssets(): Observable<Asset[]> {
return this.authService.user$.pipe(
// first() will deliver an EmptyError to the observer's error callback if the
// observable completes before any next notification was sent. If you don't
// want this behavior, use take(1) instead.
first(),
// Switch to users firebase asset stream.
switchMap(user => {
// You might have to tweak this part. I'm not exactly sure what
// listAll() returns. I guessed that it returns a promise with
// firebase asset metadata.
return from(this.afs.storage.ref(`${user.uid}/assets`).listAll());
}),
// Map to objects that contain method to get image url.
map(firebaseAssetMetadata => firebaseAssetMetadata?.items ?? []),
// Switch to parallel getDownloadUrl streams.
switchMap(assets => {
// Not an rxjs map, a regular list map. Returns a list of getAssetUrlPipes.
const parallelGetAssetUrlPipes = assets.map(asset => {
return from(asset.getDownloadUrl()).pipe(
map(url => { name: asset.name, url })
);
});
// 1) Listen to all parallel pipes.
// 2) Wait until they've all completed.
// 3) Merge all parallel data into a list.
// 4) Then move list down the pipe.
return forkJoin(parallelGetAssetUrlPipes);
}),
// Outputs all parallel pipe data as a single emission in list form.
// Set local variable to users asset data.
tap(assetObjects => this.assets = assetObjects)
);
}
// Outputs the list of user asset data.
this.getAssets().subscribe(console.log);
Good luck out there, and enjoy your Swedish meatballs!
const { from } = rxjs
const { mergeMap } = rxjs.operators
const assets = [1,2,3,4,5]
function getUrl (index) {
return new Promise((res) => {
setTimeout(() => res(`http://example.com/${index}`), Math.random() * 3 + 1000)
})
}
// add param2 1 for mergeMap === concatMap
from(assets).pipe(
mergeMap(asset => {
return getUrl(asset)
}, 1)
).subscribe(console.log)
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
use concatMap to run one by one.

JS: what's a use-case of Promise.resolve()

I am looking at https://www.promisejs.org/patterns/ and it mentions it can be used if you need a value in the form of a promise like:
var value = 10;
var promiseForValue = Promise.resolve(value);
What would be the use of a value in promise form though since it would run synchronously anyway?
If I had:
var value = 10;
var promiseForValue = Promise.resolve(value);
promiseForValue.then(resp => {
myFunction(resp)
})
wouldn't just using value without it being a Promise achieve the same thing:
var value = 10;
myFunction(10);
Say if you write a function that sometimes fetches something from a server, but other times immediately returns, you will probably want that function to always return a promise:
function myThingy() {
if (someCondition) {
return fetch('https://foo');
} else {
return Promise.resolve(true);
}
}
It's also useful if you receive some value that may or may not be a promise. You can wrap it in other promise, and now you are sure it's a promise:
const myValue = someStrangeFunction();
// Guarantee that myValue is a promise
Promise.resolve(myValue).then( ... );
In your examples, yes, there's no point in calling Promise.resolve(value). The use case is when you do want to wrap your already existing value in a Promise, for example to maintain the same API from a function. Let's say I have a function that conditionally does something that would return a promise — the caller of that function shouldn't be the one figuring out what the function returned, the function itself should just make that uniform. For example:
const conditionallyDoAsyncWork = (something) => {
if (something == somethingElse) {
return Promise.resolve(false)
}
return fetch(`/foo/${something}`)
.then((res) => res.json())
}
Then users of this function don't need to check if what they got back was a Promise or not:
const doSomethingWithData = () => {
conditionallyDoAsyncWork(someValue)
.then((result) => result && processData(result))
}
As a side node, using async/await syntax both hides that and makes it a bit easier to read, because any value you return from an async function is automatically wrapped in a Promise:
const conditionallyDoAsyncWork = async (something) => {
if (something == somethingElse) {
return false
}
const res = await fetch(`/foo/${something}`)
return res.json()
}
const doSomethingWithData = async () => {
const result = await conditionallyDoAsyncWork(someValue)
if (result) processData(result)
}
Another use case: dead simple async queue using Promise.resolve() as starting point.
let current = Promise.resolve();
function enqueue(fn) {
current = current.then(fn);
}
enqueue(async () => { console.log("async task") });
Edit, in response to OP's question.
Explanation
Let me break it down for you step by step.
enqueue(task) add the task function as a callback to promise.then, and replace the original current promise reference with the newly returned thenPromise.
current = Promise.resolve()
thenPromise = current.then(task)
current = thenPromise
As per promise spec, if task function in turn returns yet another promise, let's call it task() -> taskPromise, well then the thenPromise will only resolve when taskPromise resolves. thenPromise is practically equivalent to taskPromise, it's just a wrapper. Let's rewrite above code into:
current = Promise.resolve()
taskPromise = current.then(task)
current = taskPromise
So if you go like:
enqueue(task_1)
enqueue(task_2)
enqueue(task_3)
it expands into
current = Promise.resolve()
task_1_promise = current.then(task_1)
task_2_promise = task_1_promise.then(task_2)
task_3_promise = task_2_promise.then(task_3)
current = task_3_promise
effectively forms a linked-list-like struct of promises that'll execute task callbacks in sequential order.
Usage
Let's study a concrete scenario. Imaging you need to handle websocket messages in sequential order.
Let's say you need to do some heavy computation upon receiving messages, so you decide to send it off to a worker thread pool. Then you write the processed result to another message queue (MQ).
But here's the requirement, that MQ is expecting the writing order of messages to match with the order they come in from the websocket stream. What do you do?
Suppose you cannot pause the websocket stream, you can only handle them locally ASAP.
Take One:
websocket.on('message', (msg) => {
sendToWorkerThreadPool(msg).then(result => {
writeToMessageQueue(result)
})
})
This may violate the requirement, cus sendToWorkerThreadPool may not return the result in the original order since it's a pool, some threads may return faster if the workload is light.
Take Two:
websocket.on('message', (msg) => {
const task = () => sendToWorkerThreadPool(msg).then(result => {
writeToMessageQueue(result)
})
enqueue(task)
})
This time we enqueue (defer) the whole process, thus we can ensure the task execution order stays sequential. But there's a drawback, we lost the benefit of using a thread pool, cus each sendToWorkerThreadPool will only fire after last one complete. This model is equivalent to using a single worker thread.
Take Three:
websocket.on('message', (msg) => {
const promise = sendToWorkerThreadPool(msg)
const task = () => promise.then(result => {
writeToMessageQueue(result)
})
enqueue(task)
})
Improvement over take two is, we call sendToWorkerThreadPool ASAP, without deferring, but we still enqueue/defer the writeToMessageQueue part. This way we can make full use of thread pool for computation, but still ensure the sequential writing order to MQ.
I rest my case.

Promise callback queue with priority for some callbacks

My code needs to perform a multitude of async actions simultaneously and handle their promises in a specific sequential way.
What I mean by "specific sequential way" ==> Assume you are starting promises promise1, promise2, promise3 in that order, and that promise3 actually resolves first, and promise2 second, then what I want is to process promise3, promise2 and promise1 sequentially in that order
Let's consider a timeout async function that times out after X seconds, and fetchMyItem1, fetchMyItem2, that return promises which, when fulfilled, should execute a different code depending whether timeout has resolved or not.
For a concrete scenario, imagine there is a customer waiting at the counter for an item to be delivered, and either the customer stays and we can serve him directly at the counter by bringing one item at a time, or the customer goes away (timeout) and we have to ask a waiter to come so that when the items arrive, he/she can bring the items to him. Note here that even when one item is being delivered, the other items should still be undergoing delivery (the promises are pending), and might even arrive (be fulfilled) while the customer is being served one item or the server arrives.
Here is some code to start with
const allItemsToBeDeliveredPromises = [fetchMyItem1(), fetchMyItem2(), ...]
const customerLeavesCounterPromise = timeout()
const waiter = undefined
const allPromisesToBeFulfilled = [...allItemsToBeDeliveredPromises, customerLeavesCounterPromise]
// LOOP
const itemDeliveredOrVisitorLeft = await Promise.all(allPromisesToBeFulfilled)
if hasCustomerLeft(itemDeliveredOrCustomerLeft) {
// hasCustomerLeft allows us to detect if the promise that resolved first is `customerLeavesCounterPromise` or not
waiter = await callWaiter()
} else {
// An item has arrived
if (waiter) {
deliverItemViaWaiter(itemDeliveredOrVisitorLeft)
} else {
deliverItemAtCounter(itemDeliveredOrVisitorLeft)
}
}
// remove itemDeliveredOrCustomerLeft from allPromisesToBeFulfilled
// END loop
I am not sure how to implement a loop for this scenario. Promises must be accumulated into a queue as they resolve, but there is a priority for a specific promise in the queue (the timeout promise should be executed asap when it arrives, but after finishing the processing of the current promise if a promise fulfilment is already being processed)
My code needs to perform a multitude of async actions simultaneously and handle their promises in a specific sequential way.
You could use streams to consume the promises, as streams are essentially queues that process one message at a time.
The idea (i.e. not tested):
import { Readable, Writable } from 'stream';
let customerHasLeft = false;
/*const customerLeavesCounterPromise = */timeout() // your code...
.then(() => { customerHasLeft = true; }); // ... made boolean
// let's push all our promises in a readable stream
// (they are supposedly sorted in the array)
const input = new Readable({
objectMode: true,
read: function () { // don't use arrow function: we need `this`
const allItemsToBeDeliveredPromises = [fetchMyItem1(), fetchMyItem2(), ...]; // your code
// put everything, in the same order, in the output queue
allItemsToBeDeliveredPromises.forEach(p => this.push(p));
this.push(null); // terminate the stream after all these
}
});
// let's declare the logic to process each promise
// (depending on `timeout()` being done)
const consumer = new Writable({
write: async function (promise, uselessEncoding, callback) {
try {
const order = await promise; // wait for the current promise to be completed
} catch (e) {
/* delivery error, do something cool like a $5 coupon */
return callback(e); // or return callback() without e if you don't want to crash the pipe
}
if (customerHasLeft) { /* find a waiter (you can still `await`) and deliver `order` */ }
else { /* deliver `order` at the counter */ }
callback(); // tell the underlying queue we can process the next promise now
}
});
// launch the whole pipe
input.pipe(consumer);
// you can add listeners on all events you'd like:
// 'error', 'close', 'data', whatever...
EDIT: actually we want to process promises as they resolve, but sequentially (i.e. a single post-process for all promises)
let customerHasLeft = false;
timeout() // your code...
.then(() => { customerHasLeft = true; }); // ... made boolean
const allItemsToBeDeliveredPromises = [fetchMyItem1(), fetchMyItem2(), ...];
const postProcessChain = Promise.resolve(); // start with a promise ready to be thened
// add a next step to each promise so that as soon as one resolves, it registers
// as a next step to the post-process chain
allItemsToBeDeliveredPromises.forEach(p => p.then(order => postProcessChain.then(async () => {
// do something potentially async with the resulting order, like this:
if (customerHasLeft) { /* find a waiter (you can still `await`) and deliver `order` */ }
else { /* deliver `order` at the counter */ }
})));
one of them must have the effect that, whenever it fulfils, it should alter the processing of other fulfilled promises that have not yet been processed or are not yet being processed.
Do you run them all together? Something like:
promise1 = asyncFunc1()
promise2 = asyncFunc2()
promise3 = asyncFunc3()
Promise.all([promise1, promise2, promise3]).then(//sometthing)
If yes, it can't be done, unless the promise2 function and promise3 function aren't waiting for an event, a lock, or something handled by the promise1 function.
If this is the case is better to organize the promises like:
asyncFunc1()
.then(() => asyncFunc2(paramIfOk))
.catch(() => asyncFunc2(paramIfFail))
In your example:
const allItemsToBeDelivered = [myPromise1(), myPromise2(), ...]
myPromise1() code should wait for the item to be delivered and check if someone is waiting for it. It's a model/code design problem, not a promise one.
Another way to do it is consider some event driven logic: an entity Customer has a listener for the delivered event that will be fired by the waitItemDelivered() promise just before resolve itself.
EDIT: as requested here a little more elaboration on the event-driven solution.
Short answer: it highly depend on your software design.
It's already something released and running in production? Be carefull with changes like this. If it's a service that you are developing you're still in time to take in consideration some logic changes. A solution that doesn't change radically how is working but use an events is not that great, mixing patterns never pay off on long term.
Example:
const events = require('events');
class Customer() {
constructor(shop) {
this.emitter = new events.EventEmitter()
this.shopEventListener = shop.emitter
this.setupEventLinstening() // for keeping things clean in the constructor
// other properties here
}
setupEventLinstening() {
this.shopEventListener.on('product-delivered', (eventData) => {
// some logic
})
}
buyProduct() {
// some logic
this.emitter.emit('waiting-delivery', eventData)
}
}
class Shop() {
constructor(shop) {
this.emitter = new events.EventEmitter()
// other properties here
}
addCustomer(customer) {
customer.emitter.on('waiting-delivery', (eventData) => {
// some logic
self.waitDelivery().then(() => self.emitter.emit('product-delivered'))
})
}
waitDelivery() {
return new Promise((resolve, reject) => {
// some logic
resolve()
})
}
}
// setup
const shop = new Shop()
const customer = new Customer(shop)
shop.addCustomer(customer)
This is a new way of seeing the logic, but a similar approach can be used inside a promise:
const waitDelivery = () => new Promise((resolve, reject) => {
logisticWaitDelivery().then(() => {
someEmitter.emit('item-delivered')
resolve()
})
}
const customerPromise = () => new Promise((resolve, reject) => {
someListener.on('item-delivered', () => {
resolve()
})
}
promiseAll([waitDelivery, customerPromise])
Make 2 queues, the second starting after the first ends:
var highPriority=[ph1,ph2,...]//promises that have to be executed first
var lowPriority=[pl1,pl2,...]//promises that have to wait the high priority promises
Promise.all(highPriority).then(()=>{
Promise.all(lowPriority)
})

Wait for an async operation in onNext of RxJS Observable

I have an RxJS sequence being consumed in the normal manner...
However, in the observable 'onNext' handler, some of the operations will complete synchronously, but others require async callbacks, that need to be waited on before processing the next item in the input sequence.
...little bit confused how to do this. Any ideas? thanks!
someObservable.subscribe(
function onNext(item)
{
if (item == 'do-something-async-and-wait-for-completion')
{
setTimeout(
function()
{
console.log('okay, we can continue');
}
, 5000
);
}
else
{
// do something synchronously and keep on going immediately
console.log('ready to go!!!');
}
},
function onError(error)
{
console.log('error');
},
function onComplete()
{
console.log('complete');
}
);
Each operation you want to perform can be modeled as an observable. Even the synchronous operation can be modeled this way. Then you can use map to convert your sequence into a sequence of sequences, then use concatAll to flatten the sequence.
someObservable
.map(function (item) {
if (item === "do-something-async") {
// create an Observable that will do the async action when it is subscribed
// return Rx.Observable.timer(5000);
// or maybe an ajax call? Use `defer` so that the call does not
// start until concatAll() actually subscribes.
return Rx.Observable.defer(function () { return Rx.Observable.ajaxAsObservable(...); });
}
else {
// do something synchronous but model it as an async operation (using Observable.return)
// Use defer so that the sync operation is not carried out until
// concatAll() reaches this item.
return Rx.Observable.defer(function () {
return Rx.Observable.return(someSyncAction(item));
});
}
})
.concatAll() // consume each inner observable in sequence
.subscribe(function (result) {
}, function (error) {
console.log("error", error);
}, function () {
console.log("complete");
});
To reply to some of your comments...at some point you need to force some expectations on the stream of functions. In most languages, when dealing with functions that are possibly async, the function signatures are async and the actual async vs sync nature of the function is hidden as an implementation detail of the function. This is true whether you are using javaScript promises, Rx observables, c# Tasks, c++ Futures, etc. The functions end up returning a promise/observable/task/future/etc and if the function is actually synchronous, then the object it returns is just already completed.
Having said that, since this is JavaScript, you can cheat:
var makeObservable = function (func) {
return Rx.Observable.defer(function () {
// execute the function and then examine the returned value.
// if the returned value is *not* an Rx.Observable, then
// wrap it using Observable.return
var result = func();
return result instanceof Rx.Observable ? result: Rx.Observable.return(result);
});
}
someObservable
.map(makeObservable)
.concatAll()
.subscribe(function (result) {
}, function (error) {
console.log("error", error);
}, function () {
console.log("complete");
});
First of all, move your async operations out of subscribe, it's not made for async operations.
What you can use is mergeMap (alias flatMap) or concatMap. (I am mentioning both of them, but concatMap is actually mergeMap with the concurrent parameter set to 1.) Settting a different concurrent parameter is useful, as sometimes you would want to limit the number of concurrent queries, but still run a couple concurrent.
source.concatMap(item => {
if (item == 'do-something-async-and-wait-for-completion') {
return Rx.Observable.timer(5000)
.mapTo(item)
.do(e => console.log('okay, we can continue'));
} else {
// do something synchronously and keep on going immediately
return Rx.Observable.of(item)
.do(e => console.log('ready to go!!!'));
}
}).subscribe();
I will also show how you can rate limit your calls. Word of advice: Only rate limit at the point where you actually need it, like when calling an external API that allows only a certain number of requests per second or minutes. Otherwise it is better to just limit the number of concurrent operations and let the system move at maximal velocity.
We start with the following snippet:
const concurrent;
const delay;
source.mergeMap(item =>
selector(item, delay)
, concurrent)
Next, we need to pick values for concurrent, delay and implement selector. concurrent and delay are closely related. For example, if we want to run 10 items per second, we can use concurrent = 10 and delay = 1000 (millisecond), but also concurrent = 5 and delay = 500 or concurrent = 4 and delay = 400. The number of items per second will always be concurrent / (delay / 1000).
Now lets implement selector. We have a couple of options. We can set an minimal execution time for selector, we can add a constant delay to it, we can emit the results as soon as they are available, we can can emit the result only after the minimal delay has passed etc. It is even possible to add an timeout by using the timeout operators. Convenience.
Set minimal time, send result early:
function selector(item, delay) {
return Rx.Observable.of(item)
.delay(1000) // replace this with your actual call.
.merge(Rx.Observable.timer(delay).ignoreElements())
}
Set minimal time, send result late:
function selector(item, delay) {
return Rx.Observable.of(item)
.delay(1000) // replace this with your actual call.
.zip(Rx.Observable.timer(delay), (item, _))
}
Add time, send result early:
function selector(item, delay) {
return Rx.Observable.of(item)
.delay(1000) // replace this with your actual call.
.concat(Rx.Observable.timer(delay).ignoreElements())
}
Add time, send result late:
function selector(item, delay) {
return Rx.Observable.of(item)
.delay(1000) // replace this with your actual call.
.delay(delay)
}
Another simple example to do manual async operations.
Be aware that it is not a good reactive practice ! If you only want to wait 1000ms, use Rx.Observable.timer or delay operator.
someObservable.flatMap(response => {
return Rx.Observable.create(observer => {
setTimeout(() => {
observer.next('the returned value')
observer.complete()
}, 1000)
})
}).subscribe()
Now, replace setTimeout by your async function, like Image.onload or fileReader.onload ...

Categories