Observable.forkJoin with a for loop - javascript

I am trying to populate an array in my component called processes which is an array of process. Each process also has a list of tasks.
So currently, I am working with two api calls which are:
/processes
and
/process/{processId}/tasks
I use /processes to get all the processes and initially populate the processes array. Then I use the process id of each process to call the second API to get the tasks of that process.
Currently, my code looks something like this:
this.processes.forEach((process, index) => {
myService.getTasks().subscribe((tasks) => {
process.tasks = tasks;
})
})
I understand that I can create an array of observables, and use Observable.forkJoin() to wait for all these async calls to finish but I want to be able to define the subscribe callback function for each of the calls since I need a reference to the process. Any ideas on how I can go about approaching this issue?

Using the for loop to make multiple HTTP requests, and then subscribe to all of them separately should be avoided in order not to have many Observable connections opened.
As #Juan Mendes mentioned, Observable.forkJoin will return an array of tasks that match the index of each process in your processes array. You can also assign tasks to each process as they arrive as follows:
getTasksForEachProcess(): Observable<any> {
let tasksObservables = this.processes.map(process, processIdx) => {
return myService.getTasks(process)
.map(tasks => {
this.processes[processIdx].tasks = tasks; // assign tasks to each process as they arrive
return tasks;
})
.catch((error: any) => {
console.error('Error loading tasks for process: ' + process, 'Error: ', error);
return Observable.of(null); // In case error occurs, we need to return Observable, so the stream can continue
});
});
return Observable.forkJoin(tasksObservables);
};
this.getTasksForEachProcess().subscribe(
tasksArray => {
console.log(tasksArray); // [[Task], [Task], [Task]];
// In case error occurred e.g. for the process at position 1,
// Output will be: [[Task], null, [Task]];
// If you want to assign tasks to each process after all calls are finished:
tasksArray.forEach((tasks, i) => this.processes[i].tasks = tasksArray[i]);
}
);
Please also take a look at this post: Send multiple asynchronous HTTP GET requests

Thanks to Seid Mehmedovic for great explanation but it looks like code missing one round bracket near map. For me it worked as follows:
getTasksForEachProcess(): Observable<any> {
let tasksObservables = this.processes.map((process, processIdx) => {
return myService.getTasks(process)
.map(tasks => {
this.processes[processIdx].tasks = tasks; // assign tasks to each process as they arrive
return tasks;
})
.catch((error: any) => {
console.error('Error loading tasks for process: ' + process, 'Error: ', error);
return Observable.of(null); // In case error occurs, we need to return Observable, so the stream can continue
});
});
return Observable.forkJoin(tasksObservables);
};
this.getTasksForEachProcess().subscribe(
tasksArray => {
console.log(tasksArray); // [[Task], [Task], [Task]];
// In case error occurred e.g. for the process at position 1,
// Output will be: [[Task], null, [Task]];
// If you want to assign tasks to each process after all calls are finished:
tasksArray.forEach((tasks, i) => this.processes[i].tasks = tasksArray[i]);
}
);

Related

Resolving multiple promises inside an observable not working

I'm using Firebase Storage and I'm trying to load all assets via a function call. The only way to get an assets url is to call getDownloadURL which returns a promise. I need to call this for every asset but I can't make it wait for all promises to be done before continuing for some reason.
I thought returning a promise from mergeMap would make it wait for all of them but that doesn't seem to be the case.
I've look at a number of questions regarding promises and RXJS but I can't seem to figure out what's wrong with the code.
getAssets() {
return this.authService.user$.pipe(
first(),
switchMap(user => defer(() => from(this.afs.storage.ref(`${user.uid}/assets`).listAll()))),
switchMap(assets => from(assets.items).pipe(
mergeMap(async (asset) => {
return new Promise((res, rej) => {
asset.getDownloadURL().then(url => {
const _asset = {
name: asset.name,
url,
};
this.assets.push(_asset);
res(_asset);
})
.catch((e) => rej(e));
});
}),
)),
map(() => this.assets),
);
}
...
this.getAssets().subscribe(assets => console.log(assets)); // this runs before all asset's url has been resolved
Overview
mergeMap doesn't wait for all internal observables. It spins up n internal observable pipes that run in parallel, and spits all the values out the same coupling at the bottom of the pipe (your subscribe statement in this case) as individual emissions. Hence why this.getAssets().subscribe(assets => console.log(assets)) runs before all your parallel internal mergeMap pipes complete their individual computations, because mergeMap doesn't wait for all of them before emitting (it will emit one by one as they finish). If you want to wait for n observable pipes to finish, then you need to use forkJoin.
Fork Join
forkJoin is best used when you have a group of observables and only care about the final emitted value of each. One common use case for this is if you wish to issue multiple requests on page load (or some other event) and only want to take action when a response has been received for all. In this way it is similar to how you might use Promise.all.
Solution
getAssets(): Observable<Asset[]> {
return this.authService.user$.pipe(
// first() will deliver an EmptyError to the observer's error callback if the
// observable completes before any next notification was sent. If you don't
// want this behavior, use take(1) instead.
first(),
// Switch to users firebase asset stream.
switchMap(user => {
// You might have to tweak this part. I'm not exactly sure what
// listAll() returns. I guessed that it returns a promise with
// firebase asset metadata.
return from(this.afs.storage.ref(`${user.uid}/assets`).listAll());
}),
// Map to objects that contain method to get image url.
map(firebaseAssetMetadata => firebaseAssetMetadata?.items ?? []),
// Switch to parallel getDownloadUrl streams.
switchMap(assets => {
// Not an rxjs map, a regular list map. Returns a list of getAssetUrlPipes.
const parallelGetAssetUrlPipes = assets.map(asset => {
return from(asset.getDownloadUrl()).pipe(
map(url => { name: asset.name, url })
);
});
// 1) Listen to all parallel pipes.
// 2) Wait until they've all completed.
// 3) Merge all parallel data into a list.
// 4) Then move list down the pipe.
return forkJoin(parallelGetAssetUrlPipes);
}),
// Outputs all parallel pipe data as a single emission in list form.
// Set local variable to users asset data.
tap(assetObjects => this.assets = assetObjects)
);
}
// Outputs the list of user asset data.
this.getAssets().subscribe(console.log);
Good luck out there, and enjoy your Swedish meatballs!
const { from } = rxjs
const { mergeMap } = rxjs.operators
const assets = [1,2,3,4,5]
function getUrl (index) {
return new Promise((res) => {
setTimeout(() => res(`http://example.com/${index}`), Math.random() * 3 + 1000)
})
}
// add param2 1 for mergeMap === concatMap
from(assets).pipe(
mergeMap(asset => {
return getUrl(asset)
}, 1)
).subscribe(console.log)
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
use concatMap to run one by one.

How to call multiple dependent api calls with time intervals using RxJS

I am trying to write a code in angular 11 for a scenario like this -
I have list of files, and for every file I hit an api (say api1), i take an fileId from response and i pass it to another api (say api2),i want to keep on hitting the api2 every 3 seconds,unless i dont get the status="available" in the response. Once i get the available status, i no more need to hit the api2 for that fileId and we can start processing for the next file in loop.
This whole process for every file that I have.
I understand we can achieve this using rxjs operators like mergeMap or switchMap (as the sequence do not matter to me right now) . But i am very new to rxjs and not sure how to put it together.
This is what i am doing right now -
this.filesToUpload.forEach((fileItem) => {
if (!fileItem.uploaded) {
if (fileItem.file.size < this.maxSize) {
self.fileService.translateFile(fileItem.file).then( //hit api1
(response) => {
if (response && get(response, 'status') == 'processing') {
//do some processing here
this.getDocumentStatus(response.fileId);
}
},
(error) => {
//show error
}
);
}
}
});
getDocumentStatus(fileId:string){
this.docStatusSubscription = interval(3000) //hitting api2 for every 3 seconds
.pipe(takeWhile(() => !this.statusProcessing))
.subscribe(() => {
this.statusProcessing = false;
this.fileService.getDocumentStatus(fileId).then((response)=>{
if(response.results.status=="available"){
this.statusProcessing = true;
//action complete for this fileId
}
},(error)=>{
});
})
}
Here's how I might do this given the description of what you're after.
Create a list of observables of all the calls you want to make.
Concatenate the list together
Subscribe
The thing that makes this work is that we only subscribe once (not once per file), and we let the operators handle subscribing and unsubscribing for everything else.
Then nothing happens until we subscribe. That way concat can do the heavy lifting for us. There's no need for tracking anything ourselves with variables like this.statusProessing or anything like that. That's all handled for us! It's less error prone that way.
// Create callList. This is an array of observables that each hit the APIs and only
// complete when status == "available".
const callList = this.filesToUpload
.filter(fileItem => !fileItem.uploaded && fileItem.file.size < this.maxSize)
.map(fileItem => this.createCall(fileItem));
// concatenate the array of observables by running each one after the previous one
// completes.
concat(...callList).subscribe({
complete: () => console.log("All files have completed"),
error: err => console.log("Aborted call list due to error,", err)
});
createCall(fileItem: FileItemType): Observable<never>{
// Use defer to turn a promise into an observable
return defer(() => this.fileService.translateFile(fileItem.file)).pipe(
// If processing, then wait untill available, otherwise just complete
switchMap(translateFileResponse => {
if (translateFileResponse && get(translateFileResponse, 'status') == 'processing') {
//do some processing here
return this.delayByDocumentStatus(translateFileResponse.fileId);
} else {
return EMPTY;
}
}),
// Catch and then rethrow error. Right now this doesn't do anything, but If
// you handle this error here, you won't abort the entire call list below on
// an error. Depends on the behaviour you're after.
catchError(error => {
// show error
return throwError(() => error);
})
);
}
delayByDocumentStatus(fileId:string): Observable<never>{
// Hit getDocumentStatus every 3 seconds, unless it takes more
// than 3 seconds for api to return response, then wait 6 or 9 (etc)
// seconds.
return interval(3000).pipe(
exhaustMap(_ => this.fileService.getDocumentStatus(fileId)),
takeWhile(res => res.results.status != "available"),
ignoreElements(),
tap({
complete: () => console.log("action complete for this fileId: ", fileId)
})
);
}

Data operations are not rolled back in promisified IndexedDB transaction

When I call function saveDamage() and an exception occurs in the Promise.all() at the end of the function. I expected all preceding operations be rolled back but they're not. What am I doing wrong please? Thank you!
saveDamage(damage) {
return this.transaction('rw', [
this.DAMAGE_STORE_NAME,
this.DAMAGE_IMAGE_DATA_STORE_NAME,
this.DAMAGE_IMAGE_DATA_TEMP_STORE_NAME
], (tx) => {
let deletePromise = (damage.id)
? this._deleteDamage(tx, damage.id)
: Promise.resolve();
return deletePromise.then(() => {
// Copy temporary image data of damage into permanent location and update their IDs
let copyPromises = this._getDamageImageFields(damage).map(field => {
return this._copyObject(tx, this.DAMAGE_IMAGE_DATA_TEMP_STORE_NAME, this.DAMAGE_IMAGE_DATA_STORE_NAME, field.value.imageDataId)
.then(id => field.value.imageDataId = id)
});
return Promise.all(copyPromises)
// Save damage
/* The exception occurs inside _saveObject(). Previous DB operations are not rolled back. */
.then(() => this._saveObject(tx, this.DAMAGE_STORE_NAME, damage, damage.id))
.then(id => damage.id = id)
// Delete temporary image data
.then(() => this._clearStore(tx, this.DAMAGE_IMAGE_DATA_TEMP_STORE_NAME));
});
});
}
transaction(mode, storeNames, executorFnc) {
let tx = this._db.transaction(this._transactionStoreNames(storeNames), this._transactionMode(mode));
tx.onabort = (event) => reject("Transaction failed: " + event.target.errorCode);
return executorFnc(tx);
}
_deleteDamage(tx, id) {
// Fetch damage from DB
return this._getObject(tx, this.DAMAGE_STORE_NAME, id).then(damage => {
// Delete all big image data
let promises = this._getIdsOfDamageImageData(damage)
.map(imageDataId => this._deleteObject(tx, this.DAMAGE_IMAGE_DATA_STORE_NAME, imageDataId));
// Delete damage itself
promises.push(this._deleteObject(tx, this.DAMAGE_STORE_NAME, id));
return Promise.all(promises);
});
}
_deleteObject(tx, storeName, id) {
return new Promise((resolve, reject) => {
let result = tx.objectStore(storeName).delete(id);
result.onsuccess = (event) => resolve();
result.onerror = (event) => reject(event);
});
}
_copyObject(tx, sourceStoreName, targetStoreName, id) {
return this._getObject(tx, sourceStoreName, id)
.then(obj => this._saveObject(tx, targetStoreName, obj));
}
_saveObject(tx, storeName, object, id) {
return new Promise((resolve, reject) => {
let store = tx.objectStore(storeName);
let result = store.put(object, id);
result.onsuccess = (event) => resolve(event.target.result /* object id */);
result.onerror = (event) => reject(event);
});
}
The exception is (I know how to fix it):
Uncaught (in promise) DOMException: Failed to execute 'put' on 'IDBObjectStore': The object store uses in-line keys and the key parameter was provided.
EDIT: After some experimenting I understood that as soon as I exit success handler and there are no more unfinished success handlers then the transaction commits. It looks like operations issued in 'then' function are executed in another transaction because 'then' is executed as a result of returning resolved promise when exitting a success handler. The only way I found to execute dependent operations in a single transaction is using plain success handlers. But using plain success handlers becomes difficult when next-step-operations depends on the result of the current operation. For example when I need to delete object 'A' stored in store 'a' and it's many sub-objects 'B' stored in store 'b'. In such case I need to fetch object 'A' first, then delete sub-objects 'B' one by one, which requires generating handling code dynamically, the delete 'A'. I can't believe it's so difficult to achieve such basic thing so I suspect I'm still missing something.
The error means id is undefined, I think in _saveObject
If your promises represent individual transactions, then all transactions in the Promise.all array that were committed prior to the error will have already committed regardless of the fact that the Promise.all promise rejected. Run all database operations a single transaction to achieve a complete roll back if any one operation fails.

Typescript Observables http.post and http.get Handing Question

I am working on a project where I am building a simple front end in Angular (typescript) / Node to make call to a back end server for executing different tasks. These tasks take time to execute and thus need to be queued on the back end server. I solved this issue by following the following tutorial: https://github.com/realpython/flask-by-example and everything seems to work just fine.
Now I am finishing things up on the front end, where most of the code has been already written in Typescript using Angular and Rxjs. I am trying to replicate the following code in Typescript:
https://github.com/dimoreira/word-frequency/blob/master/static/main.js
This code consists of two functions, where first function "getModelSummary"(getResults in the example) calls a post method via:
public getModelSummary(modelSummaryParameters: ModelSummaryParameters): Observable<ModelSummary> {
return this.http.post(`${SERVER_URL}start`, modelSummaryParameters)
.map(res => res.json())
;
}
to put the job in queue and assign a jobID to that function on the back end server. The second function "listenModelSummary", ideally should get executed right after the first function with the jobId as it's input and loops in a short interval checking if the job has been completed or not:
public listenModelSummary(jobID: string) {
return this.http.get(`${SERVER_URL}results/` + jobID).map(
(res) => res.json()
);
}
Once the job is done, it needs to return the results, which would update the front end.
I am new to Typescript, Observables and rxjs and wanted to ask for the right way of doing this. I do not want to use javascript, but want to stick to Typescript as much as possible in my front end code stack. How can I use the first function to call the second function with it's output "jobID" and have the second function run via interval until the output comes back?
Observables are great, and are the type of object returned by Angular's HttpClient class, but sometimes, in my opinion, dealing with them is a lot more complicated than using promises.
Yes, there is a slight performance hit for the extra operation to convert the Observable to a Promise, but you get a simpler programming model.
If you need to wait for the first function to complete, and then hand the returned value to another function, you can do:
async getModelSummary(modelSummaryParameters: ModelSummaryParameters): Promise<ModelSummary> {
return this.http.post(`${SERVER_URL}start`, modelSummaryParameters).toPromise();
}
async doStuff(): Promise<void> {
const modelSummary = await this.getModelSummary(params);
// not sure if you need to assign this to your viewmodel,
// what's returned, etc
this.listenModelSummary(modelSummary)
}
If you're dead-set on using Observables, I would suggest using the concatMap pattern, which would go something like this:
doStuff(modelSummaryParameters: ModelSummaryParameters): Observable<ModelSummary> {
return this.http
.post(`${SERVER_URL}start`, modelSummaryParameters)
.pipe(
concatMap(modelSummary => <Observable<ModelSummary>> this.listenModelSummary(modelSummary))
);
}
Here's an article on different mapping solutions for Observables: https://blog.angularindepth.com/practical-rxjs-in-the-wild-requests-with-concatmap-vs-mergemap-vs-forkjoin-11e5b2efe293 that might help you out.
You can try the following:
getModelSummary(modelSummaryParameters: ModelSummaryParameters): Promise<ModelSummary> {
return this.http.post(`${SERVER_URL}start`, modelSummaryParameters).toPromise();
}
async someMethodInYourComponent() {
const modelSummary = await this.get(modelSummary(params);
this.listenModelSummary(modelSummary)
}
// OR
someMethodInYourComponent() {
this.get(modelSummary(params).then(() => {
this.listenModelSummary(modelSummary);
});
}
After doing more reading/researching into rxjs, I was able to make my code work and I wanted to thank you guys for the feedback and to post my code below.
In my services I created two observables:
First one is to fetch a jobId returned by queue server:
// API: GET / FUNCTION /:jobID
public getModelSummaryQueueId(modelSummaryParameters: ModelSummaryParameters): Observable<JobId>{
return this.http.post(${SERVER_URL}start, modelSummaryParameters).map(
(jobId) => jobId.json()
)
}
Use the jobId from first segment to fetch data:
// API: GET / FUNCTION /:results
public listenModelSummary(jobId: JobId): Observable <ModelSummary>{
return this.http.get(${SERVER_URL}results/+ jobId).map(
(res) => res.json()
)
}
Below is the component that works with the 2 services above:
`
this.subscription = this.developmentService.getModelSummaryQueueId(this.modelSummaryParameters)
.subscribe((jobId) => {
return this.developmentService.listenModelSummary(jobId)
// use switchMap to pull value from observable and check if it completes
.switchMap((modelSummary) =>
// if value has not changed then invoke observable again else return
modelSummary.toString() === 'Nay!'
? Observable.throw(console.log('...Processing Request...'))
// ? Observable.throw(this.modelSummary = modelSummary)
: Observable.of(modelSummary)
)
.retryWhen((attempts) => {
return Observable
// specify number of attempts
.range(1,20)
.zip(attempts, function(i) {
return(i);
})
.flatMap((res:any) => {
// res is a counter of how many attempts
console.log("number of attempts: ", res);
res = 'heartbeat - ' + res
this.getProgressBar(res);
// this.res = res;
// delay request
return Observable.of(res).delay(100)
})
})
// .subscribe(this.displayData);
// .subscribe(modelSummary => console.log(modelSummary));
.subscribe((modelSummary) => {
console.log("FINAL RESULT: ", modelSummary)
this.modelSummary = modelSummary;
this.getProgressBar('Done');
});
});
`

Node.js for loop using previous values?

Currently, I am trying to get the md5 of every value in array. Essentially, I loop over every value and then hash it, as such.
var crypto = require('crypto');
function userHash(userIDstring) {
return crypto.createHash('md5').update(userIDstring).digest('hex');
}
for (var userID in watching) {
refPromises.push(admin.database().ref('notifications/'+ userID).once('value', (snapshot) => {
if (snapshot.exists()) {
const userHashString = userHash(userID)
console.log(userHashString.toUpperCase() + "this is the hashed string")
if (userHashString.toUpperCase() === poster){
return console.log("this is the poster")
}
else {
..
}
}
else {
return null
}
})
)}
However, this leads to two problems. The first is that I am receiving the error warning "Don't make functions within a loop". The second problem is that the hashes are all returning the same. Even though every userID is unique, the userHashString is printing out the same value for every user in the console log, as if it is just using the first userID, getting the hash for it, and then printing it out every time.
Update LATEST :
exports.sendNotificationForPost = functions.firestore
.document('posts/{posts}').onCreate((snap, context) => {
const value = snap.data()
const watching = value.watchedBy
const poster = value.poster
const postContentNotification = value.post
const refPromises = []
var crypto = require('crypto');
function userHash(userIDstring) {
return crypto.createHash('md5').update(userIDstring).digest('hex');
}
for (let userID in watching) {
refPromises.push(admin.database().ref('notifications/'+ userID).once('value', (snapshot) => {
if (snapshot.exists()) {
const userHashString = userHash(userID)
if (userHashString.toUpperCase() === poster){
return null
}
else {
const payload = {
notification: {
title: "Someone posted something!",
body: postContentNotification,
sound: 'default'
}
};
return admin.messaging().sendToDevice(snapshot.val(), payload)
}
}
else {
return null
}
})
)}
return Promise.all(refPromises);
});
You have a couple issues going on here. First, you have a non-blocking asynchronous operation inside a loop. You need to fully understand what that means. Your loop runs to completion starting a bunch of non-blocking, asynchronous operations. Then, when the loop finished, one by one your asynchronous operations finish. That is why your loop variable userID is sitting on the wrong value. It's on the terminal value when all your async callbacks get called.
You can see a discussion of the loop variable issue here with several options for addressing that:
Asynchronous Process inside a javascript for loop
Second, you also need a way to know when all your asynchronous operations are done. It's kind of like you sent off 20 carrier pigeons with no idea when they will all bring you back some message (in any random order), so you need a way to know when all of them have come back.
To know when all your async operations are done, there are a bunch of different approaches. The "modern design" and the future of the Javascript language would be to use promises to represent your asynchronous operations and to use Promise.all() to track them, keep the results in order, notify you when they are all done and propagate any error that might occur.
Here's a cleaned-up version of your code:
const crypto = require('crypto');
exports.sendNotificationForPost = functions.firestore.document('posts/{posts}').onCreate((snap, context) => {
const value = snap.data();
const watching = value.watchedBy;
const poster = value.poster;
const postContentNotification = value.post;
function userHash(userIDstring) {
return crypto.createHash('md5').update(userIDstring).digest('hex');
}
return Promise.all(Object.keys(watching).map(userID => {
return admin.database().ref('notifications/' + userID).once('value').then(snapshot => {
if (snapshot.exists()) {
const userHashString = userHash(userID);
if (userHashString.toUpperCase() === poster) {
// user is same as poster, don't send to them
return {response: null, user: userID, poster: true};
} else {
const payload = {
notification: {
title: "Someone posted something!",
body: postContentNotification,
sound: 'default'
}
};
return admin.messaging().sendToDevice(snapshot.val(), payload).then(response => {
return {response, user: userID};
}).catch(err => {
console.log("err in sendToDevice", err);
// if you want further processing to stop if there's a sendToDevice error, then
// uncomment the throw err line and remove the lines after it.
// Otherwise, the error is logged and returned, but then ignored
// so other processing continues
// throw err
// when return value is an object with err property, caller can see
// that that particular sendToDevice failed, can see the userID and the error
return {err, user: userID};
});
}
} else {
return {response: null, user: userID};
}
});
}));
});
Changes:
Move require() out of the loop. No reason to call it multiple times.
Use .map() to collect the array of promises for Promise.all().
Use Object.keys() to get an array of userIDs from the object keys so we can then use .map() on it.
Use .then() with .once().
Log sendToDevice() error.
Use Promise.all() to track when all the promises are done
Make sure all promise return paths return an object with some common properties so the caller can get a full look at what happened for each user
These are not two problems: the warning you get is trying to help you solve the second problem you noticed.
And the problem is: in Javascript, only functions create separate scopes - every function you define inside a loop - uses the same scope. And that means they don't get their own copies of the relevant loop variables, they share a single reference (which, by the time the first promise is resolved, will be equal to the last element of the array).
Just replace for with .forEach.

Categories