let's say that i wanna update a serie of documents, i'm doing it using forEach like this.
students.forEach(async (name) => {
const docRef = doc(db, "students", name);
await updateDoc(docRef, {
school: "Some School",
});
});
And it's working fine, but i was wondering if that's bad for sending too many requests or something. Is there another better/smarter way to do it?
There's nothing particularly "bad" about this given what you've shared, unless you are also observing some behavior that you don't like.
If your intent is to update all of the documents atomically (up to a limit of 500 in a batch), so that any failures or interruptions won't leave the set of documents in an inconsistent state, you are better off using a batch write instead. But that won't necessarily give you any better performance or other improved runtime behavior.
I usually recommend against using await in a scenario where you are using updateDoc on a sequence of documents, as it's actually faster to let the updates run in parallel. For more on this, see What is the fastest way to write a lot of documents to Firestore?
But the await here is harmless, since using await in a forEach has no impact on the other operations in that same forEach. For more on this, see: Using async/await with a forEach loop If you were to use a for of loop though, be sure to remove the await for improved throughput.
Related
Can someone please explain what’s the use of this pattern with example?
All I'm confused is that I can have database instance wherever I want and I have flexibility to do anything by it, am I wrong? specially is
The repository pattern is a strategy for abstracting data access.
it’s like putting a universal adapter in between your application and your data so it doesn’t matter what data storage technology you use. All your app wants is having defined operations on items, it shouldn’t have to care about how it’s stored or where it comes from.
Also, there's no need to mention that all impacts of changes will be handled from one place instead of cascading all through your code!
Personally, I love this design pattern because it allows me to only have concerns about my business logics at the first steps instead of dealing with variant databases, on top of that, it solves a huge amount of my headaches when it comes to writing tests! So instead of stubbing or spying databases, which can be a headache, I can simply enjoy a mock version of operations
Now let’s implement a sample in js, it can be just as simple as below code (it's a simplified sample of course)
// userRepository.js
const userDb = [];
module.exports = {
insert: async (user) => userDb.push(user),
findAll: async () => userDb,
};
here is how I use this pattern, first I write something like below code in a 5 minute
// userRepository.js
const userDb = new Map();
module.exports = Object.freeze({
findById: async (id) => userDb.get(id),
insert: async (user) => userDb.set(user.id, user),
findAll: async () => Array.from(userDb.values()),
removeById: async (id) => userDb.delete(id),
update: async (updatedUser) => {
if (!userDb.has(updatedUser.id)) throw new Error('user not found');
userDb.set(updatedUser.id, { ...userDb.get(updatedUser.id), ...updatedUser });
},
});
Then I start to write my unit tests for repository that I’ve just written and business use-cases and so on…
anytime I’m satisfied with everything I can simply use a real database, because it’s just an IO mechanism, isn’t it? :)
So in above code I’ll replace userDb with a real database and write real data access methods, and of course expect all my tests to be passed.
This question concerns the Firestore database, but, more generally, it concerns making async requests in parallel.
Simply put, I wish to update multiple Firestore documents as quickly and efficiently as possible by mapping over an array of their document IDs.
the .set() method is async (returning a promise) and so I understand that I can wrap the multiple requests - i.e. the map() - in a Promise.all() function, which will return a single promise once all of the requests have resolved.
However, it is not at all clear to me whether I should await the set() within the map().
Whichever way I write the code (i.e. with or without await), it does not appear to make a difference to speed, but does it, or should it?
What is the correct way to achieve the greatest speed and efficiency in this instance?
const update_promises = array_of_ids.map(async id => {
const new_values = {
new_values_1: "val1",
last_update: new Date()
}
return await db.collection("my_collection").doc(id).set(new_values, { merge: true });
// OR SHOULD IT BE:
// return db.collection("my_collection").doc(id).set(new_values, { merge: true });
})
return await Promise.all(update_promises)
When get call set(), the SDK is going to immediately pipeline the write request over the a single managed connection. If you want to write a bunch of documents as fast as possible, you should kick them all off, then await the results at the end. You probably don't want to await each one individually, since you are causing the code to stop for a moment while it waits for the result before the next one can get sent. That said, the performance impact is probably negligible overall, unless you have a lot of work to be done in between each write.
My general rule is to only await an individual promise if its result is needed right away before moving on. Otherwise, collect all the promises together into an array for a single await with Promise.all().
I am having a problem where I am making a bulk insert of multiple elements into a table, then I immediatly get the last X elements from that table that were recently inserted but when I do that it seems that the elements have not yet been inserted fully even thought I am using async await to wait for the async operations.
I am making a bulk insert like
const createElements = elementsArray => {
return knex
.insert(elementsArray)
.into('elements');
};
Then I have a method to immediately access those X elements that were inserted:
const getLastXInsertedElements = (userId, length, columns=['*']) => {
return knex.select(...columns)
.from('elements').where('userId', userId)
.orderBy('createdAt', 'desc')
.limit(length);
}
And finally after getting those elements I get their ids and save them into another table that makes use of element_id of those recently added elements.
so I have something like:
// A simple helper function that handles promises easily
const handleResponse = (promise, message) => {
return promise
.then(data => ([data, undefined]))
.catch(error => {
if (message) {
throw new Error(`${message}: ${error}`);
} else {
return Promise.resolve([undefined, `${message}: ${error}`])
}
}
);
};
async function service() {
await handleResponse(createElements(list), 'error text'); // insert x elements from the list
const [elements] = await handleResponse(getLastXInsertedElements(userId, list.length), 'error text') // get last x elements that were recently added
await handleResponse(useElementsIdAsForeignKey(listMakingUseOfElementsIds), 'error text'); // Here we use the ids of the elements we got from the last query, but we are not getting them properly for some reason
}
So the problem:
Some times when I execute getLastXInsertedElements it seems that the elements are not yet finished inserting, even thought I am waiting with async/await for it, any ideas why this is? maybe something related to bulk inserts that I don't know of? an important note, all the elements always properly inserted into the table at some point, it just seems like this point is not respected by the promise (async operation that returns success for the knex.insert).
Update 1:
I have tried putting the select after the insert inside a setTimeout of 5 seconds for testing purposes, but the problem seems to persist, that is really weird, seems one would think 5 seconds is enough between the insert and the select to get all the data.
I would like to have all X elements that were just inserted accessible in the select query from getLastXInsertedElements consistently.
Which DB are you using, how big list of data are you inserting? You could also test if you are inserting and getLastXInsertedElements in a transaction if that hides your problem.
Doing those operations in transaction also forces knex to use the same connection for both queries so it might lead to a tracks where is this coming from.
Another trick to force queries to use the same connection would be to set pool's min and max configuration to be 1 (just for testing is parallelism is indeed the problem here).
Also since you have not provided complete reproduction code for this, I'm suspecting there is something else here in the mix which causes this odd behavior. Usually (but not always) this kind of weird cases that shouldn't happen are caused by user error in elsewhere using the library.
I'll update the answer if there is more information provided. Complete reproduction code would be the most important piece of information.
I am not 100% sure but I guess the knex functions do not return promise by default (but a builder object for the query). That builder has a function called then that transforms the builder into a promise. So you may try to add a call to that:
...
limit(length)
.then(x => x); // required to transform to promise
Maybe try debugging the actual type of the returned value. It might happen that this still is not a promise. In this case you may not use async await but need to use the then Syntax because it might not be real js promises but their own implementation.
Also see this issue about standard js promise in knex https://github.com/knex/knex/issues/1588
In theory, it should work.
You say "it seems"... a more clear problem explanation could be helpful.
I can argue the problem is that you have elements.length = list.length - n where n > 0; in your code there are no details about userId property in your list; a possible source of the problem could be that some elements in your list has a no properly set userId property.
How does one decide between promises, async awaits, and mapping operators like concatMap?
Here's my specific case, but I'm also curious about how you decide in general:
I am making an http call to my backend, and then I make another http call afterwards. When processing the json data from the second call, I need to use values that are returned by the first call. In this situation, is it better to use async await, a promise or concatMap? Also in general, what are the guidelines for deciding which to use?
Here is what I current have, using concatMap. (I am dynamically generating child components from my getTask http call, and each child component needs to have access to annotationFormats).
this.dashboardService.getAnnotationFormats()
.pipe(
concatMap(annotationFormats=> this.dashboardService.getTasks())
)
.subscribe(
(tasks)=>{
for(let task of tasks){
const componentFactory=this.CFR.resolveComponentFactory(DashboardItemComponent);
const componentRef=this.vc.createComponent(componentFactory);
componentRef.instance.task=task;
componentRef.instance.annotationFormats=annotationFormats;
componentRef.instance.compInteraction=this;
this.taskRef.push(componentRef);
}
}
);
Async/await and promises are basically the same with different syntax. Asynchronous code that will run once after some job has been finished.
As a rule, I would never never use none of those while using Angular. Angular comes with RxJS out of the box, which is so much more than promises. You can use RxJS for running async code once when a job has completed, but it also gives you the possibility of creating streams of data and manipulating them in so many different ways.
It does take a bit to fully understand RxJS and reactive programming but once you do you realize how much you can do with it.
In your case, I like to use the operator forkJoin, since the two requests seem independent from each other. You can give it a list of resources you want to obtain and will execute the async code in subscribe once they have all completed, which makes it perfect for http requests:
forkJoin({
annotationFormats: this.dashboardService.getAnnotationFormats(),
tasks: this.dashboardService.getTasks(),
})
.subscribe(
({tasks, annotationFormats})=>{
for(let task of tasks){
const componentFactory=this.CFR.resolveComponentFactory(DashboardItemComponent);
const componentRef=this.vc.createComponent(componentFactory);
componentRef.instance.task=task;
componentRef.instance.annotationFormats=annotationFormats;
componentRef.instance.compInteraction=this;
this.taskRef.push(componentRef);
}
}
);
Take your time to learn RxJS, I guarantee it will pay off. Whenever you are using RxJS and it feels too complex or wrong, that is because it probably is. Head to the RxJS documentation and look for something that might useful, and if you don't find anything a quick google search will probably get you the solution anyways. Point is, don't just use it blindly, always try to understand how it works.
I hope this is useful. :)
Edit:
For RxJS < 6.5, the syntax is a bit different:
forkJoin(
this.dashboardService.getTasks(),
this.dashboardService.getAnnotationFormats()
)
.subscribe(
([tasks, annotationFormats])=>{
for(let task of tasks){
const componentFactory=this.CFR.resolveComponentFactory(DashboardItemComponent);
const componentRef=this.vc.createComponent(componentFactory);
componentRef.instance.task=task;
componentRef.instance.annotationFormats=annotationFormats;
componentRef.instance.compInteraction=this;
this.taskRef.push(componentRef);
}
}
);
Notice we pass the resources as arguments, not as an object, and the result in the subscribe will be in an array form instead of an object too.
They have different use. async/await is used when you want to hold on a place where you have written some asynchronous code. while primises are tool to spot a place where async code is executed and invokes callback.
Coming from C++ and Python, I still struggle with Javascript asynchronous ubiquity. Sometimes it's very useful, but other times I just don't see how to fit it in without writing terrible code.
I have a Node.js + Express CRUD setup, and I have to do some basic check-ups before continuing with the request. I want to check that http POST fields match with database fields before running the final query. I cannot declare it an async function and use await as it must match a given Interface.
showColumns(dbTable) returns a db premise with a SHOW COLUMNS query.
The only solution that I found is:
database.showColumns(dbTable).then((columns)=>{
//But do I really need to put all my logic inside a then???
let row = Object.keys(req.body).filter({}.hasOwnProperty.bind(columns));
//... all the rest of the logic goes here
});
In your opinion, what is the cleanest/most elegant way to solve that?
database.showColumns(dbTable)
.then(columns => this.handleColumns(columns))
.then(parsedData => this.doSthElse(parsedData);
You can extract your logic to a seperate method. However, it must be called inside then as it is the callback triggered once your async operation is finished.
Alternatively, you might consider using generators, async/await functions or promises.
You can use async/await for this.
(async function() {
try {
var columns = await database.showColumns(dbTable)
let row = Object.keys(req.body).filter({}.hasOwnProperty.bind(columns));
} catch (e) {
console.log("Promise error");
}
})