Dataloader is able to batch and cache requests, but it can only be used by either calling load(key) or loadMany(keys).
The problem I am having is that sometimes I do not know they keys of the items I want to load in advance.
I am using an sql database and this works fine when the current object has a foreign key from a belongsTo relation with another model.
For example a user that belongs to a group and so has a groupId. To resolve the group you would just call groupLoader.load(groupId).
On the other hand, if I wanted to resolve the users within a group, of which there could be many I would want a query such as
SELECT * from users where user.groupId = theParticularGroupId
but a query such as this doesn't use the keys of the users and so I am not sure how make use of dataloader.
I could do another request to get the keys like
SELECT id from users where user.groupId = theParticularGroupId
and then call loadMany with those keys... But I could have just requested the data directly instead.
I noticed that dataloader has a prime(key, value) function which can be used to prime the cache, however that can only be done once the data is already fetched. At which point many queries would already have been sent, and duplicate data could have been fetched.
Another example would be the following query
query {
groups(limit: 10) {
id
...
users {
id
name
...
}
}
}
I cannot know the keys if I am searching for say the first or last 10 groups. Then once I have these 10 groups. I cannot know the keys of their users, and if each resolver would resolve the users using a query such as
SELECT * from users where user.groupId = theParticularGroupId
that query will be executed 10 times. Once the data is loaded I could now prime the cache, but the 10 requests have already been made.
Is there any way around this issue? Perhaps a different pattern or database structure or maybe dataloader isn't even the right solution.
You'll want a dataloader instance for the lookup you can do, in this case you have a group ID and you want the users:
import DataLoader from 'dataloader';
const userIdsForGroupLoader = new DataLoader(groupIds => batchGetUsersIdsForGroups(groupIds));
Now your batchGetUsersForGroups function is essentially has to convert an array of group IDs to an array of arrays of users (one array of user IDs for each group).
You'd start off with an IN query:
SELECT id from users where user.groupId in (...groupIds)
This will give you a single result set of users, which you'll have to manipulate, by grouping them by their groupId, the array should be ordered according to the original array of groupIds. Make sure you return an empty array for groupIds that don't have any users.
Note that in this we're only returning the user ids, but you can batch fetch the users in one go once you have them. You could tweak it slightly to return the users themselves, you'll have to decide for yourself if that's the right approach.
Everything I mention in this article can be achieved using clever use of Dataloader. But the key takeaway is that the values you pass to the load/loadMany functions don't have to correspond to the IDs of the objects you're trying to return.
Related
Trying to create an activation code which should be unique, but it only consists of specific characters.
So, this is solution which i build
function findByActivationId() {
return Activation
.findOne({activationId})
.lean()
.exec();
}
let activationId = buildActivationId();
while (await findByActivationId(activationId)) {
activationId = buildActivationId();
}
This makes too many db calls, is there any better way to make query to mongodb?
Well, the major problem of checking if key is unique is based on how you are creating those.
Choose the best way for you to avoid bunch of problems later.
Your own generated string as a key
Well, you can do this but it's important to understand few disclaimers
If you want to generate your own key by the code and then compare if it is unique
in the database with all other currently created it can be done. Just create key by your
algorithm then select all keys from db and check if array of selected rows contains this freshly created string
Problems of this solution
As we can see we need to select all keys from DB and then compare each one to freshly created one. Problem can appear when your database is storing big amount of data. Every time application have to "download" big amount of data and then compare it to new one so in addition this might produce some freezes.
But if you are sure that your database will store not that much amount of unique rows, it is cool to work with.
Then it is important to create those keys properly. Now we talking about complexity, more symbols key is created from, harder to get same ones.
Shall we take a look at this example?
If you are creating keys based on letters a-z and numbers 1-9
and the length of key is for example 5, the complexity of this key is 35^5
which generates more than 52 milions possibilities.
Same keys can be generated but it is like a win on a lottery, almost impossible
And then you can just check if generated key is really unique, if not. (oh cmon) Repeat.
Other ways
Use mongodb _id which is always unique
Use UNIX timestamp to create unique key
I am struggling to find good material on best practices for filtering data using firebase firestore. I want to filter my data based on the categories selected by the user. I have a collection of documents stored on my firestore database and each document have an array which has all the appropriate categories for that single document. For the sake of filtering, I'm keeping a local array with a user's preferred categories as well. All I want to do is to filter the data based on the user's preferred categories.
firestore categories field
consider I have the user's preferred categories stored as an array of strings ( ["Film", "Music"] ) .I was planning on using firestore's 'array-contains' method like
db.collection(collectioname)
.where('categoriesArray', 'array-contains', ["Film", "Music"])
Later I found out that I can't use 'array-contains' against an array itself and after investigating on this issue, I decided to change my data structure as mentioned here.
categories changed to Map
Once I changed the categories from an array to map, I thought I could use multiple where conditions to filter the documents
let query = db.collection(collectionName)
.where(somefield, '==', true)
this.props.data.filterCategories.forEach((val) => {
query = query.where(`categories.${val}`, '==', true);
});
query = query
.orderBy(someOtherField, "desc")
.limit(itemsPerPage)
const snapshot = await query.get()
Now problem number 2, firebase requires to add indexes for compound queries. The categories I have saved within each document is dynamic and there's no way I can add these indexes in advance. What would be the ideal solution in such cases? Any help would be deeply appreciated.
This is a new feature of Firebase JavaScript SDK launched at November 7, 2019:
Version 7.3.0 - November 7, 2019
array-contains-any
"array-contains-any operator to combine up to 10 array-contains clauses on the same field with a logical OR. An array-contains-any query returns documents where the given field is an array that contains one or more of the comparison values"
citiesRef.where('regions', 'array-contains-any',
['west_coast', 'east_coast']);
Instead of iterating through each category that you wish to query and appending clauses to a single query object, each iteration should be its own independent query. And you can keep the categories in an array.
<document>
- itemId: abc123
- categories: [film, music, television]
If you wish to perform an OR query, you would make n-loops where each loop would query for documents where array-contains that category. Then on your end, you would dedup (remove duplicates) from the results based on the item's identifier. So if you wanted to query film or music, you would make 2 loops where the first iteration queried documents where array-contains film and the second loop queried documents where array-contains music. The results would be placed into the same collection and then you would simply remove all duplicates with the same itemId.
This also does not pose a problem with the composite-index limit because categories is a static field. The real problem comes with pagination because you would need to keep a record of all fetched itemId in case a future page of results returns an item that was already fetched and this would create an O(N^2) scenario (more on big-o notation: https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/). And because you're deduping locally, pagination blocks as the user sees them are not guaranteed to be even. If each pagination block is set to 25 documents, for example, some pages may end up displaying 24, some 21, others 14, depending on how many duplicates were removed from each block.
Are you planning on retrieving documents with the exact category array? Say, your user preference is listed as ["Film", "Music"]. Do you wish to retrieve only those documents with Film AND Music, or do you wish to retrieve documents having Film OR music?
If it's the latter, then maybe you can query for all documents with "Film" and then query for all documents with "Music", then merge it. However, the drawback here is some redundant document reads, when such document has both "Film" and "Music" in the categoryArray field.
You can also explore using Algolia to enable full-text search. In this case, you'd probably store the category list as a string maybe separated by commas, then update the whole string when the user changes their preferences.
For the former case, I have not come across sa workable solution other than maybe storing it as a concatenated string in alphabetical order? Others might have a more solid solution than mine.
Hope this helps!
Your query includes an orderBy clause. This, in combination with any equality filter, requires that you create an index to support that query. There is no way to avoid this.
If you remove the orderBy, you will be able to have flexible, dynamic filters for equality using the map properties in the document. This is the only way you will be able to have a dynamic filter without creating an index. This of course means that you will have to order and page the query results on the client.
In my site i am conducting a survey like tests, each test has attendies sub collection look like this
When someone finishes a test i also add their uid to completed field like i drawn in the box. Now i want to query tests based on status == completed.
Here is what i tried
this.completedModulesRef$ = this.afs.collection('tests', ref =>
ref.orderBy('moduleNum', 'desc')
.where('completed.'+auth.uid+'.status','==','completed'));
this.completedModules$ = this.completedModulesRef$.valueChanges();
Then firestore asked me to add indexes, when i follow the generated link to add indexes i got this
which is pointing to completed.CurrentUserId.status. I believe this only work for current user.
I have few question
1) .where('completed.'+auth.uid+'.status','==','completed') Is this a valid query?
2) If yes how can i index it?
3) Any way to query the top collections based on sub collection value?. (this is what i really want)
Any help appreciated.
You could just instead use an array of objects (each object represents one attendie) to track all the completed attendies. Arrays are indexed in firestore.
The data structure of the array would be:
compeleted: [{}] = [
{id: string, points: number, status: string}
]
This might not be the most optimal database model depending on how and where you want to fetch the data but it will be indexed and you will be able to query it. I would consider storing the points and status in the subcollection of attendies. Have a look at grouped collection queries in firestore - new feature where you can query across all attendies subcollections at once - for any attendie with a certain id and a status of complete if you want to fetch all the tests that one attendie completed.
my receipe data in firebase
![my receipe data in firebase][1]
My receipe data looks like the following
receipe/1/12: "{"itemno":"1","receipeno":"12","receipedescript..."
How can I filter all receipes where receipeno = 12 or starting with receipeno = 12, irrespective of itemno?
I have tried the following, but got no results or errors?
Also tried startAt
this.query = this.db.database.ref("receipe").orderByChild("receipeno").equalTo("12").limitToFirst(100);
BTW: this.query = this.db.database.ref("receipe") this returns all data
and this.db.database.ref("receipe/1") returns all receipe for itemno == 1.
I have updated the data to not use opaque strings.
[Updated db so as not to use opaque strings][2]
and have tried the following.
this.query = this.db.database.ref("receipe").orderByChild("itemno").equalTo("1").limitToFirst(100);
And
this.query = this.db.database.ref("receipe").orderByChild("receipeno").equalTo("r1").limitToFirst(100);
Firebase is not going to help you with deep queries. It filters and sorts at one level. Therefore, there is no way to say, "find me all the recipes with an id (or recipeno) of 12, under any node under recipes.
If you had a single level, then you would do orderByKey().equalTo("12"). Of course this will limit you to one recipe per ID.
If you are going to have multiple recipes with the number 12 and want to do this kind of querying, you need to change your database structure--essentially getting rid of the intermediate level where you currently have keys of 1, 2 etc. Most likely in this case you would use automatically generated keys of the kind that result from push, such as:
+ recipes
+ autokey1: {itemno, recipeno, recipedescription}
+ autokey2: {itemno, recipeno, recipedescription}
Then you could say
orderByChild('recipeno').equalTo('12')
This assumes, of course, as pointed out in a comment, that you are saving the data as objects, not as stringified JSON, which Firebase will never be able to query against the insides of.
This is a good case study of the notion that you should design your Firebase database structure carefully in advance to allow you to do the kinds of querying you will need to do.
By the way, this question has nothing to do whatsoever with Angular.
Ok, so I've been reading and reading and searching and searching and strangely it doesn't seem like my scenario has been really covered anywhere.
I have an app that creates a list of products. I want a simple view that can sort the products and page through them.
Fore reference here is a simple representation of the data in Firebase.
app
stock
unique_id
name
url
imageUrl
price
When creating the list I have multiple threads using the push method on my firebase references:
new Firebase(firebaseUrl).child('stock').push({
name: "name",
price: 123
});
This gives me a lovely "hash" collection on the stock property of the app.
So what I'd now like to do is have a table to sort and page through the records that were placed in the stock hash.
I make a GET request to my server to a url like /stock?limit=10&skip=10&sort=name%20asc. This particular url would be the second page where the table contained 10 records per page and was sorted by the name property in ascending order.
Currently in my query handler I have this:
var firebaseRef = new Firebase(firebaseUrl).child('stock');
if (this.sortDesc) {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.endAt()
.limitToFirst(this.limitAmount);
} else {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.limitToFirst(this.limitAmount);
if (this.skipAmount > 0) {
firebaseRef = firebaseRef.startAt(this.skipAmount);
}
}
firebaseRef.once('value', function (snapshot) {
var results = [];
snapshot.forEach(function (childSnapshot) {
results.push(childSnapshot.val());
});
callback(null, results);
});
I'm running into a couple of problems. I'm going to split this into two cases, ascending and descending queries.
Ascending query
The orderByChild and limitToFirst seems to work correctly in the sorting ascending case. This means I can change which property has an ascending sort and how many results to return. What I am not able to get to work is skipping n records for paging to work. In the example query above I'm going to the second page. I do not get results 11-20, but I instead get the same 10 records as the first page.
Descending query
In this case I cannot begin to figure out how to tell Firebase to order by a property of the object identified by the unique key in a descending fashion. The closest I've read is to use endAt() and then limit. Docs say the limit is deprecated plus this still doesn't help me with any paging.
I tired to do doodles picturing how this would work. I came up with: order by the property, start at the 'end' of the collection, and then limit back to the page size. While this still wouldn't solve paging I would expect it to give me the last n records where n was the size of the page. I get no results.
I suppose I could say use firebaseRef = firebaseRef .orderByChild(this.sortProperty).limitToLast(this.limitAmount + this.skipAmount); and in the result callback use the forEach loop to take the first (or would it be the last; I'm not sure how that iteration would work) n records where n=this.limitAmount. This just seems inefficient. Wouldn't it be better to limit the query instead of using CPU cycles to limit data that had come over the wire or is this the relational DB query thought pattern overriding the correct thought process for NoSQL?
Further Confusion
After posting this I've still been working on a solution. I've had some things get close, but I'm also running into this filtering issue. How could I filter a set of items to one property by still sorting on another? Jeez! I want to have the ability for a user to get all the stock that isn't sold out and order it by price.
Finally
Why hasn't this basic example been fleshed out on any of the Firebase "Getting Started" pages? Being able to show tabular data, page through it, sort, and filter seem like something that EVERY web developer would come across. I'm using ng-table in an Angular app to drive the view, but it still seems that regardless of platform that the queries that I'm trying to generate would be practical on any platform that Firebase supports. Perhaps I'm missing something! Please educate me!
Firebase and NoSQL
I've come up with this simple scenario that I often run into with web applications. I want to show tabular data, filter, page, and sort it. Very simple. Very common. Writing a SQL statement for this would be dead easy. Why is the query so complicated for something like Firebase. Is this common with all NoSQL solutions? There is no relational data being stored thus the need for a relational database seems unnecessary. Yet, it seems like I could hack together a little flat file to do this storage since the ability to make Firebase do these simple tasks is not made clear in its API or Docs. FRUSTRATED!!!