In my scenario a user can "like" the profile of another user. As a result I have a subcollection called "likedBy" for each user, where I create a document for a specific user, e.g.:
users(col) -> User A(doc) -> likedBy(col) -> User B (doc), User C (doc)
So in the rare scenario of a user being deleted, I want to delete all the likes issues by that user.
I am just not sure if this is even possible (a workaround could be to just save the userId again in said document and query for that).
What I am basically looking for is something like this:
db.collectionGroup('likedBy').where(firebase.firestore.FieldPath.documentId(), '==', "User A").get();
The problem is, that I can not create an index for the documentId in the Firestore console.
UPDATE 2020-01-23:
For updates on this, see a conversation with Sam Stern on the group board:https://groups.google.com/d/msgid/google-cloud-firestore-discuss/e1b47358-b106-43a0-91fb-83c97d6244de%40googlegroups.com
Much of the discussion comes from the apparent fact that there is a SINGLE "master index" of ALL records in a database based on the FULLY QUALIFIED path to the document (hence only needing to be unique "within a collection").
To "accelerate" document references, the JS SDK actually "pre-pends" the collection path onto whatever info is passed in .doc to "conveniently" use that master index
i.e. all of these are exactly equivalent
db.doc('collection/docId/collection/docId/collection/docId")
db.collection('collection").doc("docId/collection/docId/collection/docId")
db.collection('collection").doc("docId").collection("collection").doc("docId/collection/docId")
db.collection('collection").doc("docId").collection("collection").doc("docId").collection("collection").doc("docId")
db.doc("collection/docId").collection("collection").doc("docId").collection("collection").doc("docId")
db.collection("collection/docId/collection").doc("docId").collection("collection").doc("docId")
db.doc("collection/docId/collection/docId").collection("collection").doc("docId")
db.collection("collection/docId/collection/docId/collection").doc("docId")
--- they ALL create the same index reference 'collection/docId/collection/docId/collection/docId" to find the document in the "master index".
(in my not-at-all-humble-opinion) FieldPath.documentId() was implemented (incorrectly) to "conveniently" match this behavior thus requiring the fully-qualified path, not the docId, when it should have been implemented like any other query, and required creating and maintaining a NEW INDEX to accommodate the query.
The code for this behavior was written BEFORE collectionGroups were implemented - and never documented the hack used didn't match the METHOD NAME used.
The work around is to require the Coder to copy the docId as a field in each document, and write your queries on that. I already wrote my own layer between Firestore.js and my application to abstract the behavior, and will probably simply implement this as a basic feature of the library.
But this is clearly a MISTAKE, and so far everybody keeps trying to tell me it makes sense, and that they'll change the documentation to match the existing behavior (but not the method name).
As I wrote previously, I keep getting handed a ratty bunch of daisies, and being told "See? these are roses!! The documentation calls them roses!! Roses by any other name smell as sweet, and all that!!"
No Update Expected Unless They Get Embarrassed Enough
UPDATE 2020-01-10: I have built a demo app showing the exact bug, and have sent it to Firebase support as requested. For some dang reason, the support critter considers it a "feature request", in spite of it clearly a bug. When a URL is called in the form "/ShowInfo/showID", the App signs in to Firebase Auth anonymously; then calls a query on the collectionGroup (3 levels deep) using FieldPath.documentId() "==" showID
It makes the query 3 ways:
1) Once with only the showID- which fails with the familiar "Invalid query. When querying a collection group by FieldPath.documentId(), the value provided must result in a valid document path, but 'pqIPV5I7UWne9QjQMm72'(the actual showID) is not because it has an odd number of segments (1)."
2) Once with a "Relative Path" (/Shows/showID), which doesn't have the error, but returns no document.
3) Finally with the "Full Path" (/Artists/ArtistID/Tour/tourID/Shows/showID). This doesn't have an error, and does return a document - but if I have the full path, why do I need the query on the collectionGroup? And I don't have the full path - the showID (above) comes in as part of a URL (a link to the show data, obviously) - I hand-faked it for the test.
Waiting for response.
UPDATE 2019-12-02: Firebase support reached out to ask if I still wanted this solved. Duh.
UPDATE 2019-09-27: Firebase Support has acknowledged this is a bug. No word on when it will be fixed. documentId() should, indeed, be able to be used directly against only the document Id.
documentID can be used as part of a query but, as #greg-ennis notes above, it needs an even number of segments. As it turns out, if you truly need a collectionGroup (I do), then the "Id" to be compared needs to redundantly add the collectionGroup ID/name as a segment:
db.collectionGroup('likedBy')
.where(firebase.firestore.FieldPath.documentId(), '==', "likedBy" + "/" + "User A")
.get();
I've used it and it (*sorta) works. (admittedly, it took me 2 hours to figure it out, and the above question helped focus the search)
On the other hand, this specific case is not where you want to use collectionGroup. Remember what collection group is - a way to refer to a a set of separate collections as if they were one. In this case the "collection" that holds "User A"s likes exists as a collection under "User A"s doc. Simply delete that single collection before deleting "User A"s doc. No need to bring everybody else's likes into it.
Sorta: the field path compared apparently has to be the complete path to the document. If you know the documentId, but for "reasons" you do not know the complete path, including which document the sub-collection is a part of (kinda why you were using the collectionGroup approach in the first place), this now seems a tadly dead-end. Continuing working on it.
Verified and Bug Report filed: FieldPath.documentID() does not compare against the documentId; it compares against the fully segmented document path, exactly as you would have to provide to .doc(path):
To wit: to find a document at "TopCollection/document_this/NextCollection/document_that/Travesty/document_Id_I_want"
using the collectionGroup "Travesty"
db.collectionGroup("Travesty")
.where(firebase.firestore.FieldPath.documentId(), '==', "document_id_I_want")
...will fail, but...
db.collectionGroup("Travesty")
.where(firebase.firestore.FieldPath.documentId(), '==', "TopCollection/document_this/NextCollection/document_that/Travesty/document_Id_I_want")
.get()
...will succeed. Which makes this useless, since if we had all that info, we would just use:
db.doc("TopCollection/document_this/NextCollection/document_that/Travesty/document_Id_I_want")
.get()
There is no way you can use the following query:
db.collectionGroup('likedBy').where(firebase.firestore.FieldPath.documentId(), '==', "User A").get();
And this is because collection group queries work only on document properties and not on document ids. According to the official documentation regarding collection group queries:
db.collectionGroup('landmarks').where('type', '==', 'museum');
You query the landmarks subcollection where the type property holds the value of museum.
A workaround could be to store the id of the user in an array and use array-contains but remember, for each collection group query you use, you need an index and unfortunately you cannot create such an index programmatically. Even if you can create an index in the Firebase console, it won't help you since you get those ids dynamically. So is not an option to create an index for each user separately because you'll reach the maximim number of indexes very quickly.
Maximum number of composite indexes for a database: 200
To solve this kind of problems, you should consider adding an array under each user object and use a query like this:
usersRef.where("usersWhoLikedMe", "array-contains", "someUserId")
Where usersWhoLikedMe is a property of type array.
If you add User A and B ids to the doc itself:
users(col) -> User A(doc) -> likedBy(col) -> User B ({user_id: B, profile_liked_id: A})
Then you can query using:
db.collectionGroup('likedBy').where('user_id', '==', 'B');
if the reference is a collection, the value you compare to needs to be document id (the last segment of a full path)
this is a error message if you violated the rule
Invalid query. When querying a collection by documentId(), you must
provide a plain document ID, but 'a/b/c' contains a '/' character
if the reference is a collectonGroup, the value you compare to needs to be a full document path <-- which is very redundant in my opinion
Invalid query. When querying a collection group by documentId(), the
value provided must result in a valid document path, but
'a/b/c' is not because it has an odd number of
segments (3)
tested recently
I have created a small application for keeping track of how much time I spend in different courses (as a teacher) using Angular 5 and putting my data in Firestore via AngularFire2. Most things have worked out nicely, but for the final part of the application I am having serious problems. I am quite a newbie at Angular/Firebase and most importantly (perhaps) with reactive programming.
The data that is stored in Firestore is quite simple:
Data stored in Firebase It consists of six fields for information about the course, when the course was held and most importantly what I did (called Element) and for how long (called Duration).
Now, what I would like to do is to make a report that summaries all the durations for each thing I did (Element). In essence, what I would like to do is the equivalent of a GroupBy in SQL. As I understand it Firebase does not have an operation for grouping.
My, so far failed approach to this, has been to try to create an array into which I put the data and make calculations on that. It is here that I fail.
My first attempt is based on creating an observer as most examples for retrieving data in tutorials use. WorkItem is a simple class that has the same content as in Firebase.
this.lectureDoc = this.afs.collection(`users/${this.afAuth.auth.currentUser.uid}/regTime`,
ref => ref.where('course', '==', this.selectedCourse.courseName).orderBy('element') );
this.lectureItems = this.lectureDoc.snapshotChanges().pipe(
map (courses => courses.map(a => {
const data = a.payload.doc.data() as WorkItem;
const id = a.payload.doc.id;
return { id, ...data };
})
));
this.lectureItems.subscribe(item => {
item.forEach(i => {
this.allElements.push(i);
});
});
console.error(this.allElements);
console.error(this.allElements[0]);
The two last lines show the problem, namely that the first (of the two last lines) will return the complete array while the second will return 'undefined'.
I understand that reactive programming will make asynchronous calls and that I therefore cannot know for sure when the data is filled. However, I do not understand why I can see the content of the array in the first of the two last lines, but not in the second.
My second attempt is based on getting the data from the documents of the collection itself and also, to prevent empty arrays, using .then() but again the two last lines show exactly the same output as previously.
this.lectureDoc.ref.get().then(item => {
item.forEach(i => {
this.lectures.push(i.data() as WorkItem);
});
});
console.error(this.lectures);
console.error(this.lectures[0]);
The output can be seen in this image: Output from the running program
So, to recap, what I would like to do is to collect all the data in an array of WorkItem and then calculate how much time has been spent on different tasks and then display it in list on the web page. I have not come to the display part and I suspect that I will have trouble again with the array not being populated when bound to the list. I have a hard time to understand reactive programming...
Any help would be greatly appreciated!
//Tobias
This is a pretty common behaviour of the console as far as I'm aware, at least in Chrome. When you log data that's async but haven't been populated yet, the console will still show the array once the async operation is completed.
But when you log an object in the array, at the time that you log it that object isn't there yet, therefore the log will show as undefined.
Simply do the logging inside of the forEach and you will see the data being pushed properly.
An another case, example there: Console.log behavior
Dataloader is able to batch and cache requests, but it can only be used by either calling load(key) or loadMany(keys).
The problem I am having is that sometimes I do not know they keys of the items I want to load in advance.
I am using an sql database and this works fine when the current object has a foreign key from a belongsTo relation with another model.
For example a user that belongs to a group and so has a groupId. To resolve the group you would just call groupLoader.load(groupId).
On the other hand, if I wanted to resolve the users within a group, of which there could be many I would want a query such as
SELECT * from users where user.groupId = theParticularGroupId
but a query such as this doesn't use the keys of the users and so I am not sure how make use of dataloader.
I could do another request to get the keys like
SELECT id from users where user.groupId = theParticularGroupId
and then call loadMany with those keys... But I could have just requested the data directly instead.
I noticed that dataloader has a prime(key, value) function which can be used to prime the cache, however that can only be done once the data is already fetched. At which point many queries would already have been sent, and duplicate data could have been fetched.
Another example would be the following query
query {
groups(limit: 10) {
id
...
users {
id
name
...
}
}
}
I cannot know the keys if I am searching for say the first or last 10 groups. Then once I have these 10 groups. I cannot know the keys of their users, and if each resolver would resolve the users using a query such as
SELECT * from users where user.groupId = theParticularGroupId
that query will be executed 10 times. Once the data is loaded I could now prime the cache, but the 10 requests have already been made.
Is there any way around this issue? Perhaps a different pattern or database structure or maybe dataloader isn't even the right solution.
You'll want a dataloader instance for the lookup you can do, in this case you have a group ID and you want the users:
import DataLoader from 'dataloader';
const userIdsForGroupLoader = new DataLoader(groupIds => batchGetUsersIdsForGroups(groupIds));
Now your batchGetUsersForGroups function is essentially has to convert an array of group IDs to an array of arrays of users (one array of user IDs for each group).
You'd start off with an IN query:
SELECT id from users where user.groupId in (...groupIds)
This will give you a single result set of users, which you'll have to manipulate, by grouping them by their groupId, the array should be ordered according to the original array of groupIds. Make sure you return an empty array for groupIds that don't have any users.
Note that in this we're only returning the user ids, but you can batch fetch the users in one go once you have them. You could tweak it slightly to return the users themselves, you'll have to decide for yourself if that's the right approach.
Everything I mention in this article can be achieved using clever use of Dataloader. But the key takeaway is that the values you pass to the load/loadMany functions don't have to correspond to the IDs of the objects you're trying to return.
As "collection group query" is still not implemented on Firestore I am using a map of values as a workaround. Here is the structure:
/chat/[room_id]/
user[ user_id ] = true
The query is something like:
db.collection( 'chat' )
.doc( room_id )
.where( 'user.' + user_id, "==", true )
.onSnapshot( ... )
They query works perfectly, but if a "user-map" adds or remove the "user_id" field, the listener (onSnapshot) is not called. On Firebase console the value flashes indicating that a change was made.
According to the documentation maps are automatically indexed, so it should not be a indices problem.
Any idea?
Looking at the documentation we can see that:
Queries on multiple fields with range filters require a composite
index. Unfortunately, this is not possible with the approach shown
above. Indexes must be defined on specific field paths. You would have
to create an index on each possible field path (such as
categories.cats or categories.opinion) which is impossible to do in
advance if your categories are user generated.
The "approach shown above" refers to the case where you are performing a compound query with a map of values (same example as yours). The solution they propose is to
encode all of the information for the query into the map values
Although not ideal, it might be useful for certain cases.
Ok, so I've been reading and reading and searching and searching and strangely it doesn't seem like my scenario has been really covered anywhere.
I have an app that creates a list of products. I want a simple view that can sort the products and page through them.
Fore reference here is a simple representation of the data in Firebase.
app
stock
unique_id
name
url
imageUrl
price
When creating the list I have multiple threads using the push method on my firebase references:
new Firebase(firebaseUrl).child('stock').push({
name: "name",
price: 123
});
This gives me a lovely "hash" collection on the stock property of the app.
So what I'd now like to do is have a table to sort and page through the records that were placed in the stock hash.
I make a GET request to my server to a url like /stock?limit=10&skip=10&sort=name%20asc. This particular url would be the second page where the table contained 10 records per page and was sorted by the name property in ascending order.
Currently in my query handler I have this:
var firebaseRef = new Firebase(firebaseUrl).child('stock');
if (this.sortDesc) {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.endAt()
.limitToFirst(this.limitAmount);
} else {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.limitToFirst(this.limitAmount);
if (this.skipAmount > 0) {
firebaseRef = firebaseRef.startAt(this.skipAmount);
}
}
firebaseRef.once('value', function (snapshot) {
var results = [];
snapshot.forEach(function (childSnapshot) {
results.push(childSnapshot.val());
});
callback(null, results);
});
I'm running into a couple of problems. I'm going to split this into two cases, ascending and descending queries.
Ascending query
The orderByChild and limitToFirst seems to work correctly in the sorting ascending case. This means I can change which property has an ascending sort and how many results to return. What I am not able to get to work is skipping n records for paging to work. In the example query above I'm going to the second page. I do not get results 11-20, but I instead get the same 10 records as the first page.
Descending query
In this case I cannot begin to figure out how to tell Firebase to order by a property of the object identified by the unique key in a descending fashion. The closest I've read is to use endAt() and then limit. Docs say the limit is deprecated plus this still doesn't help me with any paging.
I tired to do doodles picturing how this would work. I came up with: order by the property, start at the 'end' of the collection, and then limit back to the page size. While this still wouldn't solve paging I would expect it to give me the last n records where n was the size of the page. I get no results.
I suppose I could say use firebaseRef = firebaseRef .orderByChild(this.sortProperty).limitToLast(this.limitAmount + this.skipAmount); and in the result callback use the forEach loop to take the first (or would it be the last; I'm not sure how that iteration would work) n records where n=this.limitAmount. This just seems inefficient. Wouldn't it be better to limit the query instead of using CPU cycles to limit data that had come over the wire or is this the relational DB query thought pattern overriding the correct thought process for NoSQL?
Further Confusion
After posting this I've still been working on a solution. I've had some things get close, but I'm also running into this filtering issue. How could I filter a set of items to one property by still sorting on another? Jeez! I want to have the ability for a user to get all the stock that isn't sold out and order it by price.
Finally
Why hasn't this basic example been fleshed out on any of the Firebase "Getting Started" pages? Being able to show tabular data, page through it, sort, and filter seem like something that EVERY web developer would come across. I'm using ng-table in an Angular app to drive the view, but it still seems that regardless of platform that the queries that I'm trying to generate would be practical on any platform that Firebase supports. Perhaps I'm missing something! Please educate me!
Firebase and NoSQL
I've come up with this simple scenario that I often run into with web applications. I want to show tabular data, filter, page, and sort it. Very simple. Very common. Writing a SQL statement for this would be dead easy. Why is the query so complicated for something like Firebase. Is this common with all NoSQL solutions? There is no relational data being stored thus the need for a relational database seems unnecessary. Yet, it seems like I could hack together a little flat file to do this storage since the ability to make Firebase do these simple tasks is not made clear in its API or Docs. FRUSTRATED!!!