passing data from a model/database to a channel using presence - javascript

I have a simple chat application that I built and I want to be able to display user uploaded images (locally hosted) next to the user names on the channel's html page. Currently, I am using presence to track the users who are logged into the channel etc. I was able to override the fetch/2 function with the understanding that it would allow me to add a couple map fields to the :metas symbol with user model data.
From what I can tell based on extensive IO.inspecting of different parts of each of the functions; fetch/2, handle_info/2, and some console.logging on my JS layer, the fetch/2 function is not actually getting any data out of the database nor is it assigning it to the :metas map.
here is my current fetch/2 function:
def fetch(_topic, entries) do
query =
from u in User,
where: u.id in ^Map.keys(entries),
select: {u.id, u}
users = query |> Repo.all |> Enum.into(%{})
for {key, %{metas: metas}} <- entries, into: %{} do
{key, %{metas: metas, user: users[key]}}
end
It is basically ripped directly from the documentation. In Theory, the function above should query my User Model and grab all of the user data based on the User.id that is being passed to it through the entries map. Users[keys] comes back as empty despite users being a full map of my User model.
Also, according to the documentation, the query is only supposed to run on join so as not to overload the DB but it seems to run 4-5 times every time I refresh the page. Another thing to note, is that the user.id inside of entries seems to be a string type. Im not sure if this is important, I've tried passing a integer from the JS layer and also using Interger.parse from the actual fetch/2 function to change this to no avail.
When I inspect the users map I get this:
{"1" => %MyApp.User{__meta__: #Ecto.Schema.Metadata<:loaded, "users">,
email: "test#test.com", encrypt_pass: "$pbkdf2-
sha512$160000$ebfY956TgIXhEAF.mqLJAg$QWzBubfeiy4Xrf‌​.EsFiU0jEZAuKvV4ZO5a‌​
8QpeFr817C61DuaNfyo5‌​6WWzj6jak2homCFWAINb‌​PrFtCSXUPWTw", gravatar: %
{file_name: "logo.png", updated_at: #Ecto.DateTime<2017-04-20 22:00:08>},
id: 1, inserted_at: ~N[2017-04-20 22:00:09.071000], password: nil,
updated_at: ~N[2017-04-20 22:00:09.090000]}}
My users[key] returns an empty map like this %{} and converting the input into an integer throws an error, (Poison.EncodeError) unable to encode value: {nil, "users"} if i convert it inside of the elixir code, and where: u.id in ^["undefined"], select: {u.id, u} (elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3 – if I convert it from the JS layer.
The original fetch/2 output is an array with online_at: 1492764577562 and phx_ref: "OAyzaGE82xc=" in the :metas map and my user id or email in the users var.
What is it that I am missing here? I know that the fetch/2 function only executes as a callback to the Presence.list/1 function which I am calling in my handle_info/2 channel function. I am also calling Presence.list in my JS layer and mapping it to my presences so that I can produce the list of usernames in the HTML. Am I just misunderstanding how this works or is there some other more simple way that I should be going about this? If you need to see more code I can supply more.
edit: I have a much better understanding of what is happening here. my entries map is actually this:
%{"1" => %{metas: [%{online_at: 1492798247818, phx_ref: "ELHwA+gWF+0="}]}}
So basically, the string for the user id, "1" is being mapped onto the metas map. When I try to just take that key out of the map with the Map.keys(entries) function it isn't able to pull anything out of the DB because it is a string, however, when I change it to an integer from the JavaScript side it throws an error because for whatever reason phoenix is expecting that key to be a string type. Strangely enough if I change the id from an id to an email and try to query the db with the email it doesn't work either. despite the email in the Database being string and the metas map expecting a string key for the entries map.
I am going to rebuild this channel part of the app from the ground up and see what is causing this problem. Then I will come back and see if I was able to fix the error.

You should validate the keys in the entries map first.
ids = Map.keys(entries)
true = Enum.all?(ids, &is_integer/1)
Ecto will convert strings to integers when interpolating into a query:
iex(40)> Ecto.Query.from(u in Users, where: u.id in ^[1, 2, "3", nil], select: u.id) |> Repo.all()
outputs the following debug log:
[debug] QUERY OK source="users" db=0.8ms
SELECT u0."id" FROM "users" AS r0 WHERE (r0."id" = ANY($1)) [[1, 2, 3, nil]]
Notice it coerced the string "3" to an integer and allowed the nil.
However a map will not be so kind:
iex(42)> users = %{1 => %{name: "joe"}, 2 => %{name: "jill"}}
%{1 => %{name: "joe"}, 2 => %{name: "jill"}}
iex(43)> users["1"]
nil
So in the code where you are using the keys from entries for database lookups and map lookups, it could be producing different results.

I've already figured out that the problem has very little to do with my fetch/2 function itself, rather, it had to do with my implementation of the presence module and channel in this case. Basically, the fetch/2 function was being called 4 times every time some one entered the chat room and two out of the four times it was being called with an empty list value [].
Obviously, you can't query a Ecto model with an empty list so it was throwing an error in that case as well. I tried putting guards on the fetch function to filter out the empty list calls but it would not show me the metas map data that I was looking for even when the query succeeded.
Also, the other main problem was my implementation or lack of implementation of a token. I wouldn't have to pass around the user model data through fetch function metas map if I was using a token for joining the chat room rather then just a user (aka the just a username). After making that realization, I was able to successfully connect the user model data with the channel and show it through the JS layer and ultimately, put it on the client.
Anyways guys, thanks for the suggestions. You may not have answered the question (It was my fault for asking the wrong question), but you certainly helped me get there. And also gave me the tools to form a much better understanding of the framework in general on the way.
If/When I have any more questions, I will make sure that I am asking the correct questions before posting them to stack overflow, that way I wont be wasting time.

Related

Firestore collection group query on documentId

In my scenario a user can "like" the profile of another user. As a result I have a subcollection called "likedBy" for each user, where I create a document for a specific user, e.g.:
users(col) -> User A(doc) -> likedBy(col) -> User B (doc), User C (doc)
So in the rare scenario of a user being deleted, I want to delete all the likes issues by that user.
I am just not sure if this is even possible (a workaround could be to just save the userId again in said document and query for that).
What I am basically looking for is something like this:
db.collectionGroup('likedBy').where(firebase.firestore.FieldPath.documentId(), '==', "User A").get();
The problem is, that I can not create an index for the documentId in the Firestore console.
UPDATE 2020-01-23:
For updates on this, see a conversation with Sam Stern on the group board:https://groups.google.com/d/msgid/google-cloud-firestore-discuss/e1b47358-b106-43a0-91fb-83c97d6244de%40googlegroups.com
Much of the discussion comes from the apparent fact that there is a SINGLE "master index" of ALL records in a database based on the FULLY QUALIFIED path to the document (hence only needing to be unique "within a collection").
To "accelerate" document references, the JS SDK actually "pre-pends" the collection path onto whatever info is passed in .doc to "conveniently" use that master index
i.e. all of these are exactly equivalent
db.doc('collection/docId/collection/docId/collection/docId")
db.collection('collection").doc("docId/collection/docId/collection/docId")
db.collection('collection").doc("docId").collection("collection").doc("docId/collection/docId")
db.collection('collection").doc("docId").collection("collection").doc("docId").collection("collection").doc("docId")
db.doc("collection/docId").collection("collection").doc("docId").collection("collection").doc("docId")
db.collection("collection/docId/collection").doc("docId").collection("collection").doc("docId")
db.doc("collection/docId/collection/docId").collection("collection").doc("docId")
db.collection("collection/docId/collection/docId/collection").doc("docId")
--- they ALL create the same index reference 'collection/docId/collection/docId/collection/docId" to find the document in the "master index".
(in my not-at-all-humble-opinion) FieldPath.documentId() was implemented (incorrectly) to "conveniently" match this behavior thus requiring the fully-qualified path, not the docId, when it should have been implemented like any other query, and required creating and maintaining a NEW INDEX to accommodate the query.
The code for this behavior was written BEFORE collectionGroups were implemented - and never documented the hack used didn't match the METHOD NAME used.
The work around is to require the Coder to copy the docId as a field in each document, and write your queries on that.  I already wrote my own layer between Firestore.js and my application to abstract the behavior, and will probably simply implement this as a basic feature of the library.
But this is clearly a MISTAKE, and so far everybody keeps trying to tell me it makes sense, and that they'll change the documentation to match the existing behavior (but not the method name).
As I wrote previously, I keep getting handed a ratty bunch of daisies, and being told "See? these are roses!! The documentation calls them roses!! Roses by any other name smell as sweet, and all that!!"
No Update Expected Unless They Get Embarrassed Enough
UPDATE 2020-01-10: I have built a demo app showing the exact bug, and have sent it to Firebase support as requested. For some dang reason, the support critter considers it a "feature request", in spite of it clearly a bug. When a URL is called in the form "/ShowInfo/showID", the App signs in to Firebase Auth anonymously; then calls a query on the collectionGroup (3 levels deep) using FieldPath.documentId() "==" showID
It makes the query 3 ways:
1) Once with only the showID- which fails with the familiar "Invalid query. When querying a collection group by FieldPath.documentId(), the value provided must result in a valid document path, but 'pqIPV5I7UWne9QjQMm72'(the actual showID) is not because it has an odd number of segments (1)."
2) Once with a "Relative Path" (/Shows/showID), which doesn't have the error, but returns no document.
3) Finally with the "Full Path" (/Artists/ArtistID/Tour/tourID/Shows/showID). This doesn't have an error, and does return a document - but if I have the full path, why do I need the query on the collectionGroup? And I don't have the full path - the showID (above) comes in as part of a URL (a link to the show data, obviously) - I hand-faked it for the test.
Waiting for response.
UPDATE 2019-12-02: Firebase support reached out to ask if I still wanted this solved. Duh.
UPDATE 2019-09-27: Firebase Support has acknowledged this is a bug. No word on when it will be fixed. documentId() should, indeed, be able to be used directly against only the document Id.
documentID can be used as part of a query but, as #greg-ennis notes above, it needs an even number of segments. As it turns out, if you truly need a collectionGroup (I do), then the "Id" to be compared needs to redundantly add the collectionGroup ID/name as a segment:
db.collectionGroup('likedBy')
.where(firebase.firestore.FieldPath.documentId(), '==', "likedBy" + "/" + "User A")
.get();
I've used it and it (*sorta) works. (admittedly, it took me 2 hours to figure it out, and the above question helped focus the search)
On the other hand, this specific case is not where you want to use collectionGroup. Remember what collection group is - a way to refer to a a set of separate collections as if they were one. In this case the "collection" that holds "User A"s likes exists as a collection under "User A"s doc. Simply delete that single collection before deleting "User A"s doc. No need to bring everybody else's likes into it.
Sorta: the field path compared apparently has to be the complete path to the document. If you know the documentId, but for "reasons" you do not know the complete path, including which document the sub-collection is a part of (kinda why you were using the collectionGroup approach in the first place), this now seems a tadly dead-end. Continuing working on it.
Verified and Bug Report filed: FieldPath.documentID() does not compare against the documentId; it compares against the fully segmented document path, exactly as you would have to provide to .doc(path):
To wit: to find a document at "TopCollection/document_this/NextCollection/document_that/Travesty/document_Id_I_want"
using the collectionGroup "Travesty"
db.collectionGroup("Travesty")
.where(firebase.firestore.FieldPath.documentId(), '==', "document_id_I_want")
...will fail, but...
db.collectionGroup("Travesty")
.where(firebase.firestore.FieldPath.documentId(), '==', "TopCollection/document_this/NextCollection/document_that/Travesty/document_Id_I_want")
.get()
...will succeed. Which makes this useless, since if we had all that info, we would just use:
db.doc("TopCollection/document_this/NextCollection/document_that/Travesty/document_Id_I_want")
.get()
There is no way you can use the following query:
db.collectionGroup('likedBy').where(firebase.firestore.FieldPath.documentId(), '==', "User A").get();
And this is because collection group queries work only on document properties and not on document ids. According to the official documentation regarding collection group queries:
db.collectionGroup('landmarks').where('type', '==', 'museum');
You query the landmarks subcollection where the type property holds the value of museum.
A workaround could be to store the id of the user in an array and use array-contains but remember, for each collection group query you use, you need an index and unfortunately you cannot create such an index programmatically. Even if you can create an index in the Firebase console, it won't help you since you get those ids dynamically. So is not an option to create an index for each user separately because you'll reach the maximim number of indexes very quickly.
Maximum number of composite indexes for a database: 200
To solve this kind of problems, you should consider adding an array under each user object and use a query like this:
usersRef.where("usersWhoLikedMe", "array-contains", "someUserId")
Where usersWhoLikedMe is a property of type array.
If you add User A and B ids to the doc itself:
users(col) -> User A(doc) -> likedBy(col) -> User B ({user_id: B, profile_liked_id: A})
Then you can query using:
db.collectionGroup('likedBy').where('user_id', '==', 'B');
if the reference is a collection, the value you compare to needs to be document id (the last segment of a full path)
this is a error message if you violated the rule
Invalid query. When querying a collection by documentId(), you must
provide a plain document ID, but 'a/b/c' contains a '/' character
if the reference is a collectonGroup, the value you compare to needs to be a full document path <-- which is very redundant in my opinion
Invalid query. When querying a collection group by documentId(), the
value provided must result in a valid document path, but
'a/b/c' is not because it has an odd number of
segments (3)
tested recently

Putting Firestore data into array with Angular

I have created a small application for keeping track of how much time I spend in different courses (as a teacher) using Angular 5 and putting my data in Firestore via AngularFire2. Most things have worked out nicely, but for the final part of the application I am having serious problems. I am quite a newbie at Angular/Firebase and most importantly (perhaps) with reactive programming.
The data that is stored in Firestore is quite simple:
Data stored in Firebase It consists of six fields for information about the course, when the course was held and most importantly what I did (called Element) and for how long (called Duration).
Now, what I would like to do is to make a report that summaries all the durations for each thing I did (Element). In essence, what I would like to do is the equivalent of a GroupBy in SQL. As I understand it Firebase does not have an operation for grouping.
My, so far failed approach to this, has been to try to create an array into which I put the data and make calculations on that. It is here that I fail.
My first attempt is based on creating an observer as most examples for retrieving data in tutorials use. WorkItem is a simple class that has the same content as in Firebase.
this.lectureDoc = this.afs.collection(`users/${this.afAuth.auth.currentUser.uid}/regTime`,
ref => ref.where('course', '==', this.selectedCourse.courseName).orderBy('element') );
this.lectureItems = this.lectureDoc.snapshotChanges().pipe(
map (courses => courses.map(a => {
const data = a.payload.doc.data() as WorkItem;
const id = a.payload.doc.id;
return { id, ...data };
})
));
this.lectureItems.subscribe(item => {
item.forEach(i => {
this.allElements.push(i);
});
});
console.error(this.allElements);
console.error(this.allElements[0]);
The two last lines show the problem, namely that the first (of the two last lines) will return the complete array while the second will return 'undefined'.
I understand that reactive programming will make asynchronous calls and that I therefore cannot know for sure when the data is filled. However, I do not understand why I can see the content of the array in the first of the two last lines, but not in the second.
My second attempt is based on getting the data from the documents of the collection itself and also, to prevent empty arrays, using .then() but again the two last lines show exactly the same output as previously.
this.lectureDoc.ref.get().then(item => {
item.forEach(i => {
this.lectures.push(i.data() as WorkItem);
});
});
console.error(this.lectures);
console.error(this.lectures[0]);
The output can be seen in this image: Output from the running program
So, to recap, what I would like to do is to collect all the data in an array of WorkItem and then calculate how much time has been spent on different tasks and then display it in list on the web page. I have not come to the display part and I suspect that I will have trouble again with the array not being populated when bound to the list. I have a hard time to understand reactive programming...
Any help would be greatly appreciated!
//Tobias
This is a pretty common behaviour of the console as far as I'm aware, at least in Chrome. When you log data that's async but haven't been populated yet, the console will still show the array once the async operation is completed.
But when you log an object in the array, at the time that you log it that object isn't there yet, therefore the log will show as undefined.
Simply do the logging inside of the forEach and you will see the data being pushed properly.
An another case, example there: Console.log behavior

GraphQL Dataloader not knowing keys in advance

Dataloader is able to batch and cache requests, but it can only be used by either calling load(key) or loadMany(keys).
The problem I am having is that sometimes I do not know they keys of the items I want to load in advance.
I am using an sql database and this works fine when the current object has a foreign key from a belongsTo relation with another model.
For example a user that belongs to a group and so has a groupId. To resolve the group you would just call groupLoader.load(groupId).
On the other hand, if I wanted to resolve the users within a group, of which there could be many I would want a query such as
SELECT * from users where user.groupId = theParticularGroupId
but a query such as this doesn't use the keys of the users and so I am not sure how make use of dataloader.
I could do another request to get the keys like
SELECT id from users where user.groupId = theParticularGroupId
and then call loadMany with those keys... But I could have just requested the data directly instead.
I noticed that dataloader has a prime(key, value) function which can be used to prime the cache, however that can only be done once the data is already fetched. At which point many queries would already have been sent, and duplicate data could have been fetched.
Another example would be the following query
query {
groups(limit: 10) {
id
...
users {
id
name
...
}
}
}
I cannot know the keys if I am searching for say the first or last 10 groups. Then once I have these 10 groups. I cannot know the keys of their users, and if each resolver would resolve the users using a query such as
SELECT * from users where user.groupId = theParticularGroupId
that query will be executed 10 times. Once the data is loaded I could now prime the cache, but the 10 requests have already been made.
Is there any way around this issue? Perhaps a different pattern or database structure or maybe dataloader isn't even the right solution.
You'll want a dataloader instance for the lookup you can do, in this case you have a group ID and you want the users:
import DataLoader from 'dataloader';
const userIdsForGroupLoader = new DataLoader(groupIds => batchGetUsersIdsForGroups(groupIds));
Now your batchGetUsersForGroups function is essentially has to convert an array of group IDs to an array of arrays of users (one array of user IDs for each group).
You'd start off with an IN query:
SELECT id from users where user.groupId in (...groupIds)
This will give you a single result set of users, which you'll have to manipulate, by grouping them by their groupId, the array should be ordered according to the original array of groupIds. Make sure you return an empty array for groupIds that don't have any users.
Note that in this we're only returning the user ids, but you can batch fetch the users in one go once you have them. You could tweak it slightly to return the users themselves, you'll have to decide for yourself if that's the right approach.
Everything I mention in this article can be achieved using clever use of Dataloader. But the key takeaway is that the values you pass to the load/loadMany functions don't have to correspond to the IDs of the objects you're trying to return.

Firestore Query using a map of values does not invoke listener when the map is modified

As "collection group query" is still not implemented on Firestore I am using a map of values as a workaround. Here is the structure:
/chat/[room_id]/
user[ user_id ] = true
The query is something like:
db.collection( 'chat' )
.doc( room_id )
.where( 'user.' + user_id, "==", true )
.onSnapshot( ... )
They query works perfectly, but if a "user-map" adds or remove the "user_id" field, the listener (onSnapshot) is not called. On Firebase console the value flashes indicating that a change was made.
According to the documentation maps are automatically indexed, so it should not be a indices problem.
Any idea?
Looking at the documentation we can see that:
Queries on multiple fields with range filters require a composite
index. Unfortunately, this is not possible with the approach shown
above. Indexes must be defined on specific field paths. You would have
to create an index on each possible field path (such as
categories.cats or categories.opinion) which is impossible to do in
advance if your categories are user generated.
The "approach shown above" refers to the case where you are performing a compound query with a map of values (same example as yours). The solution they propose is to
encode all of the information for the query into the map values
Although not ideal, it might be useful for certain cases.

Sorting and Paging A Collection With Unique IDs in Firebase

Ok, so I've been reading and reading and searching and searching and strangely it doesn't seem like my scenario has been really covered anywhere.
I have an app that creates a list of products. I want a simple view that can sort the products and page through them.
Fore reference here is a simple representation of the data in Firebase.
app
stock
unique_id
name
url
imageUrl
price
When creating the list I have multiple threads using the push method on my firebase references:
new Firebase(firebaseUrl).child('stock').push({
name: "name",
price: 123
});
This gives me a lovely "hash" collection on the stock property of the app.
So what I'd now like to do is have a table to sort and page through the records that were placed in the stock hash.
I make a GET request to my server to a url like /stock?limit=10&skip=10&sort=name%20asc. This particular url would be the second page where the table contained 10 records per page and was sorted by the name property in ascending order.
Currently in my query handler I have this:
var firebaseRef = new Firebase(firebaseUrl).child('stock');
if (this.sortDesc) {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.endAt()
.limitToFirst(this.limitAmount);
} else {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.limitToFirst(this.limitAmount);
if (this.skipAmount > 0) {
firebaseRef = firebaseRef.startAt(this.skipAmount);
}
}
firebaseRef.once('value', function (snapshot) {
var results = [];
snapshot.forEach(function (childSnapshot) {
results.push(childSnapshot.val());
});
callback(null, results);
});
I'm running into a couple of problems. I'm going to split this into two cases, ascending and descending queries.
Ascending query
The orderByChild and limitToFirst seems to work correctly in the sorting ascending case. This means I can change which property has an ascending sort and how many results to return. What I am not able to get to work is skipping n records for paging to work. In the example query above I'm going to the second page. I do not get results 11-20, but I instead get the same 10 records as the first page.
Descending query
In this case I cannot begin to figure out how to tell Firebase to order by a property of the object identified by the unique key in a descending fashion. The closest I've read is to use endAt() and then limit. Docs say the limit is deprecated plus this still doesn't help me with any paging.
I tired to do doodles picturing how this would work. I came up with: order by the property, start at the 'end' of the collection, and then limit back to the page size. While this still wouldn't solve paging I would expect it to give me the last n records where n was the size of the page. I get no results.
I suppose I could say use firebaseRef = firebaseRef .orderByChild(this.sortProperty).limitToLast(this.limitAmount + this.skipAmount); and in the result callback use the forEach loop to take the first (or would it be the last; I'm not sure how that iteration would work) n records where n=this.limitAmount. This just seems inefficient. Wouldn't it be better to limit the query instead of using CPU cycles to limit data that had come over the wire or is this the relational DB query thought pattern overriding the correct thought process for NoSQL?
Further Confusion
After posting this I've still been working on a solution. I've had some things get close, but I'm also running into this filtering issue. How could I filter a set of items to one property by still sorting on another? Jeez! I want to have the ability for a user to get all the stock that isn't sold out and order it by price.
Finally
Why hasn't this basic example been fleshed out on any of the Firebase "Getting Started" pages? Being able to show tabular data, page through it, sort, and filter seem like something that EVERY web developer would come across. I'm using ng-table in an Angular app to drive the view, but it still seems that regardless of platform that the queries that I'm trying to generate would be practical on any platform that Firebase supports. Perhaps I'm missing something! Please educate me!
Firebase and NoSQL
I've come up with this simple scenario that I often run into with web applications. I want to show tabular data, filter, page, and sort it. Very simple. Very common. Writing a SQL statement for this would be dead easy. Why is the query so complicated for something like Firebase. Is this common with all NoSQL solutions? There is no relational data being stored thus the need for a relational database seems unnecessary. Yet, it seems like I could hack together a little flat file to do this storage since the ability to make Firebase do these simple tasks is not made clear in its API or Docs. FRUSTRATED!!!

Categories