We Keep Coding

Sequelize Many-to-many querying model

Have some troubles with querying data.
Have 2 models with many to many relationships.
1st - S3FileData, 2nd - Playlists, they are connected through PlaylistContent table.
Also S3FileData is connected with User with User has-many playlists.
I need to query S3Files which belongs to user, but are not presented in user playlist.
I have playlist Id,userId.
Also I need offset and limit methods, so I tried to query this using findAndCountAll
But got no luck with it.
Would be very appreciate for any help :)

You will require include(joins in SQL) that will connect all these tables.
Something like this but first you will need to intertwine your models(db tables) which will look something like this:
User.belongsToMany(Profile, { through: Grant });
Profile.belongsToMany(User, { through: Grant });
User.hasMany(Grant);
Grant.belongsTo(User);
Profile.hasMany(Grant);
Grant.belongsTo(Profile);
Reference:
https://sequelize.org/master/manual/advanced-many-to-many.html
Now, Once you are through the joins you'll require to use those joins using include keyword in your findAll (or findAllAndCount as per the requirement):
Some basic code (You will need to try and tweak accordingly, this is rough code):
S3Files.findAll({
include: [
{
model: user,
attributes: ['some columns']
include: {
model: userPlaylist,
attributes: ['some columns'],
required: false
},
where: {[Op.and]: Sequelize.where(Sequelize.col('userPlaylist.userId'), 'is not' null)},
attributes: ['some columns']
]
});
lastly for offset and limit you'll require basic SQL logic of LIMIT and OFFSET
sample snippet from official docs for offset and limit:
Project
.findAndCountAll({
where: {
title: {
[Op.like]: 'foo%'
}
},
offset: 10,
limit: 2
})
Reference for the same:
https://sequelize.org/v5/manual/models-usage.html

How best can I retrieve all comments for a given chat room in Firebase on the web?

I have a real time firebase app with chatrooms and comments. A comment belongs to a single chatroom and a chatroom can have many comments. I'd like retrieve just the comments for a given room, but right now I'm only able to get all of them.
Every time a comment is saved, I also save its id to the room to which it belongs. That way, every room has a list of its comment ids. I can retrieve a chatroom's list of child comment ids using chatRooms/${id}/commentIds.
// data structure is basically like this:
chatRooms: {
ROOMID123: {
title: "Room A",
commentIds: {
COMMENTIDABC: true,
COMMENTIDXYZ: true
}
}
},
comments: {
COMMENTIDABC: {
message: "some message",
parentRoomId: ROOMID123
},
COMMENTIDXYZ: {
message: "Another message",
parentRoomId: ROOMID123
}
}
I can get the comment ids for a given room, based on the room's id, like this:
firebase.database().ref(`chatRooms/${chatRoomId}/commentIds`).on('value',
snapshot => {
const commentsObject = snapshot.val();
const commentIdsList = Object.keys(commentsObject);
});
Would it be better for me to
a) use that list of commentIds to retrieve only the comments for a given room? If so, what query should I use?
b) use the chatRoom's id to retrieve every comment with a matching parentRoomId? If so, I don't know how to do this despite reading through the docs.
Thank you!

I'd propose a third option: store the comments for each chat room in a separate parent node. So something like:
commentsPerRoom: {
ROOMID123: {
COMMENTIDABC: {
message: "some message",
},
COMMENTIDXYZ: {
message: "Another message",
}
}
}
With the above structure you can retrieve the comments for a specific room with a single direct lookup:
firebase.database().ref(`commentsPerRoom/${chatRoomId}`).on('value',
Reasons I'd use this data structure over your current one:
Storing the comments as a single list means you'd have to query that list. And while Firebase Realtime Database scales quite well, querying for data is always going to have scalability limits. That's why the above data structure allows a direct look up, instead of requiring a query.
Loading the individual comments through the comment IDs is definitely also possible, and not nearly as slow as some developers think due to the fact that Firebase pipelines the requests over a single connection. But it seems unneeded here, since the comments already have a strong 1:n association with the room they belong to. Each comment only belongs to one room, so you might as well model that relationship in how you structure the data, and save yourself all the lookups.
Retrieving comments for a specific room is by far the most common use-case, and this data structure allows that use-case the most efficiently.

How can we search through 5,000 users in Firestore db?

We have a staffing application, built using vuejs and a firestore database with over 5,000 users. Our challenge is that we need a layout for admins to search for users in the db.
Previously, we were loading all users on our Users.vue layout and then searching/viewing them in a Vuetify data table. The problem now is that we just have too many users. That layout loads way too slowly and will even cause the app to crash on mobile browsers.
The solution we are trying to make work is to search for users in the db, and only load those results into our data table. The code below (using vuex) works, as long as the "name" is EXACT.
getUsersState({commit}, payload){
fb.usersCollection.where("name", "==", payload.search).limit(10).onSnapshot(querySnapshot => {
let usersArray = []
console.log(payload.search)
querySnapshot.forEach(doc => {
let user = doc.data()
user.id = doc.id
usersArray.push(user)
})
commit('setUsers', usersArray)
})
},
The problem is that we need it to work even if we only type in the first few letters of a name or even an email address. Firestore only offers ==, >=, <=, >, and < parameters. And "array-contains" only works with an array, not our "user" object.
On Users.vue:
created () {
console.log('getting users')
this.$store.dispatch("getUsersState", {search: this.search})
},
computed: {
...mapState(['currentUser', 'users', 'userProfile']),
isAdmin: function() {
return this.userProfile.accessLevel >= 5
},
isUsers: function() {
return this.users
}
},
watch: {
search: 'updateSearch'
},
methods: {
clearSearch () {
return this.isSearch = ''
},
updateSearch() {
this.$store.dispatch("getUsersState", {search: this.search})
},
},
Does anyone have any ideas for how we can search the users in our firestore DB by only typing in the first few letters of their name?

Integrate a full text search engine, and keep it in sync with Firestore. Nontrivial to implement. Official docs recommend Algolia: https://firebase.google.com/docs/firestore/solutions/search

The right answer is full text search, but is a big hammer for this use case. Here are some other options that can keep you going for a while:
1) First, note that Firestore has an index sitting there that looks like
Collection\User\a -> somedoc
Collection\User\aaa -> somedoc
Collection\User\aba -> somedoc
Collection\User\abc -> somedoc
Collection\User\bbc -> somedoc
If you have a username prefix like a there is nothing to say you can't run a query for user >='a' and user <= 'b' and have if fetch (in this example) {a,aaa,aba}
Similarly >= 'ab' && <= 'b' gets you {ab, abc}
So you go from fetching all 5000 users to just the users with the prefix -- which is alot smaller.
2) Stuff the things you want to autocomplete into a few documents and load them.
Imagine you have 5000 users, and you store their names into 10 documents with 500 usernames each -- you keep those documents up to date as users add or remove. To get the entire autocomplete list you fetch those 10 documents into the browser and feed the 5000 users to some sort of autocomplete widget. You could do the same thing for emails.
The browser can now do fancy instant autocomplete. This is faster/cheaper than fetching the entire collection of 5000 users -- you only ask for the data you need.

Emberjs Deep Model Associations not available in templates

I'm new to Emberjs, and I'm trying to get my head around how to get data out of my model.
Here's the data structure as its being returned from my server
candidate: {
image: [image],
user: {
name: [User Name]
},
agent: {
phone: [phone]
team: {
name: [Team Name]
}
user: {
name: [User Name]
}
}
}
I cant get ember to recognize any associations more than one level deep.
heres my candidate controller
App.CandidateController = Ember.ObjectController.extend({
location: function() {
var address = this.get('address');
var city = this.get('city');
var state = this.get('state');
var zip = this.get('zip');
return address + ' ' + city + ' ' + state + ', ' + zip;
}.property('address', 'city', 'state', 'zip')
});
App.CandidateSerializer = DS.RESTSerializer.extend(DS.EmbeddedRecordsMixin, {
attrs: {
agent: {embedded: 'always'},
user: {embedded: 'always'}
}
});
This allows me to get one level deep, but I need two or three levels of association.
Is there a different way to go about it?
I thought organizing everything on the server the way I want it displayed, and just returning the data as its supposed to be rendered to ember, but I have a feeling I'll run into problems when trying to update the models.
Any help is appreciated.
/****** UPDATE *******/
I have refactored my code and I am now returning data from the serializer like this:
{
candidate: {
// relevant fields
},
agent: {
//relevant fields
},
team: {
// relevant fields
}
user: [
{ // candidate user fields },
{ // agent user fields }
]
};
However, data is still not available in my template. In the Ember chrome extension, in the data tab, I get
candidate: (1)
agent: (0)
team: (0)
user (2)
I can see candidate and user data, but not agent or team data. In my candidate model, I have:
App.Candidate = DS.Model.extend({
// other fields
user_id: DS.attr('number'),
agent_id: DS.attr('number'),
user: DS.belongsTo('user'),
agent: DS.belongsTo('agent')
});
It doesn't seem like the belongsTo association actually does anything.
So the first issue is that I'm not getting the correct data, the second issue, and the one that makes me think I am going about this incorrectly, is that I have two users information that I need to display in the template. The first is user information associated with the candidate, and the second is user information that is associated with the agent. Both data need to appear in the same template. Since there is no hierarchy to the data, how would the template know which user info to display in each location?
Again, I think Im thinking about this whole thing incorrectly.
Thanks for your help.

Ember data expects models in a JSON response to be flat, and it's the job of the Serializer to transform your server response into that format. Associations usually are done by ids. The second level is not working because Ember needs to turn each level into an Ember.Object in order to observe property changes.
It might help to look over the JSON conventions portion of the guides. You can also plug your models into the ember data model maker and see what your responses should look like. Finally, make sure you're using the ember-inspector Chrome extension. When I am debugging ember-data related issues, it's usually easiest to just stop after the model hook returns, look in the inspector's store tab, and examine what data has been loaded.

Adding information from another mongodb collection before providing the data via meteor's publish method

What I want to accomplish on http://crowducate.me:
Display the usernames of the course authors (i.e. "owner" of a document).
Current Code:
Meteor.publish 'popularCourses', ->
# find all courses
courses = Course.find({}, {sort: {createdAt: -1}}).fetch()
for course in courses
# find each User by course owner
owner = Meteor.users.findOne({_id: course.owner})
# overwrite the ownerId with the desired username
course.owner = owner.username
return courses
If I turn autopublish on, it works. The image shows the current status (autopublish off). As seen in the image, the author's name is only rendered if the current user is the same as the author.
--
A friend suggested the following:
https://gist.github.com/wiesson/1fd93d77ed9df353b7ab
"The basic idea was to attach the username to the course before providing the data with the publish method. However, as described in Meteor MongoDB find / fetch issues, the publish method should return a curser and not an array of objects.”
Any ideas how to solve that? Putting the owner usernames in an array? If so, how?
P.S.: Sourecode can be found here (currently, has more commits than the deployed version):
https://github.com/Crowducate/crowducate.me
Thanks a lot.

There are a couple of ways you can accomplish this join. A few notes before before we begin:
As I explained in the answer to this question, sorting in the publish function has no affect on the order of documents on the client.
Using the plural form in a collection name is the accepted standard. Course just looks odd when the collection contains courses.
This question is fundamentally about joins, so I'd recommend reading Reactive Joins In Meteor.
Server Transform
The literal answer to your question is to transform the documents on the server like so:
Meteor.publish 'popularCourses', ->
transform = (fields) ->
if fields.owner
username = Meteor.users.findOne(fields.owner)?.username
fields.owner = username
fields
handle = Course.find().observeChanges
added: (id, fields) =>
#added 'course', id, transform fields
changed: (id, fields) =>
#changed 'course', id, transform fields
removed: (id) =>
#removed 'course', id
#ready()
#onStop ->
handle.stop()
Advantages
All of the work is done on the server, so the client can just use owner as if it was a username.
Disadvantages
Using observeChanges is probably more computational work than a simple join deserves.
If you publish courses somewhere else, it's entirely likely that owner will be overwritten when the documents are merged on the client. This can be countered by appending a field like ownerUsername but that would also require a more expensive observe.
This isn't helpful if you actually need the owner id somewhere on the client.
It isn't reactive if the username changes (probably rare but figured I'd point that out).
Non-Reactive Publish + Client Join
You could implement the publish like this:
CoffeeScript
Meteor.publish 'popularCourses', ->
courseCursor = Course.find()
userIds = courseCursor.map (c) -> c.owner
userCursor = Meteor.users.find {_id: $in: userIds}, {fields: username: 1}
[courseCursor, userCursor]
JavaScript
Meteor.publish('popularCourses', function() {
var courseCursor = Course.find();
var userIds = courseCursor.map(function(c) {return c.owner;});
var userCursor = Meteor.users.find(
{_id: {$in: userIds}},
{fields: {username: 1}
});
return [courseCursor, userCursor];
});
Note that I'm being careful to only publish the username and _id from userCursor (you don't want to publish the hashed password and session data by accident). Then you can join the two collections on the client like this:
Template.myTemplate.helpers
courses: ->
Course.find().map (c) ->
c.owner = Meteor.users.findOne(c.owner)?.username
c
Advantages
Computationally light-weight and simple publish function.
Reactive if the username changes.
Disadvantages
Not reactive if the owner changes.
You'll need to do the join on the client. An interesting alternative is to use something like Collection Helpers.
Finally, I'll point out that you can use a package to do a fully reactive join. However, unless the owner (or owner's username) is changing a lot then this is probably overkill.

A simple solution would be to just publish both popularCourses and owners and add the owner to each course on the client (with the exact same code you have written on the publication).