We Keep Coding

Construct objects for client in Firebase Cloud Functions

I'm working on a simple registration system using Firebase as a backend. I am successfully authenticating users and writing to the database. I have an index of courses and users with the following structure:
{
courses: { // index all the courses available
key1: {
title: "Course 1",
desc: "This is a description string.",
date: { 2018-01-01 12:00:00Z }
members: {
user1: true
...
}
},
key2 { ... },
},
users: { // track individual user registrations
user1: {
key1: true,
...
},
user2: { ... }
}
}
I have a cloud function that watches for the user to add a course and it builds an array with the corresponding courseId that will look at the courses node to return the appropriate items.
exports.listenForUserClasses = functions.database.ref('users/{userId}')
.onWrite(event => {
var userCourses = [];
var ref = functions.database.ref('users/{userId}');
for(var i=0; i<ref.length; i++) {
userCourses.push(ref[i])
}
console.log(userCourses); // an array of ids under the user's node
});
So, my question has two parts:
How can I build the updated object when the page is loaded?
How do I return the function to the client script?

Question 1: From the client side you want to get the reference to the database path. Then you want to call the child_added event. Keep it in-memory, this will be called whenever one is add then you can update your UI.
var ref = db.ref("path/to/courses");
ref.on("child_added", function(snapshot, prevChildKey) {
var newClass = snapshot.val();
});
If you are completely refreshing the page then you can always grab the data again from the database path by using the value option and calling once
Questions 2: You don't. This is considered an asynchronous function. If you wanted a response from the function then you would setup an HTTP trigger and wait for the response from that function.

Meteor js: The page taking too much time to query with mongodb if records are around 100000

I am working on meteor js, there will be huge data in mongodb database. For now it is around 50000 messages in database. I am providing the code that I am currently using. For some reason the application is taking too much time to render or load the data from database. Also one more thing, if I am doing any activity,e.g. just like the messages the app fetches the messages again from database.
Template.messages.helper({
linkMessages() {
var ids = _.pluck(Meteor.user().subscription, '_id');
var messages = Messages.find({ $or: [{ feedId: { $exists: false }, link: { $exists: true } }, { feedId: { $in: ids }, link: { $exists: true } }] }, { sort: { timestamp: 1 }, limit: Session.get("linkMessageLimit") }).fetch();
return messages;
}
})
calling publication in oncreate method
Template.roomView.onCreated(function() {
const self = this;
Deps.autorun(function() {
Meteor.subscribe('messages', Session.get('currentRoom'), Session.get('textMessageLimit'), {
onReady() {
isReady.messages = true;
if (scroll.needScroll) {
scroll.needScroll = false;
if (scroll.previousMessage) {
Client.scrollToMessageText(scroll.previousMessage);
}
} else {
Meteor.setTimeout(function() {
Client.scrollChatToBottomMsg();
}, 1000)
}
}
});
});
});`
The publication function on server:
Meteor.publish('messages', function(roomId, limit) {
check(roomId, String);
check(limit, Match.Integer);
let query = { $or: [{ link: {$exists: false} }, { feedId: { $exists: false } }] };
const thresholdMessage = Messages.findOne(query, { skip: limit, sort: { timestamp: 1 } });
if (thresholdMessage && thresholdMessage.timestamp) {
query.timestamp = { $gt: thresholdMessage.timestamp };
}
return Messages.find(query, { sort: { timestamp: -1 } });
});

It is not a good practice to allow mini-mongodb to get populated with such a huge data. Though Meteor JS is good at this too, still it will take some amount of time taking into consideration the network traffic, bandwidth etc.
Irrespective of whether it is unlimited scroll or simple pagination I would suggest you to use pagination. I have already got it accepted and it works like charm, here is the answer and entire code for pagination.
My pagination solution is server specific, so it performs good. Collection publish is limited to the limit provided from subscription.
NOTE: There is yet no such proper full fledged solution for table with search and pagination and much more facility which makes it very flexible as per our need. I suggest to create your own.
MORE INSIGHTS:
https://www.quora.com/Does-Meteor-work-well-with-large-datasets
https://forums.meteor.com/t/very-big-data-collection-in-mongodb-how-to-fetch-in-meteor-js/6571/7
https://projectricochet.com/blog/top-10-meteor-performance-problems

How to improve or avoid find / fetch cycle in meteor's publication?

TL;DR:
Chat is one collection. ChatMess another one that has messages refering to a Chat's _id. How do I get the last messages from a list of chats with the less computation possible ? Here, find / fetch cycle in a loop is way too heavy and long.
I have this publication that is used to return a set of cursor to the user :
The chats sessions he takes part in (from Chat collection)
The last message from each of the chat session referenced in the first cursor (from ChatMess collection)
Currently, the logic is to :
Get the list of chat sessions from the user profile
Find the Chat sessions and loop through it
In the loop, I findOne the last message from this chat session and store its _id in an array. In addition, I store all the other users _ids.
Then, I find the messages which _id match the ones in my array.
Here is my main problem :
Isn't there a way more faster way to get the last messages from each of my chat session ? With that algo, I easily reach the 8000ms of response time, which is a way too heavy computation time, as much of this time is spent to find / fetch the chat messages's _id (cf linked screen from Kadira).
Meteor.publish("publishNewChat", function() {
this.unblock();
// we get a list of chat _id
let chatIdList = _get_all_the_user_chats_ids(this.userId);
if (!chatList)
return ;
// get the chat sessions objects
let chats_cursor = Modules.both.queryGet({
type : 'chat',
method : 'find',
query : { _id: { $in: chatIdList } },
projection : { sort: { _id: 1 }, limit : 1000 }
});
let array_of_fetched_chats = chats_cursor.fetch();
let chat_ids = [];
// and here we loop through the chat documents in order to get the last message that's been attached to each of them
array_of_fetched_chats.forEach(function(e) {
let lastMess = Modules.both.queryGet({
type : 'chatMess',
method : 'findOne',
query : { chatId: e._id },
projection : { sort: { date: -1 } }
});
if (lastMess)
chat_ids.push(lastMess._id);
});
return ([
chats_cursor,
Modules.both.queryGet({
type : 'chatMess',
method : 'find',
query : { _id: { $in: chat_ids } },
projection : { sort: { date: -1 }, limit: 1000 }
})
]);
});
Finally, it also add latence to all my DDP request that follows. I currently use a this.unblock() to avoid that, but I'd prefer not to use it here.
FYI, I have another publish that is updated each time the client change his current active chat session : on the client, routing to a new chat add its _id in a reactive array that update my getChatMess subscription in order to get on the client the messages from every chats the user visited in this since he connected. The goal is obviously to spare the server the sending of every message from every chat session the user have visited in his life.
Unfortunately, I lack ideas to improve that algo without breaking all my chat logic :S. Have you any idea ? How would you do ?
Thanks you.
EDIT: here is a screen from kadira that clearly show the problem :

Have you considered using the reywood/publishComposite package?
With this package you can publish related data in the same method without having to do a bunch of logic to get the correct data published.
The below code should get you started:
Meteor.publishComposite("publishNewChat", function() {
return [{
find:function(){
return Users.find({ _id: this.userId },{fields:{"profile.chat":1}});
},
children:[{
find:function(user){ //this function is passed each user returned from the cursor above.
return UserChats.find({userId:user._id},{fields:{blah:1,blah:1}}); //find the user chats using whatever query
},
children:[
//if there are any children of user chats that you need to publish, do so here...
{
find:function(userchat){
return Chats.find({_id:userchat.chatId})
},
children:[
{
find:function(chat){
return ChatMess.find({chatId:chat._id},{ sort: { date: -1 } });
},
children:[
{
find:function(chatMess){
var uids = _.without(chatMess.participants, this.userId);
return Users.find({_id:{$in:uids}});
}
}
]
}
]
}
]
},
]
}]
This will publish the cursors for all of the documents related to each of the parent documents. It is pretty fast, I use this package on a production platform high traffic and large datasets with no problems. On the client you could then query the documents as normal to get the ones you need to display.
Something like:
Users.findOne({_id:Meteor.userId()});
UserChats.find({userId:Meteor.userId()});
etc...

Here is a solution I developped :
Meteor.publish("publishNewChat", function() {
this.unblock();
let user = Modules.both.queryGet({
type : 'users',
method : 'findOne',
query : { _id: this.userId },
projection : { fields: { "profile.chat": true } }
});
let thisUserschats = tryReach(user, "profile", "chat").value;
if (!thisUserschats)
return ;
thisUserschats = thisUserschats.map(function(e) { return (e.chatId); });
let chats = Modules.both.queryGet({
type : 'chat',
method : 'find',
query : { _id: { $in: thisUserschats } },
projection : { sort : { _id: 1 },
limit : 1000
}
});
let chatArray = chats.fetch(),
uids = cmid = [];
let messages_id_list = [],
i = chatArray.length;
let _parallelQuery = index => {
Meteor.setTimeout(function () {
let tmp = Modules.both.queryGet({
type : 'chatMess',
method : 'find',
query : { chatId: chatArray[index]._id },
projection: { limit: 1, sort: { date: -1 } }
});
tmp.forEach(doc => {
messages_id_list.push((doc && doc._id) ? doc._id : null);
});
}, 1);
}
while (--i >= 0)
_parallelQuery(i);
let cursors = {
chats : chats,
chatMessages : null
}
let interval = Meteor.setInterval(function () {
if (messages_id_list.length === chatArray.length)
{
Meteor.clearInterval(interval);
cursors.chatMessages = Modules.both.queryGet({
type : 'chatMess',
method : 'find',
query : { _id: { $in: messages_id_list } },
projection : { sort: { date: -1 }, limit: 1000 }
});
cursors.chats.observeChanges({
// ...
});
cursors.chatMessages.observeChanges({
// ...
});
self.ready();
self.onStop(() => subHandle.stop(); );
}
}, 10);
});
I used async function with Meteor.setTimeout to parallelize the queries and save an index refering to a chat _id to look for. Then, when a query is finished, I add the last message to an array. With a Meteor.setInterval, I check the array length to know when all the queries are done. Then, as I can't return cursors anymore, I use the Meteor publication low level API to handle the publishing of the documents.
FYI : in a first attempt, I was using 'findOne' in my _parallelQueries, which divided my computation time by 2/3. But then, thanks to a friend, I tried the cursor.foreach() function, which allowed me to divide the computation time by 2 again !
In production, the benchmarks allowed me to go from a 7/8 second response time to an average response time of 1.6 second :)
Hope this will be usefull to you people ! :)

How to add additional fields to user documents (not in profile)?

I know that the classic way to add data to user collection is in profile array, but according to this document, it is not the best way to store data.
Is there an alternative to that, for example to create a field in the root of user collection at the same level with default fields (_id, username, etc.)?

There is nothing wrong per-se with the profile field, other than the fact that a users can (currently) directly update their own profile by default.
I don't find this behavior desired, as a user could store arbitrary data in the profile.
This may become a real security risk if the developer uses that field as a source of authority; for example, stores the user's groups or roles in it.
In this case, users could set their own permissions and roles.
This is caused by this code:
users.allow({
// clients can modify the profile field of their own document, and
// nothing else.
update: function (userId, user, fields, modifier) {
// make sure it is our record
if (user._id !== userId)
return false;
// user can only modify the 'profile' field. sets to multiple
// sub-keys (eg profile.foo and profile.bar) are merged into entry
// in the fields list.
if (fields.length !== 1 || fields[0] !== 'profile')
return false;
return true;
}
});
The first thing to do is to restrict writes to it:
Meteor.users.deny({
update() {
return true;
}
});
It could then be updated using methods and other authorized code.
If you add your own fields and want to publish them to the currently logged-in user, you can do so by using an automatic publication:
Meteor.publish(null, function () {
if (this.userId) {
return Meteor.users.find({
_id: this.userId
}, {
fields: {
yourCustomField1: 1,
yourCustomField2: 1
}
});
} else {
return this.ready();
}
});
Meteor.users is just a normal Mongo.Collection, so modifying it is done just like any other Collection. There is also the creation hook, Accounts.onCreateUser which allows you to add custom data to the user object when it is first created, as mentioned in #MatthiasEckhart's answer.

You could add extra fields to user documents via the accountsServer.onCreateUser(func) function.
For example:
if (Meteor.isServer) {
Accounts.onCreateUser(function(options, user) {
_.extend(user, {
myValue: "value",
myArray: [],
myObject: {
key: "value"
}
});
});
}
Please note: By default, the following Meteor.users fields are published to the client username, emails and profile. As a consequence, you need to publish any additional fields.
For instance:
if (Meteor.isServer) {
Meteor.publish("user", function() {
if (this.userId) return Meteor.users.find({
_id: this.userId
}, {
fields: {
'myValue': 1,
'myArray': 1,
'myObject': 1
}
});
else this.ready();
});
}
if (Meteor.isClient) {
Meteor.subscribe("user");
}

Migrate simple List Array key to another key with an extra attribute in MongoDB

Sorry if I'm not getting the terminology right. Here's what I have currently my MongoDB user docs db.users:
"liked" : [
"EBMKgrD4DjZxkxvfY",
"WJzAEF5EKB5aaHWC7",
"beNdpXhYLnKygD3yd",
"RHP3hngma9bhXJQ2g",
"vN7uZ2d6FSfzYJLmm",
"NaqAsFmMmnhqNbqbG",
"EqWEY3qkeJYQscuZJ",
"6wsrFW5pFdnQfoWMs",
"W4NmGXyha8kpnJ2bD",
"8x5NWZiwGq5NWDRZX",
"Qu8CSXveQxdYbyoTa",
"yLLccTvcnZ3D3phAs",
"Kk36iXMHwxXNmgufj",
"dRzdeFAK28aKg3gEX",
"27etCj4zbrKhFWzGS",
"Hk2YpqgwRM4QCgsLv",
"BJwYWumwkc8XhMMYn",
"5CeN95hYZNK5uzR9o"
],
And I am trying to migrate them to a new key that also captures the time that a user liked the post
"liked_time" : [
{
"postId" : "5CeN95hYZNK5uzR9o",
"likedAt" : ISODate("2015-09-23T08:05:51.957Z")
}
],
I am wondering if it might be possible to simply do this within the MongoDB Shell with a command that iterates over each user doc and then iterates over the liked array and then updates and $push the new postId and time.
Or would it be better to do this in JavaScript. I am using Meteor.
I almost got it working for individual users. But want to know if I could do all users at once.
var user = Meteor.users.findOne({username:"atestuser"});
var userLiked = user.liked;
userLiked.forEach(function(entry) {
Meteor.users.update({ username: "atestuser" },
{ $push: { liked_times: { postId: entry, likedAt: new Date() }}});
console.log(entry);
});
Still a bit of a newbie to MongoDB obviously......

Here is something i made real quick you should run this on the server side just put it into a file e.g. "migrate.js" in root meteor and run the meteor app
if (Meteor.isServer) {
Meteor.startup(function () {
var users = Meteor.users.find().fetch();
users.forEach(function (doc) {
liked.forEach(function (postId) {
Meteor.users.update(doc._id, { $push: { liked_times: { postId: postId, likedAt: new Date() } } });
});
});
console.log('finished migrating');
});
}
p.s I didn't test it

If this is a one time migration i would do something like this in a one time js script.
Get all users
Iterate over each user
Get all likes
Iterate over them, get likedAt
var liked_times = _.collect(likes, function (likeId) {
return {
'postId' : likeId,
'likedAt': // get post liked time from like id.
}
});
Insert the above in the collection of choice.
Note:
The above example makes use of lodash
I would rather just save likedAt as a timestamp.