Pagination with populated data in mongo - javascript

I'm struggling with a problem of fetching data from DB and applying "pagination" functionality. My case is that I have webstore with multiple users. Every user can publish their profile and create their products. On the website, I'd like to show the products of users who did publish their profiles.
Currently, this is my code:
Product.find({ active: true, deleted: false })
.sort({ created: -1 })
.populate({
path: 'user',
match: { isPublished: true },
select: 'details slug name',
})
.exec((error, products) => {
if (error) {
return response.status(500).send({ error: 'There was an error.' });
}
if (products === null) {
return response.status(404).send({ error: 'Products not found.' });
}
const productsWithPublishedTrainerProfile = _.filter(products, product => !_.isNull(product.user));
return response.status(200).send(_.map(productsWithPublishedTrainerProfile, product => mapProduct(product)));
});
It works - It fetches all the active products, populate the user who created the product and if their profile is published then the product is not filtered out.
Now I need to implement "pagination" functionality which means that I want to use .skip() and .limit() functions but this skip/limit will be run on all the products (not only the filtered ones.) what means that the endpoint won't be returning the exact number of products (set in the limit() function.
I wonder if there is any solution for that. I read about .aggregate with setting $lookup, $skip and $limit but after hours of fighting with it I couldn't make it work.
I will really appreciate any help ✌️

Related

MongoDB update many using the values of the user

I've written a function to execute hourly which looks up a user and finds some values and then pushes those values into a history collection that records the hourly updated values. I've written this so far as a test just finding a user by their ID but now I need to roll this out to my entire database of 50,000+ users.
From what I've read using updateMany is a lot more performant but I'm not entirely sure how to retrieve the document detail of the record that is being updated at the time.
Here is my code so far, which you can see I'm first looking up the user and then grabbing their valuation details which I'd like to then push into a history collection.
exports.updateUserValuationHistoric = () => {
User.find({ _id: "609961fdd989613914ef7216" })
.populate('UserValuationHistory')
.exec((err, userDoc) => {
if (err){
console.log('[ERROR]: ', err)
}
const updatedValuationHistory = {
totalValuation: userDoc[0].valuation.totalValuation,
comicsValuation: userDoc[0].valuation.comicsValuation,
collectiblesValuation: userDoc[0].valuation.collectiblesValuation,
omiValuation: userDoc[0].valuation.omiValuation
}
UserValuationHistory.findOneAndUpdate(
{ user: userDoc[0]._id },
{ $push: {
'history': updatedValuationHistory
}},
{upsert: true, new: true}
)
.exec((error, updated) => {
if (error){
console.log('[ERROR]: Unable to update the user valuation history.')
} else {
console.log('[SUCCESS]: User historic valuation has been updated.')
}
})
})
}
Any help is greatly appreciated!
User model:
https://pastebin.com/7MWBVHf3
Historic model:
https://pastebin.com/nkTGztJY

Mongoose - Find by array of ids, but retrieve limited results (1 for each id)

i have my model called "Conversations", and my model "Messages", right now i want to retrieve all conversations with the last Message attached (only 1 message per conversation), so i filtered the conversationids and i queried the messages, but i'm not able to get this messages (last messages) for each conversation, thanks in advance.
let conversations = await ConversationModel.find({});
const conversationIds = conversations.map(conversation => conversation._id)
// ConversationIds is basically ["conversation1", "conversation2", "conversation3"]
// Te problem is here, i want to attach the las message for each conversation, if i put limit(1)
// i will get 1 record for all query, but i want the last message record for each conversation.
MessageModel.find({ _id: { "$in" : conversationIds} }, ...);
From information gathered in comments; This is possible to achieve in a case where MessageModel documents contain a time-stamp to identify latest of them.
Idea: Filter messages based on conversationIds via $match, sort them by timestamp for the next stage where $group on conversation reference (lets say conversation_id) and pick latest of them by $first accumulator.
Aggregation Query: playground link
db.collection.aggregate([
{
$match: {
conversation_id: {
$in: conversationIds
}
}
},
{
$sort: {
timestamp: -1
}
},
{
$group: {
_id: "$conversation_id",
latest_doc: {
$first: "$$ROOT"
}
}
}
]);

Query MongoDB to display only one section of data

I have created a MongoDB database and have it hooked up to a MEAN stack web application I have created. I have it able to display the most recent set of data put into it rather than all of the data but now I want to take that down to display only 1 section of that data, as there are currently 50 different sections.
I am using post.find and then sorting the data to show only the most recent record from the DB but I am struggling to break it down to show only one part of that data. Current data: https://imgur.com/a/cwKTCEa.
As you can see the data is grouped by "S0" and then data follows, then there is "S1" etc... In essence what I want is for just "S0" to be displayed when queried.
exports.list = function (req, res) {
Post.find().sort({ _id: -1 }).limit(1)
.then(function (posts) {
return res.status(200).json({
status: 200,
data: posts,
message: 'Success'
})
})
.catch(function (err) {
return res.status(400).json({
status: 400,
message: err.message
});
});
}
I believe I need to add to the find query but I am not sure how to specify that I would only like to see "S0" rather than S0-S49.
Thank you
If you want to return only specific fields from query you should use Project Fields:
Post.find({}, {"s0": 1}).sort({ _id: -1 }).limit(1)
Here is the docs Project Fields

Firebase for getting unread posts

I'm creating an app with a "Card stack" similar to Tinder, with a Firebase Realtime DB backend. Each card will be a new unread post, if the user runs out of new posts they will run out of cards. However I don't know the best way to structure the data for this. Could I store the ID of the read posts under the user, then as I watch the Posts feed I could filter out read posts client side?
That seems a bit messy and not a very good option performance wise. Are there better options?
EDIT: Rough code of what I'm thinking:
Data Example
posts:
"-KibasdkjbSAdASd": {
title: 'New Post',
body: {
...
}
},
"-KisadBVsdadSd": {
title: 'New Post 2',
body: {
..
}
}
"-KibaFNQsicaASd": {
title: 'New Post 3',
body: {
...
}
}
users :
"-KisadBVsdadSd": {
name: 'Tom',
readPosts: {
"-KibasdkjbSAdASd": {
title: 'New Post',
body: {
...
}
},
"-KisadBVsdadSd": {
title: 'New Post 2',
body: {
..
}
}
}
}
Code
const rootRef = firebase.database.ref();
const postRef = rootRef.child("posts");
const readPostRef = rootRef.child("users/"+uid+"/readPosts");
let readPosts= [];
//Get initial list of read posts
readPostRef.once("value", function(snapshot) {
readPosts = Object.keys(snapshot);
});
//Update read posts when added
readPostRef.on("child_added", function(snapshot) {
readPosts = Object.keys(snapshot);
});
//Get list of posts, filtered on read post array
urlRef.on("value", function(snapshot) {
snapshot.forEach(function(child) {
if(!readPosts.includes(child.key)){
//Unread post
}
});
});
It depends on the order in which you show the cards to the user.
If you show them in a predictable order (e.g. from newest to oldest) you can just remember the ID of the last card the user saw.
If you show them in a random or personalized order you might indeed have to track precisely what cards each user has already seen.
I'm not sure why that would be messy or perform badly. So if you want a better option, you'll have to show how you'd implement the messy/slow option.
I'm running into this same design problem. I only see two options, perhaps there are others!
1) Download every post and ignore the ones that have been read. Expensive when you have a lot of posts.
2) Save a copy of every post to every user account and allow them to delete them once they have been read. Expensive when you have a lot of users.

How can I add/remove user specific data in Meteor

A few questions about storing user data in MongoDB. What is the best place in mongo to store user specific data, such as User settings, User photo url, User friends, User events?
In Mongo, user data is stored in:
Meteor
/ Collections
/ users
/ _id
/ profile
/ services
Should I add there a new collections? In a following way:
/ events / _id's
/ friends / _id's
/ messages / _id's
/ settings
How should I publish user's private data and manipulate this collections, to be sure it's save and no one else will modify or have access to private data of another person.
You can add data to the users profile field like this:
Meteor.users.update( id, { $set: { 'profile.friends': someValue } } );
To only publish specific fields you can do something like this:
Meteor.publish( 'users', function () {
return Meteor.users.find( {}, { fields: { 'profile.friends': 1 } } );
});
Hope this helps.
Normalization
"Database normalization is the process of organizing the attributes and tables of a relational database to minimize data redundancy."
MongoDB is a non relational database. This makes normalized data hard to query for. This is why in MongoDB we denormalize data. That makes querying for it easier.
It depends on your use-case. The question is basically when to demormalize. It's mostly a matter of opinion. But objective here are some pros and cons:
Pros to demormalization
It's easier to retrieve data (Due to Mongo not beeing a relational DB)
It performs better if you are always getting the data in bulk
Cons to demormalization
It doesn't scale well for things like user.messages (You can't just publicize some messages)
In your case I'd definitly go for seperate collections for events, friends and messages. Setting can't expand infinitly. So I'd put it into the users collection.
Security
I'd use a publications and allow and deny for this. Let me make an example for Messages:
Collection
Messages = new Mongo.Collection('Messages')
Messages.insert({
sender: Meteor.userId,
recipient: Meteor.users.findOne()._id,
message: 'Hello world!'
})
Publication
Meteor.publish('userMessages', function (limit) {
return Messages.subscribe({
$or: [
{sender: this.userId},
{recipient: this.userId}
]
}, {limit: limit})
})
Allow
function ownsMessage (user, msg) {
return msg.sender === user ? true : false
}
Messages.allow({
insert: function (userId, newDoc) {
!!userId
},
update: function (userId, oldDoc, newDoc) {
if(
ownsMessage(userId, oldDoc) &&
ownsMessage(userId, newDoc)
) return true
return false
},
remove: function () {
return false
}
})
This code is untested, so it might contain small errors

Categories