I've got two tables,
Places
Products
What I am trying to achieve is to find Place By multiple products.
So basically.
find PLACE where product ids in [1,4,6]
The thing is that it looks for every place where product id = 1 OR id = 4. I want to find a place that contains all the products. So 3 conditionals must be achieved.
This is how it looks in sequelize
const eq = await Place.findAndCountAll({
include: [
{
model: PlaceProduct,
as: 'products',
where: {
productId: [1,2,5]
}
},
],
}
And it returns places which contain only of the required products.
I want all to be required.
Thanks!
You can check the count of records are exact of product_ids , if yes then thats the record you want , and if is less than that it will be ignored
const product_ids = [1, 2, 5];
const eq = await Place.findAll({
include: [{
model: PlaceProduct,
attributes : [],
as: 'products',
where: {
productId: product_ids // <----- product_ids
}
}, ],
group: ['place.id'], // <--- Not sure but , you might need to add place_product.id also here
having: {
$and: [
db.sequelize.where(db.sequelize.fn('count', db.sequelize.col('place.id')), product_ids.length), // <----- product_ids
]
}
}
I am interested in optimizing a "pagination" solution I'm working on with MongoDB. My problem is straight forward. I usually limit the number of documents returned using the limit() functionality. This forces me to issue a redundant query without the limit() function in order for me to also capture the total number of documents in the query so I can pass to that to the client letting them know they'll have to issue an additional request(s) to retrieve the rest of the documents.
Is there a way to condense this into 1 query? Get the total number of documents but at the same time only retrieve a subset using limit()? Is there a different way to think about this problem than I am approaching it?
Mongodb 3.4 has introduced $facet aggregation
which processes multiple aggregation pipelines within a single stage
on the same set of input documents.
Using $facet and $group you can find documents with $limit and can get total count.
You can use below aggregation in mongodb 3.4
db.collection.aggregate([
{ "$facet": {
"totalData": [
{ "$match": { }},
{ "$skip": 10 },
{ "$limit": 10 }
],
"totalCount": [
{ "$group": {
"_id": null,
"count": { "$sum": 1 }
}}
]
}}
])
Even you can use $count aggregation which has been introduced in mongodb 3.6.
You can use below aggregation in mongodb 3.6
db.collection.aggregate([
{ "$facet": {
"totalData": [
{ "$match": { }},
{ "$skip": 10 },
{ "$limit": 10 }
],
"totalCount": [
{ "$count": "count" }
]
}}
])
No, there is no other way. Two queries - one for count - one with limit. Or you have to use a different database. Apache Solr for instance works like you want. Every query there is limited and returns totalCount.
MongoDB allows you to use cursor.count() even when you pass limit() or skip().
Lets say you have a db.collection with 10 items.
You can do:
async function getQuery() {
let query = await db.collection.find({}).skip(5).limit(5); // returns last 5 items in db
let countTotal = await query.count() // returns 10-- will not take `skip` or `limit` into consideration
let countWithConstraints = await query.count(true) // returns 5 -- will take into consideration `skip` and `limit`
return { query, countTotal }
}
Here's how to do this with MongoDB 3.4+ (with Mongoose) using $facets. This examples returns a $count based on the documents after they have been matched.
const facetedPipeline = [{
"$match": { "dateCreated": { $gte: new Date('2021-01-01') } },
"$project": { 'exclude.some.field': 0 },
},
{
"$facet": {
"data": [
{ "$skip": 10 },
{ "$limit": 10 }
],
"pagination": [
{ "$count": "total" }
]
}
}
];
const results = await Model.aggregate(facetedPipeline);
This pattern is useful for getting pagination information to return from a REST API.
Reference: MongoDB $facet
Times have changed, and I believe you can achieve what the OP is asking by using aggregation with $sort, $group and $project. For my system, I needed to also grab some user info from my users collection. Hopefully this can answer any questions around that as well. Below is an aggregation pipe. The last three objects (sort, group and project) are what handle getting the total count, then providing pagination capabilities.
db.posts.aggregate([
{ $match: { public: true },
{ $lookup: {
from: 'users',
localField: 'userId',
foreignField: 'userId',
as: 'userInfo'
} },
{ $project: {
postId: 1,
title: 1,
description: 1
updated: 1,
userInfo: {
$let: {
vars: {
firstUser: {
$arrayElemAt: ['$userInfo', 0]
}
},
in: {
username: '$$firstUser.username'
}
}
}
} },
{ $sort: { updated: -1 } },
{ $group: {
_id: null,
postCount: { $sum: 1 },
posts: {
$push: '$$ROOT'
}
} },
{ $project: {
_id: 0,
postCount: 1,
posts: {
$slice: [
'$posts',
currentPage ? (currentPage - 1) * RESULTS_PER_PAGE : 0,
RESULTS_PER_PAGE
]
}
} }
])
there is a way in Mongodb 3.4: $facet
you can do
db.collection.aggregate([
{
$facet: {
data: [{ $match: {} }],
total: { $count: 'total' }
}
}
])
then you will be able to run two aggregate at the same time
By default, the count() method ignores the effects of the
cursor.skip() and cursor.limit() (MongoDB docs)
As the count method excludes the effects of limit and skip, you can use cursor.count() to get the total count
const cursor = await database.collection(collectionName).find(query).skip(offset).limit(limit)
return {
data: await cursor.toArray(),
count: await cursor.count() // this will give count of all the documents before .skip() and limit()
};
It all depends on the pagination experience you need as to whether or not you need to do two queries.
Do you need to list every single page or even a range of pages? Does anyone even go to page 1051 - conceptually what does that actually mean?
Theres been lots of UX on patterns of pagination - Avoid the pains of pagination covers various types of pagination and their scenarios and many don't need a count query to know if theres a next page. For example if you display 10 items on a page and you limit to 13 - you'll know if theres another page..
MongoDB has introduced a new method for getting only the count of the documents matching a given query and it goes as follows:
const result = await db.collection('foo').count({name: 'bar'});
console.log('result:', result) // prints the matching doc count
Recipe for usage in pagination:
const query = {name: 'bar'};
const skip = (pageNo - 1) * pageSize; // assuming pageNo starts from 1
const limit = pageSize;
const [listResult, countResult] = await Promise.all([
db.collection('foo')
.find(query)
.skip(skip)
.limit(limit),
db.collection('foo').count(query)
])
return {
totalCount: countResult,
list: listResult
}
For more details on db.collection.count visit this page
It is possible to get the total result size without the effect of limit() using count() as answered here:
Limiting results in MongoDB but still getting the full count?
According to the documentation you can even control whether limit/pagination is taken into account when calling count():
https://docs.mongodb.com/manual/reference/method/cursor.count/#cursor.count
Edit: in contrast to what is written elsewhere - the docs clearly state that "The operation does not perform the query but instead counts the results that would be returned by the query". Which - from my understanding - means that only one query is executed.
Example:
> db.createCollection("test")
{ "ok" : 1 }
> db.test.insert([{name: "first"}, {name: "second"}, {name: "third"},
{name: "forth"}, {name: "fifth"}])
BulkWriteResult({
"writeErrors" : [ ],
"writeConcernErrors" : [ ],
"nInserted" : 5,
"nUpserted" : 0,
"nMatched" : 0,
"nModified" : 0,
"nRemoved" : 0,
"upserted" : [ ]
})
> db.test.find()
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c8"), "name" : "forth" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c9"), "name" : "fifth" }
> db.test.count()
5
> var result = db.test.find().limit(3)
> result
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }
> result.count()
5 (total result size of the query without limit)
> result.count(1)
3 (result size with limit(3) taken into account)
Try as bellow:
cursor.count(false, function(err, total){ console.log("total", total) })
core.db.users.find(query, {}, {skip:0, limit:1}, function(err, cursor){
if(err)
return callback(err);
cursor.toArray(function(err, items){
if(err)
return callback(err);
cursor.count(false, function(err, total){
if(err)
return callback(err);
console.log("cursor", total)
callback(null, {items: items, total:total})
})
})
})
Thought of providing a caution while using the aggregate for the pagenation. Its better to use two queries for this if the API is used frequently to fetch data by the users. This is atleast 50 times faster than getting the data using aggregate on a production server when more users are accessing the system online. The aggregate and $facet are more suited for Dashboard , reports and cron jobs that are called less frequently.
We can do it using 2 query.
const limit = parseInt(req.query.limit || 50, 10);
let page = parseInt(req.query.page || 0, 10);
if (page > 0) { page = page - 1}
let doc = await req.db.collection('bookings').find().sort( { _id: -1 }).skip(page).limit(limit).toArray();
let count = await req.db.collection('bookings').find().count();
res.json({data: [...doc], count: count});
I took the two queries approach, and the following code has been taken straight out of a project I'm working on, using MongoDB Atlas and a full-text search index:
return new Promise( async (resolve, reject) => {
try {
const search = {
$search: {
index: 'assets',
compound: {
should: [{
text: {
query: args.phraseToSearch,
path: [
'title', 'note'
]
}
}]
}
}
}
const project = {
$project: {
_id: 0,
id: '$_id',
userId: 1,
title: 1,
note: 1,
score: {
$meta: 'searchScore'
}
}
}
const match = {
$match: {
userId: args.userId
}
}
const skip = {
$skip: args.skip
}
const limit = {
$limit: args.first
}
const group = {
$group: {
_id: null,
count: { $sum: 1 }
}
}
const searchAllAssets = await Models.Assets.schema.aggregate([
search, project, match, skip, limit
])
const [ totalNumberOfAssets ] = await Models.Assets.schema.aggregate([
search, project, match, group
])
return await resolve({
searchAllAssets: searchAllAssets,
totalNumberOfAssets: totalNumberOfAssets.count
})
} catch (exception) {
return reject(new Error(exception))
}
})
I had the same problem and came across this question. The correct solution to this problem is posted here.
You can do this in one query. First you run a count and within that run the limit() function.
In Node.js and Express.js, you will have to use it like this to be able to use the "count" function along with the toArray's "result".
var curFind = db.collection('tasks').find({query});
Then you can run two functions after it like this (one nested in the other)
curFind.count(function (e, count) {
// Use count here
curFind.skip(0).limit(10).toArray(function(err, result) {
// Use result here and count here
});
});
I have 2 collections.
ItemList = new Mongo.Collection('items');
BorrowerDetails = new Mongo.Collection('borrow');
ItemList.insert({
brand: "brand-Name",
type: "brand-Type",
._id: id
});
BorrowerDetails.insert({
key: "ItemList.id", //equals to .id of the ItemList Collection
name : "borrowerName"
});
Question !
How can i retrieve records from the BorrowerDetails Collection based on a certain type from the ItemList Collection.
ex. Retrieve all records from the BorrowerDetails Collection where key is equals to the id of a record on the ItemList Collection whose type is equals to "Desktop".
return BorrowerDetails.find(
{ key :
ItemList.find(
{ type : 'Desktop' },
{ fields: {'_id':1 } }
)
}
); //error!
Note that I don't have nodejs right now in my laptop, so might have several errors since I am not able to test it.
First, Create a publication file (eg. server\publications\borrowPub.js). inside the file, you should create a publication method. The logic here is simple, get the itemid array first, and then pass it as parameter in Mongo select $in.
Meteor.publish('queryBorrowerTypeDesktop', function(criteria)
{
var itemArr = ItemList.find( { type : 'Desktop' },
{ fields: {'_id':1 } });
if(itemArr){
//itemArr found, so do a select in using the array of item id we retrieved earlier
var borrowers = BorrowerDetails.find({ key: { $in: itemArr } });
return borrowers;
}else{
//found nothing- so return nothing
return []
}
});
Second, in the router :
Router.route('/test', {
name: 'test',
action: function()
{
//change accordingly.
},
waitOn: function()
{//subscribe from publisher that we just created...
return [
Meteor.subscribe('queryBorrowerTypeDesktop')
];
},
data: function() {
if(Meteor.userId())
{
// also include the sorting and limit so your page will not crash. change accordingly.
return {
borrowerDetails: BorrowerDetails.find({},{sort: {name: -1}, limit: 30}),
}
}
}
});
Note that in data, BorrowerDetails.find() does not need to filter by anything because it has been filtered during the subscribe, which has been cached in MiniMongo in your browser.
I'm building a simple messaging app with Meteor. The section I'm struggling with in unread messages. I would like to return a list, showing the username (I'm not concerned about this, please don't focus on this aspect, around reactive joins/ composites etc ) and the latest message from that user What I need to return therefore, in the publish function below, is the newest unread messages, BUT obviously only one from each unique user id.
to do this im trying to manipulate the results of a find query in my publish method, but I'm unclear as to how to manipulate the document set without breaking the reactivity as I've shown in currently in the code below, this is what i've got so far :
Meteor.publish('unreadmessages', function() {
if (!this.userId) {
throw new Meteor.Error('denied', 'not-authorized');
}
var messageQuery, messages, userGroupQuery, userGroups;
var self = this;
var user = Meteor.users.findOne(self.userId);
var userIdArr = [self.userId]; // for use where queries require an array
var contacts = user.contacts;
// get groups
userGroupQuery = Groups.find({
$or : [
{ owner : self.userId },
{ members : self.userId }
]
}, { // Projection to only return the _id field
fields : { _id:1 }
}
);
userGroups = _.pluck(userGroupQuery.fetch(), '_id'); // create an array of id's
messages = Messages.find({
$or : [
{
$and : [
{ participant : self.userId },
{ userId : { $in : contacts } },
{ readBy : { $nin : userIdArr } }
]
},
{
$and : [
{ groupId : { $in : userGroups } },
{ readBy : { $nin : userIdArr } }
]
},
]
});
// TODO : also handle groups here
uniqueMessages = _.uniq(messages.fetch(), function(msg) {
return msg.userId;
});
return uniqueMessages; // obviously an array and not a cursor - meteor errors out.
});
i realize that my underscore function is of course working with, and indeed returning, an array rather than the reactive cursor i need. i know one solution would be to simply pluck the message ids and then run another .find on messages, but is there another / better / more efficient / more natural way to return a cursor with the result set i'm looking for?
You can use observeChanges and makes this reactive. On the added, you can add fields. I'm using this amazing package: meteor-publish-composite, It saves you time.
Use pagination, otherwise you won't enjoy the performance.
The following code does publish a reactive cursor and is a viable solution, but I suppose the crux of my question is if there is a better way to manipulate the result set of of the second last Messages.find to still publish a reactive cursor (I'm thinking along the lines o the cursor.forEach or .map but I'm not sure how to approach this).
Basically - is there a better way to do this :
Meteor.publish('unreads', function() {
if (!this.userId) {
throw new Meteor.Error('denied', 'not-authorized');
}
// setup some vars
var messageQuery, messages, userGroupQuery, userGroups, uniqeMsgIds;
var self = this;
var user = Meteor.users.findOne(self.userId);
var userIdArr = [self.userId]; // for use where queries require an array
var contacts = user.contacts;
// get groups
userGroupQuery = Groups.find({
$or : [
{ owner : self.userId },
{ members : self.userId }
]
}, { // Projection to only return the _id field
fields : { _id:1 }
}
);
// create an array of group id's that belong to the user.
userGroups = _.pluck(userGroupQuery.fetch(), '_id');
messages = Messages.find({
$or : [
{ // unread direct messages
$and : [
{ participant : self.userId },
{ userId : { $in : contacts } },
{ readBy : { $nin : userIdArr } }
]
},
{ // unread group messages
$and : [
{ groupId : { $in : userGroups } },
{ readBy : { $nin : userIdArr } }
]
},
]
}, { sort : { // put newest messages first
time : -1
}
});
// returns an array of unique documents based on userId or groupId
uniqueMessages = _.uniq(messages.fetch(), function(msg) {
if (msg.groupId) {
return msg.groupId;
}
return msg.userId;
});
// Get the id's of all the latest unread messages (one per user or group)
uniqeMsgIds = _.pluck(uniqueMessages, '_id');
// finally publish a reactive cursor containing one unread message(latest) for each user/group
return Messages.find({
_id : { $in : uniqeMsgIds };
});
});
Heres the final working code. This does return a cursor as basically what I'm doing is taking the results from multiple queries / modifications and pumping a set of id's into the final .find query.
Meteor.publish('unreads', function() {
if (!this.userId) {
throw new Meteor.Error('denied', 'not-authorized');
}
// setup some vars
var messageQuery,
messages,
userGroupQuery,
userGroups,
uniqeMsgIds;
var self = this;
var user = Meteor.users.findOne(self.userId);
var userIdArr = [self.userId]; // for use where queries require an array
var contacts = user.contacts;
// get groups
userGroupQuery = Groups.find({
$or : [
{ owner : self.userId },
{ members : self.userId }
]
}, { // Projection to only return the _id field
fields : { _id:1 }
}
);
// create an array of group id's that belong to the user.
userGroups = _.pluck(userGroupQuery.fetch(), '_id');
messages = Messages.find({
$or : [
{ // unread direct messages
$and : [
{ participant : self.userId },
{ userId : { $in : contacts } },
{ readBy : { $nin : userIdArr } }
]
},
{ // unread group messages
$and : [
{ groupId : { $in : userGroups } },
{ readBy : { $nin : userIdArr } }
]
},
]
}, { sort : { // put newest messages first
time : -1
}
});
// returns an array of unique documents based on userId or groupId
uniqueMessages = _.uniq(messages.fetch(), function(msg) {
if (msg.groupId) {
return msg.groupId;
}
return msg.userId;
});
/* Get the id's of all the latest unread messages
(one per user or group) */
uniqeMsgIds = _.pluck(uniqueMessages, '_id');
/* finally publish a reactive cursor containing
one unread message(latest) for each user/group */
return Messages.find({
_id : { $in : uniqeMsgIds }
});
});