Related
I'm working on a system where I use a schema with the type Map much like this:
const Product = mongoose.model('Product', {
name: String,
data:{
type:Map,
of:String
}
});
I'm trying to query certain parts of the docs in this collection with a projection:
Product.findOne({
_id: req.params.id
}, {
name:true,
data.xy*
})
I would like to only get the fields in data starting with xy. Do you have any hint how to do that?
Since Mongoose's map type will be represented as a nested object in mongodb, you could convert using $objectToArray which will result in array of key-value pairs that allows you to query by the keys starting with xy. Something like:
Product.aggregate([
{
$project: {
data: {
$objectToArray: "$data"
}
}
},
{
"$unwind": "$data"
},
{
$match: {
"data.k": {
$regex: "^xy"
}
}
}
])
Here's a working example on mongoplayground:
https://mongoplayground.net/p/yCJLhzalOXI
I am interested in optimizing a "pagination" solution I'm working on with MongoDB. My problem is straight forward. I usually limit the number of documents returned using the limit() functionality. This forces me to issue a redundant query without the limit() function in order for me to also capture the total number of documents in the query so I can pass to that to the client letting them know they'll have to issue an additional request(s) to retrieve the rest of the documents.
Is there a way to condense this into 1 query? Get the total number of documents but at the same time only retrieve a subset using limit()? Is there a different way to think about this problem than I am approaching it?
Mongodb 3.4 has introduced $facet aggregation
which processes multiple aggregation pipelines within a single stage
on the same set of input documents.
Using $facet and $group you can find documents with $limit and can get total count.
You can use below aggregation in mongodb 3.4
db.collection.aggregate([
{ "$facet": {
"totalData": [
{ "$match": { }},
{ "$skip": 10 },
{ "$limit": 10 }
],
"totalCount": [
{ "$group": {
"_id": null,
"count": { "$sum": 1 }
}}
]
}}
])
Even you can use $count aggregation which has been introduced in mongodb 3.6.
You can use below aggregation in mongodb 3.6
db.collection.aggregate([
{ "$facet": {
"totalData": [
{ "$match": { }},
{ "$skip": 10 },
{ "$limit": 10 }
],
"totalCount": [
{ "$count": "count" }
]
}}
])
No, there is no other way. Two queries - one for count - one with limit. Or you have to use a different database. Apache Solr for instance works like you want. Every query there is limited and returns totalCount.
MongoDB allows you to use cursor.count() even when you pass limit() or skip().
Lets say you have a db.collection with 10 items.
You can do:
async function getQuery() {
let query = await db.collection.find({}).skip(5).limit(5); // returns last 5 items in db
let countTotal = await query.count() // returns 10-- will not take `skip` or `limit` into consideration
let countWithConstraints = await query.count(true) // returns 5 -- will take into consideration `skip` and `limit`
return { query, countTotal }
}
Here's how to do this with MongoDB 3.4+ (with Mongoose) using $facets. This examples returns a $count based on the documents after they have been matched.
const facetedPipeline = [{
"$match": { "dateCreated": { $gte: new Date('2021-01-01') } },
"$project": { 'exclude.some.field': 0 },
},
{
"$facet": {
"data": [
{ "$skip": 10 },
{ "$limit": 10 }
],
"pagination": [
{ "$count": "total" }
]
}
}
];
const results = await Model.aggregate(facetedPipeline);
This pattern is useful for getting pagination information to return from a REST API.
Reference: MongoDB $facet
Times have changed, and I believe you can achieve what the OP is asking by using aggregation with $sort, $group and $project. For my system, I needed to also grab some user info from my users collection. Hopefully this can answer any questions around that as well. Below is an aggregation pipe. The last three objects (sort, group and project) are what handle getting the total count, then providing pagination capabilities.
db.posts.aggregate([
{ $match: { public: true },
{ $lookup: {
from: 'users',
localField: 'userId',
foreignField: 'userId',
as: 'userInfo'
} },
{ $project: {
postId: 1,
title: 1,
description: 1
updated: 1,
userInfo: {
$let: {
vars: {
firstUser: {
$arrayElemAt: ['$userInfo', 0]
}
},
in: {
username: '$$firstUser.username'
}
}
}
} },
{ $sort: { updated: -1 } },
{ $group: {
_id: null,
postCount: { $sum: 1 },
posts: {
$push: '$$ROOT'
}
} },
{ $project: {
_id: 0,
postCount: 1,
posts: {
$slice: [
'$posts',
currentPage ? (currentPage - 1) * RESULTS_PER_PAGE : 0,
RESULTS_PER_PAGE
]
}
} }
])
there is a way in Mongodb 3.4: $facet
you can do
db.collection.aggregate([
{
$facet: {
data: [{ $match: {} }],
total: { $count: 'total' }
}
}
])
then you will be able to run two aggregate at the same time
By default, the count() method ignores the effects of the
cursor.skip() and cursor.limit() (MongoDB docs)
As the count method excludes the effects of limit and skip, you can use cursor.count() to get the total count
const cursor = await database.collection(collectionName).find(query).skip(offset).limit(limit)
return {
data: await cursor.toArray(),
count: await cursor.count() // this will give count of all the documents before .skip() and limit()
};
It all depends on the pagination experience you need as to whether or not you need to do two queries.
Do you need to list every single page or even a range of pages? Does anyone even go to page 1051 - conceptually what does that actually mean?
Theres been lots of UX on patterns of pagination - Avoid the pains of pagination covers various types of pagination and their scenarios and many don't need a count query to know if theres a next page. For example if you display 10 items on a page and you limit to 13 - you'll know if theres another page..
MongoDB has introduced a new method for getting only the count of the documents matching a given query and it goes as follows:
const result = await db.collection('foo').count({name: 'bar'});
console.log('result:', result) // prints the matching doc count
Recipe for usage in pagination:
const query = {name: 'bar'};
const skip = (pageNo - 1) * pageSize; // assuming pageNo starts from 1
const limit = pageSize;
const [listResult, countResult] = await Promise.all([
db.collection('foo')
.find(query)
.skip(skip)
.limit(limit),
db.collection('foo').count(query)
])
return {
totalCount: countResult,
list: listResult
}
For more details on db.collection.count visit this page
It is possible to get the total result size without the effect of limit() using count() as answered here:
Limiting results in MongoDB but still getting the full count?
According to the documentation you can even control whether limit/pagination is taken into account when calling count():
https://docs.mongodb.com/manual/reference/method/cursor.count/#cursor.count
Edit: in contrast to what is written elsewhere - the docs clearly state that "The operation does not perform the query but instead counts the results that would be returned by the query". Which - from my understanding - means that only one query is executed.
Example:
> db.createCollection("test")
{ "ok" : 1 }
> db.test.insert([{name: "first"}, {name: "second"}, {name: "third"},
{name: "forth"}, {name: "fifth"}])
BulkWriteResult({
"writeErrors" : [ ],
"writeConcernErrors" : [ ],
"nInserted" : 5,
"nUpserted" : 0,
"nMatched" : 0,
"nModified" : 0,
"nRemoved" : 0,
"upserted" : [ ]
})
> db.test.find()
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c8"), "name" : "forth" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c9"), "name" : "fifth" }
> db.test.count()
5
> var result = db.test.find().limit(3)
> result
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }
> result.count()
5 (total result size of the query without limit)
> result.count(1)
3 (result size with limit(3) taken into account)
Try as bellow:
cursor.count(false, function(err, total){ console.log("total", total) })
core.db.users.find(query, {}, {skip:0, limit:1}, function(err, cursor){
if(err)
return callback(err);
cursor.toArray(function(err, items){
if(err)
return callback(err);
cursor.count(false, function(err, total){
if(err)
return callback(err);
console.log("cursor", total)
callback(null, {items: items, total:total})
})
})
})
Thought of providing a caution while using the aggregate for the pagenation. Its better to use two queries for this if the API is used frequently to fetch data by the users. This is atleast 50 times faster than getting the data using aggregate on a production server when more users are accessing the system online. The aggregate and $facet are more suited for Dashboard , reports and cron jobs that are called less frequently.
We can do it using 2 query.
const limit = parseInt(req.query.limit || 50, 10);
let page = parseInt(req.query.page || 0, 10);
if (page > 0) { page = page - 1}
let doc = await req.db.collection('bookings').find().sort( { _id: -1 }).skip(page).limit(limit).toArray();
let count = await req.db.collection('bookings').find().count();
res.json({data: [...doc], count: count});
I took the two queries approach, and the following code has been taken straight out of a project I'm working on, using MongoDB Atlas and a full-text search index:
return new Promise( async (resolve, reject) => {
try {
const search = {
$search: {
index: 'assets',
compound: {
should: [{
text: {
query: args.phraseToSearch,
path: [
'title', 'note'
]
}
}]
}
}
}
const project = {
$project: {
_id: 0,
id: '$_id',
userId: 1,
title: 1,
note: 1,
score: {
$meta: 'searchScore'
}
}
}
const match = {
$match: {
userId: args.userId
}
}
const skip = {
$skip: args.skip
}
const limit = {
$limit: args.first
}
const group = {
$group: {
_id: null,
count: { $sum: 1 }
}
}
const searchAllAssets = await Models.Assets.schema.aggregate([
search, project, match, skip, limit
])
const [ totalNumberOfAssets ] = await Models.Assets.schema.aggregate([
search, project, match, group
])
return await resolve({
searchAllAssets: searchAllAssets,
totalNumberOfAssets: totalNumberOfAssets.count
})
} catch (exception) {
return reject(new Error(exception))
}
})
I had the same problem and came across this question. The correct solution to this problem is posted here.
You can do this in one query. First you run a count and within that run the limit() function.
In Node.js and Express.js, you will have to use it like this to be able to use the "count" function along with the toArray's "result".
var curFind = db.collection('tasks').find({query});
Then you can run two functions after it like this (one nested in the other)
curFind.count(function (e, count) {
// Use count here
curFind.skip(0).limit(10).toArray(function(err, result) {
// Use result here and count here
});
});
I'm trying to update an array in document by adding object if it doesn't exist, and replacing the object in array otherwise. But nothing ($push, $addToSet) except the $set parameter does anything, and $set works as expected - overwrites the whole array.
My mongoose schema:
var cartSchema = mongoose.Schema({
mail: String,
items: Array
});
The post request handler:
app.post('/addToCart', function(req, res) {
var request = req.body;
Cart.findOneAndUpdate({
"mail": request.mail
}, {
$addToSet: {
"items": request.item
}
}, {
upsert: true
},
function(err, result) {
console.log(result);
}
);
res.send(true);
});
The data that I'm sending from the client:
{
"mail":"test#gmail.com",
"item":{
"_id":"59da78db7e9e0433280578ec",
"manufacturer":"Schecter",
"referenceNo":"Daemon-412",
"type":"Gitare",
"image":"images/ba9727909d6c3c26412341907e7e12041507489988265.jpeg",
"__v":0,
"subcategories":[
"Elektricne"
]
}
}
EDIT:
I also get this log when I trigger 'addToCart' request:
{ MongoError: The field 'items' must be an array but is of type object in
document {_id: ObjectId('5a19ae2884d236048c8c91e2')}
The comparison in $addToSet would succeeded only if the existing document has the exact same fields and values, and the fields are in the same order. Otherwise the operator will fail.
So in your case, request.item always need to be exactly the same.
I would recommend creating a model of "item". Then, your cart schema would be like:
var cartSchema = mongoose.Schema({
mail: String,
items: [{
type: ObjectId,
ref: 'item',
}],
});
And let MongoDB determine if the item exist.
this should work you just need to implement objectExits function that test if the item is that one you're looking for :
Cart.findOne({ "mail": request.mail })
.exec()
.then(cart => {
var replaced = cart.items.some((item, i) => {
if (item._id == request.item._id)) {
cart.items[i] = request.item;
return true;
}
})
if (!replaced) {
cart.items.push(request.item);
}
cart.save();
return cart;
})
.catch(err => {
console.log(err)
});
I'm new to this technology and working with Node and Express server that uses Mongoose and MongoDB 3.3 . I have following schema for a document collection.
exports.schemaLivre = new mongoose.Schema({
Id : Number,
Nom: { type: String, maxlength: 50 },
Auteur: String,
Description: String,
Categorie: String,
}
exports.Livre = mongoose.model('Livre', schemas.schemaLivre);
I'm trying to run a web service having 3 parameters .
app.get('/Livres/search=:search&category:=category&auteur=:auteur', controller.RecupererLivresMultiple);
each parameter can be empty and foreach parameter I have a different kind of search logic ( for the param search , I will use regex expression to check through "Nom" and "Description" and "Auteur" fields in the same time, for category and auteur , the value submitted must be equal to the content of the field).
I will need something like that but I didn't found the right query .
exports.RecupererLivresMultiple = function (req, res) {
var re = new RegExp(req.params.search, 'i');
models.Livre.find($and: [
{
$or: [
{ 'Auteur': req.params.auteur },
]
},
{
$or: [
{ 'Category': req.params.Category },
]
}
]).or([{'Nom': {$regex: re}}, {'Auteur': {$regex: re}},{'Description':{$regex: re}}]).exec(function (err, livres) {
if (err) {
return console.error(err);
}
res.json(livres);
});
};
The question is How to check the request parameter sent from the web service is empty and How I disable it in my query when searching for records .
My second question is how to combine those logic of search together ?
I think the basic structure you want for your query is:
{
'Auteur': req.params.auteur,
'Categorie': req.params.category,
$or: [
{ 'Auteur': re},
{ 'Categorie': re},
{ 'Nom': re},
]
}
To make each parameter optional, you can do something like this to build the query:
// create empty query
var findJSON = {
};
// append criteria if parameters present
if (req.params.auteur){
findJSON.Auter = req.params.auter
}
if (req.params.category){
findJSON.Categorie = req.params.category
}
if (req.params.search){
var re = new RegExp(req.params.search, 'i');
findJSON.$or = [
{ 'Auteur': re},
{ 'Categorie': re},
{ 'Nom': re},
];
}
models.Livre.find(findJSON).exec(function (err, livres) {
if (err) {
return console.error(err);
}
res.json(livres);
});
Because each parameter is optional, you should handle the case where none is provided. You could do:
if(req.params.auteur || req.params.category || req.params.search){
models.Livre.find(findJSON).exec(function (err, livres) {
if (err) {
return console.error(err);
}
res.json(livres);
});
} else {
res.json([]);
}
You may also want to handle the error condition differently. As you have it now, you aren't actually writing to the response if your find() returns an error.
I find no doc for the sort modifier. The only insight is in the unit tests:
spec.lib.query.js#L12
writer.limit(5).sort(['test', 1]).group('name')
But it doesn't work for me:
Post.find().sort(['updatedAt', 1]);
In Mongoose, a sort can be done in any of the following ways:
Post.find({}).sort('test').exec(function(err, docs) { ... });
Post.find({}).sort([['date', -1]]).exec(function(err, docs) { ... });
Post.find({}).sort({test: 1}).exec(function(err, docs) { ... });
Post.find({}, null, {sort: {date: 1}}, function(err, docs) { ... });
This is how I got sort to work in mongoose 2.3.0 :)
// Find First 10 News Items
News.find({
deal_id:deal._id // Search Filters
},
['type','date_added'], // Columns to Return
{
skip:0, // Starting Row
limit:10, // Ending Row
sort:{
date_added: -1 //Sort by Date Added DESC
}
},
function(err,allNews){
socket.emit('news-load', allNews); // Do something with the array of 10 objects
})
As of Mongoose 3.8.x:
model.find({ ... }).sort({ field : criteria}).exec(function(err, model){ ... });
Where:
criteria can be asc, desc, ascending, descending, 1, or -1
Note: Use quotation marks or double quote
use "asc", "desc", "ascending", "descending", 1, or -1
UPDATE:
Post.find().sort({'updatedAt': -1}).all((posts) => {
// do something with the array of posts
});
Try:
Post.find().sort([['updatedAt', 'descending']]).all((posts) => {
// do something with the array of posts
});
Mongoose v5.x.x
sort by ascending order
Post.find({}).sort('field').exec(function(err, docs) { ... });
Post.find({}).sort({ field: 'asc' }).exec(function(err, docs) { ... });
Post.find({}).sort({ field: 'ascending' }).exec(function(err, docs) { ... });
Post.find({}).sort({ field: 1 }).exec(function(err, docs) { ... });
Post.find({}, null, {sort: { field : 'asc' }}), function(err, docs) { ... });
Post.find({}, null, {sort: { field : 'ascending' }}), function(err, docs) { ... });
Post.find({}, null, {sort: { field : 1 }}), function(err, docs) { ... });
sort by descending order
Post.find({}).sort('-field').exec(function(err, docs) { ... });
Post.find({}).sort({ field: 'desc' }).exec(function(err, docs) { ... });
Post.find({}).sort({ field: 'descending' }).exec(function(err, docs) { ... });
Post.find({}).sort({ field: -1 }).exec(function(err, docs) { ... });
Post.find({}, null, {sort: { field : 'desc' }}), function(err, docs) { ... });
Post.find({}, null, {sort: { field : 'descending' }}), function(err, docs) { ... });
Post.find({}, null, {sort: { field : -1 }}), function(err, docs) { ... });
For Details: https://mongoosejs.com/docs/api.html#query_Query-sort
Update
There is a better write up if this is confusing people; check out finding documents and how queries work in the mongoose manual. If you want to use the fluent api you can get a query object by not providing a callback to the find() method, otherwise you can specify the parameters as I outline below.
Original
Given a model object, per the docs on Model, this is how it can work for 2.4.1:
Post.find({search-spec}, [return field array], {options}, callback)
The search spec expects an object, but you can pass null or an empty object.
The second param is the field list as an array of strings, so you would supply ['field','field2'] or null.
The third param is the options as an object, which includes the ability to sort the result set. You would use { sort: { field: direction } } where field is the string fieldname test (in your case) and direction is a number where 1 is ascending and -1 is desceding.
The final param (callback) is the callback function which receives the collection of docs returned by the query.
The Model.find() implementation (at this version) does a sliding allocation of properties to handle optional params (which is what confused me!):
Model.find = function find (conditions, fields, options, callback) {
if ('function' == typeof conditions) {
callback = conditions;
conditions = {};
fields = null;
options = null;
} else if ('function' == typeof fields) {
callback = fields;
fields = null;
options = null;
} else if ('function' == typeof options) {
callback = options;
options = null;
}
var query = new Query(conditions, options).select(fields).bind(this, 'find');
if ('undefined' === typeof callback)
return query;
this._applyNamedScope(query);
return query.find(callback);
};
HTH
you can sort your query results by
Post.find().sort({createdAt: "descending"});
This is how I got sort to work in mongoose.js 2.0.4
var query = EmailModel.find({domain:"gmail.com"});
query.sort('priority', 1);
query.exec(function(error, docs){
//...
});
Chaining with the query builder interface in Mongoose 4.
// Build up a query using chaining syntax. Since no callback is passed this will create an instance of Query.
var query = Person.
find({ occupation: /host/ }).
where('name.last').equals('Ghost'). // find each Person with a last name matching 'Ghost'
where('age').gt(17).lt(66).
where('likes').in(['vaporizing', 'talking']).
limit(10).
sort('-occupation'). // sort by occupation in decreasing order
select('name occupation'); // selecting the `name` and `occupation` fields
// Excute the query at a later time.
query.exec(function (err, person) {
if (err) return handleError(err);
console.log('%s %s is a %s.', person.name.first, person.name.last, person.occupation) // Space Ghost is a talk show host
})
See the docs for more about queries.
app.get('/getting',function(req,res){
Blog.find({}).limit(4).skip(2).sort({age:-1}).then((resu)=>{
res.send(resu);
console.log(resu)
// console.log(result)
})
})
Output
[ { _id: 5c2eec3b8d6e5c20ed2f040e, name: 'e', age: 5, __v: 0 },
{ _id: 5c2eec0c8d6e5c20ed2f040d, name: 'd', age: 4, __v: 0 },
{ _id: 5c2eec048d6e5c20ed2f040c, name: 'c', age: 3, __v: 0 },
{ _id: 5c2eebf48d6e5c20ed2f040b, name: 'b', age: 2, __v: 0 } ]
with the current version of mongoose (1.6.0) if you only want to sort by one column, you have to drop the array and pass the object directly to the sort() function:
Content.find().sort('created', 'descending').execFind( ... );
took me some time, to get this right :(
This is how I managed to sort and populate:
Model.find()
.sort('date', -1)
.populate('authors')
.exec(function(err, docs) {
// code here
})
Post.find().sort({updatedAt: 1});
As of October 2020, to fix your issue you should add .exec() to the call. don't forget that if you want to use this data outside of the call you should run something like this inside of an async function.
let post = await callQuery();
async function callQuery() {
return Post.find().sort(['updatedAt', 1].exec();
}
Others worked for me, but this did:
Tag.find().sort('name', 1).run(onComplete);
Post.find().sort({updatedAt:1}).exec(function (err, posts){
...
});
Starting from 4.x the sort methods have been changed. If you are using >4.x. Try using any of the following.
Post.find({}).sort('-date').exec(function(err, docs) { ... });
Post.find({}).sort({date: -1}).exec(function(err, docs) { ... });
Post.find({}).sort({date: 'desc'}).exec(function(err, docs) { ... });
Post.find({}).sort({date: 'descending'}).exec(function(err, docs) { ... });
Post.find({}).sort([['date', -1]]).exec(function(err, docs) { ... });
Post.find({}, null, {sort: '-date'}, function(err, docs) { ... });
Post.find({}, null, {sort: {date: -1}}, function(err, docs) { ... });
This is what i did, it works fine.
User.find({name:'Thava'}, null, {sort: { name : 1 }})
// Ascending with updatedAt field
Post.find().sort('updatedAt').exec((err, post) => {...});
// Descending with updatedAt field
Post.find().sort('-updatedAt').exec((err, post) => {...});
Refer here: https://mongoosejs.com/docs/queries.html
you can also use aggregate() for sorting
const sortBy = req.params.sort;
const limitNum = req.params.limit;
const posts = await Post.aggregate([
{ $unset: ['field-1', 'field-2', 'field-3', 'field-4'] },
{ $match: { field-1: value} },
{ $sort: { [sortBy]: -1 } }, //-------------------> sort the result
{ $limit: Number(limitNum) },
]);
Solution :
posts.find().sort({field:1})
// for ascending and for descending order just use -1 instead of 1