Let's say I have this schema and it has two fields
someDate: Date
isPriority: boolean
and I want to make a sort query by someDate and records with isPriority: true should be always first.
So how should it look like? Right now I know how to sort by date like this:
Record.find().sort({ someDate: -1 });
I think at some point MongoDB started supporting sort (by means of other than which index(s) it uses), and the key order works to support sorts.
Per MongoDB documentation, to achieve a "stable" sort, ensure that you have a unique column as the last key.
db.restaurants.find().sort( { "borough": 1, "_id": 1 } )
So, for your particular query, use:
{isPriority: 1, someDate: -1, _id: 1}.
The wrong way, meaning, would be
{someDate: -1, isPriority: 1, _id: 1}
Related
I have a mongodb containing submissions with the unique _id parameter. I want to get all submissions that have specific parameters x = y (fictional), sort them by the creation date and then reduce the result to only contain the newest submission for each id. I think I need to use an aggregation to sort the results but I dont know how to continue.
So, currently I have:
Submission.find({x: y}).aggregate(
{
$sort: {_id: -1}
}, .....)
Submission.sample.aggregate([{$match: {x: y}}, {$sort: {createdDate: 1}}, {$project: {x: 1, createdDate: 1}}])
Here is the sample code using aggregation pipeline. First, find documents that have x property with y value, then sort by createdDate property, and finally get only attributes you want to reduce returned object size.
I have two DBs for testing and each contains thousands/hundreds of thousand of documents.
But with the same Schemas and CRUD operations.
Let's call DB1 and DB2.
I am using Mongoose
Suddenly DB1 became really slow during:
const eventQueryPipeline = [
{
$match: {
$and: [{ userId: req.body.userId }, { serverId: req.body.serverId }],
},
},
{
$sort: {
sort: -1,
},
},
];
const aggregation = db.collection
.aggregate(eventQueryPipeline)
.allowDiskUse(true);
aggregation.exect((err, result) => {
res.json(result);
});
In DB2 the same exact query runs in milliseconds up to maximum a 10 seconds
In DB1 the query never takes less than 40 seconds.
I do not understand why. What could I be missing?
I tried to confront the Documents and the Indexes and they're the same.
Deleting the collection and restrting saving the documents, brings the speed back to normal and acceptable, but why is it happening? Does someone had same experience?
Short answer:
You should create following index:
{ "userId": 1, "serverId": 1, "sort": 1 }
Longer answer
Based on your code (i see that you have .allowDiskUse(true)) it looks like mongo is trying to do in memory sort with "a lot" of data. Mongo has by default 100MB system memory limit for sort operations, and you can allow it to use temporary files on disk to store data if it hits that limit.
You can read more about it here: https://www.mongodb.com/docs/manual/reference/method/cursor.allowDiskUse/
In order to optimise the performance of your queries, you can use indexes.
Common rule that you should follow when planning indexes is ESR (Equality, Sort, Range). You can read more about it here: https://www.mongodb.com/docs/v4.2/tutorial/equality-sort-range-rule/
If we follow that rule while creating our compound index, we will add equality matches first, in your case "userId" and "serverId". After that comes the sort field, in your case "sort".
If you had a need to additionally filter results based on some range (eg. some value greater than X, or timestamp greater than yday), you would add that after the "sort".
That means your index should look like this:
schema.index({ userId: 1, serverId: 1, sort: 1 });
Additionally, you can probably remove allowDiskUse, and handle err inside aggregation.exec callback (im assuming that aggregation.exect is a typo)
I have event objects in MonogDB that look like this:
{
"start": 2010-09-04T16:54:11.216Z,
"title":"Short descriptive title",
"description":"Full description",
"score":56
}
And I need to get a query across three parameters:
Time window (event start is between two dates)
Score threshold (score is > x)
Full-text search of title and description
What's the right way to approach this efficiently? I think the first two are done with an aggregation but I'm not sure how text search would factor in.
Assuming your start field is of type date (which it should be) and not a string, here are the basic components that you'd want to play with. In fact, given the ISO 8601 structure of a MongoDB date a string based comparison would work just as well.
// create your text index
db.collection.ensureIndex({
description: "text",
title: "text"
})
// optionally create an index on your other fields
db.collection.ensureIndex({
start: 1,
score: 1
})
x = 50
lowerDate = ISODate("2010-09-04T16:54:11.216Z") // or just the string part for string fields
upperDate = ISODate("2010-09-04T16:54:11.216Z")
// simple find to retrieve your result set
db.collection.find({
start: {
$gte: lowerDate, // depending on your exact scenario, you'd need to use $gt
$lte: upperDate // depending on your exact scenario, you'd need to use $lt
},
score: { $gt: x }, // depending on your exact scenario, you'd need to use $gte
$text: { // here comes the text search
$search: "descriptive"
}
})
There is an important topic with respect to performance/indexing that needs to be understood, though, which is very well documented here: Combine full text with other index
This is why I initially wrote "components of what you'd want to play with". So depending on the rest of your application you may want to create different indexes.
I 'm facing an issue while trying to get some results from a mongoDB aggregation pipeline.
Here's what my DB look like:
var dbSchema = mongoose.Schema({
identity: Number,
parametres: {
style: {
fuel: [styleSchema],
gasoline: [styleSchema],
},
},
And here's what the styleSchema looks like:
var styleSchema = new mongoose.Schema({
date: Date,
value: Number,
type: String,
});
I'm trying to extract ALL the objects in 'fuel' and 'gasoline' which are of some kind of 'type'.
I've tried to group both in a unique array with concatArray and then match the 'type' I want by:
db.aggregate([
{$match:
{'identity':3,
}},
{$project: {all: {$concatArrays: ['$parametres.style.fuel','$parametres.style.gasoline']} }},
{$match: {'$all.type': 'example'}},
Before trying to match the second time, I've got a unique array ('all') and I try to match some things on it, but nothing works (I've tried 'all.type' also)...
I've probably misunderstood the way I have to use the 'match' query as I am a beginner, so thanks for your time and your answers,
Arthur
db.aggregate([
{
$match:{identity:3}
},
{
$project: {all: {$concatArrays: ['$parametres.style.fuel','$parametres.style.gasoline']} }
},
{$unwind: "$all"},
{$match: {"all.type": 'example'}},
{$group : {_id: null, all:{$push:"$all"}}}
])
Probably you are trying to do something like this.
In aggregate operation, the $ is used in right side of : to mention field name and in left side to mention operator.
So when you are using "$all.type" in left-hand side MongoDB is treating it as an operator which is not available in the Mongodb operator list.
Another thing is that when you do any query over an array. Mongodb sends back the full array if atleast one of the element matches the condition. So we need to use $unwind operator to deconstruct the array before doing any query.
In an AngularApp, i am getting my data from a WebAPI via OData queries.
All results have a date and an int statuscode. Statuscode can be 1, 2, 3 or 4.
I want to order my results, so all results with statuscode 1, goes to the top of the list. The rest of the results, with code 2, 3 and 4, gets sorted by the data attached to them, and not the statuscode.
Possible?
This is my code as it looks now. It just sorts by the date, and nothing else.
return Products.query({ $orderby: 'Date' });
From your description, it sounds like you want all Products with StatusCode of 1 to be treated equally. That is, they are not further sorted by Date. If you can relax this requirement a bit and allow all Products to be sorted by StatusCode and Date, there is a straightforward solution:
return Products.query({ $orderby: 'StatusCode,Date' });
The $orderby query option in OData v4 consists of "a comma-separated list of expressions whose primitive result values are used to sort the items". Property names are the most common expressions used to produce sort values.
If you want to reverse the ordering on a particular property, use the desc suffix on that property like so:
return Products.query({ $orderby: 'StatusCode,Date desc' });