We Keep Coding

How do I pick then sort MongoDB documents and then find next document to mine?

I have a collection of documents, which I need to first narrow down by set criteria, then sort alphabetically by string value inside those documents — let's say that's a "search result". I then need to find document that matches a given _id and then pick a document next to it (before or after) from the above "search result".
Background:
I use mongoose to query my database via Node.js.
I have a set of "special sections" in my blog that are comprised of all the articles that must have three particular conditions associated within the keys in the document. I can get the list of articles belonging to said section like so:
const specialSectionListQuery = Article.find({
tag: { $ne: "excluded" },
[`collections.cameras`]: { $exists: true },
status: "published",
})
To finish creating the "special section," I must sort the documents alphabetically via their title attribute:
.sort({ [`collections.cameras.as.title`]: "asc" })
Now I want to add a link to "next article within the same special section" at the bottom of such articles. I know _id and any other value needed from the current article. The above query gives me an ordered list of documents within the section so I can easily find it within that list specialSectionListQuery.findOne({ _id: "xxx" }).exec().
However, I need to find the next article within the above list. How do I do that?
My attempts thus far:
I tried to create article list via aggregation, which led me nowhere (I simply made my app do exactly the same thing — make a list for a "special sectin"):
Article.aggregate([
{
$match: {
tag: { $ne: "excluded" },
[`collections.cameras`]: { $exists: true },
status: "published",
},
},
{
$sort: {
[`collections.cameras.as.title`]: 1,
},
}
]).exec()
But I can't for the life of me figure out how to iterate to the next document in the list properly.
I have thought of saving the list in Node.js memory (variable) and then finding what I need via JavaScript but that can't be scalable.
I have considered creating a new collection and saving the above list there but that would require me to either 1) do it every time a document is altered/added/deleted via Node.js — which is a lot of code and it may break if I interact with database another way 2) rebuild the colleciton every time I run the query, but that feels like it'll lack in performance.
Please help and thank you!
P.S.:
Example collection which should cover most of the cases I'm looking to solve for:
[
{
_id: 1,
name: "Canon",
collections: { cameras: { as: { title: "Half-Frame" } } },
tag: "included",
status: "published"
},
{
_id: 2,
name: "Pentax",
collections: { cameras: { as: { title: "Full-Frame" } } },
tag: "included",
status: "published"
},
{
_id: 3,
name: "Kodak",
collections: { film: { as: { title: "35mm Film" } } },
tag: "included",
status: "published"
},
{
_id: 4,
name: "Ricoh",
collections: { cameras: { as: { title: "Full-Frame" } } },
tag: "included",
status: "published"
},
{
_id: 5,
name: "Minolta",
collections: { cameras: { as: { title: "Half-Frame Review" } } },
tag: "excluded",
status: "published"
},
{
_id: 4,
name: "FED",
collections: { cameras: { as: { title: "Full-Frame" } } },
tag: "included",
status: "draft"
}
]

One thing you can try is to extend your $sort by adding _id so that it always returns documents in deterministic order:
{
$sort: {
"collections.cameras.as.title": 1,
_id: 1
}
},
{
$limit: 1
}
Once your first query returns the document with _id: 2 and collections.cameras.as.title: Full-Frame, you can use below query to get subsequent document:
{
$match: {
$and: [
{
tag: { $ne: "excluded" },
"collections.cameras": { $exists: true },
status: "published",
},
{
$or: [
{
$and: [
{ "collections.cameras.as.title": { $eq: "Full-Frame" } },
{ "_id": { $gt: 2 } }
]
},
{ "collections.cameras.as.title": { $gt: "Full-Frame" } }
]
}
]
}
},
{
$sort: {
"collections.cameras.as.title": 1,
_id: 1
}
},
{
$limit: 1
}
In this case due to deterministic $sort you can exclude previously found document by adding additional filtering criteria and the order should be preserved.
Mongo Playground

Nodejs with Mongoose returns one object value instead of multiple in an array

data=[{
locId: '332wn',
locadetails: [
{ loc: 'ny',
status: true
},
{ loc: 'ca',
status: null
},
{ loc: 'tx',
status: null
}
]
}]
I have following query that is trying to find all the locdetails that have open value as null or false
Loc.find({'locId': id}, {'locadetails' : {$elemMatch: {'status': {$ne: true}}}}, (err, locs)=>{
if(err) {
retrun callback(err);
}
callback(null, locs)
});
Problem I have is this query will only return one value o locadetails with null while it should return two as seen in the data.
Please let me know what to do so I can get whole array of items that have status field as null or false ...Thanks

$elemMatch will return first matching element from an array based on a condition. Use Aggregation instead.
Both the $ operator and the $elemMatch operator project the first
matching element from an array based on a condition. Reference
Loc.aggregate([
{ $match: { "locId": "332wn" } },
{ $unwind: "$locadetails" },
{ $match: { "locadetails.status": { $ne: true } } },
{ $group: { _id: "$_id", locId: { $first: "$locId" }, locadetails: { $push: "$locadetails" }, } }
])

Find documents that contain certain field for sub-object MongoDb and Node.js

I have a collection with the following format:
[
{
firstname: 'Joe',
lastname: 'Blow',
emails: [
{
email: 'test#example.com',
valid: false
},
{
email: 'test2#example.com',
valid: false
}
],
password: 'abc123',
_id: 57017e173915101e0ad5d94a
},
{
firstname: 'Johnny',
lastname: 'Doe',
emails: [
{
email: 'test3#example.com',
valid: false
}
],
password: 'abc123',
_id: 57017e173915101e0ad5d87b
},
]
I am trying to find a user based on the emails.email field. Here is what I have so far:
db.collection('users').aggregate([
{$unwind: "$emails"},
{$group: {_id: "$_id",user_emails: { $push: "$emails.email" } } },
{$match: {'user_emails': { $in: ['test#example.com'] } } }
],
(error, result) => {
console.log(error);
console.log(result);
}
);
When I run this command in the mongo shell it seems to work; however when I run it in Node.js it prints null for the error and [] for the result.
What am I doing wrong? I am pretty new to MongoDB and just can't seem to figure this out.

Why do you want to unwind the entire emails? That will be a very expensive operation when your collection grows with tons of records.
The below query will return the user with the email test2#example.com. I think thats what you are looking for right?
db.email.find({emails :{$elemMatch:{email:"test2#example.com"}}})

I have re-written your code with slight changes.
var col = db.collection('collection');
if (col) {
col.aggregate([
{$unwind: "$emails"},
{$group: {_id: "$_id",user_emails: { $push: "$emails.email" } } },
{$match: {'user_emails': { $in: ['test#example.com'] } } }
], function(e, r){
if(e){
console.log(error);
}
console.log(r);
db.close();
});
}
It should work, provided you have establish connection and other requirements successfully. Provided your sample documents, it will emit:
[
{
_id: '57017e173915101e0ad5d94a',
user_emails: [
'test#example.com',
'test2#example.com'
]
}
]

Sorting and grouping nested subdocument in Mongoose

I have a schema, Comment, like the one below. It's a system of "comments" and "replies", but each comment and reply has multiple versions. When a user wants to view a comment, I want to return just the most recent version with the status of APPROVED.
const Version = new mongoose.Schema({
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
},
body: String,
created: Date,
title: String,
status: {
type: String,
enum: [ 'APPROVED', 'OPEN', 'CLOSED' ]
}
})
const Reply = new mongoose.Schema({
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
},
created: Date,
versions: [ Version ]
})
const Comment = new mongoose.Schema({
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
},
created: Date,
versions: [ Version ],
replies: [ Reply ]
})
I've gotten the parent Comment to display how I want with the code below. However, I've had trouble applying that to the sub-document, Reply.
const requestedComment = yield Comment.aggregate([
{ $match: {
query
} },
{ $project: {
user: 1,
replies: 1,
versions: {
$filter: {
input: '$versions',
as: 'version',
cond: { $eq: [ '$$version.status', 'APPROVED' ] }
}
},
}},
{ "$unwind": "$versions" },
{ $sort: { 'versions.created': -1 } },
{ $group: {
_id: '$_id',
body: { $first: '$versions.body' },
title: { $first: '$versions.title' },
replies: { $first: '$replies' }
}}
])
.exec()
Any help achieving the same result with the replies subdocuments would be appreciated. I would like to return the most recent APPROVED version of each reply in a form like this:
comment: {
body: "The comment's body.",
user: ObjectId(...),
replies: [
{
body: "The reply's body."
user: ObjectId(...)
}
]
}

Basically you just need to continue the same process on from the existing pipeline. But this time to $unwind out the "versions" per each "replies" entry and $sort them there.
So these are "additional" stages to your pipeline.
// Unwind replies
{ "$unwind": "$replies" },
// Unwind inner versions
{ "$unwind": "$replies.versions" },
// Filter for only approved
{ "$match": { "replies.versions.status": "APPROVED" } },
// Sort on all "keys" and then the "version" date
{ "$sort": {
"_id": 1,
"replies._id": 1,
"replies.versions.created": -1
}},
// Group replies to get the latest version of each
{ "$group": {
"_id": {
"_id": "$_id",
"body": "$body",
"title": "$title",
"replyId": "$replies._id",
"replyUser": "$replies.user",
"replyCreated": "$replies.created"
},
"version": { "$first": "$replies.version" }
}},
// Push replies back into an array in the main document
{ "$group": {
"_id": "$_id._id",
"body": { "$first": "$_id.body" },
"title": { "$first": "$_id.title" },
"replies": {
"$push": {
"_id": "$_id.replyId",
"user": "$_id.replyUser" },
"created": "$_id.replyCreated", // <-- Value from Reply
"body": "$version.body", // <-- Value from specific Version
"title": "$version.title"
}
}
}}
All depending of course on which fields you want, being either from ther Reply or from the Version.
Whichever fields, since you "un-wound" two arrays, you $group back "twice".
Once to get the $first items after sorting per Reply
Once more to re-construct the "replies" array using $push
That's all there is too it.
If you were still looking at ways to "sort" the array "in-place" without using $unwind, well MongoDB just does not do that yet.
Bit of advice on your design
As a note, I see where you are going with this and this is the wrong model for the type of usage that you want.
It makes little sense to store "revision history" within the embdedded structure. You are rarely going to use it in general update and query operations, and as this demonstrates, most of the time you just want the "latest".
So just do that instead, and store a "flag" indicating "revisions" if really necessary. That data can then be stored external to the main structure, and you won't have to jump through these hoops just to get the "latest accepted version" on every request.

Node mongoose populate with condition not returns the expected results

I am trying to use the query conditions in populate method.
if a condition is used then still all the records are populated but ones that don't satisfy the condition have the populated field is set to null,
For example:
var query = BookModel.find();
query.populate('author','name',{name:'author1'});
query.find(function(err,books){
console.log(books);
});
Then the output is:
[ { author: { _id: 4ea0db52e09aa6aad2e831fe, name: 'author1' },
title: 'book1',
_id: 4ea0dbebc191848704000005 },
{ author: null,
title: 'book2',
_id: 4ea0dbebc191848704000006 } ,
{ author: null,
title: 'book3',
_id: 4ea0dbebc191848704000007 } ,
{ author: null,
title: 'book4',
_id: 4ea0dbebc191848704000008 } ]
However, I expect only the 1st record in the output result. How can i solve this problem?

I had same problem which you are facing and I tried the above code but it did not help me solve the problem. I found a way to do it just like you want. You cannot do this kind of filter by using populate, you have to use raw query of mongodb to filter out your data.
// db['result'] is the name of database
db['result'].aggregate([
// this is just like populate this populate the user field
{
$lookup:{
from:'users',
localField:'user',
foreignField:'_id',
as:'user'
}
},
// unwind convert the array to objects to apply filter on it
{
$unwind:{
preserveNullAndEmptyArrays : true, // this remove the object which is null
path : "$user"
}
},
// in match you have to define your condition this check if the user has role equals to 3
{
$match:{
"user.role": 3
}
},
{
// this provide pagination
$facet: {
edges: [
{ $skip: sk },
{ $limit: lm },
],
/* pageInfo: [
{ $group: { _id: null, count: { $sum: 1 } } },
],*/
},
}
], function (err, result) {
if (err) {
// this res is send if there is some error
console.log(err);
callback(500, err)
} else {
// you get the data
console.log(result);
callback(200, result)
}
});

Look if this can helps you, I need more information about the problem, that's a possible solution.
BookModel.find()
.populate({
path: 'author',
match: { author: 'author1' },
select: 'name' // I'm suppossing you want to select "name" field, if not, delete this line.
}).find(function(err,books){
console.log(books);
});
or you can do this:
var query = BookModel.find({name: 'author1'});
query.populate('author','name');
query.find(function(err,books){
console.log(books);
});

You could add an extra query in your results to filter out the nulls:
var query = BookModel.find();
query.populate('author','name',{name:'author1'});
query.find(function(err, books){
books = books.filter(function(b){ return b.author; });
console.log(books, null, 4);
});
Console output
[
{
"author": {
"_id": "4ea0db52e09aa6aad2e831fe",
"name": "author1"
},
"title": "book1",
"_id": "4ea0dbebc191848704000005"
}
]