We Keep Coding

How to group mongoose results by status, and populate a reference field?

I have a list of task records (see schema below). I am attempting to return records for a specific projectId, group my task records by status, and populate the responsible field. However, the responsible field is not populating. I have attached a code snippet below. Can anyone advise what I am doing incorrectly?
Code Snippet:
const test = await Task.aggregate([
{
$match: { project: { $eq: mongoose.Types.ObjectId(projectId) } },
},
{
$group: {
_id: "$status",
data: {
$push: {
_id: "$status",
name: "$name",
responsible: "$responsible",
endDate: "$endDate",
},
},
},
},
{
$sort: { status: 1 },
},
]);
console.log("test1:", test);
// Populate Aggregated Data:
const tasks = await User.populate(test, { path: "data.responsible" });
console.log("test2:", tasks);
TaskSchema:
const TaskSchema = new mongoose.Schema({
name: {
type: String,
trim: true,
required: true,
},
responsible: {
type: mongoose.Schema.Types.ObjectId,
ref: "User",
},
endDate: {
type: Date,
required: false,
},
status: {
type: String,
enum: ["new", "active", "inactive", "closed"],
required: true,
},
project: {
type: mongoose.Schema.Types.ObjectId,
ref: "Project",
},
});
Desired Results:
results = [
{
_id: “new”
data: [
{
endDate: "2022-09-16T04:00:00.000Z”,
name: "test1”,
responsible: {
email: “jane.doe#ymail.com”,
firstName: “Jane”,
lastName: “Doe”
},
{
endDate: "2022-09-16T04:00:00.000Z”,
name: "test2”,
responsible: {
email: “john.doe#ymail.com”,
firstName: “John”,
lastName: “Doe”
},
]
},

]

how to get data with $gte $lte

I'm trying to get some datas by mongoose.
let datas =
await Data.find({
'$and': [
{ test_id: testId },
{ createdAt: { $gte: moment(req.query.from).startOf('day') } },
{ createdAt: { $lte: moment(req.query.to).endOf('day') } }
],
});
This is works. But, I try to change aggregate, I can't get any data.
How should I get same result?
const datas = await Data.aggregate([
{
'$match': {
'$and': [
{ test_id: testId },
{ createdAt: { $gte: moment(req.query.from).startOf('day') } },
{ createdAt: { $lte: moment(req.query.to).endOf('day') } }
],
}
}]);
My definition is like this.
const Data = new Schema({
test_id: {
type: String,
required: true,
},
user_id: {
type: String,
required: true,
},
item: {
agent_id: { type: String, ref: 'Company' },
product_no: { type: String, required: true },
user_id: { type: String, required: true, ref: 'User' },
name: { type: String, required: true },
quantity: { type: Number, required: true },1 },
standard_price: { type: Number},
designation: { type: String },
category: { type: String },
weight: { type: Number },
materials: { type: String },
remarks: { type: String },
note: { type: String },
location: String,
image: String
}
}, {
timestamps: true,
});
And when I change the condition only { test_id: testId },
I can get the data. I doubt of term conditions in the aggregate.
But I don't know how to resolve it.
mongoose version is "5.11.15".
"mongoose": "5.11.15",

Mongoose, apply `.aggregate()` on array of subdocuments and get the result with other fields of document in least number of queries

This is my Mongoose Model:
const postSchema = new Schema({
user: {
type: Schema.Types.ObjectId,
ref: 'User',
required: true
},
caption: {
type: String
},
action: {
type: [{
actionName: {
type: String,
required: true
},
user: {
type: Schema.Types.ObjectId,
ref: 'User'
}
}],
default: []
},
shares: [{
type: Schema.Types.ObjectId,
ref: 'User'
}];
});
All I want is to have a mongodb query with or without using .aggregate() to get the user & caption field as it is but instead of action and shares I want their counts for a particular document.
Sample Document
{
_id: "fgsergehegoieofgesfglesfg",
user: "dfjksjdnfkjsdfkjsdklfjglkbj",
caption: "This is the post caption",
action: [
{
actionName: 'e1', user: "sdfasdsdfasdfdsdfac951e5c"
},
{
actionName: 'e1', user: "asdfmadfadfee103c9c951e5d"
},
{
actionName: 'e2', user: "op34937cdbae0cd4160bbec"
},
{
actionName: 'e2', user: "2543ebbasdfd1750690b5b01c"
},
{
actionName: 'e3', user: "asdfcfebdb5dd1750690b5b01d"
},
],
shares: ["ewrebdb5ddadsf5069sadf1d", "asdfsdfbb85dd1750690b5b01c", "fasec92dsfasde103c9c95df5d"]
};
Desired output after query:
{
_id: "fgsergehegoieofgesfglesfg",
user: 'dfjksjdnfkjsdfkjsdklfjglkbj',
caption: 'This is the post caption',
actionCount: [{ count: 1, actionName: 'e3' },
{ count: 2, actionName: 'e2' },
{ count: 2, actionName: 'e1' }],
shareCount: 3
}
I am able do get following results using .aggregate():
Query:
let data = await Test.aggregate([
{ $match: { _id: mongoose.Types.ObjectId("fgsergehegoieofgesfglesfg") } },
{ $unwind: "$action" },
{
$group: {
_id: "$action.actionName",
count: { $sum: 1 }
}
},
{
$project: {
_id: 0,
actionName: "$_id",
count: 1
}
}
]);
Result:
[
{ count: 1, actionName: 'e3' },
{ count: 2, actionName: 'e2' },
{ count: 2, actionName: 'e1' }
]
I just want to put this in the original document and get the result. Also, doing the same for share field. It would be better if this can be done in single query. I have tried using $replaceRoot along with $mergeObjects but don't know how to correctly use them. I am very new to mongodb and mongoose.
Please help. Thank you.

Since you're aggregating a nested array you need to run $grouptwice and $first can be used to preserve original document's field values:
await Test.aggregate([
{ $match: { _id: mongoose.Types.ObjectId("fgsergehegoieofgesfglesfg") } },
{ $unwind: "$action" },
{
$group: {
_id: { _id: "$_id", actionName: "$action.actionName" },
user: { $first: "$user" },
caption: { $first: "$caption" },
count: { $sum: 1 },
shareCount: { $first: { $size: "$shares" } }
}
},
{
$group: {
_id: "$_id._id",
user: { $first: "$user" },
caption: { $first: "$caption" },
shareCount: { $first: "$shareCount" },
actionCount: {
$push: {
actionName: "$_id.actionName",
count: "$count"
}
}
}
}
])
Mongo Playground

Inner joins 2 tables in mongoose

I'm trying to do an inner join that matches, but for some reason I get a left join.
I have 2 relations tables, and I want to get the movie with the genre names.
Consider the following models:
// Movie
const MovieSchema = new mongoose.Schema({
id: {
type: Number,
default: null,
required: true,
unique: true
},
title: {
type: String,
default: null,
required: true,
unique: true,
trim: true
},
});
const Movie = mongoose.model('Movie', MovieSchema);
module.exports = Movie;
// Genre
const GenreSchema = new mongoose.Schema({
id: {
type: Number,
default: null,
required: true,
unique: true
},
name: {
type: String,
default: null,
required: false,
trim: true,
unique: true
}
});
const Genre = mongoose.model('Genre', GenreSchema);
module.exports = Genre;
// MovieGenre
const MovieGenreSchema = new mongoose.Schema({
genreId: {
type: Number,
default: null,
required: true
},
movieId: {
type: Number,
default: null,
required: true
}
});
const MovieGenre = mongoose.model('MovieGenre', MovieGenreSchema);
module.exports = MovieGenre;
I try to do the following query:
{
$lookup:
{
from: MovieGenre.collection.name,
localField: 'id',
foreignField: 'movieId',
as: 'movieGenres'
}
},
{
$lookup: {
from: Genre.collection.name,
localField: 'g.id',
foreignField: 'genreId',
as: 'genreNames'
}
},
{
$match: {
'genreNames.name': 'Action'
}
}
and I get the results:
{
_id: 5ee9b51609f44c0f38262c94,
id: 26583,
title: 'The Keeper',
__v: 0,
movieGenres: [
{
_id: 5ee8b8cf0d186c20b4bf3ccd,
genreId: 28,
movieId: 26583,
__v: 0
},
{
_id: 5ee8b8cf0d186c20b4bf3cce,
genreId: 53,
movieId: 26583,
__v: 0
}
],
genreNames: [
{ _id: 5ee8b68f0d186c20b4b03a3d, id: 35, name: 'Comedy', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a3e, id: 80, name: 'Crime', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a40, id: 18, name: 'Drama', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a42, id: 53, name: 'Thriller', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a43, id: 28, name: 'Action', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a45, id: 14, name: 'Fantasy', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a46, id: 27, name: 'Horror', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a4a, id: 10752, name: 'War', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a4b, id: 10402, name: 'Music', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a4c, id: 37, name: 'Western', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a4d, id: 36, name: 'History', __v: 0 }
]
}
but what i expected to get is:
{
_id: 5ee9b51609f44c0f38262c94,
id: 26583,
title: 'The Keeper',
__v: 0,
movieGenres: [
{
_id: 5ee8b8cf0d186c20b4bf3ccd,
genreId: 28,
movieId: 26583,
__v: 0
},
{
_id: 5ee8b8cf0d186c20b4bf3cce,
genreId: 53,
movieId: 26583,
__v: 0
}
],
genreNames: [
{ _id: 5ee8b68f0d186c20b4b03a43, id: 28, name: 'Action', __v: 0 },
{ _id: 5ee8b68f0d186c20b4b03a42, id: 53, name: 'Thriller', __v: 0 },
]
}
Can you please tell me,
What am I doing wrong?
Thanks.

You just need to correct your second lookup with genre collection, i have added 2 approaches you can use anyone,
1) Using your approach:
localField pass previous lookup result's movieGenres.genreId
foreignField pass id of genre collection
{
$lookup: {
from: Genre.collection.name,
localField: "movieGenres.genreId",
foreignField: "id",
as: "genreNames"
}
}
if you want to filter the genreNames names from above lookup by name,
$filter to iterate loop of genreNames array and filter by name: Action
{
$addFields: {
genreNames: {
$filter: {
input: "$genreNames",
cond: { $eq: ["$$this.name", "Action"] }
}
}
}
}
Your final query would be,
{
$lookup: {
from: MovieGenre.collection.name,
localField: "id",
foreignField: "movieId",
as: "movieGenres"
}
},
{
$lookup: {
from: Genre.collection.name,
localField: "movieGenres.genreId",
foreignField: "id",
as: "genreNames"
}
},
{
$match: {
"genreNames.name": "Action"
}
},
{
$addFields: {
genreNames: {
$filter: {
input: "$genreNames",
cond: { $eq: ["$$this.name", "Action"] }
}
}
}
}
Playground
2) Using lookup with pipeline approach:
The alternate way to do this using lookup with pipeline,
let to pass movieGenres.genreId from above lookup
$match to match genreId using $expr expression match and name field and combine conditions using $and operations
{
$lookup: {
from: MovieGenre.collection.name,
localField: "id",
foreignField: "movieId",
as: "movieGenres"
}
},
{
$lookup: {
from: Genre.collection.name,
let: { genreIds: "$movieGenres.genreId" },
pipeline: [
{
$match: {
$and: [
{ $expr: { $in: ["$id", "$$genreIds"] } },
{ name: "Action" }
]
}
}
],
as: "genreNames"
}
}
Playground

Mongo Multiple Level Aggregate Lookup Group

I have 3 collections, User, Dispensary and City. I want my result to look like this:
{
_id: ,
email: ,
birthdate: ,
type: ,
dispensary: {
_id: ,
schedule: ,
name: ,
address: ,
phone: ,
user:,
city: {
name:,
},
},
}
However I am getting the city object out, in the first level, and I want to get it as child of the dispensary collection.
This is my current pipeline I'm using:
User.aggregate
([
{
$match: { "_id": id }
},
{
$lookup:
{
from: Dispensary.collection.name,
localField: "dispensary",
foreignField: "_id",
as: "dispensary"
},
},
{"$unwind": {path:"$dispensary",preserveNullAndEmptyArrays: true} ,},
{
$lookup:
{
from: City.collection.name,
localField: "dispensary.city",
foreignField: "_id",
as: "city"
},
},
{"$unwind": {path:"$city",preserveNullAndEmptyArrays: true}} ,
{
"$group": {
_id: "$_id",
email : { $first: '$email' },
birthdate : { $first: '$birthdate' },
type : { $first: '$type' },
dispensary: { $push: "$dispensary" },
city: { $push: "$city" },
},
},
{"$unwind": {path:"$dispensary",preserveNullAndEmptyArrays: true}} ,
{"$unwind": {path:"$city",preserveNullAndEmptyArrays: true}} ,
], (aggErr, aggResult) => {
(aggErr) ? console.log(aggResult)
: console.log(aggResult)
})
SCHEMAS:
const CitySchema = new Schema({
name: { type: String, required: true, unique:true },
zip: { type: String, required: true },
});
const DispensarySchema = new Schema({
name: { type: String, required: true },
address: { type: String, required: true },
longitude: { type: String, required: true },
latitude: { type: String, required: true },
phone: { type: String, required: true },
user: {type: mongoose.Schema.Types.ObjectId, ref: 'User'},
schedule: [{type: mongoose.Schema.Types.ObjectId, ref: 'Schedule'}],
city: {type: mongoose.Schema.Types.ObjectId, ref: 'City'},
})
const UserSchema = new Schema({
name: { type: String, required: true },
email: { type: String, required: true, unique: true },
password: { type:String, required: true },
birthdate: { type: Date, required: true },
type: { type: String, enum: ['ADMIN','DISPENSARY','CUSTOMER'], required: true},
verificationToken: { type: String, required: false },
resetPasswordToken: { type: String, required: false },
resetPasswordExpires: { type: String, required: false },
isVerified: { type: Boolean, required: true },
isActive: { type: Boolean, required: true },
last_session: { type: Date },
last_ip_session: { type:String },
dispensary: {type: mongoose.Schema.Types.ObjectId, ref: 'Dispensary'},
},
{ timestamps: true }
)

I think you don't need to use $group at all, you can use as: "dispensary.city" in your second $lookup
[
{
"$match": {
"_id": id
}
},
{
"$lookup": {
from: Dispensary.collection.name,
localField: "dispensary",
foreignField: "_id",
as: "dispensary"
},
},
{
"$unwind": {
path: "$dispensary",
preserveNullAndEmptyArrays: true
},
},
{
"$lookup": {
from: City.collection.name,
localField: "dispensary.city",
foreignField: "_id",
as: "dispensary.city" // modify here
},
},
{
"$unwind": {
path: "$dispensary.city", // modify here
preserveNullAndEmptyArrays: true
}
}
]

You can used another lookup method using pipeline, This allow you to make more condition/sub-query inside of the lookup function. see reference: aggregate-lookup
User.aggregate([
{
$match: { "_id": id }
},
{
$lookup: {
from: Dispensary.collection.name,
let: {dispensaryId: "$dispensary"},
pipeline: [
{
$match: {
$expr: {
$eq: ["$_id", "$$dispensaryId"]
}
}
},
{
$lookup:
{
from: City.collection.name,
localField: "city",
foreignField: "_id",
as: "city"
},
},
{
$unwind: {
path:"$city",
preserveNullAndEmptyArrays: true
}
}
]
as: "dispensary",
},
},
{
$unwind: {
path:"$dispensary",
preserveNullAndEmptyArrays: true
}
},
{
"$group": {
_id: : {
_id: "$_id",
email : '$email' ,
birthdate : '$birthdate' ,
type : '$type'
dispensary: "$dispensary"
}
}
}
], (aggErr, aggResult) => {
(aggErr) ? console.log(aggResult)
: console.log(aggResult)
})
Update: Pipeline NOTE: To reference variables in pipeline stages, use the "$$" syntax.`