We Keep Coding

MongoDB Aggregation: Counting distinct fields from array

Need to count distinct tags on all registers in a mongodb query, using JS.
Register structure:
{
"_id": {
"$oid": "62e593aed8fd9808777225e8"
},
"title": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
"author": {
"name": "Albert Einstein",
"url": "https://quotes.toscrape.com/author/Albert-Einstein"
},
"tag": [
"change",
"deep-thoughts",
"thinking",
"world"
]
}

This could be useful. In addition to get the different values for the field, it returns the number of appearances:
db.collection.aggregate([
{
"$unwind": "$tag"
},
{
"$group": {
"_id": "$tag",
"total": {
"$sum": 1
}
}
},
])
You can try here: https://mongoplayground.net/p/yXLYkJKO3Wf

How can I merge two mongo collections?

I am banging my head against the wall on this...
SEE UPDATE 1 (below) !
I am merging two collections together... I looked at this example ( and ~several~ other examples here on SO ... )
https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/#lookup-single-equality
I think I am really close, but my expected results are not the same as what I would expect out of the example.
Here is the schema for 'Event'
const EventSchema = new Schema({
name: {type: String, required: true},
})
Here is some 'Event' data
[
{
"_id": "5e8e4fcf781d96df5c1f5358",
"name": "358 Event"
},
{
"_id": "5e8e55c5a0f5fc1431453b5f",
"name": "b5f Event"
}
]
Here is 'MyEvent' schema:
const MyEventSchema = new Schema({
userId: {type: Schema.Types.ObjectId, required: true},
eventId: {type: Schema.Types.ObjectId, required: true},
})
Here is some 'MyEvent' data
[
{
"_id": "5e8f4ed2ddab5e3d04ff30b3",
"userId": "5e6c2dddad72870c84f8476b",
"eventId": "5e8e4fcf781d96df5c1f5358",
}
]
Here is my code ( the code is wrapped in a promise so it returns resolve and reject with data )
var agg = [
{
$lookup:
{
from: "MyEvent",
localField: "_id",
foreignField: "eventId",
as: "userIds"
}
}
];
Event.aggregate(agg)
.then( events => {
return resolve(events);
})
.catch(err => {
return reject(null);
})
Here are my results,
[
{
"_id": "5e8e4fcf781d96df5c1f5358",
"name": "358 Event",
"__v": 0,
"UserIds": []
},
{
"_id": "5e8e55c5a0f5fc1431453b5f",
"name": "b5f Event",
"__v": 0,
"UserIds": []
}
]
I expect to see UserIds filled in for event '358 Event', like this
What am I missing ???
[
{
"_id": "5e8e4fcf781d96df5c1f5358",
"name": "358 Event",
"__v": 0,
"UserIds": [
{"userId": "5e6c2dddad72870c84f8476b"}
]
},
{
"_id": "5e8e55c5a0f5fc1431453b5f",
"name": "b5f Event",
"__v": 0,
"UserIds": []
}
]
UPDATE 1
I found a mongo playground and what I have works there, but it doesn't work in my code ??
https://mongoplayground.net/p/fy-GP_yx5j7
In case the link breaks, here is configuration: * select 'bson multiple collections'
db={
"collection": [
{
"_id": "5e8e4fcf781d96df5c1f5358",
"name": "358 Event"
},
{
"_id": "5e8e55c5a0f5fc1431453b5f",
"name": "b5f Event"
}
],
"other": [
{
"_id": "5e8f4ed2ddab5e3d04ff30b3",
"userId": "5e6c2dddad72870c84f8476b",
"eventId": "5e8e4fcf781d96df5c1f5358",
}
]
}
Here is Query:
db.collection.aggregate([
{
$lookup: {
from: "other",
localField: "_id",
foreignField: "eventId",
as: "userIds"
}
}
])
Here is the result:
[
{
"_id": "5e8e4fcf781d96df5c1f5358",
"name": "358 Event",
"userIds": [
{
"_id": "5e8f4ed2ddab5e3d04ff30b3",
"eventId": "5e8e4fcf781d96df5c1f5358",
"userId": "5e6c2dddad72870c84f8476b"
}
]
},
{
"_id": "5e8e55c5a0f5fc1431453b5f",
"name": "b5f Event",
"userIds": []
}
]
any suggestions as to why this doesn't work in my code... but works in the playground?
UPDATE 2
I found this:
Need a workaround for lookup of a string to objectID foreignField
UPDATE 3
I have changed the schema to use ObjectId for ids now
still doesn't work
And they are ObjectIds :
RESOLUTION:
So the real answer was a combination of Update 2 and Update 3 and using the right collection name in the lookup.
Update 2 is pretty much my very same question... just using different table names
Update 3 is the correct way to solve this issue.
Mohammed Yousry pointed out the collection name might be wrong... so I looked at my schema and I did have it wrong - changed the name to the right name (along with ObjectId types) and it worked !

It seems there's a typo in from property in $lookup, MyEvent maybe not the collection name
db.collection.aggregate([
{
$lookup: {
from: "MyEvent", // here is the issue I think, check the collection name and make sure that it matches the one you write here
localField: "_id",
foreignField: "eventId",
as: "userIds"
}
}
])
in mongo playground you attached in the question, if you change the 'other' in the $lookup to anything else, or make a typo in it .. like others instead of other, you will face the same issue
so check that there is no typo in the word MyEvent that you populate from

$lookup giving no results in mongodb

Note: Edits below where I tried this directly using mongo shell and correct collection names, but still the same issue.
I am currently trying to learn Node and Mongodb. I am looking to understand how to add one document with another in a query. All the documentation points back to $lookup.
I have the two following models set up, which both work perfectly on their own
var BearSchema = new Schema({
name: String
});
module.exports = mongoose.model('Bear', BearSchema);
var CommentSchema = new Schema({
creator_id : { type: String, ref: 'Bear' },
comment: String
});
module.exports = mongoose.model('Comment', CommentSchema);
I will omit other set up details and get straight to the queries.
When I run Bear.find() I get the expected result...
[
{
"_id": "585887a29b7915f437742b88",
"name": "new bear",
"__v": 0
}
]
When I run Comment.find() I get the expected result...
[
{
"_id": "585887ae9b7915f437742b89",
"creator_id": "584de876238179030d7d7916",
"comment": "yoyoyo",
"__v": 0
},
{
"_id": "585887e09b7915f437742b8a",
"creator_id": "585887a29b7915f437742b88",
"comment": "ok lets give this a go",
"__v": 0
}
]
Note the creator_id in the second comment is the same as the _id in the bear result.
I then run
Bear.aggregate([
{
$lookup: {
from: "Comment",
localField: "_id",
foreignField: "creator_id",
as: "comments"
}
}
], function (err, bears) {
if (err)
res.send(err);
res.json(bears);
});
and get the following:
[
{
"_id": "585887a29b7915f437742b88",
"name": "new bear",
"__v": 0,
"comments": []
}
]
I was hoping the following would appear:
[
{
"_id": "585887a29b7915f437742b88",
"name": "new bear",
"__v": 0,
"comments": [
{
"_id": "585887e09b7915f437742b8a",
"creator_id": "585887a29b7915f437742b88",
"comment": "ok lets give this a go",
"__v": 0
}
]
}
]
I cant understand in this situation how it would know what "Comment" is referring to.
EDIT: From the documentation I can see the from field says: Specifies the collection in the same database to perform the join with. The from collection cannot be sharded.
EDIT 2: In mongoshell I have ran the following queries and their results, as you can see the same issue is still appearing even with the correct collection name, however I can now see ObjectId() may be the issue...
> show collections
bears
comments
> db.bears.find();
{ "_id" : ObjectId("585887a29b7915f437742b88"), "name" : "new bear", "__v" : 0 }
> db.comments.find();
{ "_id" : ObjectId("585887ae9b7915f437742b89"), "creator_id" : "584de87623817903
0d7d7916", "comment" : "yoyoyo", "__v" : 0 }
{ "_id" : ObjectId("585887e09b7915f437742b8a"), "creator_id" : "585887a29b7915f4
37742b88", "comment" : "ok lets give this a go", "__v" : 0 }
> db.bears.aggregate([ { $lookup: { from: "comments", localField: "_id", foreign
Field: "creator_id", as: "comments" } } ]);
{ "_id" : ObjectId("585887a29b7915f437742b88"), "name" : "new bear", "__v" : 0,
"comments" : [ ] }

whenever you'r using $lookup, you must add an extra "s" in "from" field.
for example:
if your table name is
"register"
then you have to write
"registers"
Note: at the time of $lookup only

I resolved this. There were two issues.
The Bear Schema _id is actually an ObjectID() so it wasnt comparing the two correctly.
I misunderstood what collection names were and so Comment would not have been recognised.
Solution:
When creating the Comment Model I used Schema.ObjectId
var CommentSchema = new Schema({
creator_id : { type: Schema.ObjectId, ref: 'Bear' },
comment: String
});
When doing the query I used comments instead of Comment as this was the collection named Mongoose created.
Bear.aggregate([
{
$lookup: {
from: "comments",
localField: "_id",
foreignField: "creator_id",
as: "comments"
}
}

Keep the main document when filtered array is empty in aggregation pipeline

This is really an addition to a previously asked question. With some help from #JohnnyHK I can now remove unwanted subdocuments from an array based on a particular criteria: deleted != null. I discovered that my $unwind pipe broke when there were no items in the array so I also implemented a $cond to add in a dummy object where the array was empty. Here is the code so far:
Collection.aggregate([
{ $match:
{ _id: ObjectID(collection_id) }
},
{
$project:
{
_id: 1,
name: 1,
images: {
$cond:
[
{
$eq:
[
"$images",
[]
]
},
[
dummyImg // Variable containing dummy object
],
'$images'
]
},
}
},
{ $unwind: "$images" },
{ $match:
{ "images.deleted": null }
},
// Regroup the docs by _id to reassemble the images array
{$group: {
_id: '$_id',
name: {$first: '$name'},
images: {$push: '$images'}
}}
], function (err, result) {
if (err) {
console.log(err);
return;
}
console.log(result);
});
The problem now occurs when the array is not empty but only contains objects where deleted is null. I $unwind the images but the $match doesn't find any matches so can't the perform the final $group.
I'm thinking along the lines of pushing in the dummy object at the beginning of the pipeline and then counting the images towards the end. The dummy object will stay if it is the only one but will need removing if there are other image objects that have made it through the pipeline.
If this is a sensible route I'd be glad of some pointers. If my thinking is way off, any tips to steer me in the right direction will be gratefully received.
Thank you.

Modern MongoDB releases of course simply apply $filter and $addFields to write the filtered and possibly empty array result into the document:
Collection.aggregate([
{ "$addFields": {
"images": {
"$filter": {
"input": "$filter",
"as": "i",
"cond": { "$eq": [ "$$i.deleted", null ] }
}
}
}}
])
The answer you were given before wasn't a very modern or efficient answer and there are since other and better ways to deal with filtering the content from arrays than doing $unwind, $match and $group.
As on MongoDB 2.6 you can do this with unique array identifiers:
Collection.aggregate([
{ "$project": {
"name": 1,
"images": { "$cond": [
{ "$eq": [{ "$size": { "$ifNull": [ "$images",[]] }}, 0] },
{ "$ifNull": [ "$images", [] ] },
{ "$setDifference": [
{ "$map": {
"input": "$images",
"as": "i",
"in": { "$cond": [
{ "$eq": [ "$$i.deleted", null ] },
"$$i",
false
]}
}},
[false]
]}
]}
}}
],
The $map operator transforms arrays in place in the document by returning each inspected element evaluated by the given expression. Here you can use $cond in order to test the field value and decide whether to return the field as is or otherwise return false.
The $setDifference operation "compares" the resulting transformed array with the other singular element array [false]. This has the effect of removing all items that did not match from the array and leaves behind just that and even an empty array where there were no matches.
The following with $redact is safe as long as your document does not contain the same referenced property at multiple levels of the document. The somewhat funny looking condition is because a "deleted": null property is actually being projected ( for evaluation purposes ) at the levels where it does not exists. This is needed because $redact is being used in a "recursive" way, descending the document tree to decide what it gets rid of, or "redacts":
Collection.aggregate([
{ "$redact": {
"$cond": [
{ "$eq": [ { "$ifNull": [ "$deleted", null ] }, null ] },
"$$DESCEND",
"$$PRUNE"
]
}}
]
That really is the most simple logic to implement for your specific purpose. Just remember that you likely couldn't use this later if you added another "deleted" field in your document somehow meaning something else.
If you really are stuck with a version of MongoDB earlier than 2.6 and don't have access to those operations, then of course you need to do the $unwind, $match and $groupprocess. So care needs to be taken both at the beginning with empty or missing arrays as well as when matching arrays with no matching entries.
One approach:
Collection.aggregate([
// Cater for missing or empty arrays
{ "$project": {
"name": 1,
"images": { "$cond": [
{ "$eq": [{ "$ifNull": [ "$images", [] ] }, [] ] },
{ "$const": [{ "deleted": false }] },
"$images"
]}
}},
// Safe to unwind
{ "$unwind": "$images" },
// Just count the matched array entries first
{ "$group": {
"_id": "$_id",
"name": { "$first": "$name" },
"images": { "$push": "$images" },
"count": { "$sum": { "$cond": [
{ "$eq": [ "$images.deleted", null ] },
1,
0
]}}
}},
// Unwind again
{ "$unwind": "$images" },
// Match either non deleted or unmatched array
{ "$match": {
"$or": [
{ "images.deleted": null},
{ "count": 0 }
]
}},
// Group back with the things that were matched
{ "$group": {
"_id": "$_id",
"name": { "$first": "$name" },
"images": { "$push": "$images" },
"count": { "$first": "$count" }
}},
// Replace the un-matched arrays with empty ones
{ "$project": {
"name": 1,
"images": { "$cond": [
{ "$eq": [ "$count", 0 ] },
[],
"$images"
]}
}}
],
So there is a bit more lifting there but the general principle is to get a "count" of the matched elements only and when you are filtering you also keep the array items with a 0 "count" but just replace those whole arrays later.
You can also consider here that if you maintained and "activeCount" field on your document in the first place then you would remove the need to calculate that and drop a few stages.
And of course the other argument here is that you could save yourself the trouble of this by actually maintaining both "active" and "deleted" items in separate arrays. Doing that with each update removes any need to filter via aggregate. I suppose though it all depends on your real purposes. But food fo thought.
Of course this is all tested based on your original data with some modifications to suit the test cases:
{
"_id" : ObjectId("54ec9cac83a214491d2110f4"),
"name" : "my_images",
"images" : [
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2311026b0cb289ed04188"),
"deleted" : null,
"date_added" : ISODate("2015-02-28T21:20:16.961Z")
},
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2314a26b0cb289ed04189"),
"deleted" : ISODate("2015-02-24T15:38:14.826Z"),
"date_added" : ISODate("2015-02-28T21:21:14.910Z")
},
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2315526b0cb289ed0418a"),
"deleted" : null,
"date_added" : ISODate("2015-02-28T21:21:25.042Z")
},
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2315d26b0cb289ed0418b"),
"deleted" : null,
"date_added" : ISODate("2015-02-28T21:21:33.081Z")
}
]
},
{
"_id" : ObjectId("54fa6ca87c105bc872cc1886"),
"name" : "another",
"images" : [ ]
},
{
"_id" : ObjectId("54fa6cef7c105bc872cc1887"),
"name" : "final",
"images" : [
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2314a26b0cb289ed04189"),
"deleted" : ISODate("2015-02-24T15:38:14.826Z"),
"date_added" : ISODate("2015-02-28T21:21:14.910Z")
}
]
}
Which all versions produce with safe results:
{
"_id" : ObjectId("54ec9cac83a214491d2110f4"),
"name" : "my_images",
"images" : [
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2311026b0cb289ed04188"),
"deleted" : null,
"date_added" : ISODate("2015-02-28T21:20:16.961Z")
},
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2315526b0cb289ed0418a"),
"deleted" : null,
"date_added" : ISODate("2015-02-28T21:21:25.042Z")
},
{
"ext" : "jpeg",
"type" : "image/jpeg",
"_id" : ObjectId("54f2315d26b0cb289ed0418b"),
"deleted" : null,
"date_added" : ISODate("2015-02-28T21:21:33.081Z")
}
]
},
{
"_id" : ObjectId("54fa6ca87c105bc872cc1886"),
"name" : "another",
"images" : [ ]
},
{
"_id" : ObjectId("54fa6cef7c105bc872cc1887"),
"name" : "final",
"images" : [ ]
}

Sorting records on number of embedded documents

I am using mongodb and mongoose, And I have a situation where a user creates a rating and a review, what I want is to fetch review of the product whose review has the highest number of votes. Is it possible to do it in mongo?
The structure is
{ "product" : ObjectId("53ccfa53b502542b2f463acd"),
"_id" : ObjectId("53e8675aea39355818ec4ab2"),
"vote" : [ ],
"review" : "Blah Blah Blah",
"stars" : 3,
"email" : "user#example.com", "__v" : 0 }
Now I want to show the review that has achieved the most votes, I know if after find() I put sort() and limit() function, it can be achieved through a field present on the same document level however I do not know how to handle multiple records in this case 'vote'....

The best thing you can do is to maintain a "voteCount" on the document itself. The reasons will become self apparent in a moment.
You can maintain this as members are added or removed from the array. Let's say you are using an ObjectId and the $push and $pull update operators to do this. So you also you $inc with a bit of query logic to make sure you don't duplicate the "User ObjectId" casting the vote. Assuming a model called "Product":
Product.update(
{
"_id": ObjectId("53e8675aea39355818ec4ab2"),
"votes": { "$ne": ObjectId("53e87caaca37ffa384e5a931") }, // the user ObjectId
},
{
"$push": { "votes": ObjectId("53e87caaca37ffa384e5a931" }, // same Id
"$inc": { "voteCount": 1 }
},
function(err) {
}
);
And to remove:
Product.update(
{
"_id": ObjectId("53e8675aea39355818ec4ab2"),
"votes": ObjectId("53e87caaca37ffa384e5a931"), // the user ObjectId
},
{
"$pull": { "votes": ObjectId("53e87caaca37ffa384e5a931" }, // same Id
"$inc": { "voteCount": -1 }
},
function(err) {
}
);
Then it's just a matter of sorting on the field:
Product.find().sort({ "voteCount": -1 }).limit(1).exec(function(err,doc) {
});
But if for some reason you cannot see fit to keep the "voteCount" in the document, then you need to manually "project" this with the aggregation framework. Etiher using the $size aggregate method where you have MongoDB 2.6 or greater:
Product.aggregate(
[
{ "$project": {
"product": 1,
"vote": 1,
"review": 1,
"stars": 1,
"email": 1,
"voteCount": { "$size": "$vote" }
}},
{ "$sort": { "voteCount": -1 } },
{ "$limit": 1 }
],
function(err,result) {
}
);
Or by $unwind on the array and getting the count via $sum for earlier versions:
Product.aggregate(
[
{ "$unwind": "$vote"
{ "$group": {
"_id": "$_id",
"product": { "$first": "$product" },
"vote": { "$push": "$vote" },
"review": { "$first": "$review" },
"stars": { "$first": "$stars" },
"email": { "$first": "$email" },
"voteCount": { "$sum": 1 }
}},
{ "$sort": { "voteCount": -1 } },
{ "$limit": 1 }
],
function(err,result) {
}
);
The aggregation approach really does not make that much sense to implement unless you really need other calculations than the array length. So best just to keep it in the document.

The best way to do so with MongoDB is to add new counter field to explicitly store the number of votes:
{ "product" : ObjectId("53ccfa53b502542b2f463acd"),
"_id" : ObjectId("53e8675aea39355818ec4ab2"),
"vote" : [ {...}, {...}, {...} ],
"vote_count": 3, // <-- new field
"review" : "Blah Blah Blah",
"stars" : 3,
"email" : "user#example.com", "__v" : 0 }
Of course, you have other options, like using Aggregation Pipeline. But adding new field is the best option, because it'll allow you to build an index on this field and do indexed queries.