MongoDB Aggregate $lookup on array of object with additional field - javascript

I have two different collections (example below) methods & items. As for now, I'm using pre 3.6 vanilla aggregation query for $lookup:
MongoPlayground Example
{
$lookup: {
from: "items",
localField: "reagents._id",
foreignField: "_id",
as: "reagent_items"
}
}
The problem is that if I am using it, I miss quantity field (from methods.reagents embedded) during $lookup stage from original collection. For now, I return quantity right after lookup but as I heard, Mongo introduced from 3.6 a new syntax for lookup queries, so the question is:
Can it solve my problem for receiving the following results:
{
"_id": 1,
"name": "Test",
"reagent_items": [ // <= the exact schema of what I need after lookup
{
"_id": 1,
"name": "ItemOne",
"other": "field",
"quantity": 2 //quantity field from original {array of objects} after lookup
},
{
"_id": 2,
"name": "ItemTwo",
"other": "field",
"quantity": 4 //quantity field from original {array of objects} after lookup
}
],
"reagents": [ //original reagents field here just for example, we could remove it
{
"_id": 1,
"quantity": 2
},
{
"_id": 2,
"quantity": 4
}
]
}
methods
{
"_id": 1,
"name": "Test",
"reagents": [
{
_id: 1,
quantity: 2
},
{
_id: 2,
quantity: 4
}
]
}
items
{
"_id": 1,
"name": "ItemOne",
"other": "field"
},
{
"_id": 2,
"name": "ItemTwo",
"other": "field"
}

Use $map along with $arrayElemAt to find corresponding reagent for each reagent_items and the apply $mergeObjects to get one object:
db.methods.aggregate([
{
$lookup: {
from: "items",
localField: "reagents._id",
foreignField: "_id",
as: "reagent_items"
}
},
{
$project: {
_id:1,
name: 1,
reagents: 1,
reagent_items: {
$map: {
input: "$reagent_items",
as: "ri",
in: {
$mergeObjects: [
"$$ri",
{
$arrayElemAt: [ { $filter: { input: "$reagents", cond: { $eq: [ "$$this._id", "$$ri._id" ] } } }, 0 ]
}
]
}
}
}
}
}
])
Mongo Playground

Related

Calculate Bitwise Operators Value

I have a ranks collection with a permissions field which are bitwise operators:
[
{
"_id": "xxxx",
"name": "Rank 1",
"permissions": 1
},
{
"_id": "xxxxxxxxx",
"name": "Rank 2",
"permissions": 2
}
]
Example users:
[
{
"_id":"1234",
"ranks":[
"xxxx",
"xxxxxxxxx"
]
}
]
The users collection containts a ranks value, which stores an array of the rank ids.
I'm wanting to get the user, and their ranks and set their permissions to a value.
const users = await this.collection.aggregate([
{
$match: { userID: '123' }
},
{ $limit: 1 },
{
$lookup: {
from: 'ranks',
localField: 'rank',
foreignField: '_id',
as: 'ranks'
}
},
{
$set: {
permissions: {
$arrayElemAt: ['$rank.permissions', 0]
}
}
},
{
$unwind: {
path: '$rank',
preserveNullAndEmptyArrays: true
}
}
]).toArray();
This obviously gets 1 value from the collection, I'm wanting to get all permissions and add the bitwise operators together.
Expected Output
{
"_id": "1234",
"ranks":[
"xxxx",
"xxxxxxxxx"
]
"permissions":3
}
Any help is appreciated!
Here's one way to "or" all the rank permissions by using a server-side javascript "$function".
db.users.aggregate([
{
"$match": {
"_id": 42
}
},
{
"$lookup": {
"from": "ranks",
"localField": "ranks",
"foreignField": "_id",
"pipeline": [
{
"$project": {
"_id": 0,
"permissions": 1
}
}
],
"as": "permissions"
}
},
{
"$set": {
"permissions": {
"$function": {
"body": "function(perms) {return perms.reduce((prevV, currV) => prevV | currV, 0)}",
"args": ["$permissions.permissions"],
"lang": "js"
}
}
}
}
])
Try it on mongoplayground.net.
With sample collection...
db = {
"permissions": [
{
"_id": "xxxx",
"name": "Rank 1",
"permissions": 1
},
{
"_id": "xxxxxxxxx",
"name": "Rank 2",
"permissions": 2
},
{
"_id": "xxxxxxx",
"name": "Rank 4",
"permissions": 4
}
],
"users": [
{
"_id": "1234",
"ranks": [
"xxxx",
"xxxxxxxxx"
]
},
{
"_id": "4567",
"ranks": [
"xxxx",
"xxxxxxx"
]
}
]
}
...try the following aggregation, which...
Finds the _id for the user 1234 in the users collection.
Looks for all the corresponding ranks in the permissions collection.
Unwinds to have one result per corresponding permission.
Aggregates the permissions and ranks.
db.users.aggregate([
{
$match: {
"_id": "1234"
}
},
{
$lookup: {
from: "permissions",
localField: "ranks",
foreignField: "_id",
as: "ranks"
}
},
{
$unwind: "$ranks"
},
{
$group: {
_id: "$_id",
ranks: {
$push: "$ranks._id"
},
permissions: {
$sum: "$ranks.permissions"
}
}
}
])
See MongoDB playground at...
https://mongoplayground.net/p/BCl57dNhupH
Important Note: This query groups the permissions by sum (rather than by boolean logical OR), so you must ensure that there are no duplicate permissions. If you can't ensure unique permissions per user, then suggest that the permissions are $pushed like the ranks, and then perform some post processing on the list of permissions to reduce via logical OR...

mongodb fetch records if array exists

I have 2 collections:
Vehicles:
[
{
"_id": "a1",
"type:": "car",
"make": "Honda",
"specifications": ["1", "2"]
},
{
"_id": "a2",
"type:": "car",
"make": "Toyota",
"specifications": ["3", "4"]
},
{
"_id": "a3",
"type:": "car",
"make": "Honda",
"specifications": []
},
{
"_id": "a4",
"type:": "car",
"make": "Toyota"
}
]
Specifications:
[
{
"_id": "1",
"color": "Black"
},
{
"_id": "2",
"sunroof": "yes"
},
{
"_id": "3",
"engine": "1800 CC"
},
{
"_id": "4",
"bodyType": "Sedan"
}
]
I want to fetch those records which has at least one specification.
And also the details from specifications collections should appear in Vehicles collections somewhere.
Expected response:
[
{
"_id": "a1",
"make": "Honda",
"type:": "car",
"carSpecifications": [
{
"color": "Black"
},
{
"sunroof": "yes"
}
],
},
{
"_id": "a2",
"make": "Toyota",
"type:": "car",
"specifications": [
{
"engine": "1800 CC"
},
{
"bodyType": "Sedan"
}
]
}
]
Now what I tried so far is:
db.vehicles.find({type: "car", "specifications": {$exists: true}}, {fields: {"specifications.$": 1}}).fetch()
this query is returning all the records from Vehicles.
After getting all the records I put a loop on the records I get and check manually if specifications.length > 0 than I query from Specifications collection accordingly.
Can I achieve all this with a single query?
You should look for an aggregation query.
$match - Filter documents with "type:" "car" and specifications is not an empty array (with $ifNull, default as [] when specifications field is null or not existed).
$lookup - Vehicles collection join specifications collection (Refer to Use $lookup with an Array). Work with pipeline to return the array without the _id field (Refer to Correlated Subqueries Using Concise Syntax).
MongoDB v5 query
db.vehicles.aggregate({
$match: {
$and: [
{
"type:": "car"
},
{
$expr: {
$ne: [
{
$ifNull: [
"$specifications",
[]
]
},
[]
]
}
}
]
}
},
{
$lookup: {
from: "specifications",
localField: "specifications",
foreignField: "_id",
pipeline: [
{
$project: {
_id: 0
}
}
],
as: "specifications"
}
})
Sample Mongo Playground (v5)
MongoDB v4 query
db.vehicles.aggregate({
$match: {
$and: [
{
"type:": "car"
},
{
$expr: {
$ne: [
{
$ifNull: [
"$specifications",
[]
]
},
[]
]
}
}
]
}
},
{
$lookup: {
from: "specifications",
let: {
specifications: "$specifications"
},
pipeline: [
{
$match: {
$expr: {
$in: [
"$_id",
"$$specifications"
]
}
}
},
{
$project: {
_id: 0
}
}
],
as: "specifications"
}
})
Sample Mongo Playground (v4)

how to find array element using auto generated id in mongodb

currently, I am struggling with how the MongoDB document system works. I want to fetch array elements with an auto-generated id but how to fetch that specific data that I don't know.
my current schema is
const ItemPricesSchema = new mongoose.Schema({
_id : {
type: String
},
ItemsPrices: {
type: [{
barcode : {
type: String
},
itemName : {
type: String
},
price : {
type: String
}
}]
}
});
current data is stored in this way
{
"_id": "sha#c.c",
"ItemsPrices": [
{
"barcode": "345345",
"itemName": "maggie",
"price": "45",
"_id": "620a971e11120abbde5f4c3a"
},
{
"barcode": "356345",
"itemName": "monster",
"price": "70",
"_id": "620a971e11120abbde5f4c3b"
}
],
"__v": 0
}
what I want to achieve is that I want to find array elements through ids
if I want a specific array element with id "620a971e11120abbde5f4c3b" what should I do??
I have tried $unwind , $in, $match...
the result should be like
{
"_id": "sha#c.c",
"ItemsPrices": [
{
"barcode": "356345",
"itemName": "monster",
"price": "70",
"_id": "620a971e11120abbde5f4c3b"
}
],
"__v": 0
}
what I tried is like this from the answer
router.get('/filter/:id', async (req, res) => {
try {
const item = await ItemPricesSchema.aggregate([
{$project: {
"ItemsPrices": {
$filter: {
input: "$ItemsPrices",
as: "item",
cond: {
$eq: [
"$$item._id",
"620a8dd1c88ae3eb88a8107a"
]
}
}
}
}
}
])
res.json(item);
console.log(item);
} catch (error) {
res.status(500).json({message: error.message});
}
})
and returns something like this (Empty arrays)
[
{
"_id": "xvz#zyx.z",
"ItemsPrices": []
},
{
"_id": "zxc#xc.czx",
"ItemsPrices: []
},
{
"_id": "asd#asd.asd",
"ItemsPrices": []
},
{
"_id": "qwe#qwe.qwe",
"ItemsPrices": []
}
]
but If I search for price $$item.price
cond: {
$eq: [
"$$item.price",
"30"
]
}
it returns the perfect output
[
{
"_id": "xvz#zyx.z",
"ItemsPrices": []
},
{
"_id": "zxc#xc.czx",
"ItemsPrices: []
},
{
"_id": "asd#asd.asd",
"ItemsPrices": []
},
{
"_id": "qwe#qwe.qwe",
"ItemsPrices": [
{
"barcode":"234456345",
"price":"30",
"itemName":"monster",
"_id":"620a8dd1c88ae3eb88a8107a"
}
]
}
]
You can do an aggregation with $project and apply $filter on the array part. In mongoose you can apply the aggregation query in a more or less similar way https://mongoosejs.com/docs/api/aggregate.html
db.collection.aggregate([
{
$project: {
"ItemsPrices": {
$filter: {
input: "$ItemsPrices",
as: "item",
cond: {
$eq: [
"$$item._id",
mongoose.Types.ObjectId("620a971e11120abbde5f4c3b")
]
}
}
},
"__v": 1 //when projecting 1 means in the final result this field appears
}
}
])
more examples
demo
Option 1:
Use $filter in an aggregation query as explained by cmgchess
Option 2:
If you only want one object from array you can use $elemMatch like this:
db.collection.find({
"ItemsPrices._id": "620a971e11120abbde5f4c3b"
},
{
"ItemsPrices": {
"$elemMatch": {
"_id": "620a971e11120abbde5f4c3b"
}
}
})
Example here
But take care, using $elemMatch only the first element is returned. Check this other example where there are two objects with the desired _id but only returns one.
As said before, if you only one (or only exists one) maybe you can use find and $elemMatch to avoid a filter by the entire array. But if can be multiple values use $filter.

count product qty in sub document mongodb transaction in one month

below is just an example of my mongodb documents. and sum the product qty in every transaction documents in I expected 175 when select transaction August from product documents
the expected result in products documents
[
{
"_id":"61012014f7416b0a41db7055",
"name":"MIE",
"retailPrice":20000,
"qtyInTransaction":175
}
]
here is my document
products document
[
{
"_id":"61012014f7416b0a41db7055",
"name":"MIE",
"retailPrice":20000,
"description":"PRODUK BARU",
}
]
producutInTransaction document
[
{
"_id":"6106a3b1f1bbd62640c7b404",
"transactionDate":{"$date":"2021-08-01T00:00:00.000Z"},
"productsInTransactions":
[
{
"_id":"6106a3cff1bbd62640c7b405",
"productId":"61012014f7416b0a41db7055"},
"qty":100,
"price":5000,
"description":"asdn"
},
{
"_id":"6106a3cff1bbd62640c7b406",
"productId":"61012014f7416b0a41db7055"},
"qty":50,
"price":5000,
"description":"asdn"
}
]
},
{
"_id":"6106a3b1f1bbd62640c7b4007",
"transactionDate":{"$date":"2021-08-01T00:00:00.000Z"},
"productsInTransactions":
[
{
"_id":"6106a3cff1bbd62640c7b407",
"productId":"61012014f7416b0a41db7055"},
"qty":25,
"price":5000,
"description":"asdn"
},
]
}
]
Thanks for any help, and sorry for my English
You can use this query (adding ISODate value on whatever you want:
First $match by transactionData which has to be "greater than or equal" to desired date. Here you have to calculate your date using JS and use the variable.
Then $unwind productsInTransactions field to deconstruct the array.
$group by productId and $sum the qty value.
Later, $lookup to "join" with "product" collection and generate data field with desired values.
And least $project to shown only fields you want.
db.productInTransaction.aggregate([
{
"$match": {
"transactionDate": {
"$gte": ISODate("2021-07-01T00:00:00.000Z")
}
}
},
{
"$unwind": "$productsInTransactions"
},
{
"$group": {
"_id": "$productsInTransactions.productId",
"qty": {
"$sum": "$productsInTransactions.qty"
}
}
},
{
"$lookup": {
"from": "product",
"localField": "_id",
"foreignField": "_id",
"as": "data"
}
},
{
"$project": {
"name": {
"$arrayElemAt": [
"$data.name",
0
]
},
"retailPrice": {
"$arrayElemAt": [
"$data.retailPrice",
0
]
},
"qtyInTransaction": "$qty"
}
}
])
Example here
Result is:
[
{
"_id": "61012014f7416b0a41db7055",
"name": "MIE",
"qtyInTransaction": 175,
"retailPrice": 20000
}
]

How to improve this aggregate with many $projects

I have created an aggregate function and I feel it's pretty long and non-DRY. I'm wondering what ways I can improve it.
My Thread model has a sub-document called revisions. The function tries to get the most recent revision that has the status of APPROVED.
Here is the full model.
{
"_id": ObjectId("56dc750769faa2393a8eb656"),
"slug": "my-thread",
"title": "my-thread",
"created": 1457249482555.0,
"user": ObjectId("56d70a491128bb612c6c9220"),
"revisions": [
{
"body": "This is the body!",
"status": "APPROVED",
"_id": ObjectId("56dc750769faa2393a8eb657"),
"comments": [
],
"title": "my-thread"
}
]
}
And here is the aggregate function I want to improve.
Thread.aggregate([
{ $match: {
slug: thread
} },
{ $project: {
user: '$user',
created: '$created',
slug: '$slug',
revisions: {
$filter: {
input: '$revisions',
as: 'revision',
cond: { $eq: [ '$$revision.status', 'APPROVED' ] }
}
}
} },
{ $sort: { 'revisions.created': -1 } },
{ $project: {
user: '$user',
created: '$created',
slug: '$slug',
revisions: { $slice: ["$revisions", 0, 1] }
} },
{ $unwind: '$revisions'},
{ $project: {
body: '$revisions.body',
title: '$revisions.title',
user: '$user',
slug: '$slug',
created: '$created'
}}
])
Well you cannot really since there are $sort and $unwind stages in between on purpose. It's also basically "wrong", since the $sort cannot re-order the array until you $unwind it first.
Then it is better to use $group and $first instead, to just get the first element from the sort in each document:
Thread.aggregate([
{ "$match": {
"slug": thread
} },
{ "$project": {
"user": 1,
"created": 1,
"slug": 1,
"revisions": {
"$filter": {
"input": "$revisions",
"as": "revision",
"cond": { "$eq": [ "$$revision.status", "APPROVED" ] }
}
}
} },
// Cannot sort until you $unwind
{ "$unwind": "$revisions" },
// Now that will sort the elements
{ "$sort": { "_id": 1, "revisions.created": -1 } },
// And just grab the $first boundary for everything
{ "$group": {
"_id": "$_id",
"body": { "$first": "$revisions.body" },
"title": { "$first": "$revisions.title" },
"user": { "$first": "$user" },
"slug": { "$first": "$slug" },
"created": { "$first": "$created" }
}}
])
You could always reform the array with $push and then apply $arrayElemAt instead of the $slice to yield just a single element, but it's kind of superflous considering it would need another $project after the $group in the first place.
So even though there are "some" operations you can do without using $unwind, unfortunately "sorting" the arrays generated out of functions like $filter is not something that can be presently done, until you $unwind the array first.
If you didn't "need" the $sort on the "revisions.created" ( notably missing from your sample document ) then you can instead just use normal projection instead:
Thread.find(
{ "slug": slug, "revisions.status": "APPROVED" },
{ "revisions.$": 1 },
)
Only when sorting array elements would you need anything else, since the $ positional operator will just return the first matched element anyway.

Categories