We Keep Coding

Calculate Bitwise Operators Value

I have a ranks collection with a permissions field which are bitwise operators:
[
{
"_id": "xxxx",
"name": "Rank 1",
"permissions": 1
},
{
"_id": "xxxxxxxxx",
"name": "Rank 2",
"permissions": 2
}
]
Example users:
[
{
"_id":"1234",
"ranks":[
"xxxx",
"xxxxxxxxx"
]
}
]
The users collection containts a ranks value, which stores an array of the rank ids.
I'm wanting to get the user, and their ranks and set their permissions to a value.
const users = await this.collection.aggregate([
{
$match: { userID: '123' }
},
{ $limit: 1 },
{
$lookup: {
from: 'ranks',
localField: 'rank',
foreignField: '_id',
as: 'ranks'
}
},
{
$set: {
permissions: {
$arrayElemAt: ['$rank.permissions', 0]
}
}
},
{
$unwind: {
path: '$rank',
preserveNullAndEmptyArrays: true
}
}
]).toArray();
This obviously gets 1 value from the collection, I'm wanting to get all permissions and add the bitwise operators together.
Expected Output
{
"_id": "1234",
"ranks":[
"xxxx",
"xxxxxxxxx"
]
"permissions":3
}
Any help is appreciated!

Here's one way to "or" all the rank permissions by using a server-side javascript "$function".
db.users.aggregate([
{
"$match": {
"_id": 42
}
},
{
"$lookup": {
"from": "ranks",
"localField": "ranks",
"foreignField": "_id",
"pipeline": [
{
"$project": {
"_id": 0,
"permissions": 1
}
}
],
"as": "permissions"
}
},
{
"$set": {
"permissions": {
"$function": {
"body": "function(perms) {return perms.reduce((prevV, currV) => prevV | currV, 0)}",
"args": ["$permissions.permissions"],
"lang": "js"
}
}
}
}
])
Try it on mongoplayground.net.

With sample collection...
db = {
"permissions": [
{
"_id": "xxxx",
"name": "Rank 1",
"permissions": 1
},
{
"_id": "xxxxxxxxx",
"name": "Rank 2",
"permissions": 2
},
{
"_id": "xxxxxxx",
"name": "Rank 4",
"permissions": 4
}
],
"users": [
{
"_id": "1234",
"ranks": [
"xxxx",
"xxxxxxxxx"
]
},
{
"_id": "4567",
"ranks": [
"xxxx",
"xxxxxxx"
]
}
]
}
...try the following aggregation, which...
Finds the _id for the user 1234 in the users collection.
Looks for all the corresponding ranks in the permissions collection.
Unwinds to have one result per corresponding permission.
Aggregates the permissions and ranks.
db.users.aggregate([
{
$match: {
"_id": "1234"
}
},
{
$lookup: {
from: "permissions",
localField: "ranks",
foreignField: "_id",
as: "ranks"
}
},
{
$unwind: "$ranks"
},
{
$group: {
_id: "$_id",
ranks: {
$push: "$ranks._id"
},
permissions: {
$sum: "$ranks.permissions"
}
}
}
])
See MongoDB playground at...
https://mongoplayground.net/p/BCl57dNhupH
Important Note: This query groups the permissions by sum (rather than by boolean logical OR), so you must ensure that there are no duplicate permissions. If you can't ensure unique permissions per user, then suggest that the permissions are $pushed like the ranks, and then perform some post processing on the list of permissions to reduce via logical OR...

How to join three (multipe) collections with $lookup in mongodb?

How to join three (multipe) collections with $lookup in mongodb?
Hi I am looking to join data from three collection
users collection:
[
{
_id:0,
name:"abc",
phone:999999999
},
{
_id:1,
name:"xyz",
phone:888888888
},
]
product collection:
[
{
_id:"p01",
name:"product-name",
price:1200
},
{
_id:"p02",
name:"product-name1",
price:100
}
]
productreviews collection:
[
{
_id:"pr0",
userId:0,
productId:"p01",
star:4
},
{
_id:"pr1",
userId:1,
productId:"p01",
star:3
}
]
mongodb query:
product.aggregate([
{
$lookup: {
from: "productreviews",
localField: "_id",
foreignField: "productId",
as: "review",
},
},
{
$lookup: {
from: "users",
localField: "review.userId",
foreignField: "_id",
as: "review.userInfo",
},
},
])
I am not able to get that output which i need.
How can i get this following output:
{
product: [
{
_id: "p01",
name: "product-name",
price: 1200,
review: [
{
_id: "pr0",
userId: 0,
productId: "p01",
star: 4,
"userInfo": {
name: "abc",
phone: 999999999
}
},
{
_id: "pr1",
userId: 1,
productId: "p01",
star: 3,
"userInfo": {
"name": "xyz",
"phone": 888888888,
}
},
]
},
{
_id: "p02",
name: "product-name1",
price: 100,
},
]
}
Any help appreciated!. Thank You...

db.product.aggregate([
{
$lookup: {
from: "review",
localField: "_id",
foreignField: "productId",
as: "review",
},
},
{
$lookup: {
from: "users",
localField: "review.userId", //See here
foreignField: "_id",
as: "review.userInfo",
},
},
])
Local field name is userId in the second lookup.
playground
EDIT:
To preserve reviews also
Add a unwind stage
db.product.aggregate([
{
$lookup: {
from: "review",
localField: "_id",
foreignField: "productId",
as: "review",
},
},
{
"$unwind": "$review"
},
{
$lookup: {
from: "users",
localField: "review.userId",
foreignField: "_id",
as: "review.userInfo",
},
},
])
Update
To keep the docs where there is no match, preserve null arrays at unwind stage as below
{
$unwind: {
path: "$review",
"preserveNullAndEmptyArrays": true
}
}

Getting error, 'The field name '$acknowledged' cannot be an operator name' with aggregation query

I'm trying to fetch all latest messages between User A and any other user.
I keep running into the error ,
The field name '$acknowledged' cannot be an operator name
Not sure what I'm doing wrong here. Mongo playground.
The expected output should be the latest message exchanged between user with id 5a934e000102030405000001, and any other user.
[
{
"from": ObjectId("5a934e000102030405000002"),
"to": ObjectId("5a934e000102030405000001"),
"acknowledged": true,
date: "2020-04-17T18:26:34.353+00:00"
},
{
"from": ObjectId("5a934e000102030405000001"),
"to": ObjectId("5a934e000102030405000003"),
"acknowledged": false,
date: "2020-04-17T18:26:31.353+00:00"
},
{
"from": ObjectId("5a934e000102030405000004"),
"to": ObjectId("5a934e000102030405000001"),
"acknowledged": false,
date: "2020-04-17T18:26:29.353+00:00"
},
]

You had a typo here:
$acknowledged: { acknowledged: {
$first: "$acknowledged", --> $first: "$acknowledged"
}
},
and
then: "$responseTo", --> then: "$to",
db.Message.aggregate([
{
$match: {
$or: [
{
from: {
$in: [
ObjectId("5a934e000102030405000001")
]
}
},
{
to: {
$in: [
ObjectId("5a934e000102030405000001")
]
}
}
]
}
},
{
$sort: {
date: -1
}
},
{
$group: {
_id: {
userConcerned: {
$cond: [
{
$in: [
"$to",
[
ObjectId("5a934e000102030405000001")
]
]
},
"$to",
"$from"
]
},
interlocutor: {
$cond: [
{
$in: [
"$to",
[
ObjectId("5a934e000102030405000001")
]
]
},
"$from",
"$to"
]
}
},
id: {
$first: "$_id"
},
from: {
$first: "$from"
},
acknowledged: {
$first: "$acknowledged"
},
to: {
$first: "$to"
},
date: {
$first: "$date"
}
}
},
{
$lookup: {
from: "User",
localField: "to",
foreignField: "_id",
as: "to"
}
},
{
$unwind: "$to"
},
{
$lookup: {
from: "User",
localField: "from",
foreignField: "_id",
as: "from"
}
},
{
$unwind: "$from"
},
{
$project: {
_id: 0,
date: 1,
acknowledged: 1,
from: "$from._id",
to: "$to._id"
}
}
])
MongoPlayground

Mongoose: Group by Object id and push in one array, sort by createdAt, populate all objects and paginate it

This is an example of getList of Arrivals by 10 items per page:
router.get('/arrivals', isAuthenticated, async (request, response, next) => {
jsonPreProcessor.response = response;
const resPerPage = 10;
const page = request.query.page || 1;
Arrival.find({})
.populate({
path: 'product',
populate: {
path: 'type'
}
})
.select('-__v')
.skip((resPerPage * page) - resPerPage)
.limit(resPerPage).then(arrivals => {
Arrival.countDocuments({}).then(numberOfResults => {
return jsonPreProcessor.paginate(arrivals, page, Math.ceil(numberOfResults / resPerPage), numberOfResults);
}).catch(error => {
return jsonPreProcessor.error(error.message);
});
}).catch(error => {
return jsonPreProcessor.error(error.message);
});
});
And this is an output:
{
"time": "2020-01-16T10:11:22.588Z",
"message": "success",
"success": true,
"data": {
"list": [
{
"quantity": 1,
"discount": 0,
"_id": "5e0db80a37dd4437b4329960",
"product": {
"_id": "5e0cecaaa9a5cc2c7c62e379",
"title": "Ортопедический",
"type": {
"_id": "5ddcbc4685e53838dc564a44",
"title": "fsdkjhfs",
"createdAt": "2019-11-26T05:46:46.797Z",
"updatedAt": "2019-11-26T05:46:46.797Z",
"alt": "fsdkjhfs",
"__v": 0
},
...
"mode": "simple",
"createdAt": "2020-01-01T19:02:02.840Z",
"updatedAt": "2020-01-01T19:02:02.840Z",
"alt": "ortopedicheskij",
"__v": 0
},
...
"sellPrice": 6,
"minSellPrice": 0,
"createdAt": "2020-01-02T09:29:46.688Z",
"updatedAt": "2020-01-13T09:30:26.126Z"
},
... // And other 9 items
],
"currentPage": 1,
"pages": 2,
"numberOfResults": 16,
"incompleteResults": true
},
"type": null
}
All those are arrivals, and as you see arrivals have product. Some arrivals can have same product (by id)
My problem is to group arrivals which have same product. (sorted by first item of arrivals array) populate all objects.. and paginate. I use aggregate, and unfortunately I don't have any idea how to use it (first of all aggregate's output is random) with pagination and of course output is not what I need. I used this link as solving of my problem.
router.get('/arrivalls', isAuthenticated, async (request, response, next) => {
jsonPreProcessor.response = response;
Arrival.aggregate(
[
// This is not working (may be it's not working in array)
// {
// "$sort": {
// "createdAt": 1
// }
// },
{
"$group": {
"_id": "$product",
"arrivals": {
"$push": "$$ROOT"
}
}
},
// {
// "$lookup": {
// "from": "arrivals",
// "localField": "product",
// "foreignField": "_id",
// "as": "product"
// }
// },
// {
// "$unwind": {
// "path": "$arrivals"
// }
// }
]
).then(arrivals => {
// console.log(arrivals);
return jsonPreProcessor.success(arrivals);
}).catch(error => {
return jsonPreProcessor.error(error.message);
});
});
Output should be like:
"data": {
"list": [
{
"_id": "5e1d5dba611485397cfb0386",
"arrivals": [
{
"_id": "5e1d5e26611485397cfb0387",
"quantity": 6,
"discount": 0,
"product": {
"_id": "5e1d5dba611485397cfb0386",
... // etc
},
"sellPrice": 5000,
"minSellPrice": 4500,
"createdAt": "2020-01-14T06:22:30.366Z",
"updatedAt": "2020-01-14T09:14:13.824Z",
"__v": 0
},
{
"_id": "5e1ff4d15d059430e8405f94",
"quantity": 2,
"discount": 0,
"product": {
"_id": "5e1d5dba611485397cfb0386",
... // etc
},
"sellPrice": 7000,
"minSellPrice": 6000,
"comment": "",
"createdAt": "2020-01-16T05:29:53.907Z",
"updatedAt": "2020-01-16T05:29:53.907Z",
"__v": 0
}
]
},
{
"_id": "5e1d84884d387d2334a7e9d9",
"arrivals": [
{
// etc...
}
]
}
],
"currentPage": 1,
"pages": 2,
"numberOfResults": 16,
"incompleteResults": true
},
"type": null
}

I solved my problem.. And here is grouping, sorting, populating and paginating of list.
const resPerPage = 10;
const page = request.query.page || 1;
Arrival.aggregate(
[
{
$lookup: {
from: 'products',
localField: 'product',
foreignField: '_id',
as: 'product'
}
},
{
$unwind: {
path: "$product",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'branchdans',
localField: 'branch',
foreignField: '_id',
as: 'branch'
}
},
{
$unwind: {
path: "$branch",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'colors',
localField: 'color',
foreignField: '_id',
as: 'color'
}
},
{
$unwind: {
path: "$color",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'types',
localField: 'product.type',
foreignField: '_id',
as: 'product.type'
}
},
{
$unwind: {
path: "$product.type",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'subcategories',
localField: 'product.subcategory',
foreignField: '_id',
as: 'product.subcategory'
}
},
{
$unwind: {
path: "$product.subcategory",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'categories',
localField: 'product.subcategory.category',
foreignField: '_id',
as: 'product.subcategory.category'
}
},
{
$unwind: {
path: "$product.subcategory.category",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'countries',
localField: 'product.country',
foreignField: '_id',
as: 'product.country'
}
},
{
$unwind: {
path: "$product.country",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: 'manufacturers',
localField: 'product.manufacturer',
foreignField: '_id',
as: 'product.manufacturer'
}
},
{
$unwind: {
path: "$product.manufacturer",
preserveNullAndEmptyArrays: true
}
},
{
$group: {
"_id": "$product._id",
"sizes": {
"$first": "$product.sizes"
},
"title": {
"$first": "$product.title"
},
"type": {
"$first": "$product.type"
},
"subcategory": {
"$first": "$product.subcategory"
},
"country": {
"$first": "$product.country"
},
"manufacturer": {
"$first": "$product.manufacturer"
},
"description": {
"$first": "$product.description"
},
"comment": {
"$first": "$product.comment"
},
"mode": {
"$first": "$product.mode"
},
"createdAt": {
"$first": "$product.createdAt"
},
"updatedAt": {
"$first": "$product.updatedAt"
},
"alt": {
"$first": "$product.alt"
},
arrivals: {
$push: "$$ROOT"
},
"date": {
$last: "$createdAt"
}
},
},
{
$unset: "arrivals.product"
},
{
$sort: {
"date": 1
}
},
{
$skip: (resPerPage * page) - resPerPage
},
{
$limit: resPerPage
}
]
).then(arrivals => {
Arrival.aggregate([
{
$group: {
"_id": "$product",
arrivals: {
$push: "$$ROOT"
},
"date": {
"$last": "$createdAt"
}
},
},
{
$sort: {
"date": 1
}
},
{
$count: "numberOfResults"
}
]).then(countArrivals => {
if(countArrivals.length === 0){
return jsonPreProcessor.error('Ошибка при высчитывании прибытий товаров');
}
// Todo make more practise
return jsonPreProcessor.paginate(arrivals, page, Math.ceil(countArrivals[0].numberOfResults / resPerPage), countArrivals[0].numberOfResults);
}).catch(error => {
return jsonPreProcessor.error(error.message);
});
// return jsonPreProcessor.success(arrivals);
}).catch(error => {
return jsonPreProcessor.error(error.message);
});

Slow query performance: MongoDB with $lookup and $and/$or

I'm trying to find a way to create an engine that translates GraphQL query filters to MongoDB aggregations while keeping the performance. Our application has the requisite of limiting the results from collection A by applying filters to collection B, C and even D sometimes.
For better understanding, here's a sample about how a filter is translated to MongoDB.
This:
{
"filter": {
"return": null,
"AND": [{
"customer_WITH": {
"OR": [{
"code": "CUSTOMER NAME"
}, {
"commercialName_LIKE": "CUSTOMER NAME"
}, {
"corporateName_LIKE": "CUSTOMER NAME"
}]
}
}],
"OR": [{
"dispatcher_WITH": {
"company_WITH": {
"corporateName_LIKE": "COMPANY NAME"
}
}
}, {
"redispatcher_WITH": {
"company_WITH": {
"corporateName_LIKE": "COMPANY NAME"
}
}
}],
"reversal": null
}
}
Gets translated to this:
[{
"$match": {
"return": {
"$eq": null
},
"reversal": {
"$eq": null
},
"company": {
"$eq": ObjectId("xxxxxxxxxxxxxxxxxxxxxxxx")
}
}
}, {
"$lookup": {
"as": "dispatcher",
"from": "shippers",
"localField": "dispatcher",
"foreignField": "_id"
}
}, {
"$unwind": {
"path": "$dispatcher",
"preserveNullAndEmptyArrays": true
}
}, {
"$lookup": {
"as": "dispatcher.company",
"from": "companies",
"localField": "dispatcher.company",
"foreignField": "_id"
}
}, {
"$unwind": {
"path": "$dispatcher.company",
"preserveNullAndEmptyArrays": true
}
}, {
"$lookup": {
"as": "redispatcher",
"from": "shippers",
"localField": "redispatcher",
"foreignField": "_id"
}
}, {
"$unwind": {
"path": "$redispatcher",
"preserveNullAndEmptyArrays": true
}
}, {
"$lookup": {
"as": "redispatcher.company",
"from": "companies",
"localField": "redispatcher.company",
"foreignField": "_id"
}
}, {
"$unwind": {
"path": "$redispatcher.company",
"preserveNullAndEmptyArrays": true
}
}, {
"$lookup": {
"as": "customer",
"from": "customers",
"localField": "customer",
"foreignField": "_id"
}
}, {
"$match": {
"$or": [{
"dispatcher.company.corporateName": {
"$regex": /\sCOMPANY\sNAME/
}
}, {
"redispatcher.company.corporateName": {
"$regex": /\sCOMPANY\sNAME/
}
}],
"$and": [{
"$or": [{
"customer.code": {
"$eq": "CUSTOMER NAME"
}
}, {
"customer.commercialName": {
"$regex": /CUSTOMER\sNAME/
}
}, {
"customer.corporateName": {
"$regex": /CUSTOMER\sNAME/
}
}]
}]
}
}, {
"$unwind": {
"path": "$customer",
"preserveNullAndEmptyArrays": true
}
}, {
"$group": {
"_id": "$invoiceNo",
"__rootId": {
"$first": "$_id"
},
"company": {
"$first": "$company"
},
"customer": {
"$first": "$customer._id"
},
"dispatcher": {
"$first": "$dispatcher._id"
},
"redispatcher": {
"$first": "$redispatcher._id"
},
"driverPlate": {
"$first": "$driverPlate"
},
"key": {
"$first": "$key"
},
"activities": {
"$first": "$activities"
},
"serialNo": {
"$first": "$serialNo"
},
"invoiceNo": {
"$first": "$invoiceNo"
},
"incidents": {
"$first": "$incidents"
},
"deliveries": {
"$first": "$deliveries"
},
"return": {
"$first": "$return"
}
}
}, {
"$project": {
"_id": "$__rootId",
"company": "$company",
"customer": "$customer",
"dispatcher": "$dispatcher",
"redispatcher": "$redispatcher",
"driverPlate": "$driverPlate",
"key": "$key",
"activities": "$activities",
"serialNo": "$serialNo",
"invoiceNo": "$invoiceNo",
"incidents": "$incidents",
"deliveries": "$deliveries",
"return": "$return"
}
}, {
"$sort": {
"invoiceNo": -1
}
}, {
"$limit": 51
}]
The engine is smart enough to reallocate to the first position $match properties that don't require $lookups and right after $lookups if they do, however if they are within a $and/$or condition block, then they are reallocated after the last $lookup, regardless of what properties are there.
I could scan for what is used inside the $and and deconstruct it into new reallocated $match phases, but I need to figure how to handle the $or operator: I can't apply the same desconstruction idea on it because this would invalidate the condition.
So my question is: Is there an alternative way to use the phase $lookup along with $and/$or and improve the performance drastically?
Creating more indexes won't help because they're not used for the $lookup. Moving up $match phases, as the MongoDB team would suggest is also not possible because it would break the conditions. So I'm out of ideas now.
Best regards.