New to MongoDB, and working with NodeJS's mongodb package, I am trying to retrieve and update items based on a user's permission levels which are also stored in a collection.
I understand how to fetch all items:
const collection = db.collection('events');
// Find some documents
collection.find({}).toArray(function(err, docs) {
assert.equal(err, null);
console.log("Found the following records");
console.log(docs)
callback(docs);
});
But I wonder whether, given the following data structure:
{
events: {
1: {
id: 1,
name: "first event",
},
2: {
id: 2,
name: "2nd event",
},
3: {
id: 3,
name: "3rd event",
},
},
permissions: {
events: {
1: {
user1: "read",
user2: "write",
},
2: {
user2: "write",
},
3: {
user1: "readwrite",
user2: "write",
},
},
},
}
and given you are logged in as User1, the above fetch code can fetch only the relevant events which user1 has read access to according to the permissions specified?
For CRUDS such as updates, deletes and inserts, I can run a separate query and see whether the user has the needed access. However, get queries are common and IMHO should be straightforward so I wonder how this filtering can be done elegantly?
First of all, I would simplify your data model, since there is a thing that well might bite you in your back later down the road. You should not use values as keys, as you did in the event documents. Clearly, those are arrays, and the array elements already have an id they can be referenced by.
Furthermore, your data model (and subsequently my derived one) is only good for a couple of thousand events, as there is a 16MB limit on documents. In order to mitigate this, I will show you an updated data model and how to deal with it, too.
One document for events
Data model
Here is my updated data model. We basically got rid of the unnecessary values posing as keys.
{
"events": [{
"id": 1,
"name": "first event",
},
{
"id": 2,
"name": "2nd event",
},
{
"id": 3,
"name": "3rd event",
}
],
"permissions": [{
"event": 1,
"perms": [{
"user": "user1",
"permission": "read",
},
{
"user": "user2",
"permission": "write"
}
]
},
{
"event": 2,
"perms": [{
"user": "user2",
"permission": "write"
}]
},
{
"event": 3,
"perms": [{
"user": "user1",
"permission": "readwrite"
},
{
"user": "user2",
"permission": "write"
},
]
}
],
}
Which still gives you the possibility to search for specific events via db.events.find({"events.id":1},{"events.$":1}). This, of course works for each and every other property of an event.
Permission check
The basic question of wether a user has a permission to do a certain operation on a specific document can be rephrased as
Get me all documents with the properties I am interested in AND for which user Y has the permission to do Z.
This translates to a rather simple query:
db.events.find({
// The event we want...
"events.id": 1,
// ...shall be returned provided the following conditions are met
"permissions": {
$elemMatch: {
// The permissions for event with id 1...
"event": 1,
"perms": {
$elemMatch: {
// ...allow the user in question...
"user": "user1",
// ... to read said event.
"permission": "read"
}
}
}
}
}, {
// Subsequently, we return all matching events in all documents...
"events.$": 1
})
What we use here is that query conditions are evaluated as logical AND. If the user does not have the permission we requested ("read" in this example), no document is returned.
One document per event
Data model
However, as said, there is a 16MB size limit on MongoDB documents. So if you have a lot of events, or even when you are not really sure about the number of events you might have, we should approach the problem from a different (and even simpler) angle: One document per event. The data model gets stupidly simple:
// db.events.insertMany([
{
"_id": 1,
"name": "first event",
"permissions":[
{
"user":"user1",
"permission": "read"
},
{
"user":"user2",
"permission":"write"
}
]
},
{
"_id":2,
"name": "2nd event",
"permissions":[
{
"user": "user2",
"permission": "write"
}
]
},
{
"_id":3,
"name": "3rd event",
"permissions":[
{
"user": "user1",
"permission": "readwrite"
},
{
"user": "user2",
"permission": "write"
},
]
}
// ])
Permission check
Now, let us say we want to check wether user1 can actually read (which includes readwrite) event 3 and if so, return it. Becomes even less complicated than before:
db.events.find({
// The event we want...
"_id": 3,
// ...in case...
"permissions": {
$elemMatch: {
// ... user1...
"user": "user1",
// ..has either the permission...
$or: [{
// ...to read or...
"permission": "read"
}, {
// ... to readwrite.
"permission": "readwrite"
}]
}
}
}, {
// Either way, do not return the permissions.
permissions: 0
})
Conclusion
Regardless of which data model of the two you will choose, you can use the above queries as the query part of your update statements:
db.events.update(<queryFromAbove>,{[...]})
Needless to say, I would strongly suggest to use the "one-document-per-event" approach.
However, there are two things you need to keep in mind.
If something goes wrong, you need to do extra queries to find out what went wrong. The event might either not exist altogether, or the user might not have the necessary permission to execute the query/update. However, since this should be the exception, I personally could live with that.
The other thing to keep in mind is that you need to make sure that a user can never, ever change the permissions array. For rather obvious reasons.
Related
MongoDB Atlas Cluster version: 5.0.12
MERN Stack Application
Simple Search App that returns Specific ads when keywords are typed in an input.
Front end is handled by React-Redux.
And I am using Axios to request my server for data.
Server is using Mongo's aggregate pipeline function to search for text using $search,
and then a few different operators to fetch data from another collection.
There are two collections, the main one has a foreign key that references the second one.
Here is a sample json of both the collections
ads: {
_id: 1,
companyId: 1,
primaryText: "",
description: "",
CTA: "Sign Up",
imageUrl: "",
heading: "",
}
companies: {
_id: 1,
name: "Levi's",
url: "",
}
This is the search index that I have been using to look for keywords in the collection.
{
"mappings": {
"dynamic": true,
"fields": {
"company": {
"fields": {
"name": [
{
"dynamic": true,
"type": "document"
},
{
"type": "string"
}
]
},
"type": "document"
},
"description": [
{
"dynamic": true,
"type": "document"
},
{
"type": "string"
}
],
"heading": [
{
"dynamic": true,
"type": "document"
},
{
"type": "string"
}
],
"primaryText": [
{
"dynamic": true,
"type": "document"
},
{
"type": "string"
}
]
}
}
}
Mongo doesn't let me query $search anywhere in the pipeline except as the first operation.
So the order that works is this
$seach --> $lookup --> $project --> $unwind
This works but the only problem is that when I try to search for keyword that is present in the companies collection, like name: "Levi's", it doesn't respond with the corresponding ad.
So, In short I am trying to find a way to apply $search on a collection that has the gone through a $lookup.
Thank you, and I appreciate you spending time reading this.
I have an object which is representing a lesson
{
"_id": "62ad8ac4a5523541d78df217",
"lessonID": "35d02fc5-94dd-48e7-a315-7dcea6ac6806",
"teachers": [
"629232d2fc8fa8646b21cf3d"
],
"students": [
"62924d0a6055ad4d2533577b"
],
"subject": "English",
"start": 1655515800000,
"end": 1655517600000,
"roomNumber": "3",
"venue": "Elmwood Park",
"attendance": [
"62924d0a6055ad4d2533577b"
],
"paid": false,
"__v": 0
},
what i'm trying to do is to find all documents with the same "lessonID" and update "start" and "end" (timestamps).
I'd like to increment or decrease values by a certain amount on every document
I've been sitting on it for a while and I thought about doing this:
finding all documents that match my query
deleting all found documents
mapping through the array
inserting the mapped array of documents
it sounds extremely inefficient tho (and it causes duplicate or loss of data if any of the steps fail)... there must be a better way to do that.
As #Gibbs suggested, simply use $inc:
db.collection.update({
"lessonID": "35d02fc5-94dd-48e7-a315-7dcea6ac6806"
},
{
$inc: {start: amountStartInc, end: amountEndInc}
})
See how it works on the playground example
He there, I use to work with Relational DBS, but right now trying to implement e-commerce shop and use mongodb.
I need the product, sub-products and description (multi lang);
I prefere to separate everything by 3 collection (maybe its not a good idea because mongo use 1 collection for one entity, in my example, 3 collection for 1 entity)
"content": [{
pid: 1,
lang: "ru",
title: "Привет"
},
{
pid: 1,
lang: "en",
title: "Hello"
},
{
pid: 2,
lang: "ru",
title: "Пока"
},
{
pid: 2,
lang: "en",
title: "Bye"
}
],
"products": [{
"_id": 1,
"item": "almonds",
"price": 12,
},
{
"_id": 2,
"item": "pecans",
"price": 20,
},
],
"sub": [{
"_id": 11,
"pid": 1,
"features": {
"color": ["red"],
"size": 42
},
"qt": 5
},
{
"_id": 12,
"pid": 1,
"features": {
"color": ["red"],
"size": 43
},
"qt": 2
},
{
"_id": 13,
"pid": 1,
"features": {
"color": ["yellow"],
"size": 44
},
"qt": 3
},
{
"_id": 21,
"pid": 2,
"features": {
"color": ["yellow"],
"size": 41
},
"qt": 6
},
{
"_id": 22,
"pid": 2,
"features": {
"color": ["red"],
"size": 47
},
"qt": 10
}
]
Products should have sub-products in order to use filter, for example when i want to filter items i will seek into sub-product collection find all the yellow t-short for example with size 44, then i just $group the items by main productId and make $lookup with main products and return it.
Also in order to receive main product with description I should to do $lookup with content collection.
Is it a great idea or should i use 1 collection for product and content?
Like:
"products": [{
"_id": 1,
"item": "almonds",
"price": 12,
"content": [{
lang: "ru",
title: "Привет"
},
{
lang: "en",
title: "Hello"
},
},
]
and maybe should I include sub-item also to main product, like:
"products": [{
"_id": 1,
"item": "almonds",
"price": 12,
"content": [{
lang: "ru",
title: "Привет"
},
{
lang: "en",
title: "Hello"
},
},
"sub": [{
"features": {
"color": ["red"],
"size": 42
},
"qt": 5
},
{
"features": {
"color": ["red"],
"size": 43
},
"qt": 2
},
]
]
The main Question is it good idea to compare everything and don't care about size of the collection? And if so how should I do a filter on nested documents ('sub-products')(previously 'sub-products' collection was like a plain collection and I could make aggregation in order to find all items by color for example: {"features.color": { $in: ['red'] }}) how can i manage it with nested document, and it will not be overwhelmed operation?
Your question about the data model design decision of how to split up your entities/documents cannot be answered in a useful way with just the information given, but I can give you some aspects to think about for the decision.
MongoDb is a key-value store and as such is most useful if you can design your data in a way that uses mostly key-based lookups. This can be extended with creating additional indexes on document fields other than _id, which is indexed by default. Everything that is not indexed will result in collection scans and be very inefficient to query. Indexes can substantially increase the amount of storage required, so depending on your scenario cost may be a prohibiting factor to just index every field you want to query by later. That means when designing entities you will also have to consider estimated size of collections and the possibility to reference between collections (by guid for example).
So in order to make the right design decisions for your data model we cannot judge just based on the data schema you want to store, but rather based on the queries you are looking to perform later and the expected collection sizes / future scaling requirements. Those are aspects that you only touch very lightly in your question. For example if you plan to query all kind of complex joins and combinations of property values across all your entities, you may ask yourself if you can afford the extra storage cost of non-normalized data and additional indexes, or if a traditional (or modern) SQL-RDB, or maybe a graph database may be more suitable than a key-value store for your use case. Whereas if your database scale will be small at all times and your main concern is developer productivity these considerations may be worthless.
Your specific question about accessing "sub"-documents within an array of the parent "products" can be answered by that it is supported by using an elemmatch. For example:
db.products.find({sub: {$elemMatch: {'features.size':43, 'features.color':'red'}}})
Please note again that these queries will only be efficient if you index the fields in your query. In case of array sub-documents that means looking into multi-key indexes.
In order to acquire more knowledge for better decisioning around DB models and your questions about queries in MongoDB I recommend reading the official MongoDB data model design guide, the documentation for querying arrays, as well as googling some articles on normalization and the motivation of SQL vs noSQL in terms of scaling/sharding, ACID and eventual consistency.
I'm trying to do a POST to the Strapi API and can't seem to figure out how to attach a 'has and belongs to many' (many to many) relationship.
I've already tried the following body's:
events: ["ID", "ID"]
name: "name"
&
events: [ID, ID]
name: "name"
Which regarding the docs should be right, I think.
There's no error, I get a '200 OK' response. It adds the record but without the relations.
Event.settings.json:
{
"connection": "default",
"collectionName": "events",
"info": {
"name": "event",
"description": ""
},
"options": {
"increments": true,
"timestamps": [
"created_at",
"updated_at"
],
"comment": ""
},
"attributes": {
"name": {
"type": "string"
},
"artists": {
"collection": "artist",
"via": "events",
"dominant": true
}
}
}
Artist.settings.json:
{
"connection": "default",
"collectionName": "artists",
"info": {
"name": "artist",
"description": ""
},
"options": {
"increments": true,
"timestamps": [
"created_at",
"updated_at"
],
"comment": ""
},
"attributes": {
"name": {
"required": true,
"type": "string"
},
"events": {
"collection": "event",
"via": "artists"
}
}
}
I'm using the standard SQLite database, strapi version 3.0.0-beta.13 and tried the request through Postman, HTML & curl.
I would love to know how to attach the relation on POST
Update 23-07:
Did a fresh install of Strapi and now everything is working.
I think it's because your set you ID as a String instead of an Integer
{
events: [1, 2],
name: "My name"
}
And here 1 and 2 are the IDs of events you want to add.
Late reply. Hoping this might help someone!
Right now I am using Strapi v4.3.2 and was facing the same issue. I overcame this by overriding the default core controller for create as explained in official docs. Relations are now visible!
async create(ctx) {
const { data } = ctx.request.body;
const response = await strapi.entityService.create(
"api::collection.collection",
{
data: data,
}
);
return {response}
}
This is (still? again?) a bug in Strapi, see: https://github.com/strapi/strapi/issues/12238
As a workaround you need to add the find-permission to the user / role who is performing the request for the related content type (you want to check first if this is a security issue for your scenario or not - alternatively you might want to try Paratron's approach which is described in the comments).
I am using the Rally WSAPI 2.0p5 with and the JSON return
I am looking to get fields from multiple tables in a single response. Is this possible? For example I am trying to get the User Story and also get the Iteration.State in the same data response. I know it is possible to do client side and if that is the only way. Can someone provide and example how I handle the async response to build the table (array).
Simply add State to the list of attributes included in your fetch. Rally's WSAPI will populate the value for sub objects even if the main type being queried does not have that field.
launch: function() {
var userStories = Ext.create('Rally.data.WsapiDataStore', {
model: 'HierarchicalRequirement',
fetch: ['Iteration', 'State'],
autoLoad: true,
filters: [
{
property: 'Iteration.State',
value: 'Accepted'
}
],
limit: 10000,
listeners: { load: this._onDataLoaded, scope: this }
});
}
As a follow up for my original question. I recently came across the alpha release of the Batch Query WSAPI in Rally's WSAPI documentation. I would suggest the usage of the Batch Query to retrieve multiple Object Models in a single response.
As an example to get User Stories and get the Iteration Status in a single query.
{
"stories" : "/HierarchicalRequirement?fetch=Name,Iteration,State&query=(Iteration.State = Accepted)"
}
The result is something that is more usable and doesn't require multiple queries to the server. i.e.
"Results": [{
"_rallyAPIMajor": "1",
"_rallyAPIMinor": "40",
"_ref": "https://rally1.rallydev.com/slm/webservice/x/hierarchicalrequirement/xxxxxxxx.js",
"_objectVersion": "17",
"_refObjectName": "<user role> I would like <feature> for <benifit>",
"Name": "As a <user role> I would like <feature> for <benifit>",
"Iteration": {
"_rallyAPIMajor": "1",
"_rallyAPIMinor": "40",
"_ref": "https://rally1.rallydev.com/slm/webservice/x/iteration/xxxxxxxx.js",
"_objectVersion": "4",
"_refObjectName": "Sprint #",
"Name": "Sprint #",
"State": "Accepted",
"_type": "Iteration"
},
"Project": {
"_rallyAPIMajor": "1",
"_rallyAPIMinor": "40",
"_ref": "https://rally1.rallydev.com/slm/webservice/x/project/xxxxxxxx.js",
"_refObjectName": "Name",
"_type": "Project"
},
"_type": "HierarchicalRequirement"
},
....
]
For more information and a few resources:
https://rally1.rallydev.com/slm/doc/webservice/index.jsp?version=1.40
https://rally1.rallydev.com/slm/doc/webservice/batch.jsp
https://rally1.rallydev.com/slm/doc/webservice/BatchScratchPad.jsp