In my node app, I am saving data in to dynamic mongodb collections and those collections are named after a value in the data.
Sample code:
function saveToDb(data){
let collectionName = data.someType;
let collection = getCollection(collectionName);
return collection.save(data);
}
Please note that I do not know the collection name upfront, and the collections are being created as and when the corresponding data come for the first time.
If i want to index a few known fields in this document, I am thinking of calling collection.createIndex() after every save. But, will it hit the performance too much? Or any better ways to do this?
Finally went ahead with calling collection.createIndex() after every save. Monitoring it for a few weeks now and finding no performance problem identified with ~25K records.
Note: This doesnt mean calling collection.createIndex() after every save is the ideal solution. If someone has a better one, please feel free to comment.
Related
I have a firestore collection with a bunch of documents, each with plenty subfields. On a web page I need a list of a specific subfields from each document.
Currently I load the the entire database when you load the page and then loop through and get the wanted values. This uses way to many reads to get very little data.
Is there a way to solve this e.g. a autogenerated a collection that contains field from other collection in an array or something.
Many thanks in advance
Auto-creating such a subcollection with just the fields you need is a great way to reduce the bandwidth needed to load the data.
There is nothing built into Firestore to create those derived documents, but it's fairly easy to build something using Cloud Functions. Create a function that responds to a Firestore onWrite trigger, and write the subset of the data to its destination there. It's common to have a separate Cloud Function for each such use-case, and I regularly see projects with 100+ such functions.
I expect we'll also start seeing Firebase Extensions for this type of thing, but right now no-one seems to have built one.
I have a question about Collections - specifically, I want to have a large collection on a server, and load only small bits of it a piece at a time, in an unpredictable order, where I might stop wanting to have a local copy of any given piece at any time. Should I make a new subscription for each piece of data, and then stop it when I no longer want that piece of data? Or should I use some other method? Or should I just load large chunks of it that I won't use and leave them sitting around in my local copy of the collection?
Edit: Or should I have one subscription with a list of the ID's for each piece of data I want, and have the publication function specifically find each of those? Seems complicated, but it does keep me with only having to deal with one subscription.
Edit: Or maybe I should just skip using publications and subscriptions, and just use Methods to pass my data to the client? Loses a lot of functionality, and requires some extra work, but it does dodge most of the problems and should work just fine for my purposes.
Suppose Mongo collection ="items"
{
name:'item1',
type:'basic',
qty:40
}
you define collections on the Meteor server with
Items= new Mongo.Collection('items')
1.These collections contain all the data from the MongoDB collections, and you can run Items.find({...}) on them, which will return a cursor (a set of records, with methods to iterate through them and return them).
Meteor.publish('itemOver30', function itemPublication() {
return(Items.find({qty:{gte:10},{name:1,qty:1}));
});
This will return cursor to all the records with item qty over 30 in items collection(subset of total records, not whole collection).
2.Cursor is used to publish (send) a set of records (called a "record set"). You can optionally publish only some fields from those records. It is record sets (not collections) that clients subscribe to.
Meteor.subscribe('itemOver30');
On the client, you have Minimongo collections that partially mirror some of the records from the server. "Partially" because they may contain only some of the fields, and "some of the records" because you usually want to send to the client only the records it needs, to speed up page load, and only those it needs and has permission to access.
I'm using Meteor JS for a project so inherently I'm using MongoDB. I'm storing a user's check in and out actions. I'm currently storing them as individual docs in the collection. Each action contains 3 fields; in or out, time of action and userid. Is the best way to go though? Should I just have one doc per members and then store each action in an array? Is there another way? I anticipate several hundred members, but hopefully several thousands of members in the future. Thanks.
From experience, I can say that storing records instead of arrays is a better choice in the long run.
As far as Meteor is concerned, its reactivity handles collection records, but not individual fields in arrays. In other words, if one element gets added to the checkins array of a user object, the entire user object needs to be synchronized with the clients. If you store records instead, only the newly added record will be sent by the publication.
As far as MongoDB is concerned, there is a document size limit of 16MB. Not sure how frequent your checkins and checkouts are, but if you store them in an array, you might run into that limitation at some point.
Records are also easier to access than arrays.
For more details, see MongoDB data modeling and Database modeling in Bulletproof Meteor.
Hi Stackoverflow Family,
So this is a pretty big question; be prepared to read quite a bit.
Basically my team and I working on an Employee scheduling application (we're already well into it, so please nothing about changing our stack or anything like that). Anyways, we're building this with PHP, Mongo, JavaScript, JQuery, and Bootstrap; and I'm working more on the db side of things.
What I really want to find out after asking this question is to see if my current approach to the db is right; I've basically created several collections and documents that reference each other in order to access or reference specific data that I'm looking for. If that doesn't really make sense check out my schema below:
Employee Collection - Contains indexes such as name, employeeNumber, address, etc, employee availability, position & department.
(My issue here is I want it to reference other collections which contain their Shift information, but I can only really do that when I insert a document).
Shift Collection - Contains indexes such as shiftNumber, shiftStartTime, endTime (This collection, I basically want to reference to employee, such that for creating each employee I have their shift time connected to it).
**Schedule Collection ** - Now for the Schedule collection, this is the one that confuses me the most; I basically want our Calendar UI to be able to look through our schedule collection and be able to pull all the shifts in a certain day, or in a specific week. But I have no idea as to how I can approach this from the backend.
So far what I've done with the Schedule Collection is that I've mathematically created a Calendar year and placed that within the Schedule; basically it contains a document called Year, and in that Year it contains every day of the week with information such as day number, week number, leapYear, etc.
Anyways, I hope this is enough information; my main confusion is the with the main schedule, I think I nearly have the Employee collection functioning properly since it references the Department class with no issues. I just mainly can't figure out how to implement a full schedule in mongo!
Thanks guys!
well, i am creating a network that allows users creating posts and like them.
Asking on stackoverflow i've understood how to structure my database:
A collection which includes a document for each post.
A collection which includes a document for each like, in each of these documents there is a reference to post is referenced to.
When i want to get ALL likes about a post i can query the like collection looking for the reference to that post.
And till here i am ok. But assuming i'll have millions documents in like collection, i wondered how could i query and search among them in not too long time.
And i was advised of ensureIndex, in this case, i have to ensureindex of the field which contains reference to a post.
But when do i have to create this index? is enough to create it once (for example when i set up my database) and it will be as default in mongodb or do i have to do it during application life-time? thank you
But assuming i'll have millions documents in like collection, i wondered how could i query and search among them in not too long time.
I assume you would most likely want to do a count on the likes as an example?
You can't, instead you use optimizations to combat this. A count on millions of rows might get a bit slow.
A typical scenario are counters in SQL techs that you use to amend the parent row with a sum figure of its children.
Same applies to MongoDB.
You would aggregate important data to the top.
If you require to actually query the likes to show some who have liked it then you limit those likes. Google+ and other networks tend to limit the amount of likes they show to about 1,000.
And i was advised of ensureIndex,
Adding indexes to a database does help with actually searching for documents.
But when do i have to create this index? is enough to create it once
Yes, MongoDB will manage the index itself. You only need to ensure it once.