I'm trying to set up and update some capped collections in MongoDB using Node.js (using the native MongoDB driver).
My goal is to, upon running app.js, insert documents into a capped collection, and also to update existing documents in a capped collection. Both of these are running on setInterval(), so every few seconds.
My questions:
I want to create a collection if the collection does not already exist, but if it does I want to insert a document into it instead. What is the correct way to check this?
With capped collections, should I even be explicitly creating them first before inserting anything into them? Normally I believe you can just insert things into a collection without explicitly creating them first, but in this case I need to ensure they are capped. Once the capped collection exists I know how to insert new documents into it, the problem is I need some way to handle the app being used for the first time (on a new server) where the collection doesn't already exist, and I want to do this creation using node and not having to jump into the mongo cli.
The trick here is that the collection needs to be capped, so I can do something like: db.createCollection("collectionName", { capped : true, size : 100000, max : 5000 } ). That will create the capped collection for me, but every time I call it it will call createCollection() instead of updating or inserting - if I call createCollection(), once the collection already exists, will it completely overwrite the existing collection?
An alternative is to turn a collection into a capped one with: db.runCommand({"convertToCapped": "collectionName", size: 100000, max : 5000 });. The problem with this is that node doesn't see runCommand() as a valid function and it errors. Is there something else that I'm meant to be calling to get this to work? It works in the mongo cli but not within node
What type of query do you use to find the first document in a collection? Again, within the mongo cli I can use db.collections.find() with some query, but within node it states that find() is not a valid function
How would I use collection.update() to add some new fields to an existing document? Lets say the document is some simple object like {key1: "value", key2: "value"} but I have an object that contains {key3: "value"}. Key 3 does not exist in the current document, how would I add that to what currently exists? This is somewhat related to #4 above in that I'm not sure what to pass in as the query parameter given that find() doesn't seem to play well with node.
Regarding your questions 1 - 4 about capped collections and creating them automatically, there are several ways to do this. On the one hand, you could run a script to initialise your database so that it has the capped collections available to your client when you run it for the first time. On the other hand, you could have a check to see if there are any documents in the given collection before inserting a document. If there are, you just insert your document and if there aren't, you create the capped collection and then insert the document as a callback to that function. It would work something like this:
var host = "localhost",
port = 27017,
dbName = "so";
var MongoClient = require('mongodb').MongoClient, Server = require('mongodb').Server;
var mongoclient = new MongoClient(new Server(host, port));
var db = mongoclient.db(dbName);
db.open(function(err, db) {
if(err) throw err;
// Capped collection.
var capped = db.collection('capped');
// Document to be inserted.
var document = { "foo": 1, "bar": 1 }
capped.find().count(function(err, count) {
if(err) throw err;
if (count === 0) {
console.log("Creating collection...");
db.createCollection("capped",
{ "capped": true,
"size": 100000,
"max": 5000 },
function(err, collection) {
if(err) throw err;
// Insert a document here.
console.log("Inserting document...");
collection.insert(document, function(err, result) {
if (err) throw err;
});
});
} else {
// Insert your document here without creating collection.
console.log("Inserting document without creating collection...");
capped.insert(document, function(err, result) {
if (err) throw err;
});
}
});
});
Regarding question 5, you can use findOne() to find a document in the collection, though this is not necessarily the first or last. If you want to guarantee the first or last, you can run a find() with a sort() and limit() of 1. Sorting by _id ascending should give you the first document. More information here.
// Sort 1 for ascending, -1 for descending.
capped.find().sort([["_id", 1]]).limit(1).nextObject(function(err, item) {
console.log(item);
});
Finally for question 6, you just use the $set operator with the update() method. More information here.
capped.update({ "foo": 1 }, { "$set": { "bar": 2 } }, {}, function(err, result) {
console.log(result);
});
Note that you can only update documents in place for capped collections, so you cannot do the insert of the extra field you mention. There are other restrictions enumerated here that you might want to be aware of.
[EDIT: Add updating nested fields in last document.]
If you want to update a nested field in the first or last document (use 1 or -1 in the sort, respectively), you can fetch the document, extract the _id, then perform an atomic update on that document. Something like this:
capped.find().sort([["_id", -1]]).limit(1).nextObject(function(err, item) {
if(err) throw err;
capped.update({ "_id": item._id },
{ "$set": { "timeCollected": 15, "publicIP.ip" : "127.0.0.1" }},
function(err, result) {
if(err) throw err;
console.log(result);
});
});
Note that even when updating a field that exists in a document in a capped collection, you need to ensure that the new value fits in the space allocated for the document. So, for example, updating a string value from "1" to "127.0.0.1" will not necessarily work.
Related
I have been working on an authentication system lately that is supposed to either create a new user if no user with a given ID exists OR loads in a user with an existing ID. I also already figured out that you can return whether or not a user exists with in ID by using
User.count({_id: id}, function (err, count){
if(count > 0){
// User exists, continue with loading data
} else {
// User does not exist, create one
}
});
with User being an exported Mongoose Model mongoose.model('User', UserSchema);
If I created a function existsWithId(id) in my app.js now, it would work perfectly fine (judging by the fact that the previously mentioned code works perfectly fine), but I want to move this function to my user.js Model (which contains the Mongoose / MongoDB Code) instead so that I have it only for that specific model. Whenever I try to do that by using
UserSchema.statics.existsWithId = function(id) {
this.count({_id: id}, function (err, count){
return count > 0;
});
}
or similar methods, though, it will either just return undefined or an error once I run it inside of my app.js.
NOTE: I'm sorry if this is a kind of simple or even foolish question, but I am pretty new considering DB management with Mongoose and while I do have SOME experience with JS, I am not exactly a professional at it.
If any of you need additional code provided to answer the question, feel free to ask me and I will do so. Also, I can't +rep answers yet, but I always appreciate 'em :)
Your existsWithId method returns a value asynchronously so you can give a callback to it to get the result back like this :
UserSchema.statics.existsWithId = function(id, cb) {
this.count({ _id: id }, function(err, count) {
cb(err, count > 0);
});
}
User.existsWithId("5882c3be5aad09028d4d8b46", function(err, res) {
console.log(res);
});
I have the following query:
Section.find({"clause": sub.section}, function(err, docs){
var detail = parseInt(sub.detail);
docs[0].subClauses[0].additionalDetails[detail-1].percentile = sub.percentile;
docs[0].markModified("subClauses");
docs[0].save(function(err, doc){
if(err) { return callback(err) };
callback();
})
})
When I log the doc inside the .save function it appears to have saved successfully. However on querying the db elsewhere the save has not happened. The object its trying to save to is 3 deep like this:
[
{
"clause": "1",
"subClauses:
[
{
"clause":"1.1",
"additionalDetails:
[
{
"detail": "1",
"content": "whatever"
}
]
}
]
}
]
With several items inside each array.
Am I not able to save data to the nested array object using Mongo?
If I understand what you are trying to do, then I think what you are wanting to use is the Mongo Position Operator ($). This is used to index nested arrays in a document. In your current code you are only referencing the first object in the array. Using the position operator you will be able to reference other positions based of some other data, such as a query.
Here is the documentation for the operator: http://docs.mongodb.org/manual/reference/operator/update/positional/
And here is another Stack Overflow response which may help a bit also:
MongoDB $ operator
I have a MEAN stack based application and recently, I was trying to implement some caching mechanism for caching the query results. I implemented mongoose-cache.
mongoose-cache configuration
require('mongoose-cache').install(mongoose, {max: 150, maxAge:1000*60*10});
I have two document in sample collection say
{name:'dale', dep:2},
{name:'john', dep:4}
I run a query with mongoose-cache enabled and the maxAge is say 10 minutes.
sample.find().cache().exec(function (err, doc) {
// returns 2 document
});
Next, I inserted one more document say
{name:'rasig', dep:4} and execute the same query
sample.find().cache().exec(function (err, doc) {
// returns 2 document instead of 3
});
I executed the same query twice within 10 minutes, though there was a change in the collection, I got the previous result. Is there any way to drop the cached result once there is a change in the collection. If not, can you suggest something else to implement the same.
I am the author of a new Mongoose module called Monc
Using Monc is quite easy to clean up or purge the whole cache or even the the associated Query objects simple by using:
sample.find().clean().cache().exec(function (err, doc) {});
I have a bunch of mongodb collections as the below and they are created on the different date and use the date as their names:
2015-06-01
2015-06-02
2015-06-03
...
2015-06-30
total 30 collections
Let's say I ONLY want to keep the latest 10 collections today and today is 2015-06-30. Based on their names, you should know I only want to keep
2015-06-21
2015-06-22
...
2015-06-30
10 collections
Then tomorrow is 2015-07-01, I still want to keep the latest 10 collections and at the same time, 2015-07-01 collection is created.
So I have to delete 2015-06-21 to keep the total collection number equals to 10.
Then I will repeat this process every day to keep total 10 days collections in the mongodb.
How to achieve that using Mongoose API and Node.JS (JavaScript)?
setInterval and db.dropCollection will be used, I assume.
So presuming you have a list of collections in your database by your naming of the date like so:
[ '2015-01-01',
'2015-01-02',
'2015-01-03',
'2015-01-04',
'2015-01-05',
'2015-01-06',
'2015-01-07',
'2015-01-08',
'2015-01-09',
'2015-01-10',
'2015-01-11',
'2015-01-12' ]
Then if you want to remove all but the last 10 collections that are older than "today" then you would take this approach:
var async = require('async'),
crontab = require('node-crontab'),
mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/crontest');
var conn = mongoose.connection;
conn.once("open",function() {
var jobId = crontab.scheduleJob("0 0 * * *",function() {
var today = new Date().toISOString().substr(0,10);
var db = conn.db;
db.listCollections({ "name": { "$lt": today } })
.toArray(function(err,items) {
var names = items.map(function(item) {
return item.name;
})
async.eachLimit(
names.slice(0,names.length-10),10,function(name,callback) {
db.dropCollection(name,callback);
},function(err) {
if (err) throw err;
console.log('done');
}
);
});
});
});
I'm just putting the whole thing in node-crontab but shedule however you want.
The first thing you need to do is grab a handle to underlying native Db object from your mongoose connection:
var conn = mongoose.connection;
{ ... }
var db = conn.db;
And make sure that the connection is "actually open" when you run. It will be if other "mongoose methods" on your models have actually invoked, but really you should be sure, hence:
conn.once("open",function() {
// everything in here
});
Then it's simply just a matter of getting the ISOString for todays date in order to apply to the "filter" query called on .listCollections() which is going to return your collection names ( and other properties ) based on that filter. You always don't want to remove the current day.
Then it's just a matter of "slicing" that array in order to get only those collections that are not in the last 10 items. So slice with .length-10 to get the position to get from the starting index of the array.
Then just pass those into a loop, using async.eachLimit because the .dropCollection() call is async, also to "limit" how many tasks to run in parallel.
Once those are done then the job is complete and the collections are removed. Also there is no point in "creating" a collection ( add if you want ) as MongoDB will just do that as soon as you write something to your new name.
I have a alerts collection in my mongodb..
I wish to get the attributes of the object present in my alerts model, i have written the following code....
Alert.find({},function(err,alerts){
if(err)
console.log(err);
console.log(alerts);
alerts.forEach(function(result){
console.log(result.name);
console.log(result._id);
});
});
In first console, i am getting all the objects of alerts model,
but in second console it is 'Undefined' even result.name is present in the model..
But the _id is generated by mongodb itself and im able to get that id..
From 1st console:
{ __v: 0,
_id: 55128d3a1b35c366eb974285,
brand: 'SanDisk',
name: 'SanDisk Cruzer Blade 8 GB Pen Drives Black (Combo of 2)',
source:
{ name: 'Snapdeal',
productId: '141951',
inStock: true}
}
From 2nd console:
Undefined
From 3rd console
55128d3a1b35c366eb974285
Cannot able to get the problem, what is the correct way of doing this..
Thanks in advance
There is no problem here but you have to understand that MongoDB find() method returns a cursor to the documents that match the query criteria. So what you see in the console.log(alerts) is actually the cursor returned. When the find() method “returns documents,” the method is actually returning a cursor to the documents. So what you see in the first and third console outputs are cursor objects and properties.
You need to add a toArray() method to the resulting cursor from the Alert.find() operation, e.g.
Alert.find({}).toArray(function(err, alerts){
if(err)
console.log(err);
console.log(alerts);
alerts.forEach(function (result) {
console.log(result.name);
console.log(result._id);
});
});
The toArray() method returns an array that contains all the documents from a cursor. The method iterates completely the cursor, loading all the documents into RAM and exhausting the cursor.
I was using mongoose..
and the solution for the above is applying lean() function
i.e
Alert.find({}).lean().exec(function(err,alerts){
if(err)
console.log(err);
console.log(alerts);
alerts.forEach(function(result){
console.log(result.name);
console.log(result._id);
});
});