We Keep Coding

How to solve Mongo Error: E11000 duplicate key error collection [duplicate]

I was wondering if there is way to force a unique collection entry but only if entry is not null.
e
Sample schema:
var UsersSchema = new Schema({
name : {type: String, trim: true, index: true, required: true},
email : {type: String, trim: true, index: true, unique: true}
});
'email' in this case is not required but if 'email' is saved I want to make sure that this entry is unique (on a database level).
Empty entries seem to get the value 'null' so every entry wih no email crashes with the 'unique' option (if there is a different user with no email).
Right now I'm solving it on an application level, but would love to save that db query.
thx

As of MongoDB v1.8+ you can get the desired behavior of ensuring unique values but allowing multiple docs without the field by setting the sparse option to true when defining the index. As in:
email : {type: String, trim: true, index: true, unique: true, sparse: true}
Or in the shell:
db.users.ensureIndex({email: 1}, {unique: true, sparse: true});
Note that a unique, sparse index still does not allow multiple docs with an email field with a value of null, only multiple docs without an email field.
See http://docs.mongodb.org/manual/core/index-sparse/

tl;dr
Yes, it is possible to have multiple documents with a field set to null or not defined, while enforcing unique "actual" values.
requirements:
MongoDB v3.2+.
Knowing your concrete value type(s) in advance (e.g, always a string or object when not null).
If you're not interested in the details, feel free to skip to the implementation section.
longer version
To supplement #Nolan's answer, starting with MongoDB v3.2 you can use a partial unique index with a filter expression.
The partial filter expression has limitations. It can only include the following:
equality expressions (i.e. field: value or using the $eq operator),
$exists: true expression,
$gt, $gte, $lt, $lte expressions,
$type expressions,
$and operator at the top-level only
This means that the trivial expression {"yourField"{$ne: null}} cannot be used.
However, assuming that your field always uses the same type, you can use a $type expression.
{ field: { $type: <BSON type number> | <String alias> } }
MongoDB v3.6 added support for specifying multiple possible types, which can be passed as an array:
{ field: { $type: [ <BSON type1> , <BSON type2>, ... ] } }
which means that it allows the value to be of any of a number of multiple types when not null.
Therefore, if we want to allow the email field in the example below to accept either string or, say, binary data values, an appropriate $type expression would be:
{email: {$type: ["string", "binData"]}}
implementation
mongoose
You can specify it in a mongoose schema:
const UsersSchema = new Schema({
name: {type: String, trim: true, index: true, required: true},
email: {
type: String, trim: true, index: {
unique: true,
partialFilterExpression: {email: {$type: "string"}}
}
}
});
or directly add it to the collection (which uses the native node.js driver):
User.collection.createIndex("email", {
unique: true,
partialFilterExpression: {
"email": {
$type: "string"
}
}
});
native mongodb driver
using collection.createIndex
db.collection('users').createIndex({
"email": 1
}, {
unique: true,
partialFilterExpression: {
"email": {
$type: "string"
}
}
},
function (err, results) {
// ...
}
);
mongodb shell
using db.collection.createIndex:
db.users.createIndex({
"email": 1
}, {
unique: true,
partialFilterExpression: {
"email": {$type: "string"}
}
})
This will allow inserting multiple records with a null email, or without an email field at all, but not with the same email string.

Just a quick update to those researching this topic.
The selected answer will work, but you might want to consider using partial indexes instead.
Changed in version 3.2: Starting in MongoDB 3.2, MongoDB provides the
option to create partial indexes. Partial indexes offer a superset of
the functionality of sparse indexes. If you are using MongoDB 3.2 or
later, partial indexes should be preferred over sparse indexes.
More doco on partial indexes: https://docs.mongodb.com/manual/core/index-partial/

Actually, only first document where "email" as field does not exist will get save successfully. Subsequent saves where "email" is not present will fail while giving error ( see code snippet below). For the reason look at MongoDB official documentation with respect to Unique Indexes and Missing Keys here at http://www.mongodb.org/display/DOCS/Indexes#Indexes-UniqueIndexes.
// NOTE: Code to executed in mongo console.
db.things.ensureIndex({firstname: 1}, {unique: true});
db.things.save({lastname: "Smith"});
// Next operation will fail because of the unique index on firstname.
db.things.save({lastname: "Jones"});
By definition unique index can only allow one value to be stored only once. If you consider null as one such value it can only be inserted once! You are correct in your approach by ensuring and validating it at application level. That is how it can be done.
You may also like to read this http://www.mongodb.org/display/DOCS/Querying+and+nulls

Find all realted matched docs in an array of string

I have a mongoose schema model that have a field name tags , and it is an array of strings to store some tags in it for each document. I want something for example if I have an array of tags like ["test", "testimonials", "test Doc"] tags in it, when i search for test, it returns all documents with tags that they are testimonials or test doc , it should be work for example like wildcards (test*) .... can anyone help ?
this is the model
tags: {
type: [
{
type: String,
},
],
},

First of all, I'd tweak the Schema if possible. Your schema could be changed to this:
tags: [String]
This also just means an array of strings. You don't need to always need to use/specify the type key unless you're planning to add more fields to the tag schema, but it doesn't look like it from the question.
You can do the following to select all documents with a specific tag. Since I don't know what the name of your model is, I'll just call it "Model".
await Model.find({ tags: "tagName" })
OR
await Model.find({ tags: { $elemMatch: { someKey: someValue } } })
The later is only if you have other mongodb documents inside the array. Since you only have strings in the array, use the first method.

Mongodb check If field exists in an sub-document of an array

I am trying to check If a field exists in a sub-document of an array and if it does, it will only provide those documents in the callback. But every time I log the callback document it gives me all values in my array instead of ones based on the query.
I am following this tutorial
And the only difference is I am using the findOne function instead of find function but it still gives me back all values. I tried using find and it does the same thing.
I am also using the same collection style as the example in the link above.
Example
In the image above you can see in the image above I have a document with a uid field and a contacts array. What I am trying to do is first select a document based on the inputted uid. Then after selecting that document then I want to display the values from the contacts array where contacts.uid field exists. So from the image above only values that would be displayed is contacts[0] and contacts[3] because contacts1 doesn't have a uid field.
Contact.contactModel.findOne({$and: [
{uid: self.uid},
{contacts: {
$elemMatch: {
uid: {
$exists: true,
$ne: undefined,
}
}
}}
]}

You problems come from a misconception about data modeling in MongoDB, not uncommon for developers coming from other DBMS. Let me illustrate this with the example of how data modeling works with an RDBMS vs MongoDB (and a lot of the other NoSQL databases as well).
With an RDBMS, you identify your entities and their properties. Next, you identify the relations, normalize the data model and bang your had against the wall for a few to get the UPPER LEFT ABOVE AND BEYOND JOIN™ that will answer the questions arising from use case A. Then, you pretty much do the same for use case B.
With MongoDB, you would turn this upside down. Looking at your use cases, you would try to find out what information you need to answer the questions arising from the use case and then model your data so that those questions can get answered in the most efficient way.
Let us stick with your example of a contacts database. A few assumptions to be made here:
Each user can have an arbitrary number of contacts.
Each contact and each user need to be uniquely identified by something other than a name, because names can change and whatnot.
Redundancy is not a bad thing.
With the first assumption, embedding contacts into a user document is out of question, since there is a document size limit. Regarding our second assumption: the uid field becomes not redundant, but simply useless, as there already is the _id field uniquely identifying the data set in question.
The use cases
Let us look at some use cases, which are simplified for the sake of the example, but it will give you the picture.
Given a user, I want to find a single contact.
Given a user, I want to find all of his contacts.
Given a user, I want to find the details of his contact "John Doe"
Given a contact, I want to edit it.
Given a contact, I want to delete it.
The data models
User
{
"_id": new ObjectId(),
"name": new String(),
"whatever": {}
}
Contact
{
"_id": new ObjectId(),
"contactOf": ObjectId(),
"name": new String(),
"phone": new String()
}
Obviously, contactOf refers to an ObjectId which must exist in the User collection.
The implementations
Given a user, I want to find a single contact.
If I have the user object, I have it's _id, and the query for a single contact becomes as easy as
db.contacts.findOne({"contactOf":self._id})
Given a user, I want to find all of his contacts.
Equally easy:
db.contacts.find({"contactOf":self._id})
Given a user, I want to find the details of his contact "John Doe"
db.contacts.find({"contactOf":self._id,"name":"John Doe"})
Now we have the contact one way or the other, including his/her/undecided/choose not to say _id, we can easily edit/delete it:
Given a contact, I want to edit it.
db.contacts.update({"_id":contact._id},{$set:{"name":"John F Doe"}})
I trust that by now you get an idea on how to delete John from the contacts of our user.
Notes
Indices
With your data model, you would have needed to add additional indices for the uid fields - which serves no purpose, as we found out. Furthermore, _id is indexed by default, so we make good use of this index. An additional index should be done on the contact collection, however:
db.contact.ensureIndex({"contactOf":1,"name":1})
Normalization
Not done here at all. The reasons for this are manifold, but the most important is that while John Doe might have only have the mobile number of "Mallory H Ousefriend", his wife Jane Doe might also have the email address "janes_naughty_boy#censored.com" - which at least Mallory surely would not want to pop up in John's contact list. So even if we had identity of a contact, you most likely would not want to reflect that.
Conclusion
With a little bit of data remodeling, we reduced the number of additional indices we need to 1, made the queries much simpler and circumvented the BSON document size limit. As for the performance, I guess we are talking of at least one order of magnitude.

In the tutorial you mentioned above, they pass 2 parameters to the method, one for filter and one for projection but you just passed one, that's the difference. You can change your query to be like this:
Contact.contactModel.findOne(
{uid: self.uid},
{contacts: {
$elemMatch: {
uid: {
$exists: true,
$ne: undefined,
}
}
}}
)

The agg framework makes filtering for existence of a field a little tricky. I believe the OP wants all docs where a field exists in an array of subdocs and then to return ONLY those subdocs where the field exists. The following should do the trick:
var inputtedUID = "0"; // doesn't matter
db.foo.aggregate(
[
// This $match finds the docs with our input UID:
{$match: {"uid": inputtedUID }}
// ... and the $addFields/$filter will strip out those entries in contacts where contacts.uid does NOT exist. We wish we could use {cond: {$zz.name: {$exists:true} }} but
// we cannot use $exists here so we need the convoluted $ifNull treatment. Note we
// overwrite the original contacts with the filtered contacts:
,{$addFields: {contacts: {$filter: {
input: "$contacts",
as: "zz",
cond: {$ne: [ {$ifNull:["$$zz.uid",null]}, null]}
}}
}}
,{$limit:1} // just get 1 like findOne()
]);
show(c);
{
"_id" : 0,
"uid" : 0,
"contacts" : [
{
"uid" : "buzz",
"n" : 1
},
{
"uid" : "dave",
"n" : 2
}
]
}

Assign key and variable without need to assign in Schema

I have the following mongodb / mongoose Schema
var forumReplySchema = new Schema({
userid : String,
username: String,
forumid: {type: String, default: '1'},
message: {type: String, default: ''},
time: Number,
});
module.exports = mongoose.model('forum_replies', forumReplySchema);
with following query:
forum_replies.findOne({forumid: forumid}).then(function (reply) {
currentReply.username = user.username;
currentReply.rank = result.rank;
});
Currently username is assigned cause i have username property in the Schema.
Rank is not assigned cause its not in the Schema.
But is there a way for me to assign rank, without having it defined in the Schema?
Edit: aka, assign rank to the forum replies object without the need to save in Db.

you cannot add properties to mongoose document. If you want to do so, you will need to convert it into plain object first. There are couple of ways you could go about it. Following is one of them.
reply.toObject();
Then you can add properties to it.
//you have currentReply in your code, I just want to show general idea here
reply.rank = result.rank;
Is this what you are looking for?
Update:
Thanks for accepting answer :). Also look into lean() option, this returns you plain JS objects without you having to do the conversion manually. lean() also has performance benefit.

Removing Mongo document and all of its references?

At a high level, I'm looking to remove a document from Mongo as well as its referenced documents. Take a look at the following example Schemas:
var studentSchema = mongoose.Schema({
email: String,
name: String,
_class: { type: String, ref: 'Class' },
books: [{ type: Schema.Types.ObjectId, ref: 'Book' }]
});
module.exports = mongoose.model('Student', studentSchema);
var classSchema = mongoose.Schema({
gradeLevel: Number,
students: [{ type: Schema.Types.ObjectId, ref: 'Student' }]
});
module.exports = mongoose.model('Class', classSchema);
var bookSchema = mongoose.Schema({
author: String,
subject: String,
pages: Number
});
module.exports = mongoose.model('Book', bookSchema);
Now, my question is two fold:
1. If I want to .remove() a Class document from my database AND all Student documents referenced to it, what is the best way of doing this? //currently looping through students with the given class id and individually removing, then finally removing the class document.
2. If I want to .remove() a Class document from the db AND all Students AND their Books, is there a way to do this simply through a special remove statement in mongo? //Currently finding all students, removing their books, then remove themselves, then remove their referenced class.
Ideally, I would like a statement the can remove a mongo document, and anything referenced to it, along with any sub-references that sub-document may have. (e.g: Remove a Class and have mongo auto-remove all Students and their Books) Is this possible?

there is no statement that does what you want in mongoDb or mongoose. MongoDB is not the best choice if such operations that span multiple collections are very important for your application ("no automatically supported joins and referential integrity in MongoDB"). You also might need to model your data more "mongo-like" to achieve what you want.
You can do it more efficiently than looping if you have backreferences. The model.find() function returns a query object, that has a .remove(cb) method
mongooose.model('Student').find({_class: myClassToRemove._id}).remove(function(err) {});
For your books this will not work, because Book does not have a reference to Student. But if you do not share books between different students, then you should just store the books within the student object as "embedded documents" instead of using a different model and references. Then they will be deleted automatically when you delete a student.
If you do share one book instance between multiple students, you can not automatically delete the book when you delete a student, because you don't know whether another student uses the same book instance.