I have a Javascript Array filled with mean Values and I want to insert them into a collection with a field named "mean". The Field "mean" already exists and has already values in them and now I want to update them with the values of the Array. To be more specific: I want the first Value of the Array to be in the first Document under the field "mean" and so on. I have 98 Documents and the Array has also a length of 98.
The Collection looks like this with the name "cmean":
{
"_id" : "000",
"mean" : 33.825645389680915
}
{
"_id" : "001",
"mean" : 5.046005719077798
}
and the Array:
[
33.89923155012405,
5.063347068609219
]
You can use the forEach method on the array to iterate it and update the collection. Use the index to get the _id to be used in the update query, something like the following:
meansArray.forEach(function(mean, idx) {
var id = db.cmean.find({}).skip(idx).limit(1).toArray()[0]["_id"];
db.cmean.updateOne(
{ "_id": id },
{ "$set": { "mean": mean } },
{ "upsert": true }
);
});
For large collections, you can streamline your db performance using bulkWrite as follows:
var ops = [];
meansArray.forEach(function(mean, idx) {
var id = db.cmean.find({}).skip(idx).limit(1).toArray()[0]["_id"];
ops.push({
"updateOne": {
"filter": { "_id": id },
"update": { "$set": { "mean": mean } },
"upsert": true
}
});
if (ops.length === 1000 ) {
db.cmean.bulkWrite(ops);
ops = [];
}
})
if (ops.length > 0)
db.cmean.bulkWrite(ops);
update({"_id": id}, $set: {"mean": myArray}, function(res, err) { ... });
If you are using mongoose, you also have to change data model from string to array.
Related
In MongoDB, is it possible to update the value of a field using the value from another field? The equivalent SQL would be something like:
UPDATE Person SET Name = FirstName + ' ' + LastName
And the MongoDB pseudo-code would be:
db.person.update( {}, { $set : { name : firstName + ' ' + lastName } );
The best way to do this is in version 4.2+ which allows using the aggregation pipeline in the update document and the updateOne, updateMany, or update(deprecated in most if not all languages drivers) collection methods.
MongoDB 4.2+
Version 4.2 also introduced the $set pipeline stage operator, which is an alias for $addFields. I will use $set here as it maps with what we are trying to achieve.
db.collection.<update method>(
{},
[
{"$set": {"name": { "$concat": ["$firstName", " ", "$lastName"]}}}
]
)
Note that square brackets in the second argument to the method specify an aggregation pipeline instead of a plain update document because using a simple document will not work correctly.
MongoDB 3.4+
In 3.4+, you can use $addFields and the $out aggregation pipeline operators.
db.collection.aggregate(
[
{ "$addFields": {
"name": { "$concat": [ "$firstName", " ", "$lastName" ] }
}},
{ "$out": <output collection name> }
]
)
Note that this does not update your collection but instead replaces the existing collection or creates a new one. Also, for update operations that require "typecasting", you will need client-side processing, and depending on the operation, you may need to use the find() method instead of the .aggreate() method.
MongoDB 3.2 and 3.0
The way we do this is by $projecting our documents and using the $concat string aggregation operator to return the concatenated string.
You then iterate the cursor and use the $set update operator to add the new field to your documents using bulk operations for maximum efficiency.
Aggregation query:
var cursor = db.collection.aggregate([
{ "$project": {
"name": { "$concat": [ "$firstName", " ", "$lastName" ] }
}}
])
MongoDB 3.2 or newer
You need to use the bulkWrite method.
var requests = [];
cursor.forEach(document => {
requests.push( {
'updateOne': {
'filter': { '_id': document._id },
'update': { '$set': { 'name': document.name } }
}
});
if (requests.length === 500) {
//Execute per 500 operations and re-init
db.collection.bulkWrite(requests);
requests = [];
}
});
if(requests.length > 0) {
db.collection.bulkWrite(requests);
}
MongoDB 2.6 and 3.0
From this version, you need to use the now deprecated Bulk API and its associated methods.
var bulk = db.collection.initializeUnorderedBulkOp();
var count = 0;
cursor.snapshot().forEach(function(document) {
bulk.find({ '_id': document._id }).updateOne( {
'$set': { 'name': document.name }
});
count++;
if(count%500 === 0) {
// Excecute per 500 operations and re-init
bulk.execute();
bulk = db.collection.initializeUnorderedBulkOp();
}
})
// clean up queues
if(count > 0) {
bulk.execute();
}
MongoDB 2.4
cursor["result"].forEach(function(document) {
db.collection.update(
{ "_id": document._id },
{ "$set": { "name": document.name } }
);
})
You should iterate through. For your specific case:
db.person.find().snapshot().forEach(
function (elem) {
db.person.update(
{
_id: elem._id
},
{
$set: {
name: elem.firstname + ' ' + elem.lastname
}
}
);
}
);
Apparently there is a way to do this efficiently since MongoDB 3.4, see styvane's answer.
Obsolete answer below
You cannot refer to the document itself in an update (yet). You'll need to iterate through the documents and update each document using a function. See this answer for an example, or this one for server-side eval().
For a database with high activity, you may run into issues where your updates affect actively changing records and for this reason I recommend using snapshot()
db.person.find().snapshot().forEach( function (hombre) {
hombre.name = hombre.firstName + ' ' + hombre.lastName;
db.person.save(hombre);
});
http://docs.mongodb.org/manual/reference/method/cursor.snapshot/
Starting Mongo 4.2, db.collection.update() can accept an aggregation pipeline, finally allowing the update/creation of a field based on another field:
// { firstName: "Hello", lastName: "World" }
db.collection.updateMany(
{},
[{ $set: { name: { $concat: [ "$firstName", " ", "$lastName" ] } } }]
)
// { "firstName" : "Hello", "lastName" : "World", "name" : "Hello World" }
The first part {} is the match query, filtering which documents to update (in our case all documents).
The second part [{ $set: { name: { ... } }] is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline). $set is a new aggregation operator and an alias of $addFields.
Regarding this answer, the snapshot function is deprecated in version 3.6, according to this update. So, on version 3.6 and above, it is possible to perform the operation this way:
db.person.find().forEach(
function (elem) {
db.person.update(
{
_id: elem._id
},
{
$set: {
name: elem.firstname + ' ' + elem.lastname
}
}
);
}
);
I tried the above solution but I found it unsuitable for large amounts of data. I then discovered the stream feature:
MongoClient.connect("...", function(err, db){
var c = db.collection('yourCollection');
var s = c.find({/* your query */}).stream();
s.on('data', function(doc){
c.update({_id: doc._id}, {$set: {name : doc.firstName + ' ' + doc.lastName}}, function(err, result) { /* result == true? */} }
});
s.on('end', function(){
// stream can end before all your updates do if you have a lot
})
})
update() method takes aggregation pipeline as parameter like
db.collection_name.update(
{
// Query
},
[
// Aggregation pipeline
{ "$set": { "id": "$_id" } }
],
{
// Options
"multi": true // false when a single doc has to be updated
}
)
The field can be set or unset with existing values using the aggregation pipeline.
Note: use $ with field name to specify the field which has to be read.
Here's what we came up with for copying one field to another for ~150_000 records. It took about 6 minutes, but is still significantly less resource intensive than it would have been to instantiate and iterate over the same number of ruby objects.
js_query = %({
$or : [
{
'settings.mobile_notifications' : { $exists : false },
'settings.mobile_admin_notifications' : { $exists : false }
}
]
})
js_for_each = %(function(user) {
if (!user.settings.hasOwnProperty('mobile_notifications')) {
user.settings.mobile_notifications = user.settings.email_notifications;
}
if (!user.settings.hasOwnProperty('mobile_admin_notifications')) {
user.settings.mobile_admin_notifications = user.settings.email_admin_notifications;
}
db.users.save(user);
})
js = "db.users.find(#{js_query}).forEach(#{js_for_each});"
Mongoid::Sessions.default.command('$eval' => js)
With MongoDB version 4.2+, updates are more flexible as it allows the use of aggregation pipeline in its update, updateOne and updateMany. You can now transform your documents using the aggregation operators then update without the need to explicity state the $set command (instead we use $replaceRoot: {newRoot: "$$ROOT"})
Here we use the aggregate query to extract the timestamp from MongoDB's ObjectID "_id" field and update the documents (I am not an expert in SQL but I think SQL does not provide any auto generated ObjectID that has timestamp to it, you would have to automatically create that date)
var collection = "person"
agg_query = [
{
"$addFields" : {
"_last_updated" : {
"$toDate" : "$_id"
}
}
},
{
$replaceRoot: {
newRoot: "$$ROOT"
}
}
]
db.getCollection(collection).updateMany({}, agg_query, {upsert: true})
(I would have posted this as a comment, but couldn't)
For anyone who lands here trying to update one field using another in the document with the c# driver...
I could not figure out how to use any of the UpdateXXX methods and their associated overloads since they take an UpdateDefinition as an argument.
// we want to set Prop1 to Prop2
class Foo { public string Prop1 { get; set; } public string Prop2 { get; set;} }
void Test()
{
var update = new UpdateDefinitionBuilder<Foo>();
update.Set(x => x.Prop1, <new value; no way to get a hold of the object that I can find>)
}
As a workaround, I found that you can use the RunCommand method on an IMongoDatabase (https://docs.mongodb.com/manual/reference/command/update/#dbcmd.update).
var command = new BsonDocument
{
{ "update", "CollectionToUpdate" },
{ "updates", new BsonArray
{
new BsonDocument
{
// Any filter; here the check is if Prop1 does not exist
{ "q", new BsonDocument{ ["Prop1"] = new BsonDocument("$exists", false) }},
// set it to the value of Prop2
{ "u", new BsonArray { new BsonDocument { ["$set"] = new BsonDocument("Prop1", "$Prop2") }}},
{ "multi", true }
}
}
}
};
database.RunCommand<BsonDocument>(command);
MongoDB 4.2+ Golang
result, err := collection.UpdateMany(ctx, bson.M{},
mongo.Pipeline{
bson.D{{"$set",
bson.M{"name": bson.M{"$concat": []string{"$lastName", " ", "$firstName"}}}
}},
)
I want to create new field in my document, lets call it "shelf", it will be an object.
Next I want to make two $set operations - I want to put arrays named "Tom" and "Anna" into my "shelf".
The problem is that I can't match correct query to do that.
I'm using nodejs MongoDB driver.
var myid = 'Tom-Anna'
var TomArray = ["Tom"]
var AnnaArray = ["Anna"]
await db.collection('people').updateOne(
{ pairid: myid },
{ $set: { shelf: TomArray } },
{ upsert: true }
)
await db.collection('people').updateOne(
{ pairid: myid },
{ $set: { shelf: AnnaArray } },
{ upsert: true }
)
Finally, the "shelf" document containing only "AnnaArray", because it's overwriting previously added "TomArray".
I can't add "Tom" and "Anna" array to "shelf" at the same time because content of arrays are generated separately.
I was trying this code:
var name = 'Tom'
var array = ['Tom']
await db.collection('people').updateOne(
{ pairid: myid },
{ $set: { shelf[name]: array } },
{ upsert: true }
)
But it's throwing following error:
{ $set: { shelf[name]: array } },
^
SyntaxError: Unexpected token [
My goal is to set my field like JSON:
"shelf": { "Tom": ["Tom"], "Anna": ["Anna"] }
You can use dot notation to specify nested key name:
var name = 'Tom'
var array = ['Tom']
db.people.update({ pairid: 1 }, { $set: { [`shelf.${name}`]: array } })
i have the following document , it has two array's , one inside the other ,
attachment array and files array inside attachment array .
i want to delete an element inside files array using this element _id . but its not working with me , i tried this code , it return
{ n: 144, nModified: 0, ok: 1 }
Invoice.update({}, {
$pull: {
"attachment":
{
"files":
{
$elemMatch:
{ _id: ObjectId("5b7937014b2a961d082de9bf") }
}
}
}
}, { multi: true })
.then(result => {
console.log("delete", result);
});
this is how the document looks like
You can try below update query in 3.6 version.
Invoice.update(
{},
{"$pull":{"attachment.$[].files":{_id:ObjectId("5b7969ac8fb15f3e5c8e844e")}}},
{"multi": true}, function (err, result) {console.log(result);
});
Use db.adminCommand( { setFeatureCompatibilityVersion: 3.6 or 4.0 depending on your version } ) if your are upgrading from old version.
For Mongodb version prior to 3.6
There is only one nested level here so you can simply use $ positional operator.
Invoice.update(
{ "attachment.files._id": mongoose.Types.ObjectId("5b7937014b2a961d082de9bf") },
{ "$pull": { "attachment.$.files": { "_id": mongoose.Types.ObjectId("5b7937014b2a961d082de9bf") }}},
{ "multi": true }
)
For Mongodb version 3.6 and above
If you want to update multiple elements inside attachement array then you can use $[] the all positional operator.
const mongoose = require("mongoose")
Invoice.update(
{ "attachment.files._id": mongoose.Types.ObjectId("5b7937014b2a961d082de9bf") },
{ "$pull": { "attachment.$[].files": { "_id": mongoose.Types.ObjectId("5b7937014b2a961d082de9bf") }}},
{ "multi": true }
)
And If you want to update single element inside the attachment array then you can use $[<identifier>] that identifies the array elements that match the arrayFilters conditions.
Suppose you want to update only an element inside attachment having _id equal to ObjectId(5b7934f54b2a961d081de9ab)
Invoice.update(
{ "attachment.files._id": mongoose.Types.ObjectId("5b7937014b2a961d082de9bf") },
{ "$pull": { "attachment.$[item].files": { "_id": mongoose.Types.ObjectId("5b7937014b2a961d082de9bf") } } },
{ "arrayFilters": [{ "item._id": mongoose.Types.ObjectId("5b7934f54b2a961d081de9ab") }], "multi": true }
)
Trying to delete an object from an array list, after being .splice() then save it to DB. Though it's splicing the correct index but not saving it correctly, if I splice index 0, its actually deleting index 1 in the DB, probably because of the object has been spliced first therefore it's saving the wrong object.
var user =
{
"requests" : [
{
"name" : "Test1",
"id" : "590e6c94b2d6e52674992d16",
"email" : "kram#kram.com",
"user" : "test",
"accepted" : false
},
{
"name" : "Test2",
"id" : "590e6c8bb2d6e52674992d15",
"email" : "test#test.com",
"user" : "_test",
"accepted" : false
}
]
}
accepOrdeny object example if user test1 sent a request to the xUser -
{
"name" : "Test1",
"id" : "590e6c94b2d6e52674992d16",
"email" : "kram#kram.com",
"user" : "test",
"accepted" : false
}
if the request has been accepted, accepted property boolean will switch to true and if denied then accepted: false.
{
"name" : "Test1",
"id" : "590e6c94b2d6e52674992d16",
"email" : "kram#kram.com",
"user" : "test",
"accepted" : true
}
Splicing it and saving
let xd = user.requests.indexOf(acceptOrdeny);
user.requests.splice(xd, 1);
user.save(callback);
full code logic --
User.findOne({ _id: currUser._id }, function (err, user) {
if (err) throw err;
/* acceptOrdeny is the req.body object
checking if the key accepted==true then push acceptOrdeny object to friends[], splice from request[] then save to DB
*/
if (acceptOrdeny.accepted == true) {
user.friends.push(acceptOrdeny);
let indexReq = user.requests.indexOf(acceptOrdeny);
user.requests.splice(indexReq, 1);
user.save(callback);
console.log('accepted', indexReq)
}
/*
checking if the key accepted==false remove from requests[], then save current array[] list
*/
else {
let xd = user.requests.indexOf(acceptOrdeny);
user.requests.splice(xd, 1);
user.save(callback);
}
});
I have tested splice the object manual like, this is fine but splicing them dynamically using indexOf is not quite right.
user.requests.splice(0, 1); // delete first obj in the array 'test1'
user.requests.splice(1, 1); // delete 2nd obj in the array 'test2'
Is there a better way of deleting the object in the obj[] list then save it back to the DB?
EDIT
output object, its doing the filter just fine but not deleting from the requestsp[].
Request { user_added: '5910a77a0f03b33dbc482bfe',
name: 'Mark',
username: '_markie',
email: 'mark#mark.com',
req_sent_by: 'Mark',
accepted: false }
accept/deny obj { user_added: '5910a77a0f03b33dbc482bfe',
name: 'Mark',
username: '_markie',
email: 'mark#mark.com',
req_sent_by: 'Mark',
accepted: true }
What I believe is happening is that indexOf is not finding the correct item, and is instead returning -1.
indexOf uses the equals operator and there might be a conflict between the mongoose retrieval and express body object, resulting in something like this:
var toFind = {'a': 1}
var arr = [{'a': 1}, {'b':2}]
var idx = arr.indexOf(toFind)
//here, idx = -1, but you think idx = 0
You need to implement your own comparator and use filter
user.requests = user.requests.filter(request => !(request.id.equals(acceptOrdeny.id)))
Same thing, but more verbose:
user.requests = user.requests.filter(function(request){
return !(request.id.equals(acceptOrdeny.id))
}
You might need to call id.toString() on one or both of the strings, depending on your data format.
Also, you are using duplicate logic, and could simplify by doing:
User.findOne({ _id: currUser._id }, function (err, user) {
if (err) throw err;
/* acceptOrdeny is the req.body object
checking if the key accepted==true then push acceptOrdeny object to friends[], splice from request[] then save to DB
*/
if (acceptOrdeny.accepted == true) {
user.friends.push(acceptOrdeny);
}
/*
checking if the key accepted==false remove from requests[], then save current array[] list
*/
user.requests = user.requests.filter(request => !(request.id.equals(acceptOrdeny.id)));
user.save(callback);
});
I need to replace a string in certain documents. I have googled this code, but it unfortunately does not change anything. I am not sure about the syntax on the line bellow:
pulpdb = db.getSisterDB("pulp_database");
var cursor = pulpdb.repos.find();
while (cursor.hasNext()) {
var x = cursor.next();
x['source']['url'].replace('aaa', 'bbb'); // is this correct?
db.foo.update({_id : x._id}, x);
}
I would like to add some debug prints to see what the value is, but I have no experience with MongoDB Shell. I just need to replace this:
{ "source": { "url": "http://aaa/xxx/yyy" } }
with
{ "source": { "url": "http://bbb/xxx/yyy" } }
It doesn't correct generally: if you have string http://aaa/xxx/aaa (yyy equals to aaa) you'll end up with http://bbb/xxx/bbb.
But if you ok with this, code will work.
To add debug info use print function:
var cursor = db.test.find();
while (cursor.hasNext()) {
var x = cursor.next();
print("Before: "+x['source']['url']);
x['source']['url'] = x['source']['url'].replace('aaa', 'bbb');
print("After: "+x['source']['url']);
db.test.update({_id : x._id}, x);
}
(And by the way, if you want to print out objects, there is also printjson function)
The best way to do this if you are on MongoDB 2.6 or newer is looping over the cursor object using the .forEach method and update each document usin "bulk" operations for maximum efficiency.
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
db.collection.find().forEach(function(doc) {
print("Before: "+doc.source.url);
bulk.find({ '_id': doc._id }).update({
'$set': { 'source.url': doc.source.url.replace('aaa', 'bbb') }
})
count++;
if(count % 200 === 0) {
bulk.execute();
bulk = db.collection.initializeOrderedBulkOp();
}
// Clean up queues
if (count > 0)
bulk.execute();
From MongoDB 3.2 the Bulk() API and its associated methods are deprecated you will need to use the db.collection.bulkWrite() method.
You will need loop over the cursor, build your query dynamically and $push each operation to an array.
var operations = [];
db.collection.find().forEach(function(doc) {
print("Before: "+doc.source.url);
var operation = {
updateOne: {
filter: { '_id': doc._id },
update: {
'$set': { 'source.url': doc.source.url.replace('aaa', 'bbb') }
}
}
};
operations.push(operation);
})
operations.push({
ordered: true,
writeConcern: { w: "majority", wtimeout: 5000 }
})
db.collection.bulkWrite(operations);
Nowadays,
starting Mongo 4.2, db.collection.updateMany (alias of db.collection.update) can accept an aggregation pipeline, finally allowing the update of a field based on its own value.
starting Mongo 4.4, the new aggregation operator $replaceOne makes it very easy to replace part of a string.
// { "source" : { "url" : "http://aaa/xxx/yyy" } }
// { "source" : { "url" : "http://eee/xxx/yyy" } }
db.collection.updateMany(
{ "source.url": { $regex: /aaa/ } },
[{
$set: { "source.url": {
$replaceOne: { input: "$source.url", find: "aaa", replacement: "bbb" }
}}
}]
)
// { "source" : { "url" : "http://bbb/xxx/yyy" } }
// { "source" : { "url" : "http://eee/xxx/yyy" } }
The first part ({ "source.url": { $regex: /aaa/ } }) is the match query, filtering which documents to update (the ones containing "aaa")
The second part ($set: { "source.url": {...) is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
$set is a new aggregation operator (Mongo 4.2) which in this case replaces the value of a field.
The new value is computed with the new $replaceOne operator. Note how source.url is modified directly based on the its own value ($source.url).
Note that this is fully handled server side which won't allow you to perform the debug printing part of your question.
MongoDB can do string search/replace via mapreduce. Yes, you need to have a very special data structure for it -- you can't have anything in the top keys but you need to store everything under a subdocument under value. Like this:
{
"_id" : ObjectId("549dafb0a0d0ca4ed723e37f"),
"value" : {
"title" : "Top 'access denied' errors",
"parent" : "system.admin_reports",
"p" : "\u0001\u001a%"
}
}
Once you have this neatly set up you can do:
$map = new \MongoCode("function () {
this.value['p'] = this.value['p'].replace('$from', '$to');
emit(this._id, this.value);
}");
$collection = $this->mongoCollection();
// This won't be called.
$reduce = new \MongoCode("function () { }");
$collection_name = $collection->getName();
$collection->db->command([
'mapreduce' => $collection_name,
'map' => $map,
'reduce' => $reduce,
'out' => ['merge' => $collection_name],
'query' => $query,
'sort' => ['_id' => 1],
]);