We Keep Coding

how to replace first 5 occurrences in string using mongodb query

How to replace the first n occurance of string in mongodb database.
sample JSON :
[{
"NAME":"Peter Parker",
"NUMBER":"6882859392"
},
{
"NAME":"Jhon Ripper",
"NUMBER":"9285346361"
},
{
"NAME":"James",
"NUMBER":"8754859390"
}]
i want to update 1st 5 occurence of NUMBER field of whole JSON
Expected output:
[{
"NAME":"Peter Parker",
"NUMBER":"*****59392"
},
{
"NAME":"Jhon Ripper",
"NUMBER":"*****46361"
},
{
"NAME":"James",
"NUMBER":"*****59390"
}]

How about this? without regex
Mongodb playground
db.collection.aggregate([
{
$set: {
NUMBER: {
$concat: [
"*****",
{
$substrCP: [
"$NUMBER",
5,
{
$strLenCP: "$NUMBER"
}
]
},
]
}
}
}
])

In MongoDB, you can use the updateMany() method to update multiple documents that match a specific condition. In your case, you want to update the first 5 occurrences of the "NUMBER" field.
Here is an example of how you can use the updateMany() method to update the first 5 occurrences of the "NUMBER" field in your JSON:
// Define a filter to find the first 5 occurrences of the "NUMBER" field
var filter = { NUMBER: { $regex: /^[0-9]{5}/ } };
// Define the update operation
var update = { $set: { NUMBER: "*****" + "$" + "NUMBER" } };
// Use the updateMany() method to update the documents that match the filter
db.yourCollectionName.updateMany(filter, update);
The filter { NUMBER: { $regex: /^[0-9]{5}/ } } uses a regular expression to find the documents where the value of the "NUMBER" field starts with 5 digits.
The update { $set: { NUMBER: "*****" + "$" + "NUMBER" } } uses the $set operator to update the "NUMBER" field to a string that starts with "*****" and concatenates the rest of the original number using the $ operator.
The updateMany() method updates all the documents that match the filter, in your case the first 5 occurrences of NUMBER field.
Please keep in mind that you need to be connected to a MongoDB server and a specific database before running this code.

1st i'm storing all numbers with respective _id to the variable a, then iterate the collection with forEach function and updating the document.
This code help me out in this problem statement
var a = db.coll.find({},{"NUMBER":1})
a.forEach(function( row ) {
var dict = '*****'+row.NUMBER.substring(5,row.NUMBER.length);
db.coll.updateOne({"_id" : row._id},{$set : {"NUMBER" : dict}});
});

MongoDB $graphLookup inside update query [duplicate]

In MongoDB, is it possible to update the value of a field using the value from another field? The equivalent SQL would be something like:
UPDATE Person SET Name = FirstName + ' ' + LastName
And the MongoDB pseudo-code would be:
db.person.update( {}, { $set : { name : firstName + ' ' + lastName } );

The best way to do this is in version 4.2+ which allows using the aggregation pipeline in the update document and the updateOne, updateMany, or update(deprecated in most if not all languages drivers) collection methods.
MongoDB 4.2+
Version 4.2 also introduced the $set pipeline stage operator, which is an alias for $addFields. I will use $set here as it maps with what we are trying to achieve.
db.collection.<update method>(
{},
[
{"$set": {"name": { "$concat": ["$firstName", " ", "$lastName"]}}}
]
)
Note that square brackets in the second argument to the method specify an aggregation pipeline instead of a plain update document because using a simple document will not work correctly.
MongoDB 3.4+
In 3.4+, you can use $addFields and the $out aggregation pipeline operators.
db.collection.aggregate(
[
{ "$addFields": {
"name": { "$concat": [ "$firstName", " ", "$lastName" ] }
}},
{ "$out": <output collection name> }
]
)
Note that this does not update your collection but instead replaces the existing collection or creates a new one. Also, for update operations that require "typecasting", you will need client-side processing, and depending on the operation, you may need to use the find() method instead of the .aggreate() method.
MongoDB 3.2 and 3.0
The way we do this is by $projecting our documents and using the $concat string aggregation operator to return the concatenated string.
You then iterate the cursor and use the $set update operator to add the new field to your documents using bulk operations for maximum efficiency.
Aggregation query:
var cursor = db.collection.aggregate([
{ "$project": {
"name": { "$concat": [ "$firstName", " ", "$lastName" ] }
}}
])
MongoDB 3.2 or newer
You need to use the bulkWrite method.
var requests = [];
cursor.forEach(document => {
requests.push( {
'updateOne': {
'filter': { '_id': document._id },
'update': { '$set': { 'name': document.name } }
}
});
if (requests.length === 500) {
//Execute per 500 operations and re-init
db.collection.bulkWrite(requests);
requests = [];
}
});
if(requests.length > 0) {
db.collection.bulkWrite(requests);
}
MongoDB 2.6 and 3.0
From this version, you need to use the now deprecated Bulk API and its associated methods.
var bulk = db.collection.initializeUnorderedBulkOp();
var count = 0;
cursor.snapshot().forEach(function(document) {
bulk.find({ '_id': document._id }).updateOne( {
'$set': { 'name': document.name }
});
count++;
if(count%500 === 0) {
// Excecute per 500 operations and re-init
bulk.execute();
bulk = db.collection.initializeUnorderedBulkOp();
}
})
// clean up queues
if(count > 0) {
bulk.execute();
}
MongoDB 2.4
cursor["result"].forEach(function(document) {
db.collection.update(
{ "_id": document._id },
{ "$set": { "name": document.name } }
);
})

You should iterate through. For your specific case:
db.person.find().snapshot().forEach(
function (elem) {
db.person.update(
{
_id: elem._id
},
{
$set: {
name: elem.firstname + ' ' + elem.lastname
}
}
);
}
);

Apparently there is a way to do this efficiently since MongoDB 3.4, see styvane's answer.
Obsolete answer below
You cannot refer to the document itself in an update (yet). You'll need to iterate through the documents and update each document using a function. See this answer for an example, or this one for server-side eval().

For a database with high activity, you may run into issues where your updates affect actively changing records and for this reason I recommend using snapshot()
db.person.find().snapshot().forEach( function (hombre) {
hombre.name = hombre.firstName + ' ' + hombre.lastName;
db.person.save(hombre);
});
http://docs.mongodb.org/manual/reference/method/cursor.snapshot/

Starting Mongo 4.2, db.collection.update() can accept an aggregation pipeline, finally allowing the update/creation of a field based on another field:
// { firstName: "Hello", lastName: "World" }
db.collection.updateMany(
{},
[{ $set: { name: { $concat: [ "$firstName", " ", "$lastName" ] } } }]
)
// { "firstName" : "Hello", "lastName" : "World", "name" : "Hello World" }
The first part {} is the match query, filtering which documents to update (in our case all documents).
The second part [{ $set: { name: { ... } }] is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline). $set is a new aggregation operator and an alias of $addFields.

Regarding this answer, the snapshot function is deprecated in version 3.6, according to this update. So, on version 3.6 and above, it is possible to perform the operation this way:
db.person.find().forEach(
function (elem) {
db.person.update(
{
_id: elem._id
},
{
$set: {
name: elem.firstname + ' ' + elem.lastname
}
}
);
}
);

I tried the above solution but I found it unsuitable for large amounts of data. I then discovered the stream feature:
MongoClient.connect("...", function(err, db){
var c = db.collection('yourCollection');
var s = c.find({/* your query */}).stream();
s.on('data', function(doc){
c.update({_id: doc._id}, {$set: {name : doc.firstName + ' ' + doc.lastName}}, function(err, result) { /* result == true? */} }
});
s.on('end', function(){
// stream can end before all your updates do if you have a lot
})
})

update() method takes aggregation pipeline as parameter like
db.collection_name.update(
{
// Query
},
[
// Aggregation pipeline
{ "$set": { "id": "$_id" } }
],
{
// Options
"multi": true // false when a single doc has to be updated
}
)
The field can be set or unset with existing values using the aggregation pipeline.
Note: use $ with field name to specify the field which has to be read.

Here's what we came up with for copying one field to another for ~150_000 records. It took about 6 minutes, but is still significantly less resource intensive than it would have been to instantiate and iterate over the same number of ruby objects.
js_query = %({
$or : [
{
'settings.mobile_notifications' : { $exists : false },
'settings.mobile_admin_notifications' : { $exists : false }
}
]
})
js_for_each = %(function(user) {
if (!user.settings.hasOwnProperty('mobile_notifications')) {
user.settings.mobile_notifications = user.settings.email_notifications;
}
if (!user.settings.hasOwnProperty('mobile_admin_notifications')) {
user.settings.mobile_admin_notifications = user.settings.email_admin_notifications;
}
db.users.save(user);
})
js = "db.users.find(#{js_query}).forEach(#{js_for_each});"
Mongoid::Sessions.default.command('$eval' => js)

With MongoDB version 4.2+, updates are more flexible as it allows the use of aggregation pipeline in its update, updateOne and updateMany. You can now transform your documents using the aggregation operators then update without the need to explicity state the $set command (instead we use $replaceRoot: {newRoot: "$$ROOT"})
Here we use the aggregate query to extract the timestamp from MongoDB's ObjectID "_id" field and update the documents (I am not an expert in SQL but I think SQL does not provide any auto generated ObjectID that has timestamp to it, you would have to automatically create that date)
var collection = "person"
agg_query = [
{
"$addFields" : {
"_last_updated" : {
"$toDate" : "$_id"
}
}
},
{
$replaceRoot: {
newRoot: "$$ROOT"
}
}
]
db.getCollection(collection).updateMany({}, agg_query, {upsert: true})

(I would have posted this as a comment, but couldn't)
For anyone who lands here trying to update one field using another in the document with the c# driver...
I could not figure out how to use any of the UpdateXXX methods and their associated overloads since they take an UpdateDefinition as an argument.
// we want to set Prop1 to Prop2
class Foo { public string Prop1 { get; set; } public string Prop2 { get; set;} }
void Test()
{
var update = new UpdateDefinitionBuilder<Foo>();
update.Set(x => x.Prop1, <new value; no way to get a hold of the object that I can find>)
}
As a workaround, I found that you can use the RunCommand method on an IMongoDatabase (https://docs.mongodb.com/manual/reference/command/update/#dbcmd.update).
var command = new BsonDocument
{
{ "update", "CollectionToUpdate" },
{ "updates", new BsonArray
{
new BsonDocument
{
// Any filter; here the check is if Prop1 does not exist
{ "q", new BsonDocument{ ["Prop1"] = new BsonDocument("$exists", false) }},
// set it to the value of Prop2
{ "u", new BsonArray { new BsonDocument { ["$set"] = new BsonDocument("Prop1", "$Prop2") }}},
{ "multi", true }
}
}
}
};
database.RunCommand<BsonDocument>(command);

MongoDB 4.2+ Golang
result, err := collection.UpdateMany(ctx, bson.M{},
mongo.Pipeline{
bson.D{{"$set",
bson.M{"name": bson.M{"$concat": []string{"$lastName", " ", "$firstName"}}}
}},
)

How to find longest and shortest length of a value for a field in mongoDb?

The data type of the field is String. I would like to find the length of the longest and shortest value for a field in mongoDB.
I have totally 500000 documents in my collection.

In modern releases MongoDB has the $strLenBytes or $strLenCP aggregation operators than allow you to simply do:
Class.collection.aggregate([
{ "$group" => {
"_id" => nil,
"max" => { "$max" => { "$strLenCP" => "$a" } },
"min" => { "$min" => { "$strLenCP" => "$a" } }
}}
])
Where "a" is the string property in your document you want to get the min and max length from.
To output the minimum and maximum length, the best approach available is to use mapReduce with a few tricks to just keep the values.
First you define a mapper function which is just really going to output a single item from your collection to reduce the load:
map = Q%{
function () {
if ( this.a.length < store[0] )
store[0] = this.a.length;
if ( this.a.length > store[1] )
store[1] = this.a.length;
if ( count == 0 )
emit( null, 0 );
count++;
}
}
Since this is working mostly with a globally scoped variable keeping the min and max lengths you just want to substitute this in a finalize function on the single document emitted. There is no reduce stage, but define a "blank" function for this even though it is not called:
reduce = Q%{ function() {} }
finalize = Q%{
function(key,value) {
return {
min: store[0],
max: store[1]
};
}
}
Then call the mapReduce operation:
Class.map_reduce(map,reduce).out(inline: 1).finalize(finalize).scope(store: [], count: 0)
So all the work is done on the server and not by iterating results sent to the client application. On a small set like this:
{ "_id" : ObjectId("543e8ee7ddd272814f919472"), "a" : "this" }
{ "_id" : ObjectId("543e8eedddd272814f919473"), "a" : "something" }
{ "_id" : ObjectId("543e8ef6ddd272814f919474"), "a" : "other" }
You get a result like this (shell output, but much the same for the driver ):
{
"results" : [
{
"_id" : null,
"value" : {
"min" : 4,
"max" : 9
}
}
],
"timeMillis" : 1,
"counts" : {
"input" : 3,
"emit" : 1,
"reduce" : 0,
"output" : 1
},
"ok" : 1
}
So mapReduce allows the JavaScript processing on the server to do this fairly quickly, reducing your network traffic. There is no other native way at present for MongoDB to return a string length right now, so the JavaScript processing is necessary on the server.

You can use a mongo shell script. Note that it will perform a full table scan.
function findMinMax() {
var max = 0;
var min = db.collection.findOne().fieldName.length;
db.collection.find().forEach(function(doc) {
var currentLength = doc.fieldName.length;
if (currentLength > max) {
max = currentLength;
}
if (currentLength < min) {
min = currentLength;
}
});
print(max);
print(min);
}
use <databaseName>
findMinMax();
You can save the function in a file say c:\minMax.js and run the file as,
c:\mongodb\bin> mongo dbName < c:\minMax.js
Note: you may need to supply the necessary hostname, user name, password to connect to your database.
c:\mongodb\bin> mongo --host hostName --port portNumber -u userName -p password dbName < c:\minMax.js

For getting the longest value for a field
db.entities.aggregate([{ $match:{ condition } },{
$addFields: {
"length": { $strLenCP: "$feildName" }
}},
{ "$sort": { "length": -1 } },
{$limit:1}
])
Change the { "$sort": { "length": -1 } } to { "$sort": { "length": 1 } } for the shortest value for a field

How to get the latest and oldest record in mongoose.js (or just the timespan between them)

Basic problem
I have a bunch of records and I need to get latest (most recent) and the oldest (least recent).
When googling I found this topic where I saw a couple of queries:
// option 1
Tweet.findOne({}, [], { $orderby : { 'created_at' : -1 } }, function(err, post) {
console.log( post );
});
// option 2
Tweet.find({}, [], {sort:[['arrival',-1]]}, function(err, post) {
console.log( post );
});
Unfortunatly they both error:
TypeError: Invalid select() argument. Must be a string or object.
The link also has this one:
Tweet.find().sort('_id','descending').limit(15).find(function(err, post) {
console.log( post );
});
and that one errors:
TypeError: Invalid sort() argument. Must be a string or object.
So how can I get those records?
Timespan
Even more ideally I just want the difference in time (seconds?) between the oldest and the newest record, but I have no clue on how to start making a query like that.
This is the schema:
var Tweet = new Schema({
body: String
, fid: { type: String, index: { unique: true } }
, username: { type: String, index: true }
, userid: Number
, created_at: Date
, source: String
});
I'm pretty sure I have the most recent version of mongoDB and mongoose.
EDIT
This is how I calc the timespan based on the answer provided by JohnnyHK:
var calcDays = function( cb ) {
var getOldest = function( cb ) {
Tweet.findOne({}, {}, { sort: { 'created_at' : 1 } }, function(err, post) {
cb( null, post.created_at.getTime() );
});
}
, getNewest = function( cb ) {
Tweet.findOne({}, {}, { sort: { 'created_at' : -1 } }, function(err, post) {
cb( null, post.created_at.getTime() );
});
}
async.parallel({
oldest: getOldest
, newest: getNewest
}
, function( err, results ) {
var days = ( results.newest - results.oldest ) / 1000 / 60 / 60 / 24;
// days = Math.round( days );
cb( null, days );
}
);
}

Mongoose 3.x is complaining about the [] parameter in your findOne calls as the array format is no longer supported for the parameter that selects the fields to include.
Try this instead to find the newest:
Tweet.findOne({}, {}, { sort: { 'created_at' : -1 } }, function(err, post) {
console.log( post );
});
Change the -1 to a 1 to find the oldest.
But because you're not using any field selection, it's somewhat cleaner to chain a couple calls together:
Tweet.findOne().sort({created_at: -1}).exec(function(err, post) { ... });
Or even pass a string to sort:
Tweet.findOne().sort('-created_at').exec(function(err, post) { ... });

Fast and Simple - One Line Solution
Get 10 latest documents
MySchema.find().sort({ _id: -1 }).limit(10)
Get 10 oldest documents
MySchema.find().sort({ _id: 1 }).limit(10)
In case you want sorting based on some other property i.e. createdAt and get the oldest or latest. It is similar to the above query.
MySchema.find().sort({ createdAt: -1 }).limit(10) // 10 latest docs
MySchema.find().sort({ createdAt: 1 }).limit(10) // 10 oldest docs

for version ~3.8 mongoose
to find the last entry
model.findOne().sort({ field: 'asc', _id: -1 }).limit(1)
or using
model.findOne().sort({ field: -_id }).limit(1)

collectionName.findOne().sort({$natural: -1}).limit(1).exec(function(err, res){
if(err){
console.log(err);
}
else{
console.log(res);
}
}
This will give you the last document recorded on the database. Just follow the same concept.

await Model.find().sort({$natural:-1}).limit(1); //for the latest record
await Model.find().sort({$natural:1}).limit(1); //for the oldest record
This one works for me. using mongodb natural order https://docs.mongodb.com/manual/reference/operator/meta/natural/

We have method called sort using that we can able to get first element(old document) which means 1 for sort field or last element(new document) which means -1 for sort field of collection.

The best way is to have an async function like that:
async function findLastElement () {
return await Mymodel.findOne().sort('-_id');
}
this way you get the last element and you ensure reusability.

Here is the answer with async - await
const olderDoc: any = await Model.findOne().sort({ createdAt: 1 }).lean().exec()
console.log('olderDoc', olderDoc)
const newerDoc: any = await Model.findOne().sort({ createdAt: -1 }).lean().exec()
console.log('newerDoc', newerDoc)

How to replace string in all documents in Mongo

I need to replace a string in certain documents. I have googled this code, but it unfortunately does not change anything. I am not sure about the syntax on the line bellow:
pulpdb = db.getSisterDB("pulp_database");
var cursor = pulpdb.repos.find();
while (cursor.hasNext()) {
var x = cursor.next();
x['source']['url'].replace('aaa', 'bbb'); // is this correct?
db.foo.update({_id : x._id}, x);
}
I would like to add some debug prints to see what the value is, but I have no experience with MongoDB Shell. I just need to replace this:
{ "source": { "url": "http://aaa/xxx/yyy" } }
with
{ "source": { "url": "http://bbb/xxx/yyy" } }

It doesn't correct generally: if you have string http://aaa/xxx/aaa (yyy equals to aaa) you'll end up with http://bbb/xxx/bbb.
But if you ok with this, code will work.
To add debug info use print function:
var cursor = db.test.find();
while (cursor.hasNext()) {
var x = cursor.next();
print("Before: "+x['source']['url']);
x['source']['url'] = x['source']['url'].replace('aaa', 'bbb');
print("After: "+x['source']['url']);
db.test.update({_id : x._id}, x);
}
(And by the way, if you want to print out objects, there is also printjson function)

The best way to do this if you are on MongoDB 2.6 or newer is looping over the cursor object using the .forEach method and update each document usin "bulk" operations for maximum efficiency.
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
db.collection.find().forEach(function(doc) {
print("Before: "+doc.source.url);
bulk.find({ '_id': doc._id }).update({
'$set': { 'source.url': doc.source.url.replace('aaa', 'bbb') }
})
count++;
if(count % 200 === 0) {
bulk.execute();
bulk = db.collection.initializeOrderedBulkOp();
}
// Clean up queues
if (count > 0)
bulk.execute();
From MongoDB 3.2 the Bulk() API and its associated methods are deprecated you will need to use the db.collection.bulkWrite() method.
You will need loop over the cursor, build your query dynamically and $push each operation to an array.
var operations = [];
db.collection.find().forEach(function(doc) {
print("Before: "+doc.source.url);
var operation = {
updateOne: {
filter: { '_id': doc._id },
update: {
'$set': { 'source.url': doc.source.url.replace('aaa', 'bbb') }
}
}
};
operations.push(operation);
})
operations.push({
ordered: true,
writeConcern: { w: "majority", wtimeout: 5000 }
})
db.collection.bulkWrite(operations);

Nowadays,
starting Mongo 4.2, db.collection.updateMany (alias of db.collection.update) can accept an aggregation pipeline, finally allowing the update of a field based on its own value.
starting Mongo 4.4, the new aggregation operator $replaceOne makes it very easy to replace part of a string.
// { "source" : { "url" : "http://aaa/xxx/yyy" } }
// { "source" : { "url" : "http://eee/xxx/yyy" } }
db.collection.updateMany(
{ "source.url": { $regex: /aaa/ } },
[{
$set: { "source.url": {
$replaceOne: { input: "$source.url", find: "aaa", replacement: "bbb" }
}}
}]
)
// { "source" : { "url" : "http://bbb/xxx/yyy" } }
// { "source" : { "url" : "http://eee/xxx/yyy" } }
The first part ({ "source.url": { $regex: /aaa/ } }) is the match query, filtering which documents to update (the ones containing "aaa")
The second part ($set: { "source.url": {...) is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
$set is a new aggregation operator (Mongo 4.2) which in this case replaces the value of a field.
The new value is computed with the new $replaceOne operator. Note how source.url is modified directly based on the its own value ($source.url).
Note that this is fully handled server side which won't allow you to perform the debug printing part of your question.

MongoDB can do string search/replace via mapreduce. Yes, you need to have a very special data structure for it -- you can't have anything in the top keys but you need to store everything under a subdocument under value. Like this:
{
"_id" : ObjectId("549dafb0a0d0ca4ed723e37f"),
"value" : {
"title" : "Top 'access denied' errors",
"parent" : "system.admin_reports",
"p" : "\u0001\u001a%"
}
}
Once you have this neatly set up you can do:
$map = new \MongoCode("function () {
this.value['p'] = this.value['p'].replace('$from', '$to');
emit(this._id, this.value);
}");
$collection = $this->mongoCollection();
// This won't be called.
$reduce = new \MongoCode("function () { }");
$collection_name = $collection->getName();
$collection->db->command([
'mapreduce' => $collection_name,
'map' => $map,
'reduce' => $reduce,
'out' => ['merge' => $collection_name],
'query' => $query,
'sort' => ['_id' => 1],
]);