We Keep Coding

How to avoid a JSON object with numbers as keys getting interpreted as an array when uploading to Firebase Real Time Database?

I am using Typescript with Enums to create some dynamic object structure. These enums are represented as numbers in the object, not as their string value, which I want to keep as it is. When I am uploading these objects to the Firebase RTDB, the objects are misinterpreted as arrays.
Enum example in typescript
enum TableState {
Standby = 0,
Idle,
In_Game,
Maintenance,
}
Javascript object I want to upload to the Google RTDB
configuration_per_state: {
[TableState.Idle]: {
brightness: 10,
},
[TableState.In_Game]: {
brightness: 10,
duration: 20 * 60, //20 minutes,
}
}
which is the following stringified JSON:
{
"configuration_per_state": {
"1": {
"brightness": 10
},
"2": {
"brightness": 10,
"duration": 120
}
}
}
I now faced the issue that when uploading to the Firebase Real Time Database, the object gets interpreted as an array. So when I want to retrieve the data the object structure was changed by the DB.
Code to upload the object:
const createInitialConfigInDb = (
tableConfig: InitialTableState): Promise<any> => {
return new Promise((resolve, reject) => {
const userId = auth.currentUser?.uid
db.ref(`config/${userId}/general`).set(tableConfig).then((data) => {
//success callback
resolve(data)
}).catch((error) => {
//error callback
reject(error)
})
})
};
Resulting JSON after uploading to the database and downloaded from the RTDB:
{
"configuration_per_state" : [ null, {
"brightness" : 10,
}, {
"brightness" : 10,
"duration" : 1200,
} ],
}
Firstly I don't get why there is an entry "null" as the first array entry ...
And secondly is there an easy way to avoid this behaviour?

Firebase interprets you data as an array, and the null it adds indicates that there's no element at index 0 in the array.
Firebase Realtime Database's array coercion/interpretation cannot be configured.
The only way to prevent it from interpreting your data as an array, is by not using numeric keys.
My common solution is to prefix such keys with a short, constant string value. For example:
{
"configuration_per_state" : {
"key_1":
"brightness" : 10,
},
"key_2": {
"brightness" : 10,
"duration" : 1200,
}
}
}

If the specific field is double, convert it to Int [duplicate]

I am trying to change the type of a field from within the mongo shell.
I am doing this...
db.meta.update(
{'fields.properties.default': { $type : 1 }},
{'fields.properties.default': { $type : 2 }}
)
But it's not working!

The only way to change the $type of the data is to perform an update on the data where the data has the correct type.
In this case, it looks like you're trying to change the $type from 1 (double) to 2 (string).
So simply load the document from the DB, perform the cast (new String(x)) and then save the document again.
If you need to do this programmatically and entirely from the shell, you can use the find(...).forEach(function(x) {}) syntax.
In response to the second comment below. Change the field bad from a number to a string in collection foo.
db.foo.find( { 'bad' : { $type : 1 } } ).forEach( function (x) {
x.bad = new String(x.bad); // convert field to string
db.foo.save(x);
});

Convert String field to Integer:
db.db-name.find({field-name: {$exists: true}}).forEach(function(obj) {
obj.field-name = new NumberInt(obj.field-name);
db.db-name.save(obj);
});
Convert Integer field to String:
db.db-name.find({field-name: {$exists: true}}).forEach(function(obj) {
obj.field-name = "" + obj.field-name;
db.db-name.save(obj);
});

Starting Mongo 4.2, db.collection.update() can accept an aggregation pipeline, finally allowing the update of a field based on its own value:
// { a: "45", b: "x" }
// { a: 53, b: "y" }
db.collection.updateMany(
{ a : { $type: 1 } },
[{ $set: { a: { $toString: "$a" } } }]
)
// { a: "45", b: "x" }
// { a: "53", b: "y" }
The first part { a : { $type: 1 } } is the match query:
It filters which documents to update.
In this case, since we want to convert "a" to string when its value is a double, this matches elements for which "a" is of type 1 (double)).
This table provides the code representing the different possible types.
The second part [{ $set: { a: { $toString: "$a" } } }] is the update aggregation pipeline:
Note the squared brackets signifying that this update query uses an aggregation pipeline.
$set is a new aggregation operator (Mongo 4.2) which in this case modifies a field.
This can be simply read as "$set" the value of "a" to "$a" converted "$toString".
What's really new here, is being able in Mongo 4.2 to reference the document itself when updating it: the new value for "a" is based on the existing value of "$a".
Also note "$toString" which is a new aggregation operator introduced in Mongo 4.0.
In case your cast isn't from double to string, you have the choice between different conversion operators introduced in Mongo 4.0 such as $toBool, $toInt, ...
And if there isn't a dedicated converter for your targeted type, you can replace { $toString: "$a" } with a $convert operation: { $convert: { input: "$a", to: 2 } } where the value for to can be found in this table:
db.collection.updateMany(
{ a : { $type: 1 } },
[{ $set: { a: { $convert: { input: "$a", to: 2 } } } }]
)

For string to int conversion.
db.my_collection.find().forEach( function(obj) {
obj.my_value= new NumberInt(obj.my_value);
db.my_collection.save(obj);
});
For string to double conversion.
obj.my_value= parseInt(obj.my_value, 10);
For float:
obj.my_value= parseFloat(obj.my_value);

db.coll.find().forEach(function(data) {
db.coll.update({_id:data._id},{$set:{myfield:parseInt(data.myfield)}});
})

all answers so far use some version of forEach, iterating over all collection elements client-side.
However, you could use MongoDB's server-side processing by using aggregate pipeline and $out stage as :
the $out stage atomically replaces the existing collection with the
new results collection.
example:
db.documents.aggregate([
{
$project: {
_id: 1,
numberField: { $substr: ['$numberField', 0, -1] },
otherField: 1,
differentField: 1,
anotherfield: 1,
needolistAllFieldsHere: 1
},
},
{
$out: 'documents',
},
]);

To convert a field of string type to date field, you would need to iterate the cursor returned by the find() method using the forEach() method, within the loop convert the field to a Date object and then update the field using the $set operator.
Take advantage of using the Bulk API for bulk updates which offer better performance as you will be sending the operations to the server in batches of say 1000 which gives you a better performance as you are not sending every request to the server, just once in every 1000 requests.
The following demonstrates this approach, the first example uses the Bulk API available in MongoDB versions >= 2.6 and < 3.2. It updates all
the documents in the collection by changing all the created_at fields to date fields:
var bulk = db.collection.initializeUnorderedBulkOp(),
counter = 0;
db.collection.find({"created_at": {"$exists": true, "$type": 2 }}).forEach(function (doc) {
var newDate = new Date(doc.created_at);
bulk.find({ "_id": doc._id }).updateOne({
"$set": { "created_at": newDate}
});
counter++;
if (counter % 1000 == 0) {
bulk.execute(); // Execute per 1000 operations and re-initialize every 1000 update statements
bulk = db.collection.initializeUnorderedBulkOp();
}
})
// Clean up remaining operations in queue
if (counter % 1000 != 0) { bulk.execute(); }
The next example applies to the new MongoDB version 3.2 which has since deprecated the Bulk API and provided a newer set of apis using bulkWrite():
var bulkOps = [];
db.collection.find({"created_at": {"$exists": true, "$type": 2 }}).forEach(function (doc) {
var newDate = new Date(doc.created_at);
bulkOps.push(
{
"updateOne": {
"filter": { "_id": doc._id } ,
"update": { "$set": { "created_at": newDate } }
}
}
);
})
db.collection.bulkWrite(bulkOps, { "ordered": true });

To convert int32 to string in mongo without creating an array just add "" to your number :-)
db.foo.find( { 'mynum' : { $type : 16 } } ).forEach( function (x) {
x.mynum = x.mynum + ""; // convert int32 to string
db.foo.save(x);
});

What really helped me to change the type of the object in MondoDB was just this simple line, perhaps mentioned before here...:
db.Users.find({age: {$exists: true}}).forEach(function(obj) {
obj.age = new NumberInt(obj.age);
db.Users.save(obj);
});
Users are my collection and age is the object which had a string instead of an integer (int32).

You can easily convert the string data type to numerical data type.
Don't forget to change collectionName & FieldName.
for ex : CollectionNmae : Users & FieldName : Contactno.
Try this query..
db.collectionName.find().forEach( function (x) {
x.FieldName = parseInt(x.FieldName);
db.collectionName.save(x);
});

I need to change datatype of multiple fields in the collection, so I used the following to make multiple data type changes in the collection of documents. Answer to an old question but may be helpful for others.
db.mycoll.find().forEach(function(obj) {
if (obj.hasOwnProperty('phone')) {
obj.phone = "" + obj.phone; // int or longint to string
}
if (obj.hasOwnProperty('field-name')) {
obj.field-name = new NumberInt(obj.field-name); //string to integer
}
if (obj.hasOwnProperty('cdate')) {
obj.cdate = new ISODate(obj.cdate); //string to Date
}
db.mycoll.save(obj);
});

demo change type of field mid from string to mongo objectId using mongoose
Post.find({}, {mid: 1,_id:1}).exec(function (err, doc) {
doc.map((item, key) => {
Post.findByIdAndUpdate({_id:item._id},{$set:{mid: mongoose.Types.ObjectId(item.mid)}}).exec((err,res)=>{
if(err) throw err;
reply(res);
});
});
});
Mongo ObjectId is just another example of such styles as
Number, string, boolean that hope the answer will help someone else.

I use this script in mongodb console for string to float conversions...
db.documents.find({ 'fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.fwtweaeeba = parseFloat( obj.fwtweaeeba );
db.documents.save(obj); } );
db.documents.find({ 'versions.0.content.fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.versions[0].content.fwtweaeeba = parseFloat( obj.versions[0].content.fwtweaeeba );
db.documents.save(obj); } );
db.documents.find({ 'versions.1.content.fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.versions[1].content.fwtweaeeba = parseFloat( obj.versions[1].content.fwtweaeeba );
db.documents.save(obj); } );
db.documents.find({ 'versions.2.content.fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.versions[2].content.fwtweaeeba = parseFloat( obj.versions[2].content.fwtweaeeba );
db.documents.save(obj); } );
And this one in php)))
foreach($db->documents->find(array("type" => "chair")) as $document){
$db->documents->update(
array('_id' => $document[_id]),
array(
'$set' => array(
'versions.0.content.axdducvoxb' => (float)$document['versions'][0]['content']['axdducvoxb'],
'versions.1.content.axdducvoxb' => (float)$document['versions'][1]['content']['axdducvoxb'],
'versions.2.content.axdducvoxb' => (float)$document['versions'][2]['content']['axdducvoxb'],
'axdducvoxb' => (float)$document['axdducvoxb']
)
),
array('$multi' => true)
);
}

The above answers almost worked but had a few challenges-
Problem 1: db.collection.save no longer works in MongoDB 5.x
For this, I used replaceOne().
Problem 2: new String(x.bad) was giving exponential number
I used "" + x.bad as suggested above.
My version:
let count = 0;
db.user
.find({
custID: {$type: 1},
})
.forEach(function (record) {
count++;
const actualValue = record.custID;
record.custID = "" + record.custID;
console.log(`${count}. Updating User(id:${record._id}) from old id [${actualValue}](${typeof actualValue}) to [${record.custID}](${typeof record.custID})`)
db.user.replaceOne({_id: record._id}, record);
});
And for millions of records, here are the output (for future investigation/reference)-

Nodejs, MongoDB - Determine if Document in Collection was Updated or Inserted

I am calling the mongoDB update function with {upsert:true} to insert a new document if the given _id does not exist. I would like to determine if the document was inserted or update. Much like this question I found using java but only using Nodejs.
how to check if an document is updated or inserted in MongoDB
Here is my DB call.
app.post('/mongoSubmit', function(req, res) {
console.log("This is the req.body" + JSON.stringify(req.body, null, 4));
var updateCustomer = function(db, callback){
db.collection('customers1').update(
{_id:req.body.email},
{ first: req.body.firstName,
last: req.body.lastName,
phone: req.body.phone,
email: req.body.email,
subjectIndex: req.body.subject,
messageIndex: req.body.message
},
{ upsert: true},
function(err, result){
if(err){console.log("database error" + err)}
callback(result);
}
);
}
MongoClient.connect(url, function(err, db){
updateCustomer(db, function(result){
console.log("these are the results" + JSON.stringify(result, null, 4));
/*
** Return Either
*
these are the results{
"ok": 1,
"nModified": 0,
"n": 1,
"upserted": [
{
"index": 0,
"_id": "sjr6asdfsadfsadf28#gmail.com"
}
]
}
/*
*
* or
these are the results{
"ok": 1,
"nModified": 1,
"n": 1
}
//BUT *************** Problem using this Value *********************
console.log("this is the value of Modified" + result.nModified);
/*
** Returns undefined
*/
if(result.nModified == 1){
console.log("Updated document");
}
else{
console.log("Inserted document");
}
db.close();
res.render('applications', {
title:"Title"
});
});
});
});
I have also tried for test doing
if(result.hasOwnProperty('upserted'){
//log an Insert
if(result.upserted == true {
//log an Insert
if(result.nModified == 1){
// log an update
if(result.nModified == true){
//log an update
and also adding upserted as a parameter to the callback which I found from a different forum.
function(err, result, upserted){
//callback function
//upserted was undefined
})
My result is confusing. How I could log an object with a property value but when I try to log that specific property it comes up undefined?
Could anyone explain why this might happen in javascript?
or
Suggest another solution for determining if a document in a collection was updated or inserted?
Thank You

The result is a structure that contatins it's own property "result" which has the sub-properties. So you need to inspect at the right level:
var async = require('async'),
mongodb = require('mongodb'),
MongoClient = mongodb.MongoClient;
MongoClient.connect('mongodb://localhost/test',function(err,db) {
db.collection('uptest').update(
{ "a": 1 },
{ "$set": { "b": 2 } },
{ "upsert": true },
function(err,result) {
if (err) throw err;
if (result.result.hasOwnProperty('upserted') ) {
console.log( JSON.stringify( result.result.upserted, undefined, 2 ) );
}
console.log( "matched: %d, modified: %d",
result.result.n,
result.result.nModified
);
}
);
});
On a first run you will get the "array" of "upserted" like this:
[
{
"index": 0,
"_id": "55a4c3cfbe78f212535e2f6a"
}
]
matched: 1, modified: 0
On a second run with the same values then nothing is added or modified:
matched: 1, modified: 0
Change the value of "b" and "modified" is counted since the data actually changed.

How to find longest and shortest length of a value for a field in mongoDb?

The data type of the field is String. I would like to find the length of the longest and shortest value for a field in mongoDB.
I have totally 500000 documents in my collection.

In modern releases MongoDB has the $strLenBytes or $strLenCP aggregation operators than allow you to simply do:
Class.collection.aggregate([
{ "$group" => {
"_id" => nil,
"max" => { "$max" => { "$strLenCP" => "$a" } },
"min" => { "$min" => { "$strLenCP" => "$a" } }
}}
])
Where "a" is the string property in your document you want to get the min and max length from.
To output the minimum and maximum length, the best approach available is to use mapReduce with a few tricks to just keep the values.
First you define a mapper function which is just really going to output a single item from your collection to reduce the load:
map = Q%{
function () {
if ( this.a.length < store[0] )
store[0] = this.a.length;
if ( this.a.length > store[1] )
store[1] = this.a.length;
if ( count == 0 )
emit( null, 0 );
count++;
}
}
Since this is working mostly with a globally scoped variable keeping the min and max lengths you just want to substitute this in a finalize function on the single document emitted. There is no reduce stage, but define a "blank" function for this even though it is not called:
reduce = Q%{ function() {} }
finalize = Q%{
function(key,value) {
return {
min: store[0],
max: store[1]
};
}
}
Then call the mapReduce operation:
Class.map_reduce(map,reduce).out(inline: 1).finalize(finalize).scope(store: [], count: 0)
So all the work is done on the server and not by iterating results sent to the client application. On a small set like this:
{ "_id" : ObjectId("543e8ee7ddd272814f919472"), "a" : "this" }
{ "_id" : ObjectId("543e8eedddd272814f919473"), "a" : "something" }
{ "_id" : ObjectId("543e8ef6ddd272814f919474"), "a" : "other" }
You get a result like this (shell output, but much the same for the driver ):
{
"results" : [
{
"_id" : null,
"value" : {
"min" : 4,
"max" : 9
}
}
],
"timeMillis" : 1,
"counts" : {
"input" : 3,
"emit" : 1,
"reduce" : 0,
"output" : 1
},
"ok" : 1
}
So mapReduce allows the JavaScript processing on the server to do this fairly quickly, reducing your network traffic. There is no other native way at present for MongoDB to return a string length right now, so the JavaScript processing is necessary on the server.

You can use a mongo shell script. Note that it will perform a full table scan.
function findMinMax() {
var max = 0;
var min = db.collection.findOne().fieldName.length;
db.collection.find().forEach(function(doc) {
var currentLength = doc.fieldName.length;
if (currentLength > max) {
max = currentLength;
}
if (currentLength < min) {
min = currentLength;
}
});
print(max);
print(min);
}
use <databaseName>
findMinMax();
You can save the function in a file say c:\minMax.js and run the file as,
c:\mongodb\bin> mongo dbName < c:\minMax.js
Note: you may need to supply the necessary hostname, user name, password to connect to your database.
c:\mongodb\bin> mongo --host hostName --port portNumber -u userName -p password dbName < c:\minMax.js

For getting the longest value for a field
db.entities.aggregate([{ $match:{ condition } },{
$addFields: {
"length": { $strLenCP: "$feildName" }
}},
{ "$sort": { "length": -1 } },
{$limit:1}
])
Change the { "$sort": { "length": -1 } } to { "$sort": { "length": 1 } } for the shortest value for a field

How to replace string in all documents in Mongo

I need to replace a string in certain documents. I have googled this code, but it unfortunately does not change anything. I am not sure about the syntax on the line bellow:
pulpdb = db.getSisterDB("pulp_database");
var cursor = pulpdb.repos.find();
while (cursor.hasNext()) {
var x = cursor.next();
x['source']['url'].replace('aaa', 'bbb'); // is this correct?
db.foo.update({_id : x._id}, x);
}
I would like to add some debug prints to see what the value is, but I have no experience with MongoDB Shell. I just need to replace this:
{ "source": { "url": "http://aaa/xxx/yyy" } }
with
{ "source": { "url": "http://bbb/xxx/yyy" } }

It doesn't correct generally: if you have string http://aaa/xxx/aaa (yyy equals to aaa) you'll end up with http://bbb/xxx/bbb.
But if you ok with this, code will work.
To add debug info use print function:
var cursor = db.test.find();
while (cursor.hasNext()) {
var x = cursor.next();
print("Before: "+x['source']['url']);
x['source']['url'] = x['source']['url'].replace('aaa', 'bbb');
print("After: "+x['source']['url']);
db.test.update({_id : x._id}, x);
}
(And by the way, if you want to print out objects, there is also printjson function)

The best way to do this if you are on MongoDB 2.6 or newer is looping over the cursor object using the .forEach method and update each document usin "bulk" operations for maximum efficiency.
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
db.collection.find().forEach(function(doc) {
print("Before: "+doc.source.url);
bulk.find({ '_id': doc._id }).update({
'$set': { 'source.url': doc.source.url.replace('aaa', 'bbb') }
})
count++;
if(count % 200 === 0) {
bulk.execute();
bulk = db.collection.initializeOrderedBulkOp();
}
// Clean up queues
if (count > 0)
bulk.execute();
From MongoDB 3.2 the Bulk() API and its associated methods are deprecated you will need to use the db.collection.bulkWrite() method.
You will need loop over the cursor, build your query dynamically and $push each operation to an array.
var operations = [];
db.collection.find().forEach(function(doc) {
print("Before: "+doc.source.url);
var operation = {
updateOne: {
filter: { '_id': doc._id },
update: {
'$set': { 'source.url': doc.source.url.replace('aaa', 'bbb') }
}
}
};
operations.push(operation);
})
operations.push({
ordered: true,
writeConcern: { w: "majority", wtimeout: 5000 }
})
db.collection.bulkWrite(operations);

Nowadays,
starting Mongo 4.2, db.collection.updateMany (alias of db.collection.update) can accept an aggregation pipeline, finally allowing the update of a field based on its own value.
starting Mongo 4.4, the new aggregation operator $replaceOne makes it very easy to replace part of a string.
// { "source" : { "url" : "http://aaa/xxx/yyy" } }
// { "source" : { "url" : "http://eee/xxx/yyy" } }
db.collection.updateMany(
{ "source.url": { $regex: /aaa/ } },
[{
$set: { "source.url": {
$replaceOne: { input: "$source.url", find: "aaa", replacement: "bbb" }
}}
}]
)
// { "source" : { "url" : "http://bbb/xxx/yyy" } }
// { "source" : { "url" : "http://eee/xxx/yyy" } }
The first part ({ "source.url": { $regex: /aaa/ } }) is the match query, filtering which documents to update (the ones containing "aaa")
The second part ($set: { "source.url": {...) is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
$set is a new aggregation operator (Mongo 4.2) which in this case replaces the value of a field.
The new value is computed with the new $replaceOne operator. Note how source.url is modified directly based on the its own value ($source.url).
Note that this is fully handled server side which won't allow you to perform the debug printing part of your question.

MongoDB can do string search/replace via mapreduce. Yes, you need to have a very special data structure for it -- you can't have anything in the top keys but you need to store everything under a subdocument under value. Like this:
{
"_id" : ObjectId("549dafb0a0d0ca4ed723e37f"),
"value" : {
"title" : "Top 'access denied' errors",
"parent" : "system.admin_reports",
"p" : "\u0001\u001a%"
}
}
Once you have this neatly set up you can do:
$map = new \MongoCode("function () {
this.value['p'] = this.value['p'].replace('$from', '$to');
emit(this._id, this.value);
}");
$collection = $this->mongoCollection();
// This won't be called.
$reduce = new \MongoCode("function () { }");
$collection_name = $collection->getName();
$collection->db->command([
'mapreduce' => $collection_name,
'map' => $map,
'reduce' => $reduce,
'out' => ['merge' => $collection_name],
'query' => $query,
'sort' => ['_id' => 1],
]);