We Keep Coding

Mongoose search by array entries ($in) behavior not aligned with MongoDB Atlas

I'm running a Node.js server, connecting to a MongoDB database with mongoose.
Inside my controller, I have several methods that make operations to the database. One of them is this one:
async findMultiple(req, res) {
const [baseSkillsArray] = Array(req.body);
try {
// if there is not baseSkillsArray, skip
if (!baseSkillsArray) {
return res.status(200).send([]);
}
// find all baseSkills using the ids in the baseSkillsArray
const allBaseSkills = await BaseSkill.find({
_id: { $in: [baseSkillsArray.baseSkillArray] } //
});
console.log('test ' + allBaseSkills);
res.status(200).send(allBaseSkills);
} catch (error) {
console.error(error.message);
res.status(500).send('Server error find BaseSkills');
}
}
However, this returns me nothing. I did some debugging and I found the reason is the find id $in the array. So I tried hard coding a value, like '2', for instance.
// find all baseSkills using the ids in the baseSkillsArray
const allBaseSkills = await BaseSkill.find({ _id: { $in: ['2'] } });
No success. So I went to MongoDB Atlas, where my DB is stored. I tried filtering using the same line of code in my collections.
{ _id: { $in: ['2'] } }
Surprisingly, it returns my document as I wanted!
The issue is that I need to make it work with mongoose. Any ideas? Is this a known bug?

There is nothing wrong with the query, nor a bug regarding $in.
In fact, what's wrong is the actual collection name. I manually created a collection in MongoDB Atlas, called "baseSkills". However, mongoose by default transforms your collection name into lowercase and adds an "s" if your collection's name is not in the plural.
So every time I started my server, I noticed that there was a new collection called "baseskills". I assumed it was a bug and deleted it. Only after making this post that I realized the collection was there again.
So I exported the documents to this collection and my query was working fine.
FYI, there is a way to enforce the collection's name in mongoose. When you declare you model, add a second parameter to the Schema function called "collection". Here is an example:
const BaseSkillSchema = new mongoose.Schema({
_id: {
type: String,
required: true
}, ...
}, { collection: 'baseSkills' })
That's it! Sorry for the mess and thank you for your help!

you want to query over mongo db object ids. So you should create a new ObjectId to do that.
import {Types} from 'mongoose';
{ _id: { $in: [new Types.Object("2")] } }
Or if you have 2 ids one generated and one custom created as id then you can query without creating a new object.
{ id: { $in: ['2'] } }

Populate last element of an array in Mongoose

I have a collection with an array field ("array") that stores _ids which reference another collection. I am using mongoose's .populate and can populate a specific array element using the string dot notation, e.g.
.populate({path: "array.4"})
but I would now like to populate the last element of the array. In an ideal world, "array.-1" would work but it does not. I have tried using populate's match property with something like:
.populate({
path: "array",
match: {
$arrayElemAt: {
$subtract: [
{
$size: "array"
},
1
]
}
},
})
but this doesn't work either (and I don't think is how it's supposed to be used at all!)
Is there any way to achieve this?

Try this.
User.find({},{"arrayObject": {$slice: -1})
.populate('arrayObject')

Not sure if this is precisely what you're looking for but there is an easy way to do that for a specific document, which could help you do that for multiple. Let's imagine that your document's name is User and the array field that stores those ObjectIds is hobbies. So a sample doc may look like this:
{
"_id": ObjectId("5a934e000102030405000000"),
"hobbies": [
ObjectId("5a934e000102030405000001"),
ObjectId("5a934e000102030405000002")
],
"name": "James"
}
You could populate the last element of the hobbies array for a specific user doing this:
User.findById('5a934e000102030405000000')
.then(user => {
user.populate(`hobbies.${user.hobbies.length - 1}`, (err, u) => {
console.log(u);
});
})
If you have multiple docs you want populated, I would do it like this fully realizing that depending on how many docs you have, this make be time consuming:
User.find()
.then(users => {
const promises = users.map(user =>
user.populate(`hobbies.${user.hobbies.length - 1}`).execPopulate()
)
Promise.all(promises).then(pop => {
console.log(pop); //All populated with only last element of hobbies array
})
})

Firebase adding item to array [duplicate]

I'm currently trying Firestore, and I'm stuck at something very simple: "updating an array (aka a subdocument)".
My DB structure is super simple. For example:
proprietary: "John Doe",
sharedWith:
[
{who: "first#test.com", when:timestamp},
{who: "another#test.com", when:timestamp},
],
I'm trying (without success) to push new records into shareWith array of objects.
I've tried:
// With SET
firebase.firestore()
.collection('proprietary')
.doc(docID)
.set(
{ sharedWith: [{ who: "third#test.com", when: new Date() }] },
{ merge: true }
)
// With UPDATE
firebase.firestore()
.collection('proprietary')
.doc(docID)
.update({ sharedWith: [{ who: "third#test.com", when: new Date() }] })
None works. These queries overwrite my array.
The answer might be simple, but I could'nt find it...

Firestore now has two functions that allow you to update an array without re-writing the entire thing.
Link: https://firebase.google.com/docs/firestore/manage-data/add-data, specifically https://firebase.google.com/docs/firestore/manage-data/add-data#update_elements_in_an_array
Update elements in an array
If your document contains an array field, you can use arrayUnion() and
arrayRemove() to add and remove elements. arrayUnion() adds elements
to an array but only elements not already present. arrayRemove()
removes all instances of each given element.

Edit 08/13/2018: There is now support for native array operations in Cloud Firestore. See Doug's answer below.
There is currently no way to update a single array element (or add/remove a single element) in Cloud Firestore.
This code here:
firebase.firestore()
.collection('proprietary')
.doc(docID)
.set(
{ sharedWith: [{ who: "third#test.com", when: new Date() }] },
{ merge: true }
)
This says to set the document at proprietary/docID such that sharedWith = [{ who: "third#test.com", when: new Date() } but to not affect any existing document properties. It's very similar to the update() call you provided however the set() call with create the document if it does not exist while the update() call will fail.
So you have two options to achieve what you want.
Option 1 - Set the whole array
Call set() with the entire contents of the array, which will require reading the current data from the DB first. If you're concerned about concurrent updates you can do all of this in a transaction.
Option 2 - Use a subcollection
You could make sharedWith a subcollection of the main document. Then
adding a single item would look like this:
firebase.firestore()
.collection('proprietary')
.doc(docID)
.collection('sharedWith')
.add({ who: "third#test.com", when: new Date() })
Of course this comes with new limitations. You would not be able to query
documents based on who they are shared with, nor would you be able to
get the doc and all of the sharedWith data in a single operation.

Here is the latest example from the Firestore documentation:
firebase.firestore.FieldValue.ArrayUnion
var washingtonRef = db.collection("cities").doc("DC");
// Atomically add a new region to the "regions" array field.
washingtonRef.update({
regions: firebase.firestore.FieldValue.arrayUnion("greater_virginia")
});
// Atomically remove a region from the "regions" array field.
washingtonRef.update({
regions: firebase.firestore.FieldValue.arrayRemove("east_coast")
});

You can use a transaction (https://firebase.google.com/docs/firestore/manage-data/transactions) to get the array, push onto it and then update the document:
const booking = { some: "data" };
const userRef = this.db.collection("users").doc(userId);
this.db.runTransaction(transaction => {
// This code may get re-run multiple times if there are conflicts.
return transaction.get(userRef).then(doc => {
if (!doc.data().bookings) {
transaction.set({
bookings: [booking]
});
} else {
const bookings = doc.data().bookings;
bookings.push(booking);
transaction.update(userRef, { bookings: bookings });
}
});
}).then(function () {
console.log("Transaction successfully committed!");
}).catch(function (error) {
console.log("Transaction failed: ", error);
});

Sorry Late to party but Firestore solved it way back in aug 2018 so If you still looking for that here it is all issues solved with regards to arrays.
https://firebase.googleblog.com/2018/08/better-arrays-in-cloud-firestore.htmlOfficial blog post
array-contains, arrayRemove, arrayUnion for checking, removing and updating arrays. Hope it helps.

To build on Sam Stern's answer, there is also a 3rd option which made things easier for me and that is using what Google call a Map, which is essentially a dictionary.
I think a dictionary is far better for the use case you're describing. I usually use arrays for stuff that isn't really updated too much, so they are more or less static. But for stuff that gets written a lot, specifically values that need to be updated for fields that are linked to something else in the database, dictionaries prove to be much easier to maintain and work with.
So for your specific case, the DB structure would look like this:
proprietary: "John Doe"
sharedWith:{
whoEmail1: {when: timestamp},
whoEmail2: {when: timestamp}
}
This will allow you to do the following:
var whoEmail = 'first#test.com';
var sharedObject = {};
sharedObject['sharedWith.' + whoEmail + '.when'] = new Date();
sharedObject['merge'] = true;
firebase.firestore()
.collection('proprietary')
.doc(docID)
.update(sharedObject);
The reason for defining the object as a variable is that using 'sharedWith.' + whoEmail + '.when' directly in the set method will result in an error, at least when using it in a Node.js cloud function.

#Edit (add explanation :) )
say you have an array you want to update your existing firestore document field with. You can use set(yourData, {merge: true} ) passing setOptions(second param in set function) with {merge: true} is must in order to merge the changes instead of overwriting. here is what the official documentation says about it
An options object that configures the behavior of set() calls in DocumentReference, WriteBatch, and Transaction. These calls can be configured to perform granular merges instead of overwriting the target documents in their entirety by providing a SetOptions with merge: true.
you can use this
const yourNewArray = [{who: "first#test.com", when:timestamp}
{who: "another#test.com", when:timestamp}]
collectionRef.doc(docId).set(
{
proprietary: "jhon",
sharedWith: firebase.firestore.FieldValue.arrayUnion(...yourNewArray),
},
{ merge: true },
);
hope this helps :)

addToCart(docId: string, prodId: string): Promise<void> {
return this.baseAngularFirestore.collection('carts').doc(docId).update({
products:
firestore.FieldValue.arrayUnion({
productId: prodId,
qty: 1
}),
});
}

i know this is really old, but to help people newbies with the issue
firebase V9 provides a solution using the arrayUnion and arrayRemove
await updateDoc(documentRef, {
proprietary: arrayUnion( { sharedWith: [{ who: "third#test.com", when: new Date() }] }
});
check this out for more explanation

Other than the answers mentioned above. This will do it.
Using Angular 5 and AngularFire2. or use firebase.firestore() instead of this.afs
// say you have have the following object and
// database structure as you mentioned in your post
data = { who: "third#test.com", when: new Date() };
...othercode
addSharedWith(data) {
const postDocRef = this.afs.collection('posts').doc('docID');
postDocRef.subscribe( post => {
// Grab the existing sharedWith Array
// If post.sharedWith doesn`t exsit initiated with empty array
const foo = { 'sharedWith' : post.sharedWith || []};
// Grab the existing sharedWith Array
foo['sharedWith'].push(data);
// pass updated to fireStore
postsDocRef.update(foo);
// using .set() will overwrite everything
// .update will only update existing values,
// so we initiated sharedWith with empty array
});
}

We can use arrayUnion({}) method to achive this.
Try this:
collectionRef.doc(ID).update({
sharedWith: admin.firestore.FieldValue.arrayUnion({
who: "first#test.com",
when: new Date()
})
});
Documentation can find here: https://firebase.google.com/docs/firestore/manage-data/add-data#update_elements_in_an_array

Consider John Doe a document rather than a collection
Give it a collection of things and thingsSharedWithOthers
Then you can map and query John Doe's shared things in that parallel thingsSharedWithOthers collection.
proprietary: "John Doe"(a document)
things(collection of John's things documents)
thingsSharedWithOthers(collection of John's things being shared with others):
[thingId]:
{who: "first#test.com", when:timestamp}
{who: "another#test.com", when:timestamp}
then set thingsSharedWithOthers
firebase.firestore()
.collection('thingsSharedWithOthers')
.set(
{ [thingId]:{ who: "third#test.com", when: new Date() } },
{ merge: true }
)

If You want to Update an array in a firebase document.
You can do this.
var documentRef = db.collection("Your collection name").doc("Your doc name")
documentRef.update({
yourArrayName: firebase.firestore.FieldValue.arrayUnion("The Value you want to enter")});

Although firebase.firestore.FieldValue.arrayUnion() provides the solution for array update in firestore, at the same time it is required to use {merge:true}. If you do not use {merge:true} it will delete all other fields in the document while updating with the new value. Here is the working code for updating array without loosing data in the reference document with .set() method:
const docRef = firebase.firestore().collection("your_collection_name").doc("your_doc_id");
docRef.set({yourArrayField: firebase.firestore.FieldValue.arrayUnion("value_to_add")}, {merge:true});

If anybody is looking for Java firestore sdk solution to add items in array field:
List<String> list = java.util.Arrays.asList("A", "B");
Object[] fieldsToUpdate = list.toArray();
DocumentReference docRef = getCollection().document("docId");
docRef.update(fieldName, FieldValue.arrayUnion(fieldsToUpdate));
To delete items from array user: FieldValue.arrayRemove()

If the document contains a nested object in the form of an array, .dot notation can be used to reference and update nested fields.
Node.js example:
const users = {
name: 'Tom',
surname: 'Smith',
favorites: {
sport: 'tennis',
color: 'red',
subject: 'math'
}
};
const update = await db.collection('users').doc('Tom').update({
'favorites.sport': 'snowboard'
});
or Android sdk example:
db.collection("users").document("Tom")
.update(
'favorites.sport': 'snowboard'
);

There is a simple hack in firestore:
use path with "." as property name:
propertyname.arraysubname.${id}:

db.collection("collection")
.doc("docId")
.update({arrayOfObj: fieldValue.arrayUnion({...item})})

Cannot change _id field in Mongo DB from int to objectId using foreach [duplicate]

I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?

You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})

To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use

In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.

Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])

You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.

As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.

Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter

Migrate simple List Array key to another key with an extra attribute in MongoDB

Sorry if I'm not getting the terminology right. Here's what I have currently my MongoDB user docs db.users:
"liked" : [
"EBMKgrD4DjZxkxvfY",
"WJzAEF5EKB5aaHWC7",
"beNdpXhYLnKygD3yd",
"RHP3hngma9bhXJQ2g",
"vN7uZ2d6FSfzYJLmm",
"NaqAsFmMmnhqNbqbG",
"EqWEY3qkeJYQscuZJ",
"6wsrFW5pFdnQfoWMs",
"W4NmGXyha8kpnJ2bD",
"8x5NWZiwGq5NWDRZX",
"Qu8CSXveQxdYbyoTa",
"yLLccTvcnZ3D3phAs",
"Kk36iXMHwxXNmgufj",
"dRzdeFAK28aKg3gEX",
"27etCj4zbrKhFWzGS",
"Hk2YpqgwRM4QCgsLv",
"BJwYWumwkc8XhMMYn",
"5CeN95hYZNK5uzR9o"
],
And I am trying to migrate them to a new key that also captures the time that a user liked the post
"liked_time" : [
{
"postId" : "5CeN95hYZNK5uzR9o",
"likedAt" : ISODate("2015-09-23T08:05:51.957Z")
}
],
I am wondering if it might be possible to simply do this within the MongoDB Shell with a command that iterates over each user doc and then iterates over the liked array and then updates and $push the new postId and time.
Or would it be better to do this in JavaScript. I am using Meteor.
I almost got it working for individual users. But want to know if I could do all users at once.
var user = Meteor.users.findOne({username:"atestuser"});
var userLiked = user.liked;
userLiked.forEach(function(entry) {
Meteor.users.update({ username: "atestuser" },
{ $push: { liked_times: { postId: entry, likedAt: new Date() }}});
console.log(entry);
});
Still a bit of a newbie to MongoDB obviously......

Here is something i made real quick you should run this on the server side just put it into a file e.g. "migrate.js" in root meteor and run the meteor app
if (Meteor.isServer) {
Meteor.startup(function () {
var users = Meteor.users.find().fetch();
users.forEach(function (doc) {
liked.forEach(function (postId) {
Meteor.users.update(doc._id, { $push: { liked_times: { postId: postId, likedAt: new Date() } } });
});
});
console.log('finished migrating');
});
}
p.s I didn't test it

If this is a one time migration i would do something like this in a one time js script.
Get all users
Iterate over each user
Get all likes
Iterate over them, get likedAt
var liked_times = _.collect(likes, function (likeId) {
return {
'postId' : likeId,
'likedAt': // get post liked time from like id.
}
});
Insert the above in the collection of choice.
Note:
The above example makes use of lodash
I would rather just save likedAt as a timestamp.