We Keep Coding

Mongoose search by array entries ($in) behavior not aligned with MongoDB Atlas

I'm running a Node.js server, connecting to a MongoDB database with mongoose.
Inside my controller, I have several methods that make operations to the database. One of them is this one:
async findMultiple(req, res) {
const [baseSkillsArray] = Array(req.body);
try {
// if there is not baseSkillsArray, skip
if (!baseSkillsArray) {
return res.status(200).send([]);
}
// find all baseSkills using the ids in the baseSkillsArray
const allBaseSkills = await BaseSkill.find({
_id: { $in: [baseSkillsArray.baseSkillArray] } //
});
console.log('test ' + allBaseSkills);
res.status(200).send(allBaseSkills);
} catch (error) {
console.error(error.message);
res.status(500).send('Server error find BaseSkills');
}
}
However, this returns me nothing. I did some debugging and I found the reason is the find id $in the array. So I tried hard coding a value, like '2', for instance.
// find all baseSkills using the ids in the baseSkillsArray
const allBaseSkills = await BaseSkill.find({ _id: { $in: ['2'] } });
No success. So I went to MongoDB Atlas, where my DB is stored. I tried filtering using the same line of code in my collections.
{ _id: { $in: ['2'] } }
Surprisingly, it returns my document as I wanted!
The issue is that I need to make it work with mongoose. Any ideas? Is this a known bug?

There is nothing wrong with the query, nor a bug regarding $in.
In fact, what's wrong is the actual collection name. I manually created a collection in MongoDB Atlas, called "baseSkills". However, mongoose by default transforms your collection name into lowercase and adds an "s" if your collection's name is not in the plural.
So every time I started my server, I noticed that there was a new collection called "baseskills". I assumed it was a bug and deleted it. Only after making this post that I realized the collection was there again.
So I exported the documents to this collection and my query was working fine.
FYI, there is a way to enforce the collection's name in mongoose. When you declare you model, add a second parameter to the Schema function called "collection". Here is an example:
const BaseSkillSchema = new mongoose.Schema({
_id: {
type: String,
required: true
}, ...
}, { collection: 'baseSkills' })
That's it! Sorry for the mess and thank you for your help!

you want to query over mongo db object ids. So you should create a new ObjectId to do that.
import {Types} from 'mongoose';
{ _id: { $in: [new Types.Object("2")] } }
Or if you have 2 ids one generated and one custom created as id then you can query without creating a new object.
{ id: { $in: ['2'] } }

Why $ne: null returning the empty arrays?

I`m trying to return the arrays are that have contents but $ne: null or $exsists: true still returning the empty arrays..
const userSchema = new mongoose.Schema({
email: String,
password: String,
googleId: String,
facebookId: String,
secret: Array
});
app.get("/secrets", function(req, res) {
User.find({secret: {$ne: null} }, function(err, secrets) {
if (err) {
console.log(err);
} else {
if (secrets) {
console.log(secrets);
res.render("secrets", {
secrets: secrets
});
};
};
});
});
I googled quite a bit about it and I do understand that $ne: null would return every document where the secret array doesn't exists, but if its an empty array then why? Any suggestion how to overcome this rookie problem? Im new here be kind! :)

This is because null != [].
You will have to explicitly write your condition to handle both.
$nin is for not in. So find users where secret value is not in the given array. The given array can hold your multiple values.
User.find({secret: {$nin: [null, [] ] } },...
Also, with $exists you will even get docs where the field is null. The only docs you will not get are the ones where the secret field does not exist at all.

If the documents contain values for the secret field that include missing/undefined, null, empty array, and populated array, you have a few options to only match populated arrays:
{$eval:{$gt:[0,{$size:"$secret"}]}}- get the size for thesecrets` array, and only match arrays that are not empty.
{"secret.0":{$exists:true}} - only match the document if there is a first element in the secret array (implicitly doesn't match non-array fields)
If the elements in the secret array are all the same type, a type-based query can be used. For example, if the elements in the secret array are strings, {"secret.0":{$gte:""}} will match any array whose first element is a string. This could also be optimized by creating an index on {"secret.0":1}

Why am I not being able to write into objects I fetched from database?

What I am trying to do: I am fetching data from MongoDB(doc). I has property files which is an array of objects. Schema of the mentioned object:
{
fileName : {type : String, required : true},
fileSize : {type : Number, required : true},
fileOriginalName : String,
sentEarlier : Boolean,
}
and before sending this array(in doc.files) to my frontend, I want to append a property(downloadLink), to each object. The whole code where I encountered the problem:
Router.post('/fetch-more-files', (req, res) => {
let {uuid} = req.body;
File.findOne({uuid}, (err, doc) => {
if(err) {
return res.send({'error' : 'Something went wrong. Refresh the page.'});
}
if(!doc) {
return res.send({'error' : 'Invalid link or link expired'});
}
let newFiles = [];
doc.files.forEach(file => {
file.downloadLink = `${process.env.APP_BASE_URL}/files/download/${uuid}/${file.fileName}`;
});
doc.files.forEach(file => {
if(!file.sentEarlier) newFiles.push(file);
});
doc.save((err, savedDoc) => {
res.json({newFiles});
});
});
});
PROBLEM: The newFiles array which I am populating with all the files(objects inside doc.files array), those that do not have a 'sentEarlier' key true to them, when sent as response, not a single object that array has the 'downloadLink' property to them. I tried so hard and for so long to debug, but failed. Hope to find some insights as to what I could be doing wrong here. Thanks in advance!!!

SOLUTION: After hours of lingering and loitering over this problem, I found a solution. If you want the technical answer to why it didn't work, this is not the place for you. But if you just want your problem solved, so you can finally go pee, read on.
The answer is to use toObject() method which apparently converts the mongoose document to a plain JS object, which is now at your disposal.
toObject() in mongooseJS documentation:
Mongoose v5.11.9:API docs | document.prototype.toObject()
Other answers are most welcome.

Mongoose queries return an instance of the Mongoose Document class. if you want to modify your query response just try to call your mongoose query in a different way.
File.findOne({uuid}, {new: true}).lean().exec((err, doc) => {
if(err) {
return res.send({'error' : 'Something went wrong. Refresh the page.'});
}
// your logic
});
here you get more idea about lean()

Exclude a field in Mongoose wile using assignment operator for Query

I am aware that this question has been asked before. Sites often give the solution using .select('-queryData')however I do not know how to use it for my scenario.
I have a DB with the following schema
let TestSchema = new Schema ({
..
test: {type: String, required: true},
arrayField: {type: Array, required: true},
..
});
I have to query the arrayField where it should not include 'data1'. If it includes this value then that object should not be shown.
However I am using assignment operator for query object, instead of calling the function '.select()'. Hence I tried adding it in the query JSON itself, but it did not work.
index.js
let query = {};
query.test = "Hello";
query.arrayField = '-data1'; //Change needed here
TestSchema.find(query, function(err, result){
if(err){
console.log(err);
}else{
console.log(result);
}
});
I tried checking various sites for a specific solution to such a scenario, however I could not get any conclusive solution.

You can use $exists mongodb operator for your query.
TestModel.find({ test: "Hello", "arrayField.data1": { $exists: false } })
Select method has a different use case which is to select or limit the fields returned as a result of a query. And you need to execute find method on TestModel not TestSchema.
const TestModel = mongoose.model("TestModel", TestSchema);

You should use the $exists operator, here's an example:
TestModel.find({
test: "Hello",
"arrayField.$.data1": {
$exists: false
}
}, function(err, result){
if(err){
console.log(err);
}else{
console.log(result);
}
});

an error about tailable cursor for a query in mongoose

User model contian SubscriptionSchema and AccessToken Schema, I had defined this two plugin schemas with {capped : 234556} too.
var User = new Schema({
email : String
, firstName : String
, password : String
, isAdmin : Boolean
, lastSeen : Date
, subscriptions : [ SubscriptionSchema ]
, accessTokens : [ AccessToken ]
}, {
toObject : { virtuals : true }
, toJSON : { virtuals : true }
, capped : 234556
});
var streamTest = User.find().limit(1).tailable().stream();
When I try to run the above code, I still get the error:
MongoError: tailable cursor requested on non capped collection

That doesn't look like a correct usage of a capped collection or tailable stream. But perhaps a little code first to demonstrate a working example:
var async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
var userSchema = new Schema({
email: String,
},{
capped: 2048
});
var User = mongoose.model( "User", userSchema );
mongoose.connect('mongodb://localhost/atest');
var stream;
async.waterfall(
[
function(callback) {
var user = new User({ email: "existing" });
user.save(function(err,doc) {
if (err) throw err;
callback();
});
},
function(callback) {
User.find({}).sort({ "$natural": -1 }).limit(1).exec(function(err,docs) {
if (err) throw err;
console.log( docs );
callback(err,docs[0]);
});
},
function(doc,callback) {
stream = User.find({ "_id": { "$gt": doc._id } }).tailable().stream();
stream.on("data",function(doc) {
console.log( "Streamed:\n%s", doc );
});
callback();
},
function(callback) {
async.eachSeries(
['one','two','three'],
function(item,callback) {
var user = new User({ email: item });
user.save(function(err,doc) {
if (err) throw err;
console.log( "Saved:\n%s", doc );
callback();
});
},
function(err) {
if (err) throw err;
callback();
}
);
}
]
);
First things first, there really needs to be something in the capped collection for anything to work. This presumes that the collection does not exist and it is going to be initialized as a capped collection. Then the first step is making sure something is there.
Generally when you want to "tail", you just want the new documents that are inserted to appear. So before setting up the tailable cursor you want to find the "last" document in the collection.
When you know the last document in the collection, the "tailable stream" is set up to look for anything that is "greater than" that document, which are the new documents. If you did not do this, your first "data" event on the stream would empty all of the current collection items. So the options to .sort() and .limit() here do not apply. Tailing cursors initialize and "follow".
Now that the streaming interface is set up and an listener established, you can add items to the stream. These will then log accordingly, yet as this is "evented", there is no particular sequence to the logging here as either the "save" or the "stream" data event may actually fire first.
Now onto your implementation. These two lines stand out:
, subscriptions : [ SubscriptionSchema ]
, accessTokens : [ AccessToken ]
Those contain embedded arrays, they are not "external" documents in another collection, even though it would not matter if it even did.The general problem here is that you are (at least in some way) introducing an array, which seems to imply some concept of "growth".
Unless your intention is to never "grow" these arrays and only ever insert the content with the new document and never update it, then this will cause problems with capped collections.
Documents in capped collections cannot "grow" beyond their initial allocated size. Trying to update where this happens will result in errors. If you think you are going to be smart about it and "pad" your documents, then this is likely to fail when replicated secondary hosts "re-sync". All documented with capped collections.