We Keep Coding

Sequelize Op.notIn with Sequelize Model

Hello i have a mysql query which is working fine in sequelize.query and the query is
select list_name from lists l where l.list_id not in
(SELECT sub.list_id from list_sub_activities sub left join.
Activities a on a.list_act_id = sub.list_act_id where a.agency_id = 2)
and i want to do the same using the sequelize model, i have tried but i think i am missing something.
List of Package ---> lists
List_of_Packages.findAll({
attributes: ['list_name'],
where: {
list_id: {
[Op.notIn]: [List_sub_Activities.findAll({
attributes: ['list_id'],
include: {
model: Activities,
required: false,
where: {
agency_id: 2
}
}
})
]
}
}
}).then((response) => {
console.log(response);
})
I appreciate that if you help me.
Thank you !!!

The findAll() (and other query methods) are asynchronous so you will need to resolve the promise (or use a callback) to resolve the value before you can pass the list_ids to Op.notIn. It will also return an array of objects with a property of list_id, so you will need to map this to an array of integers before you can use it. You can also pass in raw: true so that it will not generate Sequelize Instances from your results and will instead return plain javascript objects - this is more efficient than creating objects just to fetch a single property.
By setting required: false on the Activities include you will be returning all List_sub_Activities and not filtering on them (some will be null in your results). This is likely not what you intended.
This example uses async/await for clarity instead of thenables. Note that this is not the most efficient as it requires multiple database queries, the ideal solution would be to use a LEFT JOIN and then remove items where the package.list_id IS NULL (see second example).
// get an array of Activities with the list_id set
const activities = await List_sub_Activities.findAll({
attributes: ['list_id'],
include: {
model: Activities,
// don't use required: false to only return results where List_sub_Activities.Activities is not null
// required: false,
where: {
agency_id: 2,
},
},
raw: true,
});
// map the property to an array of just the IDs
const activityIds = activities.map((activity) => activity.list_id);
// now you can pass the activityIds to Op.notIn
const packages = await List_of_Packages.findAll({
attributes: ['list_name'],
where: {
list_id: {
[Op.notIn]: activityIds,
},
},
});
With thenables.
List_sub_Activities.findAll(...)
.then((activities) => activities.map((activity) => activity.list_id))
.then((activityIds) => List_of_Packages.findAll(...))
.then((packages) => {
console.log(packages);
});
This example LEFT JOINs List_of_Packages to List_sub_Activities which is JOINed to Activities with a WHERE setting the agency_id to 2, then only returns results from List_of_Packages where the List_sub_Activities.list_id is NULL (nothing was matched on the LEFT JOIN). This should return the same results as above in a single query.
// Get List_of_Packages where there is no match in List_sub_Activities after
// it is joined to Activities with the agency_id set.
const agencyId = 2;
const packages = await List_of_Packages.findAll({
attributes: ['list_name'],
include: {
model: List_sub_Activities,
// we don't need to actually fetch the list_id
attributes: [],
include: {
model: Activities,
where: {
agency_id: agencyId,
},
},
// uses a LEFT JOIN
required: false,
},
// only return results where the List_sub_Activities.list_id is null
where: sequelize.where(sequelize.col('List_sub_Activities.list_id'), 'IS', null),
});

mongoose check if id exists but that id is nested inside an array

When i fetch new alerts, i want to check if the ID of the new alert was already recorded. The issue is that that ID is nested inside an array. There's the alertsDetails array, which contains objects and those objects have an _ID filed which is what i want to check. I am not sure how to achieve that. I got the code below but then i have to iterate over the result to check the exists value. Im sure there must be a better way.
const mongoose = require('mongoose');
const { Schema } = mongoose;
const G2AlertsSchema = new Schema(
{
status: { type: String, required: true },
openDate: { type: Date, required: true },
alertType: { type: Array, required: true },
severity: { type: Array, required: true },
locationName: { type: Array, required: true },
history: { type: Array, required: true },
alertDetails: { type: Array, required: false },
assignedTo: { type: Schema.Types.ObjectId, ref: 'user' },
},
{
timestamps: true,
},
);
const G2Alerts = mongoose.model('G2Alert', G2AlertsSchema);
module.exports = G2Alerts;
This is the code i found on mongodb's website. I just want to see if the ID exists only. Basically when i fetch the new alerts i get an array and i iterate over it, i want to check each item's ID against what's inside the Database. If it's there, skip and go to the next. If it's new, then create a new alert and save it.
const exists = await G2Alerts.aggregate([
{
$project: {
exists: {
$in: ['5f0b4f508bda3805754ab343', '$alertDetails._id'],
},
},
},
]);
EDIT: Another thing. I am getting a eslint warning saying i should use array iteration instead of a for loop. The issue is, i need to use await when looking up the Alert ID. If i use, reduce or filter, i can't use await. If i use async inside the reduce or filter function, then it will return promises in or just an empty array.
This below works, based on the answer provided by Tom Slabbaert
const newAlertsData = [];
for (let item of alertData.data.items) {
const exists = await G2Alerts.find({ 'alertDetails._id': `${item._id}` });
if (exists.length === 0) {
newAlertsData.push(item);
}
}
if (newAlertsData.length !== 0) {......
But this does not
const filteredAlerts = alertData.data.items.reduce((filtered, item) => {
const exists = await G2Alerts.find({ 'alertDetails._id': `${item._id}` });
if (exists.length === 0) {
filtered.push(item);
}
return filtered;
}, []);

You're not far off, here is an example using the correct syntax:
const exists = await G2Alerts.findOne({"alertDetails._id": '5f0b4f508bda3805754ab343'}});
if (!exists) {
... do something
}
This can also be achieve using aggregate with a $match stage instead of a $project stage or even better countDocuments which just returns the count instead of the entire object if you do not require it.
One more thing I'd like to add is that make sure alertDetails._id is string type as you're using string in you're $in. otherwise you'll need to cast them to ObjectId type in mongoose like so:
new mongoose.Types.ObjectId('5f0b4f508bda3805754ab343')
And for Mongo:
import {ObjectId} from "mongodb"
...
new ObjectId('5f0b4f508bda3805754ab343')
EDIT
Try something like this?
let ids = alertData.data.items.map(item => item._id.toString());
let existing = await G2Alerts.distinct("alertsDetails._id", {"alertsDetails._id": {$in: ids}});
const filteredAlerts = alertData.data.items.reduce((filtered, item) => {
if (!existing.includes(item._id.toString())) {
return [item].concat(filtered)
}
return filtered;
}, []);
This way you only need to call the db once and not multiple times.
Final code based on the provided answer.
const ids = alertData.data.items.map(item => item._id);
const existing = await G2Alerts.find({ 'alertDetails._id': { $in: ids } }).distinct(
'alertDetails._id',
(err, alerts) => {
if (err) {
res.send(err);
}
return alerts;
},
);
const filteredAlerts = alertData.data.items.reduce((filtered, item) => {
if (!existing.includes(item._id.toString()) && item.openDate > dateLimit) {
return [item].concat(filtered);
}
return filtered;
}, []);

How to populate an array inside a map function in js and send it to the server?

This is my ObjectIds array -
obj_ids = [
"5ee71cc94be8d0180c1b63db",
"5ee71c884be8d0180c1b63d9",
"5ee71c494be8d0180c1b63d6",
"5ee71bfd4be8d0180c1b63d4"
]
I am using these objectids to serach whether they exist in the db or not and based on that I want to send the response to server.
This is the code I am trying but I dont know how to populate the array and send it to the server.
var msg = [];
obj_ids.map((ele) => {
Lead.find({ _id: ele._id }, async function (error, docs) {
if (docs.length) {
msg.push(
`Lead already exist for Lead id - ${ele._id} assgined to ${docs[0].salesPerson}`
);
} else {
msg.push(`Lead doesn't exist for Lead id: ${ele._id}`);
const newDuty = new AssignedDuty({
duty: ele._id,
salesPerson: req.body.salesPerson,
});
await newDuty.save();
}
});
});
res.json(msg);
By doing this approach I am getting an empty array. I cannot put res.json(msg) inside the loop. If it is possible by using async-await, please guide me through.

You don't need to make multiple queries to find whether given object ids exist in the database.
Using $in operator, you can make one query that will return all the documents where the _id is equal to one of the object id in the list.
const docs = await Lead.find({
_id: {
$in: [
"5ee71cc94be8d0180c1b63db",
"5ee71c884be8d0180c1b63d9",
"5ee71c494be8d0180c1b63d6",
"5ee71bfd4be8d0180c1b63d4"
]
}
});
After this query, you can check which object id is present in the docs array and which is absent.
For details on $in operator, see $in comparison operator
Your code can be simplified as shown below:
const obj_ids = [
"5ee71cc94be8d0180c1b63db",
"5ee71c884be8d0180c1b63d9",
"5ee71c494be8d0180c1b63d6",
"5ee71bfd4be8d0180c1b63d4"
];
const docs = await Lead.find({
_id: { $in: obj_ids }
});
const msg = [];
obj_ids.forEach(async (id) => {
const doc = docs.find(d => d._id == id);
if (doc) {
msg.push(
`Lead already exist for Lead id - ${doc._id} assgined to ${doc.salesPerson}`
);
}
else {
msg.push(`Lead doesn't exist for Lead id: ${id}`);
const newDuty = new AssignedDuty({
duty: id,
salesPerson: req.body.salesPerson
});
await newDuty.save();
}
});
res.json(msg);

If the specific field is double, convert it to Int [duplicate]

I am trying to change the type of a field from within the mongo shell.
I am doing this...
db.meta.update(
{'fields.properties.default': { $type : 1 }},
{'fields.properties.default': { $type : 2 }}
)
But it's not working!

The only way to change the $type of the data is to perform an update on the data where the data has the correct type.
In this case, it looks like you're trying to change the $type from 1 (double) to 2 (string).
So simply load the document from the DB, perform the cast (new String(x)) and then save the document again.
If you need to do this programmatically and entirely from the shell, you can use the find(...).forEach(function(x) {}) syntax.
In response to the second comment below. Change the field bad from a number to a string in collection foo.
db.foo.find( { 'bad' : { $type : 1 } } ).forEach( function (x) {
x.bad = new String(x.bad); // convert field to string
db.foo.save(x);
});

Convert String field to Integer:
db.db-name.find({field-name: {$exists: true}}).forEach(function(obj) {
obj.field-name = new NumberInt(obj.field-name);
db.db-name.save(obj);
});
Convert Integer field to String:
db.db-name.find({field-name: {$exists: true}}).forEach(function(obj) {
obj.field-name = "" + obj.field-name;
db.db-name.save(obj);
});

Starting Mongo 4.2, db.collection.update() can accept an aggregation pipeline, finally allowing the update of a field based on its own value:
// { a: "45", b: "x" }
// { a: 53, b: "y" }
db.collection.updateMany(
{ a : { $type: 1 } },
[{ $set: { a: { $toString: "$a" } } }]
)
// { a: "45", b: "x" }
// { a: "53", b: "y" }
The first part { a : { $type: 1 } } is the match query:
It filters which documents to update.
In this case, since we want to convert "a" to string when its value is a double, this matches elements for which "a" is of type 1 (double)).
This table provides the code representing the different possible types.
The second part [{ $set: { a: { $toString: "$a" } } }] is the update aggregation pipeline:
Note the squared brackets signifying that this update query uses an aggregation pipeline.
$set is a new aggregation operator (Mongo 4.2) which in this case modifies a field.
This can be simply read as "$set" the value of "a" to "$a" converted "$toString".
What's really new here, is being able in Mongo 4.2 to reference the document itself when updating it: the new value for "a" is based on the existing value of "$a".
Also note "$toString" which is a new aggregation operator introduced in Mongo 4.0.
In case your cast isn't from double to string, you have the choice between different conversion operators introduced in Mongo 4.0 such as $toBool, $toInt, ...
And if there isn't a dedicated converter for your targeted type, you can replace { $toString: "$a" } with a $convert operation: { $convert: { input: "$a", to: 2 } } where the value for to can be found in this table:
db.collection.updateMany(
{ a : { $type: 1 } },
[{ $set: { a: { $convert: { input: "$a", to: 2 } } } }]
)

For string to int conversion.
db.my_collection.find().forEach( function(obj) {
obj.my_value= new NumberInt(obj.my_value);
db.my_collection.save(obj);
});
For string to double conversion.
obj.my_value= parseInt(obj.my_value, 10);
For float:
obj.my_value= parseFloat(obj.my_value);

db.coll.find().forEach(function(data) {
db.coll.update({_id:data._id},{$set:{myfield:parseInt(data.myfield)}});
})

all answers so far use some version of forEach, iterating over all collection elements client-side.
However, you could use MongoDB's server-side processing by using aggregate pipeline and $out stage as :
the $out stage atomically replaces the existing collection with the
new results collection.
example:
db.documents.aggregate([
{
$project: {
_id: 1,
numberField: { $substr: ['$numberField', 0, -1] },
otherField: 1,
differentField: 1,
anotherfield: 1,
needolistAllFieldsHere: 1
},
},
{
$out: 'documents',
},
]);

To convert a field of string type to date field, you would need to iterate the cursor returned by the find() method using the forEach() method, within the loop convert the field to a Date object and then update the field using the $set operator.
Take advantage of using the Bulk API for bulk updates which offer better performance as you will be sending the operations to the server in batches of say 1000 which gives you a better performance as you are not sending every request to the server, just once in every 1000 requests.
The following demonstrates this approach, the first example uses the Bulk API available in MongoDB versions >= 2.6 and < 3.2. It updates all
the documents in the collection by changing all the created_at fields to date fields:
var bulk = db.collection.initializeUnorderedBulkOp(),
counter = 0;
db.collection.find({"created_at": {"$exists": true, "$type": 2 }}).forEach(function (doc) {
var newDate = new Date(doc.created_at);
bulk.find({ "_id": doc._id }).updateOne({
"$set": { "created_at": newDate}
});
counter++;
if (counter % 1000 == 0) {
bulk.execute(); // Execute per 1000 operations and re-initialize every 1000 update statements
bulk = db.collection.initializeUnorderedBulkOp();
}
})
// Clean up remaining operations in queue
if (counter % 1000 != 0) { bulk.execute(); }
The next example applies to the new MongoDB version 3.2 which has since deprecated the Bulk API and provided a newer set of apis using bulkWrite():
var bulkOps = [];
db.collection.find({"created_at": {"$exists": true, "$type": 2 }}).forEach(function (doc) {
var newDate = new Date(doc.created_at);
bulkOps.push(
{
"updateOne": {
"filter": { "_id": doc._id } ,
"update": { "$set": { "created_at": newDate } }
}
}
);
})
db.collection.bulkWrite(bulkOps, { "ordered": true });

To convert int32 to string in mongo without creating an array just add "" to your number :-)
db.foo.find( { 'mynum' : { $type : 16 } } ).forEach( function (x) {
x.mynum = x.mynum + ""; // convert int32 to string
db.foo.save(x);
});

What really helped me to change the type of the object in MondoDB was just this simple line, perhaps mentioned before here...:
db.Users.find({age: {$exists: true}}).forEach(function(obj) {
obj.age = new NumberInt(obj.age);
db.Users.save(obj);
});
Users are my collection and age is the object which had a string instead of an integer (int32).

You can easily convert the string data type to numerical data type.
Don't forget to change collectionName & FieldName.
for ex : CollectionNmae : Users & FieldName : Contactno.
Try this query..
db.collectionName.find().forEach( function (x) {
x.FieldName = parseInt(x.FieldName);
db.collectionName.save(x);
});

I need to change datatype of multiple fields in the collection, so I used the following to make multiple data type changes in the collection of documents. Answer to an old question but may be helpful for others.
db.mycoll.find().forEach(function(obj) {
if (obj.hasOwnProperty('phone')) {
obj.phone = "" + obj.phone; // int or longint to string
}
if (obj.hasOwnProperty('field-name')) {
obj.field-name = new NumberInt(obj.field-name); //string to integer
}
if (obj.hasOwnProperty('cdate')) {
obj.cdate = new ISODate(obj.cdate); //string to Date
}
db.mycoll.save(obj);
});

demo change type of field mid from string to mongo objectId using mongoose
Post.find({}, {mid: 1,_id:1}).exec(function (err, doc) {
doc.map((item, key) => {
Post.findByIdAndUpdate({_id:item._id},{$set:{mid: mongoose.Types.ObjectId(item.mid)}}).exec((err,res)=>{
if(err) throw err;
reply(res);
});
});
});
Mongo ObjectId is just another example of such styles as
Number, string, boolean that hope the answer will help someone else.

I use this script in mongodb console for string to float conversions...
db.documents.find({ 'fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.fwtweaeeba = parseFloat( obj.fwtweaeeba );
db.documents.save(obj); } );
db.documents.find({ 'versions.0.content.fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.versions[0].content.fwtweaeeba = parseFloat( obj.versions[0].content.fwtweaeeba );
db.documents.save(obj); } );
db.documents.find({ 'versions.1.content.fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.versions[1].content.fwtweaeeba = parseFloat( obj.versions[1].content.fwtweaeeba );
db.documents.save(obj); } );
db.documents.find({ 'versions.2.content.fwtweaeeba' : {$exists : true}}).forEach( function(obj) {
obj.versions[2].content.fwtweaeeba = parseFloat( obj.versions[2].content.fwtweaeeba );
db.documents.save(obj); } );
And this one in php)))
foreach($db->documents->find(array("type" => "chair")) as $document){
$db->documents->update(
array('_id' => $document[_id]),
array(
'$set' => array(
'versions.0.content.axdducvoxb' => (float)$document['versions'][0]['content']['axdducvoxb'],
'versions.1.content.axdducvoxb' => (float)$document['versions'][1]['content']['axdducvoxb'],
'versions.2.content.axdducvoxb' => (float)$document['versions'][2]['content']['axdducvoxb'],
'axdducvoxb' => (float)$document['axdducvoxb']
)
),
array('$multi' => true)
);
}

The above answers almost worked but had a few challenges-
Problem 1: db.collection.save no longer works in MongoDB 5.x
For this, I used replaceOne().
Problem 2: new String(x.bad) was giving exponential number
I used "" + x.bad as suggested above.
My version:
let count = 0;
db.user
.find({
custID: {$type: 1},
})
.forEach(function (record) {
count++;
const actualValue = record.custID;
record.custID = "" + record.custID;
console.log(`${count}. Updating User(id:${record._id}) from old id [${actualValue}](${typeof actualValue}) to [${record.custID}](${typeof record.custID})`)
db.user.replaceOne({_id: record._id}, record);
});
And for millions of records, here are the output (for future investigation/reference)-

MongoDb Node.js Querying Nested Objects not working?

I need to query for 2 dynamic properties using the node.js driver for mongodb.
Here's the data structure:
{
"_id":"123456",
"dateAdded":"2017-09-20T08:36:40.325Z",
"followers":{
"name1":{
"followedOn":"2017-09-20T08:36:40.325Z",
"unfollowedOn":null
},
"name2":{
"followedOn":"2017-09-20T08:36:40.325Z",
"unfollowedOn":null
}
}
}
Here's my code:
//Working but not dynamic
collections.find({ '_id': '123456', 'followers.name1': { $exists: false } })
//My failed attempt at making it dynamic
const id = "123456"
const username = "name1"
let query = {}
query['followers.'+username] = { $exists: true }
collections.find( { "_id": id, query }
Note that this is not a duplicate of "how to make a dynamic key in an object literal". The .find() method of node.js mongodb driver does not accept object literals. I can't find documentation of what it accepts exactly.

Your _id property needs to be within query object, not separate.
Here's how to do it:
let query = { _id: id };
query['followers.'+username] = { $exists: true }
collections.find(query);