We Keep Coding

Mongoose gives errors when trying to save object into array [duplicate]

I wrote a service that analyses videos with Google Cloud Video Intelligence
And I save the analysis results to the MongoDB with mongoose
This is the model I use (I've simplified everything to avoid confusion):
// Video.js
const mongoose = require('mongoose');
const videoSchema = new mongoose.Schema({
analysis_progress: {
percent: { type: Number, required: true },
details: {}
},
status: {
type: String,
enum: ['idle', 'processing', 'done', 'failed'],
default: 'idle'
}
});
module.exports = mongoose.model('Video', videoSchema);
When analyse operation ends, I call the function below and run update like this:
function detectFaces(video, results) {
//Build query
let update = {
$set: {
'analysis_results.face_annotations': results.faceDetectionAnnotations // results is the the test result
}
};
Video.findOneAndUpdate({ _id: video._id }, update, { new: true }, (err, result) => {
if (!err)
return console.log("Succesfully saved faces annotiations:", video._id);
throw err // This is the line error thrown
});
}
And this is the error I get:
Error: cyclic dependency detected
at serializeObject (C:\Users\murat\OneDrive\Masaüstü\bycape\media-analysis-api\node_modules\bson\lib\bson\parser\serializer.js:333:34)
at serializeInto (C:\Users\murat\OneDrive\Masaüstü\bycape\media-analysis-api\node_modules\bson\lib\bson\parser\serializer.js:947:17)
...
Solutions I tried:
Added {autoIndex: false} inside db config.
mongoose.connect(process.env.DB_CONNECTION, {useNewUrlParser: true, useUnifiedTopology: true, useFindAndModify: false, autoIndex: false });
Removing retryWrites=true from Mongo URI structure. (I didn't have that parameter in my connection URI already)
So, I think the source of the problem is that I am saving the whole test result but I don't have any other option to do that. I need to save as it is.
I am open to all kinds of suggestions.

Just as I guessed, the problem was that there was a cyclic dependency in the object that came to me from google.
With help of my colleague:
Then since JSON.stringify() changes an object into simple types:
string, number, array, object, boolean it is not capable of storing
references to objects therefor by using stringify and then parse you
destroy the information that stringify cannot convert.
Another way would be knowing which field held the cyclic reference and
then unsetting, or deleting that field.
I couldn't find which field has cycylic dependency so I used I JSON.stringfy() and JSON.parse() to remove it.
let videoAnnotiations = JSON.stringify(operationResult.annotationResults[0]);
videoAnnotiations = JSON.parse(videoAnnotiations);

MongoDB Atlas with "Full Document" enabled returns only the changed item, not the full document

I am trying to use Triggers with MongoDB Atlas to notify on changes to a document in my collection. I want to receive the full document that had any data in it change, and use that full document upon receipt of the change notification. In the triggers configuration, there is a slider to enable/disable "Full Document" which has the following description:
By turning on Full Document, you will receive the document created or
modified in your change event. For Delete operations, the full
document will not exist.
However, with or without that slider enabled, I get the same results.
Here is my change listener code:
const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true })
await client.connect().then(db => {
const collection = client.db("database_name").collection("collection_name")
const changeStream = collection.watch()
changeStream.on("change", async data => {
console.log("Detected database change on", Date())
console.log(data) // only returns changed data
})
})
Here is the example output, which as stated doesn't include the full document despite the trigger configuration:
{
_id: {
_data: '82606C926E000000012B022C0100296E5A10049008F3458DF14719A9225DF7AB403CEC46645F69640064606C76124670930F9A1F657C0004'
},
operationType: 'update',
clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1617728110 },
ns: { db: 'database_name', coll: 'collection_name' },
documentKey: { _id: 606c76124670930f9a1f657c },
updateDescription: {
updatedFields: {
my_data: 'my changed data',
created: 1617728109618
},
removedFields: []
}
}
The full document should look like this:
{
_id: 606c76124670930f9a1f657c,
myKey: 'something that never changed',
my_data: 'my changed data',
created: 1617728109618,
expires: 'some time in seconds that never changed',
other_info: 'something that never changed'
}
Any help would be appreciated... e.g. do I need to add a function in the trigger configuration to return the full document? If so, what would such a function look like?

I typically don't like answering my own questions, but I figured I'll post the answer here anyway in the event someone else needs it.
After a bit more digging, it looks like collection.watch() can accept options. With Full Document enabled in Atlas, my code needed this:
const changeStream = collection.watch([], { fullDocument: 'updateLookup' })
Now the full document is returned as expected. The documentation I found comes from here, and here.

TypeORM findOne with nested relations

I am having some issues performing a nested find query with TypeORM. Here's the basic code:
const { completionId } = req?.params;
const user = req.user;
const retrievedCompletion = await getRepository(
CompletionGoogleSearch
).findOne({
relations: ['run', 'run.user'],
where: {
id: completionId,
// run: { user: { id: user.id } }, // This is the code that breaks the function
},
});
console.log(retrievedCompletion?.run.user.id);
console.log(user.id);
It looks to me like there's nothing out of order, and that the query should run. Any idea on what I am doing wrong? I know I can get around this issue by writing a querybuilder query or using raw SQL–I am just curious to understand if there's a flaw in my code.

typeorm added the ability to use nested object
userRepository.find({
relations: {
profile: true,
photos: true,
videos: {
videoAttributes: true,
},
},
});
on this way, you can fetch the data without using eager.
You can find more information here

The feature you're asking about doesn't supported on typeorm yet (Feb 2021).
Checkout this issue that was opened on 2018.

the Solution is use eager:true in run.user entity :
#OneToOne(() => User, User=> User.run, {
eager:true
})
user: User;
and next time you search in CompletionGoogleSearch do just relations: ['run'] and user will come with it.

Save Or Update Mongoose

Is there a nice way of either saving, or updating a document in mongoose? Something like what I'm after below
let campaign = new Campaign({
title: req.body.title,
market: req.body.market,
logo: req.body.logo,
additional_question_information: question,
status: status
});
campaign.saveOrUpdate().then(function() { ... }
Thanks for the help all

I think what you're looking for is called an 'upsert'.
You can do this by using findOneAndUpdate and passing the { upsert: true } option, something like the below example:
let campaign = new Campaign({
title: req.body.title,
market: req.body.market,
logo: req.body.logo,
additional_question_information: question,
status: status
});
Campaign.findOneAndUpdate({
_id: mongoose.Types.ObjectId('CAMPAIGN ID TO SEARCH FOR')
}, campaign, { upsert: true }, function(err, res) {
// Deal with the response data/error
});
The first parameter to findOneAndUpdate is the query used to see if you're saving a new document or updating an existing one. If you want to return the modified document in the response data then you can also add the { new: true } option.
Documentation here for findOneAndUpdate: http://mongoosejs.com/docs/api.html#model_Model.findOneAndUpdate

You can use the MongoDB's findAndModify function. In mongoose this is natively supported by calling findOneAndUpdate(). Here is the documentation for it. http://mongoosejs.com/docs/api.html#query_Query-findOneAndUpdate
Notice that in the third argument it awaits for an object to be passed with options. You want to use { upsert : true } in there to create a new document if one does not exist.

Mongoose - Upserting Documents with Nested Models

I have a basic document with a 'checked_in' flag in my express app:
module.exports = Book= mongoose.model('Book', new Schema({
name : String,
checked_in : Boolean
},{ collection : 'Book' }));
I wanted to keep a log of when books are checked in and out so I came up with another schema:
var action = new Schema({
checked_in: Boolean,
});
module.exports = Activity = mongoose.model('Activity', new Schema({
book_id: String,
actions: [action]
},{ collection : 'Activity' }));
The 'book_id' should be the document id of a book and when I update a book I need to either create or update the activity log for that book with a new item inside of actions:
exports.update = function(req, res){
return Book.findById(req.params.id, function(err, book) {
var activity = new Activity({book_id: book.id});
activity.actions.push({
checked_in: req.body.checked_in,
});
Activity.update({ book_id: book.id}, activity.toObject(), { upsert: true }));
book.checked_in = req.body.checked_in;
return device.save(function(err) {
return res.send(book);
});
});
};
The problem I am having is that nothing gets inserted into the Activity collection. If I use .save() then i just get lots of duplicates in the collection.
UPDATE
I've started re-working things with the advice given below but am still not having any luck with this. Here's what I have now:
module.exports = Activity = mongoose.model('Activity', new Schema({
book_id: Schema.ObjectId,
actions: [new Schema({
checked_in: Boolean,
last_user: String
})]
},{ collection : 'Activity' }));
Here's the update code now:
exports.update = function(req, res){
// TODO: Check for undefined.
return book.findById(req.params.id, function(err, book) {
if(!err) {
// Update the book.
book.checked_in = req.body.checked_in;
book.last_user = req.body.last_user;
book.save();
// If there's no associated activity for the book, create one.
// Otherwise update and push new activity to the actions array.
Activity.findById(book._id, function (err, activity) {
activity.actions.push({
checked_in: req.body.checked_in,
last_user: req.body.last_user
})
activity.save();
});
}
});
};
What I want to end up with is a document for each book with an array of check outs/ins that gets updated each time someone checks a book in or out. i.e:
{
book_id: "5058c5ddeeb0a3aa253cf9d4",
actions: [
{ checked_in: true, last_user: 'ralph' },
{ checked_in: true, last_user: 'gonzo' },
{ checked_in: true, last_user: 'animal' }
]
}
Eventually I will have a time stamp within each entry.

There are a couple problems:
You're trying to find the book's activity doc using findById using the book's id instead of the activity's id.
You're not handling the case where the book's activity doc doesn't exist yet.
Try this instead:
Activity.findOne({book_id: book._id}, function (err, activity) {
if (!activity) {
// No Activity doc for the book yet, create one.
activity = new Activity({book_id: book._id});
}
activity.actions.push({
checked_in: req.body.checked_in,
last_user: req.body.last_user
});
activity.save();
});

I see a few things that can be improved...
The book_id field in the Activity model should be Schema.ObjectId instead of a String. You will then be able to use populate if you wish.
You aren't doing any error checking in exports.update. If the user passes in an invalid id, you will want to check if book is undefined or not, as well as the common if (err) return next(err) (this requires your function params to be res, res, next).
When you create the activity in exports.update, you want to use book._id instead of book.id
All the return statements are not needed
The device variable is not declared anywhere, I'm not sure what you are trying to save... I think you meant book there.
You can then just .save() the activity instead of doing the Activity.update.