I have a list of images that I get with Angular using $http.get() from MongoDB using Mongoose and Expressjs.
What I did works fine but I have a doubt about performance.
So far I found two ways to do it:
using skip
using $nin
The queries looks like this:
// Using $nin:
var skip = req.query.skip || [];
User.find({ _id : { $nin: skip }})
.sort({ _id: -1 })
.limit(15)
.exec(function(err, users) {
if (err) res.json({ 'msg': 'Error loading users' });
res.json({
users: users
});
});
and:
// Using skip
User.find({})
.sort({ _id: -1 })
.skip(15)
.limit(15)
.exec(function(err, users) {
if (err) res.json({ 'msg': 'Error loading users' });
res.json({
users: users
});
});
Googleling around it seems like using skip lack of performance after a while...
But looking at the $nin option I found after scrolling and scrolling a very long query... with plenty of _id...
Which of the 2 solution should be better to use?
Or there is a third way much better than those, maybe I'm doing something wrong?
Thanks
Ideally you want to use $nin with a list of previously seen _id values but combine that with a $gte or $lte (depending on order) operator on something you are sorting on.
This is generally the case for most "otherwise sorted queries", but in the case where the _id field is the one you are sorting on ( and in decending order ) it just becomes a matter of working with $lt to find values that are less than the "last seen value" from the previous page.
So when iterating, store the "last seen value" from the last item in your "page limit" results, then use the $lt operator in subsequent queries:
ie:
var lastSeen = null; // declare in a global or session or something
User.find({})
.sort({ "_id": -1 })
.limit(15)
.exec(function(err,docs) {
lastSeen = docs.slice(-1).id;
});
And then:
User.find({ "_id": { "$lt": lastSeen })
.sort({ "_id": -1 })
.limit(15)
.exec(function(err,docs) {
lastSeen = docs.slice(-1).id;
});
Related
I've written a function to execute hourly which looks up a user and finds some values and then pushes those values into a history collection that records the hourly updated values. I've written this so far as a test just finding a user by their ID but now I need to roll this out to my entire database of 50,000+ users.
From what I've read using updateMany is a lot more performant but I'm not entirely sure how to retrieve the document detail of the record that is being updated at the time.
Here is my code so far, which you can see I'm first looking up the user and then grabbing their valuation details which I'd like to then push into a history collection.
exports.updateUserValuationHistoric = () => {
User.find({ _id: "609961fdd989613914ef7216" })
.populate('UserValuationHistory')
.exec((err, userDoc) => {
if (err){
console.log('[ERROR]: ', err)
}
const updatedValuationHistory = {
totalValuation: userDoc[0].valuation.totalValuation,
comicsValuation: userDoc[0].valuation.comicsValuation,
collectiblesValuation: userDoc[0].valuation.collectiblesValuation,
omiValuation: userDoc[0].valuation.omiValuation
}
UserValuationHistory.findOneAndUpdate(
{ user: userDoc[0]._id },
{ $push: {
'history': updatedValuationHistory
}},
{upsert: true, new: true}
)
.exec((error, updated) => {
if (error){
console.log('[ERROR]: Unable to update the user valuation history.')
} else {
console.log('[SUCCESS]: User historic valuation has been updated.')
}
})
})
}
Any help is greatly appreciated!
User model:
https://pastebin.com/7MWBVHf3
Historic model:
https://pastebin.com/nkTGztJY
I have a Documents in a Collection that have a field that is an Array (foo). This is an Array of other subdocuments. I want to set the same field (bar) for each subdocument in each document to the same value. This value comes from a checkbox.
So..my client-side code is something like
'click #checkAll'(e, template) {
const target = e.target;
const checked = $(target).prop('checked');
//Call Server Method to update list of Docs
const docIds = getIds();
Meteor.call('updateAllSubDocs', docIds, checked);
}
I tried using https://docs.mongodb.com/manual/reference/operator/update/positional-all/#positional-update-all
And came up with the following for my Server helper method.
'updateAllSubDocs'(ids, checked) {
Items.update({ _id: { $in: ids } }, { $set: { "foo.$[].bar": bar } },
{ multi: true }, function (err, result) {
if (err) {
throw new Meteor.Error('error updating');
}
});
}
But that throws an error 'foo.$[].bar is not allowed by the Schema'. Any ideas?
I'm using SimpleSchema for both the parent and subdocument
Thanks!
Try passing an option to bypass Simple Schema. It might be lacking support for this (somewhat) newer Mongo feature.
bypassCollection2
Example:
Items.update({ _id: { $in: ids } }, { $set: { "foo.$[].bar": bar } },
{ multi: true, bypassCollection2: true }, function (err, result) {
if (err) {
throw new Meteor.Error('error updating');
}
});
Old answer:
Since you say you need to make a unique update for each document it sounds like bulk updating is the way to go in this case. Here's an example of how to do this in Meteor.
if (docsToUpdate.length < 1) return
const bulk = MyCollection.rawCollection().initializeUnorderedBulkOp()
for (const myDoc of docsToUpdate) {
bulk.find({ _id: myDoc._id }).updateOne({ $set: update })
}
Promise.await(bulk.execute()) // or use regular await if you want...
Note we exit the function early if there's no docs because bulk.execute() throws an exception if there's no operations to process.
If your data have different data in the $set for each entry on array, I think you need a loop in server side.
Mongo has Bulk operations, but I don't know if you can call them using Collection.rawCollection().XXXXX
I've used rawCollection() to access aggregate and it works fine to me. Maybe work with bulk operations.
Im having a real headache with a mongoose query, I am still quite new to it and wondered if anyone could suggest a solution.
I have the following fields in a gallery collection.
status (value of 'public' or 'restricted'),
restriction (value of 'canview' or 'cantview'),
canview (an array of user ids (ObjectIds) of users permitted to view gallery)
I then check fields against the current userID, to see what items are viewable.
how would it be possible to select ALL of the following items
a) all items with 'status' = 'public'
b) all items with 'status' = 'restricted' WHERE 'restricted' = 'canview' AND 'canview' contains the UserID
I wasnt able to do this using $and and $or, so tried with $where instead. Using the following global function
var existsInArray = function(str,arr) {
// nb: remember we have typeof ObjectID, so lets convert to strings first!
var newarr = arr.toString().split(',');
// now compare..
if(newarr.inArray(str)) {
return true;
}
return false;
};
I was hoping to do something like this...
exports.listviewable = function(req, res) {
var reqID = req.user.id;
Gallery
.find()
.$where(function () {
return this.status === "public" || ( this.status === "restricted" && this.restriction === "canview" && existsInArray(reqID, this.canview));
})
.sort('-created')
.exec(function(err, galleries) {
if(err) {
return res.status(400).send({message:getErrorMessage(err)})
} else {
res.json(galleries);
}
});
};
but this wasnt working - it appears I wasnt able to use the global existsInArray function in the $where clause ?
Is it possible to do it like this ( similar unresolved question here how to call external function in mongoose $where and passing this variable), or is there a better way to do it with AND and OR ?
Hey I'd recommend hitting the mongo docs - they have alot of info on this.
a)
Gallery.find({status: 'public'}, function(err, data){
console.log(data)
});
b)
Gallery.find({status: 'restricted', restricted: 'canview', canview: reqID }, function(err, data){
});
or both together with sorting...
Gallery.find({$or:[
{status: 'public'},
{status: 'restricted', restricted: 'canview', canview: reqID }
]}, {
sort:{created:-1}
}, function(err, data){
console.log(data)
});
A $where function can't refer to your local existsInArray function because the $where function is executed on the server where that local function doesn't exist.
But you shouldn't be using $where anyway, as you can do with with a simple $or query that takes advantage of the fact that you can directly query array fields like canview with "contains" behavior:
Gallery
.find({
$or: [
{status: 'public'},
{status: 'restricted', restriction: 'cantview', canview: reqId}
]
})
.sort('-created')
.exec(function(err, galleries) {...});
When I try to save my document, I'm getting a VersionError: No matching document found error, similar to this SO question.
After reading this blog post, it seems that the problem is with the versioning of my document. That I'm messing with an array and so I need to update the version.
However, calling document.save() doesn't work for me. When I log out the document before and after the call to save(), document._v is the same thing.
I also tried doing document._v = document._v++ which also didn't work.
Code
exports.update = function(req, res) {
if (req.body._id) { delete req.body._id; }
User.findById(req.params.id, function(err, user) {
if (err) return handleError(res, err);
if (!user) return res.send(404);
var updated = _.extend(user, req.body); // doesn't increment the version number. causes problems with saving. see http://aaronheckmann.blogspot.com/2012/06/mongoose-v3-part-1-versioning.html
console.log('pre increment: ', updated);
updated.increment();
// updated._v = updated._v++;
console.log('post increment: ', updated);
updated.save(function(err) {
if (err) return handleError(res, err);
return res.json(200, user);
});
});
};
Output
pre increment: { _id: 5550baae1b571aafa52f070c,
provider: 'local',
name: 'Adam',
email: 'azerner3#gmail.com',
hashedPassword: '/vahOqXwCwKQKtcV3KBQeFge/YB0xtqOj+YDyck7gzyALA/IP7u7BfqQhlVHBQT26//XfBTkaOCK2bQXg65OzA==',
salt: 'MvzXW7D4xuyGQBJNeFRoUg==',
__v: 32,
drafts: [],
starredSkims: [],
skimsCreated: [ 5550cfdab8dcacd1a7892aa4 ],
role: 'user' }
post increment: { _id: 5550baae1b571aafa52f070c,
provider: 'local',
name: 'Adam',
email: 'azerner3#gmail.com',
hashedPassword: '/vahOqXwCwKQKtcV3KBQeFge/YB0xtqOj+YDyck7gzyALA/IP7u7BfqQhlVHBQT26//XfBTkaOCK2bQXg65OzA==',
salt: 'MvzXW7D4xuyGQBJNeFRoUg==',
__v: 32,
drafts: [],
starredSkims: [],
skimsCreated: [ 5550cfdab8dcacd1a7892aa4 ],
role: 'user' }
The issue here has to do with using __v and trying to update it manually. .increment does not actually perform an increment immediately, but it does set an internal flag for the model to handle incrementing. I can't find any documentation on .increment, so I assume it is probably for use internally. The problem stems from trying to combine .extend with an object that already has __v (there are two underscores by the way, not that document.__v++ affects the model internally anyway) in addition to using .increment.
When you use _.extend it copies the __v property directly onto the object which seems to cause problems because Mongoose cannot find the old version internally. I didn't dig deep enough to find why this is specifically, but you can get around it by also adding delete req.body.__v.
Rather than finding and saving as two steps, you can also use .findByIdAndUpdate. Note that this does not use __v or increment it internally. As the other answer and linked bug indicate, if you want to increment the version during an update you have to do so manually.
Versioning was implemented to mitigate the doc.save() by design (not Model.update etc). But if you want you can try the following instead:
{$set: {dummy: [2]}, $inc: { __v: 1 }}
However this was a confirmed-bug according to the link
Please validate your mongoose version from the milestone of the above issue.
Thanks :)
I'm adding ObjectId to an array from another array that I receive as the body.
exports.updateBasket = function (req, res) {
Basket.findOne({ _id: req.params.id }, function (err, basket) {
for(var i=0, len=req.body.length; i < len; i++) {
basket.update({$addToSet: { "items": req.body[i] } }, { upsert: true, safe: true });
}
if (err) {
res.send(err);
}
else {
res.json({ message: 'Successfully added' });
}
});
};
I have 2 questions concerning this :
Is there any upside to do the loop in angular and have multiple PUT?
What is the way to update this same array but when removing ObjectId?
One way that I thought of was to loop ObjectId that have to be removed and look if they are in the array of the object, if yes, delete them.
Another way would be to clear the array when PUT is called and update with the new ObjectId list (which would be the ones that were there minus the one user removed).
Both doesn't feel right ...
thanks
You code looks a bit odd. You are fetching asynchronously on the req.params._id but you are queuing up req.body.length potential worth of updates, but you send 'success' before you even get a response back from the updated results.
If you wanted to filter on arrays, look at lodash, if you want to process multiple updates asynchronously and get those response use async modules.