I'm using Meteor framework with Blaze. How can I fetch data from an API and only insert new data in my MongoDB collection and not duplicates?
Fetching data from the API.
if (Meteor.isServer) {
Meteor.methods({
fetchApiData: function () {
this.unblock();
return Meteor.http.call('GET','http://jsonplaceholder.typicode.com/posts');},
Insert data into database:
populateDatabaseApi: function () {
Meteor.call('fetchApiData', function(error, result) {
myCollection.insert({
//upsert: true,
A: result.data.title,
B: result.data.userId,
C: result.data.id });
});
},
When using "myCollection.update" with "upsert: true" it does not insert new entries obviously. What is best practice to go about checking the API for data and inserting ONLY new entries with no duplicates and updating existing entries?
Thank you.
here's how i handle what i call reference data at startup. it's driven off of JSON data. you have to pick a field that serves as your "reference" for each JSON object, so you can see if it's already in the db.
_.each(ItemData.items, function(q) {
check(q, ItemsSchema);
Items.upsert({
item: q.item
}, {
$set: {
item: q.item,
}
}, function(error, result) {
if (error) {
let errMsg = 'Error while writing item data';
console.error(errMsg, error);
throw new Meteor.Error('500', errMsg);
}
});
});
i use an upsert to handle insert vs update.
I'm not familiar with your specific framework, so I can't help with syntax, but you should be able to find all documents with the same properties as the document you're trying to insert (there should be only one). If there is one, then save it using upsert. If there isn't, then the object you're saving is unique, and you should save a new one.
Using only "vanilla" Meteor, assuming your api object have unique ids and that you have the proper data access (ie, if item exist findOne would find it), I'd use :
populateDatabaseApi: function () {
Meteor.call('fetchApiData', function(error, result) {
var item = myCollection.findOne({A : result.data.id})
if(item){
//do nothing, this item already is in the db
}else{
myCollection.insert({
A: result.data.title,
B: result.data.userId,
C: result.data.id });
});
}
},
Related
I'm running a Node.js server, connecting to a MongoDB database with mongoose.
Inside my controller, I have several methods that make operations to the database. One of them is this one:
async findMultiple(req, res) {
const [baseSkillsArray] = Array(req.body);
try {
// if there is not baseSkillsArray, skip
if (!baseSkillsArray) {
return res.status(200).send([]);
}
// find all baseSkills using the ids in the baseSkillsArray
const allBaseSkills = await BaseSkill.find({
_id: { $in: [baseSkillsArray.baseSkillArray] } //
});
console.log('test ' + allBaseSkills);
res.status(200).send(allBaseSkills);
} catch (error) {
console.error(error.message);
res.status(500).send('Server error find BaseSkills');
}
}
However, this returns me nothing. I did some debugging and I found the reason is the find id $in the array. So I tried hard coding a value, like '2', for instance.
// find all baseSkills using the ids in the baseSkillsArray
const allBaseSkills = await BaseSkill.find({ _id: { $in: ['2'] } });
No success. So I went to MongoDB Atlas, where my DB is stored. I tried filtering using the same line of code in my collections.
{ _id: { $in: ['2'] } }
Surprisingly, it returns my document as I wanted!
The issue is that I need to make it work with mongoose. Any ideas? Is this a known bug?
There is nothing wrong with the query, nor a bug regarding $in.
In fact, what's wrong is the actual collection name. I manually created a collection in MongoDB Atlas, called "baseSkills". However, mongoose by default transforms your collection name into lowercase and adds an "s" if your collection's name is not in the plural.
So every time I started my server, I noticed that there was a new collection called "baseskills". I assumed it was a bug and deleted it. Only after making this post that I realized the collection was there again.
So I exported the documents to this collection and my query was working fine.
FYI, there is a way to enforce the collection's name in mongoose. When you declare you model, add a second parameter to the Schema function called "collection". Here is an example:
const BaseSkillSchema = new mongoose.Schema({
_id: {
type: String,
required: true
}, ...
}, { collection: 'baseSkills' })
That's it! Sorry for the mess and thank you for your help!
you want to query over mongo db object ids. So you should create a new ObjectId to do that.
import {Types} from 'mongoose';
{ _id: { $in: [new Types.Object("2")] } }
Or if you have 2 ids one generated and one custom created as id then you can query without creating a new object.
{ id: { $in: ['2'] } }
lets suppose I have a model with a field called draftFields. It is an object(but it can be an array).
I will create a PUT request to add data into draftFields. My question is: can I add data to draftFields and preserve the previous vale?
Lets say that I have added the first data to draftFields. E.g:
draftFields = {
someRandomValue: 'hi'
}
and after that I'm going to make another PUT request and it should look like this:
draftFields = {
someRandomValue: "hi",
anotherRandomValue: "hey"
}
How can I do that? Everytime I updated my draftFields obj it will remove the previous value. I had to save it in my frontend state to be able to save the previous value. Is there any workaround or method to preserve the values from the backend?
This is my code atm:
app.put('/api/save-draft/:id', function (req, res) {
User.findOneAndUpdate(
{ _id: req.params.id },
{ $set: { draftFields: req.body.draftFields } },
{ new: true },
(err, doc) => {
if (err) {
console.log('Something wrong when updating data!');
res.status(400).send('Error');
}
res.status(200).send('All good!');
console.log(doc);
},
);
});
I'm using Javascript(ReactJS) and NodeJS if this is relevant.
I can use the $push method and change from object to array.
https://docs.mongodb.com/manual/reference/operator/update/push/
I have failed searching, sorry.
I have a Documents in a Collection that have a field that is an Array (foo). This is an Array of other subdocuments. I want to set the same field (bar) for each subdocument in each document to the same value. This value comes from a checkbox.
So..my client-side code is something like
'click #checkAll'(e, template) {
const target = e.target;
const checked = $(target).prop('checked');
//Call Server Method to update list of Docs
const docIds = getIds();
Meteor.call('updateAllSubDocs', docIds, checked);
}
I tried using https://docs.mongodb.com/manual/reference/operator/update/positional-all/#positional-update-all
And came up with the following for my Server helper method.
'updateAllSubDocs'(ids, checked) {
Items.update({ _id: { $in: ids } }, { $set: { "foo.$[].bar": bar } },
{ multi: true }, function (err, result) {
if (err) {
throw new Meteor.Error('error updating');
}
});
}
But that throws an error 'foo.$[].bar is not allowed by the Schema'. Any ideas?
I'm using SimpleSchema for both the parent and subdocument
Thanks!
Try passing an option to bypass Simple Schema. It might be lacking support for this (somewhat) newer Mongo feature.
bypassCollection2
Example:
Items.update({ _id: { $in: ids } }, { $set: { "foo.$[].bar": bar } },
{ multi: true, bypassCollection2: true }, function (err, result) {
if (err) {
throw new Meteor.Error('error updating');
}
});
Old answer:
Since you say you need to make a unique update for each document it sounds like bulk updating is the way to go in this case. Here's an example of how to do this in Meteor.
if (docsToUpdate.length < 1) return
const bulk = MyCollection.rawCollection().initializeUnorderedBulkOp()
for (const myDoc of docsToUpdate) {
bulk.find({ _id: myDoc._id }).updateOne({ $set: update })
}
Promise.await(bulk.execute()) // or use regular await if you want...
Note we exit the function early if there's no docs because bulk.execute() throws an exception if there's no operations to process.
If your data have different data in the $set for each entry on array, I think you need a loop in server side.
Mongo has Bulk operations, but I don't know if you can call them using Collection.rawCollection().XXXXX
I've used rawCollection() to access aggregate and it works fine to me. Maybe work with bulk operations.
I have this schema:
var UserSchema = mongoose.Schema({
analytic: {
type: Object,
default: {
today:[],
weekly:[],
monthly:[],
yearly:[],
allTime:[]
}
}
});
let User = mongoose.model("bloger", UserSchema);
module.exports = {User};
and I am trying to save some data into one of the arrays like so:
User.findOne({username:username}, (e, user) => {
if (e) {
res.send('error fetching post')
}
else if (!user) {
res.send('no user found')
}
else if (user) {
user.analytic.today.push(req.body.visitor) // push the data object to the array
user.save((e, doc) => {
if (e) {
res.send(e)
}
if (doc) {
console.log('user saved')
res.send(doc)
}
})
}
})
})
I am getting the doc object on save() and not the e so I though it should have save it but it wasn't.
I have had a similar issue before this is because I am not defining a new Model I am just passing a JSON object.
Instead of saving the object you need to create a new model and save that.
Try creating a new model passing the save into it like below;
var newUser = new User(user);
newUser.save((e, doc) {
if (e) {
res.send(e)
}
if (doc) {
console.log('user saved')
res.send(doc)
}
});
Making sure you require the User Model inside the script.
Performing deep modifications in objects not in your schema makes Mongoose oblivious to those changes, preventing it from knowing what to save (and from making efficient atomic updates). The end result is that, when calling .save, Mongoose thinks there's nothing modified and therefore does nothing.
In your particular scenario, you have two options:
1. Add your analytics sub-arrays to your schema
This is the best option and allows for finer control of everything:
const UserSchema mongoose.Schema({
analytic: {
today: [{}],
weekly: [{}],
monthly: [{}],
yearly: [{}],
allTime: [{}],
}
});
With this, those arrays are now known to Mongoose and everything should work correctly.
Note that you don't need defaults in this case, as Mongoose is smart enough to create the arrays as needed.
2. Manually mark modified object as modified
If for any reason you don't want or can't modify the schema, you can manually mark the analytic object as modifies so Mongoose knows of it:
user.analytic.today.push(req.body.visitor) // push the data object to the array
user.markModified('analytic'); // mark field as modified
user.save(...);
This signals Mongoose that analytic or any of its children have changed and triggers an update when calling .save. Note however that Mongoose views this as a full change in the object, while with option 1 it can use $push instead.
Sorry if I'm not getting the terminology right. Here's what I have currently my MongoDB user docs db.users:
"liked" : [
"EBMKgrD4DjZxkxvfY",
"WJzAEF5EKB5aaHWC7",
"beNdpXhYLnKygD3yd",
"RHP3hngma9bhXJQ2g",
"vN7uZ2d6FSfzYJLmm",
"NaqAsFmMmnhqNbqbG",
"EqWEY3qkeJYQscuZJ",
"6wsrFW5pFdnQfoWMs",
"W4NmGXyha8kpnJ2bD",
"8x5NWZiwGq5NWDRZX",
"Qu8CSXveQxdYbyoTa",
"yLLccTvcnZ3D3phAs",
"Kk36iXMHwxXNmgufj",
"dRzdeFAK28aKg3gEX",
"27etCj4zbrKhFWzGS",
"Hk2YpqgwRM4QCgsLv",
"BJwYWumwkc8XhMMYn",
"5CeN95hYZNK5uzR9o"
],
And I am trying to migrate them to a new key that also captures the time that a user liked the post
"liked_time" : [
{
"postId" : "5CeN95hYZNK5uzR9o",
"likedAt" : ISODate("2015-09-23T08:05:51.957Z")
}
],
I am wondering if it might be possible to simply do this within the MongoDB Shell with a command that iterates over each user doc and then iterates over the liked array and then updates and $push the new postId and time.
Or would it be better to do this in JavaScript. I am using Meteor.
I almost got it working for individual users. But want to know if I could do all users at once.
var user = Meteor.users.findOne({username:"atestuser"});
var userLiked = user.liked;
userLiked.forEach(function(entry) {
Meteor.users.update({ username: "atestuser" },
{ $push: { liked_times: { postId: entry, likedAt: new Date() }}});
console.log(entry);
});
Still a bit of a newbie to MongoDB obviously......
Here is something i made real quick you should run this on the server side just put it into a file e.g. "migrate.js" in root meteor and run the meteor app
if (Meteor.isServer) {
Meteor.startup(function () {
var users = Meteor.users.find().fetch();
users.forEach(function (doc) {
liked.forEach(function (postId) {
Meteor.users.update(doc._id, { $push: { liked_times: { postId: postId, likedAt: new Date() } } });
});
});
console.log('finished migrating');
});
}
p.s I didn't test it
If this is a one time migration i would do something like this in a one time js script.
Get all users
Iterate over each user
Get all likes
Iterate over them, get likedAt
var liked_times = _.collect(likes, function (likeId) {
return {
'postId' : likeId,
'likedAt': // get post liked time from like id.
}
});
Insert the above in the collection of choice.
Note:
The above example makes use of lodash
I would rather just save likedAt as a timestamp.