I am retrieving data about books from the Amazon API and storing it as a JSON object. I then want to insert this data into a Meteor collection. However, some fields in the JSON data begin with the "$" character, which leads to this error:
MongoError: The dollar ($) prefixed field '$' in 'result.results.0.ItemAttributes.0.ManufacturerMaximumAge.0.$' is not valid for storage.
I would like help to insert data into the collection given that the data contains "$" signs. If that is not possible,is there a way to remove all occurrences of "$" in the JSON data and THEN store it into the Meteor collection?
Here is my code, given a Book title and author, that searches the Amazon API and returns information about the first 10 results: (SearchResult is the reference to the collection)
Meteor.methods({
itemSearch(appUUID,title, author) {
var result = {};
result.search = {
appUUID: appUUID,
title: title,
author: author
}
client.itemSearch({
title: title,
author: author,
searchIndex: 'Books',
responseGroup: 'ItemAttributes, Images'
}).then(function(results){
result.results = results;
var upsertResult = SearchResult.upsert( {
appUUID: appUUID,
title: title,
author: author,
},
{
$set: {result: result}
}
);
}).catch(function(err){
SearchResult.upsert({
appUUID: appUUID,
title: title,
author: author,
},
{
$set: {result: result }
}
);
});
}
});
This is the JSON data, and the dollar sign before "Units" is what is causing the issue.
"Width": [
{
"_": "810",
"$": {
"Units": "hundredths-inches"
}
}
]
Simple answer is, no, you cannot use the $ in fields in MongoDB (from the docs):
The field names cannot start with the dollar sign ($) character.
There are many possible solutions to change your data into a format that MongoDB can accept, but to give you the simplest solution I can think of, you can just map the keys of the object. For example, with lodash's mapKeys function:
data = _.mapKeys(data, (value, key) => {
if(key === '$') {
return 'some_custom_key'
} else {
return key
}
})
This would change all your $ into some_custom_key. As long as you have hooks (such as Mongoose's pre save middleware and post read middleware, docs) that can convert this for you under-the-hood, it should be a somewhat workable solution.
I don't know if it's possible, but a better solution would be to get Amazon API to give you the data without $, but I can't speculate about that since I'm unfamiliar with that API.
Related
I'm trying to use Mongoose (MongoDB JS library) to create a basic database, but I can't figure out how to delete the documents / items, I'm not sure what the technical term for them is.
Everything seems to work fine, when I use Item.findById(result[i].id), it returns a valid id of the item, but when I use Item.findByIdAndDelete(result[i].id), the function doesn't seem to start at all.
This is a snippet the code that I have: (Sorry in advance for bad indentation)
const testSchema = new schema({
item: {
type: String,
required: true
},
detail: {
type: String,
required: true
},
quantity: {
type: String,
required: true
}
})
const Item = mongoose.model("testitems", testSchema)
Item.find()
.then((result) => {
for (i in result) {
Item.findByIdAndDelete(result[i].id), function(err, result) {
if (err) {
console.log(err)
}
else {
console.log("Deleted " + result)
}
}
}
mongoose.connection.close()
})
.catch((err) => {
console.log(err)
})
I'm not sure what I'm doing wrong, and I haven't been able to find anything on the internet.
Any help is appreciated, thanks.
_id is a special field on MongoDB documents that by default is the type ObjectId. Mongoose creates this field for you automatically. So a sample document in your testitems collection might look like:
{
_id: ObjectId("..."),
item: "xxx",
detail: "yyy",
quantity: "zzz"
}
However, you retrieve this value with id. The reason you get a value back even though the field is called _id is because Mongoose creates a virtual getter for id:
Mongoose assigns each of your schemas an id virtual getter by default which returns the document's _id field cast to a string, or in the case of ObjectIds, its hexString. If you don't want an id getter added to your schema, you may disable it by passing this option at schema construction time.
The key takeaway is that when you get this value with id it is a string, not an ObjectId. Because the types don't match, MongoDB will not delete anything.
To make sure the values and types match, you should use result[i]._id.
What I am trying to do
I am creating a social media app with react native and firebase. I am trying to call a function, and have that function return a list of posts from off of my server.
Problem
Using the return method on a firebase query gives me a hard to use object array:
Array [
Object {
"-L2mDBZ6gqY6ANJD6rg1": Object {
//...
},
},
]
I don't like how there is an object inside of an object, and the whole thing is very hard to work with. I created a list inside my app and named it items, and when pushing all of the values to that, I got a much easier to work with object:
Array [
Object {
//...
"key": "-L2mDBZ6gqY6ANJD6rg1",
},
]
This object is also a lot nicer to use because the key is not the name of the object, but inside of it.
I would just return the array I made, but that returns as undefined.
My question
In a function, how can I return an array I created using a firebase query? (to get the objects of an array)
My Code
runQ(group){
var items = [];
//I am returning the entire firebase query...
return firebase.database().ref('posts/'+group).orderByKey().once ('value', (snap) => {
snap.forEach ( (child) => {
items.push({
//post contents
});
});
console.log(items)
//... but all I want to return is the items array. This returns undefined though.
})
}
Please let me know if I'm getting your question correctly. So, the posts table in database looks like this right now:
And you want to return these posts in this manner:
[
{
"key": "-L1ELDwqJqm17iBI4UZu",
"message": "post 1"
},
{
"key": "-L1ELOuuf9hOdydnI3HU",
"message": "post 2"
},
{
"key": "-L1ELqFi7X9lm6ssOd5d",
"message": "post 3"
},
{
"key": "-L1EMH-Co64-RAQ1-AvU",
"message": "post 4"
}
...
]
Is this correct? If so, here's what you're suppose to do:
var items = [];
firebase.database().ref('posts').orderByKey().once('value', (snapshot) => {
snapshot.forEach((child) => {
// 'key' might not be a part of the post, if you do want to
// include the key as well, then use this code instead
//
// const post = child.val();
// const key = child.key;
// items.push({ ...post, key });
//
// Otherwise, the following line is enough
items.push(child.val());
});
// Then, do something with the 'items' array here
})
.catch(() => { });
Off the topics here: I see that you're using firebase.database().... to fetch posts from the database, are you using cloud functions or you're fetching those posts in your App, using users' devices to do so? If it's the latter, you probably would rather use cloud functions and pagination to fetch posts, mainly because of 2 reasons:
There might be too many posts to fetch at one time
This causes security issues, because you're allowing every device to connect to your database (you'd have to come up with real good security rules to keep your database safe)
I'm using Meteor framework with Blaze. How can I fetch data from an API and only insert new data in my MongoDB collection and not duplicates?
Fetching data from the API.
if (Meteor.isServer) {
Meteor.methods({
fetchApiData: function () {
this.unblock();
return Meteor.http.call('GET','http://jsonplaceholder.typicode.com/posts');},
Insert data into database:
populateDatabaseApi: function () {
Meteor.call('fetchApiData', function(error, result) {
myCollection.insert({
//upsert: true,
A: result.data.title,
B: result.data.userId,
C: result.data.id });
});
},
When using "myCollection.update" with "upsert: true" it does not insert new entries obviously. What is best practice to go about checking the API for data and inserting ONLY new entries with no duplicates and updating existing entries?
Thank you.
here's how i handle what i call reference data at startup. it's driven off of JSON data. you have to pick a field that serves as your "reference" for each JSON object, so you can see if it's already in the db.
_.each(ItemData.items, function(q) {
check(q, ItemsSchema);
Items.upsert({
item: q.item
}, {
$set: {
item: q.item,
}
}, function(error, result) {
if (error) {
let errMsg = 'Error while writing item data';
console.error(errMsg, error);
throw new Meteor.Error('500', errMsg);
}
});
});
i use an upsert to handle insert vs update.
I'm not familiar with your specific framework, so I can't help with syntax, but you should be able to find all documents with the same properties as the document you're trying to insert (there should be only one). If there is one, then save it using upsert. If there isn't, then the object you're saving is unique, and you should save a new one.
Using only "vanilla" Meteor, assuming your api object have unique ids and that you have the proper data access (ie, if item exist findOne would find it), I'd use :
populateDatabaseApi: function () {
Meteor.call('fetchApiData', function(error, result) {
var item = myCollection.findOne({A : result.data.id})
if(item){
//do nothing, this item already is in the db
}else{
myCollection.insert({
A: result.data.title,
B: result.data.userId,
C: result.data.id });
});
}
},
I aggregated some data and published it, but I'm not sure how/where to access the subscribed data. Would I be able to access WeeklyOrders client collection (which is defined as client-only collection i.e WeeklyOrders = new Mongo.Collection(null);)?
Also, I see "self = this;" being used in several examples online and I just used it here, but not sure why. Appreciate anyone explaining that as well.
Here is publish method:
Meteor.publish('customerOrdersByWeek', function(customerId) {
check(customerId, String);
var self = this;
var pipeline = [
{ $match: {customer_id: customerId} },
{ $group: {
_id : { week: { $week: "$_created_at" }, year: { $year: "$_created_at" } },
weekly_order_value: { $sum: "$order_value" }
}
},
{ $project: { week: "$_id.week", year: "$_id:year" } },
{ $limit: 2 }
];
var result = Orders.aggregate(pipeline);
result.forEach(function(wo) {
self.added('WeeklyOrders', objectToHash(wo._id), {year: wo.year, week: wo.week, order_value: wo.weekly_order_value});
});
self.ready();
});
Here is the route:
Router.route('/customers/:_id', {
name: 'customerOrdersByWeek',
waitOn: function() {
return [
Meteor.subscribe('customerOrdersByWeek', this.params._id)
];
},
data: function() { return Customers.findOne(this.params._id); }
});
Here is my template helper:
Template.customerOrdersByWeek.helpers({
ordersByWeek: function() {
return WeeklyOrders.find({});
}
});
You want var self = this (note the var!) so that call to self.added works. See this question for more details. Alternatively you can use the new es6 arrow functions (again see the linked question).
There may be more than one issue where, but in your call to added you are giving a random id. This presents two problems:
If you subscribe N times, you will get N of the same document sent to the client (each with a different id). See this question for more details.
You can't match the document by id on the client.
On the client, you are doing a Customers.findOne(this.params._id) where this.params._id is, I assume, a customer id... but your WeeklyOrders have random ids. Give this a try:
self.added('WeeklyOrders', customerId, {...});
updated answer
You'll need to add a client-only collection as a sort-of mailbox for your publisher to send WeeklyOrders to:
client/collections/weekly-orders.js
WeeklyOrders = new Meteor.Collection('WeeklyOrders');
Also, because you could have multiple docs for the same user, you'll probably need to:
Forget what I said earlier and just use a random id, but never subscribe more that once. This is an easy solution but somewhat brittle.
Use a compound index (combine the customer id + week, or whatever is necessary to make them unique).
Using (2) and adding a customerId field so you can find the docs on the client, results in something like this:
result.forEach(function (wo) {
var id = customerId + wo.year + wo.week;
self.added('WeeklyOrders', id, {
customerId: customerId,
year: wo.year,
week: wo.week,
order_value: wo.weekly_order_value,
});
});
Now on the client you can find all of the WeeklyOrders by customerId via WeeklyOrders.find({customerId: someCustomerId}).
Also note, that instead of using pub/sub you could also just do all of this in a method call. Both are no-reactive. The pub/sub gets you collection semantics (the ability to call find etc.), but it adds the additional complexity of having to deal with ids.
I'm trying to do like search into mongodb with Javascript, but i havent yet figured it out how to do it. Any help is appreciated.
I have a search parameter called "search" at request body (req.body.search). I would like find all programs which name contains that search criteria. Ie. "harry" would return "Harry Potter", "Potter Harry", "How merry Harry Really is?" and other movies which contains harry in its name.
exports.programs = function(db, req) {
return function(req, res) {
var collection = db.get('programs');
collection.find({name: /req.body.search/ },{},function(e,docs){
res.json(docs);
});
};
};
I'm not familiar with the monk library but if the parameter in that find is a MongoDB Query Document this is the syntax you are looking for:
var query = {
name: {
$regex: req.body.search,
$options: 'i' //i: ignore case, m: multiline, etc
}
};
collection.find(query, {}, function(e,docs){});
Chekcout MongoDB $regex Documentaiton.