My database has three collections, challenges, users and entries. Challenges have fields like title, description and challenge id. Entries are the completed challenges and contain fields like user id, challenge id and image. I want to join the data in entry collection to it's corresponding challenge so I could have a document containing challenge title, description, challenge id and image.
I am trying to query the challenges collection based on an array of id's gotten from entries collection and then adding the entry query result as a new field to the document.
I have implemented a for loop, which allows me to query with a different id each time. I would like to add the result of the query to an array, but sometimes it skips the results and only some of the queries are present in the resulting array. For example, when I send the API call for the first time, the server returns 2 JSON-objects in an array, but the next time it only returns one. I think there is something wrong with the synchronization of the for loop and the queries. How can I make it return the correct documents every time? Also, is there a better way to join the two collections together without a for loop?
I've tried countless of different ways to finish the for loop without skipping any queries or returning the finished array too early,but have failed to do so. This current implementation works on the first API call, but on the next one it fails. I am using MongoDB (and MERN stack) and I have a REST API where I send calls from my React front end.
exports.getDoneChallenges = [check("userId").isLength({ min: 24 }),
function(req, res) {
var myPromise = () =>
new Promise((resolve, reject) => {
// Find all challenges the user has completed.
Entry.find({ userId: req.params.id }, { _id: 0 })
.sort({ challengeId: -1 })
.exec()
.then(result => {
// Check if the user hasn't completed any challenges.
if (!result) {
console.log("Zero completed challenges.");
res
.status(401)
.json({ message: "No completed challenges found." });
} else {
// Save the completed challenge's identifiers in an array.
var ids = new Array();
for (var i = 0; i < result.length; i++) {
// Cast identifiers to ObjectID
ids.push(ObjectID(result[i].challengeId));
}
// Array of completed challenges + images relating to each.
var challenge_arr = new Array();
for (let i = 0; i < result.length; i++) {
// Match the corresponding challenge id's from entries to
challenges and add image as a new field.
Challenge.aggregate([
{ $match: { challengeId: ids[i] } },
{ $addFields: { image: result[i] } }
])
.exec()
.then(challenge => {
/* Create a new object, which has the needed fields for
the response.*/
var challenge_obj = new Object();
challenge_obj.title = challenge[0].title;
challenge_obj.challengeId = challenge[0].challengeId;
challenge_obj.description = challenge[0].description;
challenge_obj.date = challenge[0].image.date;
challenge_obj.img = challenge[0].image.img;
// Save the challenges into the challenge array.
challenge_arr.push(challenge_obj);
console.log(i)
/* If the loop is in the last round, return the filled
array.*/
if (i == result.length - 1) {
// Return the filled array.
return challenge_arr;
}
})
.then(challenge_arr => {
// Check that the array isn't undefined.
if (typeof challenge_arr !== "undefined") {
// Resolve the promise.
resolve(challenge_arr);
}
});
}
}
});
});
// Call promise function and send a response after resolving it.
myPromise().then(data => {
res.status(200).json({ data: data });
});
}
];
var EntrySchema = new Schema({
challengeId: ObjectId,
userId: ObjectId,
date: { type: Date, default: Date.now},
img: { data: Buffer, contentType: String}
})
var ChallengeSchema = new Schema({
challengeId: mongoose.SchemaTypes.ObjectId,
title: String,
description: String,
date: {type: Date}
})
I have two entries in the Entries collection, which have the same challenge id's as two of the challenges in challenge collection. I query the challenge collection with the id's of the entries, and I am supposed to get 2 documents which have the corresponding entry field added. Sometimes I get the documents correctly, but most of the time it only returns some of them. For example, from 4 expected documents, it returns {chall 1, null, chall 2, chall 3}.
Promise.all can help you orchestrate multiple promises rather than using a for loop. The way you currently have it, you end up calling resolve when the last loop is done with its work, and not when every loop is done with its work.
It might look something like:
const promises = result.map((image, i) =>
Challenge.aggregate([
{ $match: { challengeId: ids[i] } },
{ $addFields: { image } }
]).exec());
Promise.all(promises)
.then((promise_results) => ...);
Using async/await can generally make code like this much simpler to write and understand:
for (const result of results) {
const challenge = await Challenge.aggregate(...
}
Related
So the document contains an array of objects, each object containing it's own array. So how would I go about updating one of the elements in the array that's inside the object which is inside another array. I've read some things with $. But I don't understand completely how to use it to call a position. I know the position of the element. But I can't just say $[] because the position is defined in a variable and not a string...
I've tried doing a simple
db.collection.findOne({...}, (err, data) => {...});
and then changing the arrays in the objects in the array in there with a simple:
data.arr[x].type[y] = z; data.save().catch(err => {console.log(err)});
But it doesn't save the new values I set for for the element of the array.
Sample structure after proposed solution from #Tom Slabbaert:
Data.findOne({
userID: 'CMA'
}, (err, doc) => {
if(err) {console.log(err)}
if(doc) {
for(var i = 0; i<CMA.stockMarket.length; i++) {
if(CMA.stockMarket[i].name == data.userID) {
for(var z = 0; z<CMA.stockMarket[i].userStock.length; z++) {
if(z == company) {
var updateAmount = CMA.stockMarket[i].userStock[z]+args[1]
var updateKey = `stockMarket.${i}.userStock.${z}`
Data.updateOne({userID: 'CMA'}, {'$set': {[updateKey]: updateAmount}})
}
}
}
}
}
});
-------------------------EDIT-------------------------
So I tried changing some things around in the data base to see if that would fix the problem I was having. I modified the updated code that was provided by #Tom Slabbaert. But nothing seems to work for some reason :/ Here's what I have so far, at this point I hope it's just a syntax error somewhere. Cause this is really frustrating at this point. Note that I'm still using the for loops here to find if the info exists. And if not, push that info into the database. This might only be temporary until I find a better way / if there is a better way.
for(var i = 0; i<CMA.userStocks.length; i++) {
if(CMA.userStocks[i].name == data.userID) {
for(var z = 0; z<CMA.userStocks[i].shares.length; z++) {
//console.log(CMA.userStocks[i].shares[z].companyName)
if(CMA.userStocks[i].shares[z].companyName == args[0]) {
var updateKey = `CMA.userStocks.$[elem1].shares.$[elem2].amount`
Data.updateOne(
{userID: 'CMA'},
{
"$inc": {
[updateKey]: args[1]
}
},
{
arrayFilters: [
{
"elem1.name": data.userID,
"elem2.companyName": args[0]
}
]
}
)
purchaseComplete(); return;
}
}
CMA.userStocks[i].shares.push({companyName: args[0], amount: parseInt(args[1])})
CMA.save().catch(err => {console.log(err)});
purchaseComplete(); return;
}
}
CMA.userStocks.push({name: data.userID, shares: [{companyName: args[0], amount: parseInt(args[1])}]});
CMA.save().catch(err => {console.log(err)});
purchaseComplete(); return;
The data I'm trying to find and change is structured like the following:
And what I'm trying to change in the end is the 'amount' (which is an integer)
_id: (Not relavent in this question)
userID: 'CMA'
stockMarket: [...] (Not relavent in this question)
userStocks: [
Object: (position 0 in userStocks array)
name: 'string' (equal to data.userID in the code)
shares: [
Object: (position 0 in shares array)
companyName: 'string' (this is args[0] in the code)
amount: integer
]
]
You can just prepare the "key" ahead of time. like so:
const updateKey = `arr.${x}.type.${y}`
db.collection.updateOne(
{...},
{
"$set": {
[updateKey]: z
}
})
Mongo Playground
Using Mongo's positional operators ($ and $[]) are usually required when you don't know the position in the array and want to use a condition to update the element.
------ EDIT-----
After given your sample code you just have a minor syntax error:
var updateKey = `stockMarket.${i}.userStock.${z}`
Should just be:
var updateKey = `CMA.stockMarket.${i}.userStock.${z}`
However After seeing your code I recommend you execute the following solution which uses a single update with arrayFilters, it just cleans up the code quite a bit:
const updateKey = `CMA.stockMarket.$[elem1].userStock.${company}`;
db.collection.update(
{userID: 'CMA'},
{
"$inc": {
[updateKey]: args[1]
}
},
{
arrayFilters: [
{
"elem1.name": data.userID
}
]
})
Mongo Playground
Well I found something that worked. Apparently it didn't save the db.collection.updateMany unless I made a .then() function on the end? I have no idea why, but it's the same with an aggregate I made. (It basically does the same as a Data.findOne and save it too, but it isn't limited by the parallel save error)
Solution I found with aggregation:
<collection field> = <new data for collection field>
Data.aggregate([
{
$match: { //This is used to create a filter
['<insert field>']: <insert filter>
}
}, {
$addFields: { //This is used to update existing data, or create a new field containing the data if the field isn't found
['<collection field>']: <new data for collection field>
}
}, {
$merge: { //This is used to merge the new data / document with the rest of the collection. Thus having the same effect as a standard save
into: {
db: '<insert database name>',
coll: '<insert collection name>'
}
}
}
]).then(() => {
//After it's done, do something here. Or do nothing at all it doesn't matter as long as the .then() statement remains. I found that not having this part will break the code and make it not save / work for some reason.
}); return;
Solution I found with db.collection.updateMany
db.collection.updateMany(
{<insert field>: filter}, {$set: {'<insert field>': <new data>}}
).then(() => {
//This .then() statment in my case was needed for the updateMany function to work correctly. It wouldn't save data without it for some reason, it does not need to contain any actual info in this part. As long as it's here.
});
With this new info I could simply access and change the data that I was trying to before using the previous instructions provided by #Tom Slabbaert and my new method of actually making it save the changes made into the document.
I'm using Axios to scrape JSON product data from a website. The script below first requests category data from server and loops through three hierarchies of categories, subcategories, and (I guess) sub-subcategories. In the third loop, the sub-subcategories query parameter is used as a parameter in a function call, which calls an asynchronous function which uses this query parameter to concatenate onto the URL to make the new URL, make a GET request for the product data related to the query parameter, and loop this until the the pageCounter iteration variable (which is used as a pagination query parameter to get N pages for each new URL) is equal to the pagination value within the current new url containing the sub-subcategory query parameter.
for (let i=0; i<obj.length; i++) {
let subcats = obj[i].subcategories.length;
for(let n=0; n<subcats; n++) {
if(obj[i].subcategories[n].facetValueData) {
let subsubcats = obj[i].subcategories[n].facetValueData.length
for(let p=0; p<subsubcats; p++) {
let productData = []
obj[i].subcategories[n].facetValueData[p].productData = productData
const scrapedData = await scrapeData(obj[i].subcategories[n].facetValueData.code)
obj[i].subcategories[n].facetValueData[p].productData.push(scrapedData)
}
} else {
console.log(`No facet values present - ${obj[i].subcategories[n]}`)
}
}
}
This is the async function that the third loop calls
async function scrapeData(queryParam) {
var product = []
var pagination;
try {
do {
var nextPageLink = `currentPage=${pageCounter}&query=::cagCollectionPoint:Departure+from+Singapore:cagCategory:${queryParam}`
var nextUrl = url.concat(nextPageLink)
const response = await axios({
method: "GET",
url: nextUrl,
withCredentials: true,
headers: headers
})
product = response.data["products"]
pagination = response.data["pagination"]["totalPages"]
pageCounter++;
//this logs all of the correct current queries
console.log(response.data.currentQuery.query.value)
} while (pageCounter<=pagination)
return product
} catch (error) {
console.error(error)
}
}
At first glance it appears as if it is working, as it populates the first few sub-subcategory objects with an array of products scraped, however some of these are not populated, i.e. productData = []
The console log function in the scrapeData function returns all of the correct current queries per iteration, however when they are returned it only returns the first few reponses.
I'm guessing the product array needs to wait? But the axios request is already awaited, so i dont see why this is happening.
If I'm understanding your intent on this piece of code correctly:
var product = [];
do {
// ...
const response = await axios(/* ... */);
product = response.data["products"]
// ...
} while (pageCounter<=1)
return product
it seems like there are multiple pages, and you want to get all of them into the product array?
But in reality, each page you are replacing the product array. I would assume that on the empty ones your last fetch gets no results, and you are just losing all of the rest of them.
What you probably want to do is just change the product = response.data["products"] to:
product.push(...response.data["products"])
Schema:
This is how my schema looks
Current Implementation:
for (let i=0; i<data.length; i++) {
try
{
var ifPresent = db.collection("Safes-Hardware").doc(data[i]['Mac address Check']);
ifPresent.get()
.then(async (doc)=>{
if (!doc.exists)
{
// Do stuff
}
else
{
//Do stuff
}
return { message: "Success is within the palm of our hands." }
}
}
Problem:
Even though this code does the job, for each data in the array I'm doing a lookup, and this results in a socket hang-up.(sometimes)
So I'm thinking I'll get all the documents in the collection in one go, store it locally and look up if a documents exists locally instead of querying the database every time.
Question:
How do I implement this?
You can just use collection("Safes-Hardware").get().then() and you can save the data locally.
let collection = []
db.collection("Safes-Hardware").get().then(function(querySnapshot) {
collection = querySnapshot.map((doc) => ({
id: doc.id,
...doc.data()
}))
});
then you can use collection to search for what you want, maybe like this
data.forEach( doc => {
let x = collection.find(v => v.id === doc['Mac address Check'])
if(x){
//it exists
}else{
// not exists
}
})
But take care you are compromising bandwidth or number of requests with o(n^2) operation in the client side
I see that someone has given me a minus 1. I am a 55 year old mother who has no experience. I have many skills but this is not one of them. I am absolutely desperate and have bust myself to get this far. If you cannot help, I accept that, but please do not be negative towards me. I am now crying. Some encouragement would be much appreciated.
I have a page which displays items from a database on a repeater. The code searches the items using several drop down filters, which are populated from the database. Intermittently, seemingly randomly (no pattern is emerging despite extensive testing) the code is failing to populate random drop down filters (one or more of the drop down filters show the default settings rather than those self populated from the database). I discovered this by either repeatedly visiting the page or by repeatedly refreshing the page. Often the code works, then every 3 or 4 times, one or more of the drop down filters shows its default settings rather than those self populated from the database (then the next time it goes wrong, it might be the same or a different one or set of filters which do not work)
This is the code. On this page, there are 3 drop down filters but I have several pages like this, each displaying and searching a different database, with up to 10 drop down filters on each page, and they all have this intermittent problem...
import wixData from "wix-data";
$w.onReady(function () {
$w('#iTitle')
$w('#iCounty')
$w('#iGeog')
$w('#dataset1')
$w('#text102')
});
let lastFilterTitle;
let lastFilterCounty;
let lastFilterGeog;
export function iTitle_change(event, $w) {
filter($w('#iTitle').value, lastFilterCounty, lastFilterGeog);
}
export function iCounty_change(event, $w) {
filter(lastFilterTitle, $w('#iCounty').value, lastFilterGeog);
}
export function iGeog_change(event, $w) {
filter(lastFilterTitle, lastFilterCounty, $w('#iGeog').value);
}
function filter(title, county, geog) {
if (lastFilterTitle !== title || lastFilterCounty !== county || lastFilterGeog !== geog) {
let newFilter = wixData.filter();
if (title)
newFilter = newFilter.eq('title', title);
if (county)
newFilter = newFilter.eq('county', county);
if (geog)
newFilter = newFilter.eq('geog', geog);
$w('#dataset1').setFilter(newFilter)
.then(() => {
if ($w('#dataset1').getTotalCount() ===0) {
$w('#text102').show();
}
else {
$w('#text102').hide();
}
})
.catch((err) => {
console.log(err);
});
lastFilterTitle = title;
lastFilterCounty = county;
lastFilterGeog = geog;
}
}
// Run a query that returns all the items in the collection
wixData.query("Psychologists")
// Get the max possible results from the query
.limit(1000)
.ascending("title")
.distinct("title")
.then(results => {
let distinctList = buildOptions(results.items);
// unshift() is like push(), but it prepends an item at the beginning of an array
distinctList.unshift({ "value": '', "label": 'All Psychologists'});
//Call the function that builds the options list from the unique titles
$w("#iTitle").options = distinctList
});
function buildOptions(items) {
return items.map(curr => {
//Use the map method to build the options list in the format {label:uniqueTitle, valueuniqueTitle}
return { label: curr, value: curr };
})
}
// Run a query that returns all the items in the collection
wixData.query("Psychologists")
// Get the max possible results from the query
.limit(1000)
.ascending("county")
.distinct("county")
.then(results => {
let distinctList = buildOptions(results.items);
// unshift() is like push(), but it prepends an item at the beginning of an array
distinctList.unshift({ "value": '', "label": 'All Counties'});
//Call the function that builds the options list from the unique titles
$w("#iCounty").options = distinctList
});
function buildOptions1(items) {
return items.map(curr => {
//Use the map method to build the options list in the format {label:uniqueTitle1, valueuniqueTitle1}
return { label: curr, value: curr };
})
}
// Run a query that returns all the items in the collection
wixData.query("Psychologists")
// Get the max possible results from the query
.limit(1000)
.ascending("geog")
.distinct("geog")
.then(results => {
let distinctList = buildOptions(results.items);
// unshift() is like push(), but it prepends an item at the beginning of an array
distinctList.unshift({ "value": '', "label": 'All Regions'});
//Call the function that builds the options list from the unique titles
$w("#iGeog").options = distinctList
});
function buildOptions2(items) {
return items.map(curr => {
//Use the map method to build the options list in the format {label:uniqueTitle2, valueuniqueTitle2}
return { label: curr, value: curr };
})
}
export function button45_click(event, $w) {
//Add your code for this event here:
filter($w('#iTitle').value='', $w('#iCounty').value='', $w('#iGeog').value='');
}
My experience and knowledge is very limited, so the answer may well be very simple. Any help would be much appreciated as I will have to abandon my project if I can't find a solution.Thank you
I am trying to add hashtags in the post's hashtag[] array as a object with a num:1 variable to the users hashtagseen[] array if it is not already in it else add 1 the num if the hashtag is already in the hashtagseen[] array. How do I fix my code? Here is the code, thanks in advanced.
edit: I think I am not finding post.hashtag with this.hashtag and that is why it will not go to else. Just a guess.
The user object
Accounts.createUser({
username: username,
password: password,
email: email,
profile: {
hashtagsl:[],
}
});
collections/post.js
var post = _.extend(_.pick(postAttributes, 'title', 'posttext','hashtags'), {
userId: user._id,
username: user.username,
submitted: new Date().getTime(),
commentsCount: 0,
upvoters: [], votes: 0,
});
calling it
Meteor.call('addposthashtags',this.hashtags,Meteor.user().profile.hashtagsl);
lib/usershash
Meteor.methods({
addposthashtags: function (hashtags,hashtagsl) {
//supposed to make hashtagseen a array with the names from the hashtagsl object in it
var hashtagseen = _.pluck(hashtagsl, 'name');
//supposed to run once per each hashtag in the posts array.
for (var a = 0; a < hashtags.length; a++) {
//supposed set hashtagnumber to the number indexOf spits out.
var hashnumber=hashtagseen.indexOf(hashtags[a]);
//supposed to check if the current hashtag[a] === a idem in the hashtagseen.
if(hashnumber===-1){
var newhashtag = this.hashtags[a];
//supposed to make the object with a name = to the current hashtags
Meteor.users.update({"_id": this.userId},{"$push":{"profile.hashtagsl": {name: newhashtag, num: 1}}})
} else {
var hashi = hashtagseen[hashnumber];
//supposed to ad one to the num variable within the current object in hashtagsl
Meteor.users.update({"_id": this.userId, "profile.hashtagsl.name":hashi},{"$inc":{"profile.hashtagsl.num":1}});
}
}
}
});
Your addposthashtags function is full of issues. You also haven't provided a "schema" for hashtag objects.
addposthashtags: function () {
for (a = 0; a < this.hashtag.length; a++) {
// Issue1: You're querying out the user for every iteration of the loop!?
for (i = 0; i < Meteor.user().profile.hashtagseen.length; i++) {
// Issue2: You're comparing two _objects_ with ===
// Issue3: Even if you use EJSON.equals - the `num` property wont match
// Issue4: You're querying out the user again?
if (this.hashtag[a] === Meteor.user().profile.hashtagseen[i]) {
// Issue5 no `var` statement for hashtagseeni?
// Issue6 You're querying out the user again??
hashtagseeni = Meteor.user().profile.hashtagseen[i];
//Issue7 undefined hashtagsli?
//Issue8 Calling multiple methods for the one action (eg in a loop) is a waste of resources.
Meteor.call('addseen', hashtagsli);
} else {
//Issue9 no `var` statement for newhashtag?
newhashtag = this.hashtag[a];
newhashtag.num = 1;
//Issue8b Calling multiple methods for the one action (eg in a loop) is a waste of resources.
Meteor.call('updateUser', newhashtag, function (err, result) {
if (err)
console.log(err);
});
}
}
}
}
Also, the method has similiar issues:
addseen: function (hashtagseeni) {
// Issue10: var `profile` is undefined
// Issue11: should use `this.userId`
// Issue12: hashtagseeni wouldn't match profile.hashtagseen due to "num" field.
Meteor.users.update({"_id": Meteor.userId, "profile.hashtagseen": profile.hashtagseeni}, {"$inc":{"profile.hashtagseen.$.num":1}});
}
New issues with your new set of code:
Meteor.methods({
addposthashtags: function (hashtags,hashtagsl) {
//Issue1 `hashtag` is undefined, guessing you mean `hashtags`
//Issue2 no `var` for a
for (a = 0; a < hashtag.length; a++) {
//Issue3 no `var` for i
//Issue4 Why are you looping through both?
// don't you just want to check if hashtag[a] is in hashtagsl?
for (i = 0; i < hashtagsl.length; i++) {
if (hashtags[a] === hashtagsl[i].name) {
var hashi = hashtagsl[i].name;
//supposed to ad one to the num variable within the current object in hashtagsl.
// Issue5: This query wont do what you think. Test until you've got it right.
Meteor.users.update({"_id": Meteor.userId, 'profile.hashtagsl':hashi}, {"$inc":{"num":1}});
} else {
// Issue6 `this.hashtag` isn't defined. guessing you mean `hashtags[a]`
var newhashtag = this.hashtag[a];
// Issue7 superfluous statement
var newhashtagnum = num = 1;
// Issue8 Obvious syntax errors
// Perhaps try Meteor.users.update({"_id": this.userId},{"$push":{"profile.hashtagsl": {name: newhashtag, num: 1}}})
Meteor.users.update({"_id": Meteor.userId, 'profile'},{"$addToSet":{"hashtagsl"[newhashtag]=newhashtagnum}})
};
};
};
};
});
I'd suggest you trying the following
1) Assuming that after newhashtag=hashtag[a] you get a JSON object in newhashtag variable, try replacing newhashtag:{num:1}; with newhashtag.num = 1 - this will add the num variable to the object and set the value.
1.a) For debugging purposes try adding some console.log(JSON.stringify(newhashtag)); after each of the two lines where you're setting and changing the newhashtag variable - this way you'll know exactly what you're trying to add to the mongoDB document.
2) The update to increment the views also doesn't seem to me that will work. Couple of things to note here - $set:{'profile.hashtagseen[i]':num++} - MongoDB won't be able to identify the 'i' in 'profile.hashtagseen[i]' and 'num++' is not how increments are done in Mongo.
I'd suggest you look into the $inc and to the positional update documentation of MongoDB.
Your final increment update statement will look something like
Meteor.users.update({"_id": Meteor.userId, "profile.hashtagseen": profile.hashtagseen[i]}, {"$inc":{"profile.hashtagseen.$.num":1}});
I see that executing addposthashtags is in the client, and you must to pay attention because this function will execute in minimongo and doesn't work all operations. First you try execute this operation under mongo if it's work you must to create one function inside the folder server.
Add text of the documentation of Minimongo
In this release, Minimongo has some limitations:
$pull in modifiers can only accept certain kinds of selectors.
findAndModify, aggregate functions, and map/reduce aren't supported.
All of these will be addressed in a future release. For full Minimongo
release notes, see packages/minimongo/NOTES in the repository.
Minimongo doesn't currently have indexes. It's rare for this to be an
issue, since it's unusual for a client to have enough data that an
index is worthwhile.
You try create one method on the server, with the same operation.
Server:
Meteor.methods({
updateUser: function (newhashtag) {
Meteor.users.update(this.userId,
{
$addToSet: {'profile.$.hashtagseen': newhashtag}
});
}
});
Client:
Meteor.call('updateUser',newhashtag,function(err,result){
if (err)
console.log(err);// there you can print the erro if there are
});
Minimongo doesn't support alls operation, for test you can to execute in the console for testing the method if supported. After that you can to execute the operation under mongo directly, that clears your doubts.