We Keep Coding

Getting only needed elements form array in MongoDb Schema

I want to get only those element from notes Array that match the given tag from tags array in notes.
For example : If I want to get only those notes from notes Array having lrn tag in tags array but I am getting either the whole object or only the first matching note.
My Schema :
const mongoose = require('mongoose');
const noteSchema = new mongoose.Schema({
user : {
type: String,
required: true,
unique : true
},
notes : {
type : [{
title : {
type: String,
required: true
},
tags : {
type : [String],
required: true
},
note: {
type: String,
required: true
}
}],
required: true
}
});
const Notebook = mongoose.model('NOTEBOOK', noteSchema);
module.exports = Notebook;
My sample data :
{ _id: ObjectId("621c3a41fa2fe3c07f43edc9"),
user: '620c81434d8a65c1aa36e0d4',
notes:
[ { title: 'Eraning',
tags: [ 'ern', 'lrn', 'dik', 'pik', 'sik' ],
note: 'Prior to his return, Craig worked at NeXT, followed by Apple, and then spent a decade at Ariba, an internet e-commerce pioneer where he held several roles including chief technology officer.\nPrior to his return, Craig worked at NeXT, followed by Apple, and then spent a decade at Ariba, an internet e-commerce pioneer where he held several roles including chief technology officer.\n',
_id: ObjectId("62205a32bab7aed6315195e9") },
{ title: 'Lerningngs',
tags: [ 'lrn', 'eat', 'drink' ],
note: 'Prior to his return, Craig worked at NeXT, followed by Apple, and then spent a decade at Ariba, an internet e-commerce pioneer where he held several roles including chief technology officer.\n',
_id: ObjectId("6220583cbab7aed63151958b") },
{ title: 'Learning',
tags: [ 'learn', 'eran', 'buy', 'sell' ],
note: 'Prior to his return, Craig worked at NeXT, followed by Apple, and then spent a decade at Ariba, an internet e-commerce pioneer where he held several roles including chief technology officer.Solving DSA from Maths.in and SolveMaths.org using great techniques by INDIA and RUSSIA.\n',
_id: ObjectId("6220532dbab7aed631519531") },
{ title: 'Biology',
tags: [ 'tissue', 'cell', 'myto', 'energy', 'glycogen', 'hydrogen' ],
note: 'Prior to his return, Craig worked at NeXT, followed by Apple, and then spent a decade at Ariba, an internet e-commerce pioneer where he held several roles including chief technology officer.Solving DSA from Maths.in and SolveMaths.org using great techniques by INDIA and RUSSIA.\n\n',
_id: ObjectId("621f88d4c189c6b5501c3d5f") },
{ title: 'Chemistry',
tags: [ 'this', 'is', 'great', 'thing', 'to', 'do', 'by', 'time' ],
note: ' Prior to his return, Craig worked at NeXT, followed by Apple, \n and then spent a decade at Ariba, an internet e-commerce pioneer \n where he held several roles including chief technology officer.\n',
_id: ObjectId("621f85e9c189c6b5501c3d49") },
{ title: 'Physics',
tags:
[ 'newtonlawsofmotion',
'ktg',
'thermo',
'fluid',
'mechanice',
'bernaulii' ],
note: 'P\' + hrg + 1/2rv^2 = constant',
_id: ObjectId("621f8520c189c6b5501c3d3f") },
{ title: 'Maths',
tags: [ 'trignometry', 'llp', 'continuity', 'tags', 'circle' ],
note: 'Solving DSA from Maths.in and SolveMaths.org using great techniques by INDIA and RUSSIA.',
_id: ObjectId("621f84d7c189c6b5501c3d37") },
{ title: 'DSA from Net',
tags: [ 'dp', 'stack', 'queue', 'heaps', 'sorting' ],
note: 'Solving DSA from Leetcode.in and GeeksforGeeks.org',
_id: ObjectId("621c3a41fa2fe3c07f43edca") } ],
__v: 7 }

You can use $filter to get documents of given tag as shown below:
Notebook.findOne({
user: "620c81434d8a65c1aa36e0d4" // user ID
},
{
notes: {
$filter: {
input: "$notes",
as: "note",
cond: {
$in: [
"ern", // Tag to search
"$$note.tags"
]
}
}
}
})
Mongo Playground

Retreiving specific data from JSON repsonse only in certain array indexes

Hey sorry if this has been asked before, I haven't found an answer that has helped me.
I am working on building a personal application using the free Ticketmaster application but I'm having trouble with the JSON response.
An example of the response is:
data: {
classifications: [
type: {
id: 1,
name: "Dont want this"
}
],
[
type: {
id: 2,
name: "Dont want this"
}
],
[
category: {
id: 3,
name: "Food & Drink"
}
],
[
category: {
id: 4,
name: "Music"
]
}
I am trying to retrieve category names such as "Family", "Food & Drink", "Music" etc but only at some indexes does the array have category objects, the others have type.
Can someone tell me how to go through this array and only get the name when it is at category.name?
Any help is appreciated :)

How should I do an aggregate?

I have an array with products(objects) inside. It looks like:
[
{_id: 123, name:'first', category:'Vape', property:[
{key:'brand', values:'Eleaf'},
{key:'mechmode', values:'some'},
{key:'color', values:['red', 'blue']},
{key:'batrtype', values:'some2'},
]},
{_id: 1234, name:'second', category:'Vape1', property:[
{key:'brand', values:'Eleaf1'},
{key:'mechmode', values:'some'},
{key:'color', values:['black']},
{key:'batrtype', values:'some2'}
]},
{_id: 12345, name:'third', category:'Vape3', property:[
{key:'brand', values:'Eleaf3'},
{key:'mechmode', values:'some'},
{key:'color', values:['green', 'yellow']},
{key:'batrtype', values:'some2'}
]},
{_id: 123456, name:'fourth', category:'Vape', property:[
{key:'brand', values:'Eleaf'},
{key:'mechmode', values:'some'},
{key:'color', values:['red', 'green']},
{key:'batrtype', values:'some2'}
]}
]
Request from client
{category:'Vape', body:{brand:'Eleaf', color:'red'}}
How should I do an aggregate in mongo to get products which have brand = 'Eleaf'
and color = red in according to request?
Can somebody help please?

bgraham's answer is exactly right if you are using find. To do the same thing in an aggregation pipeline, use $match, such as:
db.aggregate([{$match:{
category: "Vape",
$and: [
{ property: { $elemMatch: { key: "brand", value: "Eleaf" },
{ property: { $elemMatch: { key: "color", value: "red" }
]
}}])
You could then add to the array any additional pipeline stages you need.

I think this should do the trick:
db.products.find({
category: "Vape",
$and: [
{ property: { $elemMatch: { key: "brand", value: "Eleaf1" },
{ property: { $elemMatch: { key: "color", value: "red" }
]})
I'm not sure the structure of your data is best practice. It might be easier if they were fields called "brand" with a value of "color" instead of key/value pairs.

Mongoose distinct condition not working

I'm using mongoose to query a MongoDB using following scheme:
var File = mongoose.model('File',{
size: Number,
type: String,
filename: String,
path: String,
taxonomies:[{
tags:[{
name:String,
'tag':String
}],
name: String
}]
});
Now I want to do a distinct query as follows:
File.distinct('taxonomies.tags.tag', {'taxonomies.name':'AM'},function(error, tags){
if(err)
res.send(err);
console.log("All distinct tags", tags);
})
to get all unique tags from all files where the name of the taxonomy is 'AM'. The problem is that the query above returns all tags, including those with a different taxonomies.name. Without the condition, it just works as it should.
Do I have a syntactic error in here, or am I misunderstanding how distinct works?
Update (More examples)
Each document has a taxonomy with name SM and one with name AM, something like
taxonomies: [{
tags: [
{
name:"Type",
'tag':kind
}, {
name:"Language",
'tag':lang
},
{
name:"Code",
"tag":code
}],
name:'AM'
}, {
name:'SM',
tags:[{
name:"Sales",
'tag':'has to be sold'
},{
name:"Personal filter",
'tag':'temp'
}]
}]
When I execute the query mentioned above, I get as a result:
All distinct tags [ '4007', 'fr', 'has to be sold', 'temp', 'wol', '16104', 'en' ]
while 'temp' and 'has to be sold are from SM, not AM.
I want as a result only the ones wehre taxonomies.name = 'AM', across all documents, without duplicates

The query selection is selecting documents, not elements from the array:
{'taxonomies.name':'AM'}
That criteria will select the entire document.
As a result, all items in both arrays are considered as valid for the distinct operation. If you only want subsets to be considered, you'll either need to use the aggregation framework to filter, or modify your documents so that the two taxonomies are not stored in the same document.
If you used the aggregation framework instead of the distinct command, there are a lot of options you could use to gather the distinct values:
db.test.aggregate({$unwind: "$taxonomies" },
{ $group :
{ _id: "$taxonomies.name",
tags: { $addToSet: "$taxonomies.tags" } }} )
That should group on both SM and AM (and any others):
[
{
"_id" : "SM",
"tags" : [
[
"d1",
"e1"
],
[
"d",
"e"
]
]
},
{
"_id" : "AM",
"tags" : [
[
"a1",
"b1",
"c1"
],
[
"a",
"b",
"c"
]
]
}
]
You could use $match to filter on a particular name as well.

Prevent Javascript function running out of memory because too many objects

I'm building a web scraper in nodeJS that uses request and cheerio to parse the DOM. While I am using node, I believe this is more of a general javascript question.
tl;dr - creating ~60,000 - 100,000 objects, uses up all my computer's RAM, get an out of memory error in node.
Here's how the scraper works. It's loops within loops, I've never designed anything this complex before so there might be way better ways to do this.
Loop 1: Creates 10 objects in array called 'sitesArr'. Each object represents one website to scrape.
var sitesArr = [
{
name: 'store name',
baseURL: 'www.basedomain.com',
categoryFunct: '(function(){ // do stuff })();',
gender: 'mens',
currency: 'USD',
title_selector: 'h1',
description_selector: 'p.description'
},
// ... x10
]
Loop 2: Loops through 'sitesArr'. For each site it goes to the homepage via 'request' and gets a list of category links, usually 30-70 URLs. Appends these URLs to the current 'sitesArr' object to which they belong, in an array property whose name is 'categories'.
var sitesArr = [
{
name: 'store name',
baseURL: 'www.basedomain.com',
categoryFunct: '(function(){ // do stuff })();',
gender: 'mens',
currency: 'USD',
title_selector: 'h1',
description_selector: 'p.description',
categories: [
{
name: 'shoes',
url: 'www.basedomain.com/shoes'
},{
name: 'socks',
url: 'www.basedomain.com/socks'
} // x 50
]
},
// ... x10
]
Loop 3: Loops through each 'category'. For each URL it gets a list of products links and puts them in an array. Usually ~300-1000 products per category
var sitesArr = [
{
name: 'store name',
baseURL: 'www.basedomain.com',
categoryFunct: '(function(){ // do stuff })();',
gender: 'mens',
currency: 'USD',
title_selector: 'h1',
description_selector: 'p.description',
categories: [
{
name: 'shoes',
url: 'www.basedomain.com/shoes',
products: [
'www.basedomain.com/shoes/product1.html',
'www.basedomain.com/shoes/product2.html',
'www.basedomain.com/shoes/product3.html',
// x 300
]
},// x 50
]
},
// ... x10
]
Loop 4: Loops through each of the 'products' array, goes to each URL and creates an object for each.
var product = {
infoLink: "www.basedomain.com/shoes/product1.html",
description: "This is a description for the object",
title: "Product 1",
Category: "Shoes",
imgs: ['http://foo.com/img.jpg','http://foo.com/img2.jpg','http://foo.com/img3.jpg'],
price: 60,
currency: 'USD'
}
Then, for each product object I'm shipping them off to a MongoDB function which does an upsert into my database
THE ISSUE
This all worked just fine, until the process got large. I'm creating about 60,000 product objects every time this script runs, and after a little while all of my computer's RAM is being used up. What's more, after getting about halfway through my process I get the following error in Node:
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
I'm very much of the mind that this is a code design issue. Should I be "deleting" the objects once I'm done with them? What's the best way to tackle this?