I need to perform some modification on an API that returns something like this:
[
{
"_id": "0000000000000000001",
"name": "Random name",
"categories": [
"5eb8548330550e017f8902e6",
"5eb2548630550e117f8909eb",
"5eb6548930550e017f8909e9"
]
},
{...},
]
each results' categories are returned as ids of their respective documents.
I need to manipulate the result so each object's categories field has an array of objects and each object has the id and the name of its category.
I make an example of what the outcome should be:
[
{
"_id": "0000000000000000001",
"name": "Random name",
"categories": [
{"_id": "5eb8548330550e017f8902e6",
"name": "Category 1"},
{"_id": "5eb2548630550e117f8909eb",
"name": "Category 2"},
{"_id": "5eb6548930550e017f8909e9",
"name": "Category 3"},
]
},
{...},
]
I need to do this using plain JS and for so far this is what I have done but it returns an array of undefined:
const resultArray = await Promise.all(
searchResults.map(async (item) => {
item.categories.map(async (categoryId) => {
return await CategoryType.find({ _id: categoryId});
});
})
);
For the moment I am trying to get each category document for each id in the categories field.
I bet the reason why I am getting an array of undefined is that I am handling async in the wrong way but cannot figure out how.
To strictly answer your question: you are missing synchronization (because Array.prototype.map 'ignores' async):
const resultArray = await Promise.all(
searchResults.map(async (item) => {
const promises = item.categories.map(async (categoryId) => {
// you dont want find, but findOne btw
return await CategoryType.findOne({ _id: categoryId});
});
const categories = await Promise.all(promises)
item.categories = categories
return item
})
);
This can be simplified down as
const resultArray = await Promise.all(
searchResults.map(async item => {
item.categories = await Promise.all(item.categories.map(categoryId => {
return CategoryType.findOne({ _id: categoryId})
}))
return item
})
);
But the proper way of doing it is likely to use populate
const mongoose = require('mongoose')
mongoose.connect('mongodb://localhost:27017/dummy')
const Category = mongoose.model('Category', {
name:String,
}, 'categories');
const X = mongoose.model('X', {
name:String,
categories: [{type: mongoose.Types.ObjectId, ref: 'Category'}]
}, 'xs');
;(async()=>{
try {
mongoose.set('debug', true)
const xs = await X.find().populate('categories')
console.log('xs : ', xs[0])
} finally {
mongoose.disconnect()
}
})()
You will notice by the way, that mongoose, under the hood uses find({ _id: {$in:[]}}) which makes only one request (so better) than doing multiple findOne (as you do)
Related
I tried to scrape a website using Node.JS + Cheerio + Axios, I've get all the things I need, but the problem is I don't know how to return the data from different scopes to receive it (I can only receive the url, not the data inside another scope).
The only data I can receive is the url, but all the data in another scope, I'm quite of can't figure out how to receive it together with the url
How's my module work, it scrapes multiple url, and inside of each url contains things like title, description, subtitle, etc, so that's why I have to map 2 times.
Here's my code:
The services that I'm using to scrape:
exports.getSlides = async () => {
const { data } = await client.get("/")
const $ = cheerio.load(data)
return $(".MovieListTop .TPostMv")
.toArray()
.map((element) => {
const listItem = $(element)
const url = listItem.find("a").attr("href")
axios(url).then((res) => {
const new$ = cheerio.load(res.data)
new$(".TpRwCont")
.toArray()
.map((element) => {
const item = new$(element)
const title = item.find(".Title").first().text().trim()
const subTitle = item.find(".SubTitle").first().text().trim()
const description = item.find(".Description").first().text().trim()
const time = item.find(".Time").first().text().trim()
const date = item.find(".Date").first().text().trim()
const view = item.find(".View").first().text().trim()
// console.log({ title, subTitle, description, time, date, view })
return { data: { title, subTitle, description, time, date, view } }
})
})
return { url }
})
}
The controller that I'm using to receive the data:
const movieServices = require("../services/index")
exports.getSlides = async (req, res, next) => {
const data = await movie.getSlides()
try {
res.json({
message: "Success",
data: data,
})
} catch (err) {
next(err)
}
}
What I'm expected:
{
"message:": "Success",
"data": [
{
"url": "url1",
"data": {
"title": "titleA",
"subTitle": "subTitleA",
...key : value
}
},
{
"url": "url2",
"data": {
"title": "titleB",
"subTitle": "subTitleB",
...key : value
}
},
{
"url": "url3",
"data": {
"title": "titleC",
"subTitle": "subTitleC"
...key : value
},
more objects
}
]
}
Here's a reworked version that uses async/await in order to serialize the requests, organize the data and return the data in a promise. The caller can then use await or .then() to get the data out of the promise.
I'm not entirely sure I understood what result you wanted because what you described in your question and comments doesn't quite match with what the code produces. This code gets a top level array of URLs and then for each URL, there is an array of data objects for each newsElement that URL has. So, there's an array of objects where each object has a url and an array of data. The data is an array of newsElement objects in the url's page like this:
[
{
url: url1,
data: [
{
title: someTitle1,
subTitle: someSubTitle1,
description: someDescription1,
time: someTime1,
date: someDate1,
view: someView1
},
{
title: someTitle2,
subTitle: someSubTitle2,
description: someDescription2,
time: someTime2,
date: someDate2,
view: someView2
}
]
},
{
url: url2,
data: [
{
title: someTitle3,
subTitle: someSubTitle3,
description: someDescription3,
time: someTime3,
date: someDate3,
view: someView3
},
{
title: someTitle4,
subTitle: someSubTitle4,
description: someDescription4,
time: someTime4,
date: someDate4,
view: someView4
}
]
},
]
And, here's the code:
exports.getSlides = async () => {
const { data } = await client.get("/");
const $ = cheerio.load(data);
const elements = $(".MovieListTop .TPostMv").toArray();
const results = [];
for (let element of elements) {
const listItem = $(element);
const url = listItem.find("a").attr("href");
// for each url, we collect an array of objects where
// each object has title, subTitle, etc.. from a newsElement
const urlData = [];
const res = await axios(url);
const new$ = cheerio.load(res.data);
const newsElements = new$(".TpRwCont").toArray();
for (let newsElement of newsElements) {
const item = new$(newsElement);
const title = item.find(".Title").first().text().trim()
const subTitle = item.find(".SubTitle").first().text().trim()
const description = item.find(".Description").first().text().trim()
const time = item.find(".Time").first().text().trim()
const date = item.find(".Date").first().text().trim()
const view = item.find(".View").first().text().trim()
// console.log({ title, subTitle, description, time, date, view })
urlData.push({ title, subTitle, description, time, date, view });
}
results.push({ url, data: urlData });
}
return results;
}
If you want to data collected slightly differently, you should be able to modify this code to change how it organizes the data.
Suppose I have 3 MongDB entries in a movie database:
"books": [
{
"_id": "xxxxxx",
"title": "Fast Five",
"rating": 6
},
{
"_id": "yyyyyy",
"title": "Kill Bill",
"rating": 8
},
{
"_id": "zzzzzzzz",
"title": "Jumanji",
"rating": 5
}
]
I use GraphQL to retrieve the id of multiple movies if their title and rating match the criteria:
query{
getMovies(itemFilterInput: {titles:["Fast Five", "Jumanji"], rating:6}){
_id
title
}
}
This should return Fast Five.
I want this to translate into a mongoDB query like this:
{$and: [{rating:6}, {$or: [{title:"Fast Five", title:"Jumanji"}]}]}
In the backend, I use NodeJS and Mongoose to handle this query and I use something like this:
getMovies: async args => {
try{
let d = args.itemFilterInput;
let filters = new Array();
const keys = Object.keys(d);
keys.forEach((key) => {
// The titles case
if(d[key] instanceof Array){
var sObj = {};
var sObjArray = [];
d[key].map(prop => {
sObj[key] = prop;
sObjArray.push(sObj);
});
filters.push({$or: sObjArray});
}else{
// The rating case
var obj = {};
obj[key] = d[key];
filters.push(obj);
}
});
const items = await Item.find({$and: filters});
return items.map(item => {
return item;
});
} catch (err) {
throw err;
}
}
Unfortunately, this approach does not work.
What is the correct way to nest $or parameters in $and params?
Answer:
First of all, it is better to use the $in operator instead of a $and/$or combination.
And to solve the error: use quotes everywhere to declare the query.
So I've got this part of a code where I'm creating response for my project. Now I've managed to create data, but I've got response that I need to changes.
First here is my code:
exports.getById = (req, res) => {
const id = req.params.a_id;
articleService
.getById(id)
.then((article) => {
bankService
.getRates()
.then((list) => {
let prr = article.price;
let price = parseFloat(prr.replace(/\.| ?€$/g, '').replace(',', '.'));
let mjeseci = req.body.months;
let ratanks = list.map((rata) =>
LoanJS.Loan(price, !mjeseci ? 60 : mjeseci, rata.NKS)
);
const kreditNKS = ratanks.map((index) => index.sum);
const rataNKS = ratanks.map(
(index) => index.installments[0].installment
);
let eks = list.map((stopa) => stopa.EKS);
let name = list.map((ime) => ime.bank.name);
let nks = list.map((stopa) => stopa.NKS);
let type = list.map((ime) => ime.interest_type.name);
res.status(200).json({
kredit: {
kreditNKS: kreditNKS,
rataNKS: rataNKS,
stopaEKS: eks,
stopaNKS: nks,
tip: type,
ime: name,
}
})
.catch((err) => {
res.status(500).send('Error 1 ->' + err);
});
})
.catch((err) => {
res.status(500).send('Error ->' + err);
});
};
Explain of what it does: So I'm fetching single article from my DB which has price inside it, then I'm getting data about loan also from DB. Now I'm using that data from DB, using .map function to get values one by one and calculating for that values my final loan(that is ratanks part). Now I'm also extracting some other data that I need present to the user on the frontend.
Now my problem: It's sending my res as an object with one object, who has key:value pairs and values are array of data inside it. But I want it to be an array with multiple objects.
My response in postman right now:
{
"kredit": {
"kreditNKS": [
118406.54,
118348.2,
119400.33,
118022.46,
118262.44,
118811.84
],
"rataNKS": [
19734.42,
19724.7,
19900.05,
19670.41,
19710.41,
19801.97
],
"stopaEKS": [
"6.24",
"5.65",
"8.26",
"3.13",
"4.03",
"5.68"
],
"stopaNKS": [
"4.11",
"3.94",
"7",
"2.99",
"3.69",
"5.29"
],
"tip": [
"Fiksna",
"Promjenjiva",
"Fiksna",
"Promjenjiva",
"Fiksna",
"Fiksna"
],
"ime": [
"ZiraatBank",
"ZiraatBank",
"UniCredit",
"Raiffeisen Bank",
"Raiffeisen Bank",
"ASA Banka"
]
}
}
Where I need it to be something like this:
[
{
"kreditNKS":118406.54,
"rataNKS": 19734.42,
"stopaEKS": "6.24",
"stopaNKS": "4.11",
"tip": "Fiksna",
"ime": "ZiraatBank"
},
{
"kreditNKS":118348.2,
"rataNKS": 19724.7,
"stopaEKS": "5.65",
"stopaNKS": "3.94",
"tip": "Promjenjiva",
"ime": "ZiraatBank"
},
{
"kreditNKS":119400.33,
"rataNKS": 19900,05,
"stopaEKS": "8.26",
"stopaNKS": "7",
"tip": "Fiksna",
"ime": "UniCredit"
}
etc.....
]
Is it possible to modify something like this?
Any tips are welcome!
Thanks!
it looks like you're mapping your list into 4 arrays and then putting them inside a single object, where each array is a property of the said object.
let eks = list.map((stopa) => stopa.EKS);
let name = list.map((ime) => ime.bank.name);
let nks = list.map((stopa) => stopa.NKS);
let type = list.map((ime) => ime.interest_type.name);
res.status(200).json({
kredit: {
kreditNKS: kreditNKS,
rataNKS: rataNKS,
stopaEKS: eks,
stopaNKS: nks,
tip: type,
ime: name,
}
})
The way someArray.map(eachThing => doSomethingWithThing(thing)) works is that you iterate the entire array "someArray" and execute a function for each thing inside of it.
This means that instead of doing LoanJS.Loan(price, !mjeseci ? 60 : mjeseci, rata.NKS) for all the items of the list and write that to a new array called "ratanks", you can write your own function for all the items of the list, and during the iteration of each item do something like const loan = LoanJS.Loan(price, !mjeseci ? 60 : mjeseci, eachItem.NKS).
This being said, you should be able to get the object you want by mapping your list into an array of objects like this
const mappedKredits = list.map((eachObject) => {
const computedLoan = LoanJS.Loan(price, !mjeseci ? 60 : mjeseci, eachObject.NKS);
const kreditNKS = computedLoan.sum;
const rataNKS = computedLoan.installments[0].installment;
return {
"kreditNKS": kreditNKS,
"rataNKS": rataNKS,
"stopaEKS": eachObject.EKS,
"stopaNKS": eachObject.NKS,
"tip": eachObject.interest_type.name,
"ime": eachObject.bank.name,
}
});
res.status(200).json(mappedKredits);
Btw, try to use more descriptive names for the variables, if they're in English it'll be even better, otherwise, it makes it a bit harder for folks like me to understand what the code is doing, thus making it harder to help you.
I'm pretty new to learning to code. So i might get a lot of basics wrong.
Basically i am downloading API content from two different accounts via request-promise and want to merge them into a bigger array. I'm struggling with escaping my local data from the request-promise function and also combining it with the second array
Here's what i got so far:
//request the site and do some stuff with the data
rp(rpOptions)
.then(function (parsedBody) {
let incomingData1 = (parsedBody); //turning data into a value to change it a little
incomingData1.forEach((incomingData1) => {incomingData1.yearsRetired = 0}); //to add a new property
incomingData1 = JSON.stringify(parsedBody, ["favFood", "age", "work", "yearsRetired"], 2); //to filter only relevant properties into a JSON thing (i eventually want to save it to a txt file)
});
i'd then do the same for the second account and then try to get that data outside of the function and merge it into a single array so that it looks like this:
{
"first_account_name": {
"individual1": {
"favFood": 'fries',
"age": 23,
"work": 'astronaut'
"yearsRetired": 0
},
"individual2": {
"favFood": 'banana',
"age": 55,
"work": 'zookeeper'
"yearsRetired": 0
{
...
}
},
"second_account_name": { ... }
"individual6": {
"favFood": 'apple',
"age": 49,
"work": 'dinosaur'
"yearsRetired": 0
"individual7": {
"favFood": 'sausage',
"age": 33,
"work": 'doctor'
"yearsRetired": 0
{
...
}
how do i get my data into a variable outside of rp? and how do i set it up so that it ends up like a nested array?
Thanks a lot and sorry for being confusing :P
What you are looking for is a global array that gets data pushed into it on every Promise request called right. So firstly, create a simple array and place it on top of the page or if you are using a class just insert it into the respective fields.
Let accountDetails = [];
Next, inside then function call this variable like so,
rp(rpOptions)
.then(function (parsedBody) {
let incomingData1 = (parsedBody);
incomingData1.forEach((incomingData1) => {incomingData1.yearsRetired = 0});
incomingData1 = JSON.stringify(parsedBody, ["favFood", "age", "work", "yearsRetired"], 2);
accountDetails.push({
"individual1" : incomingData1
})
});
If you're using ES6
const processData = (data) => {
return data.map((item) => ({
favFood: item.favFood,
age: item.age,
work: item.work,
yearsRetired: 0
}))
}
// any value returned by then will be wrapped in promise
// and can be `await` ed
// you can also use
// const [ data1, data2 ] = await Promise.all([
// rp(requestOption1).then(data => processData(data)),
// rp(requestOption2).then(data => processData(data))
// ])
// if you want it to be executed parallely
const data1 = await rp(requestOption1).then(data => processData(data));
const data2 = await rp(requestOption2).then(data => processData(data));
const mergedData = [
...data1,
...data2
];
If you don't have async await
const processData = (data) => {
return data.map((item) => ({
favFood: item.favFood,
age: item.age,
work: item.work,
yearsRetired: 0
}))
}
Promise.all(
rp(requestOption1).then(data => processData(data)),
rp(requestOption2).then(data => processData(data))
).then(results => {
const mergedData = results.reduce((collection, result) => {
return collection.concat(result);
}, []);
})
Note:
I wrote the function name processData because I don't know what is being processed. I suggest you to be more specific on the function name. (e.g. what it does)
Probably a silly issue, but why is the Array.find method not working as expected when working in this case? I'm trying to query a specific comment, which involves fetching the post document that has a comments property from the DB. It is from this comments array that I'd like to extract said comment object. For whatever reason, the code below doesn't work. Why?
Below are the code snippets
// Post document from which the comments are extracted
const post = await Post.findById(postId).populate({
path: "comments",
select: "addedBy id"
});
// Resulting post.comments array
[
{ "id": "5d9b137ff542a30f2c135556", "addedBy": "5b8528131719dc141cf95c99" },
{ "id": "5d9b0ba2f28afc5c3013d4df", "addedBy": "5b8528131719dc141cf95c99" },
{ "id": "5d9b0c26f28afc5c3013d4e0", "addedBy": "5b8528131719dc141cf95c99" }
];
// For instance if commentId is '5d9b137ff542a30f2c135556'
// the resulting comment object should be {"id":"5d9b137ff542a30f2c135556","addedBy":"5b8528131719dc141cf95c99"}
// However, commentToDelete is undefined
const commentId = "5d9b137ff542a30f2c135556";
const commentToDelete = comments.find(comment => comment["id"] === commentId);
Edit: Here's the full deleteComment controller code
async function deleteComment(req, res, userId, postId, commentId) {
const post = await Post.findById(postId).populate({
path: 'comments',
select: 'addedBy id',
});
const commentToDelete = post.comments.find(
comment => comment['id'] === commentId
);
if (commentToDelete.addedBy !== userId) {
return res
.status(403)
.json({ message: 'You are not allowed to delete this comment' });
}
await Comment.findByIdAndDelete(commentId);
const updatedPost = await Post.findByIdAndUpdate(
post.id,
{ $pull: { comments: { id: commentId } } },
{ new: true, safe: true, upsert: true }
).populate(populateFields);
return res.status(200).json({ updatedPost });
}
comment => comment['id'] === commentId
Your comment subdocument comes from MongoDB/Mongoose, so comment['id'] will likely be of type ObjectID, which is never equal a string. Explicitly call the toString() function (or use some other approach for transforming to a string) before comparing:
comment => comment['id'].toString() === commentId
works fine in the below snippet, copied from your post!
I am assuming it is posts.comments in your case and not comments.find? Check for typos
const comments = [
{ "id": "5d9b137ff542a30f2c135556", "addedBy": "5b8528131719dc141cf95c99" },
{ "id": "5d9b0ba2f28afc5c3013d4df", "addedBy": "5b8528131719dc141cf95c99" },
{ "id": "5d9b0c26f28afc5c3013d4e0", "addedBy": "5b8528131719dc141cf95c99" }
];
// For instance if commentId is '5d9b137ff542a30f2c135556'
// the resulting comment object should be {"id":"5d9b137ff542a30f2c135556","addedBy":"5b8528131719dc141cf95c99"}
const commentId = "5d9b137ff542a30f2c135556";
// However, commentToDelete is undefined
const commentToDelete = comments.find(comment => comment["id"] === commentId);
console.log(commentToDelete);
you can use this :
const result = comments.find(
({ id }) => id === commentId,
);
console.log(result)
// should return { id: '5d9b137ff542a30f2c135556', addedBy: '5b8528131719dc141cf95c99' }