We Keep Coding

Javascript : Filter a JSON object from API call to only get the infos I need

I'm trying to create a small project to work on API calls. I have created an async that recovers infos about a track using the MusicBrainz API. You can check the result of the request by clicking there : https://musicbrainz.org/ws/2/recording/5935ec91-8124-42ff-937f-f31a20ffe58f?inc=genres+ratings+releases+artists&fmt=json (I chose Highway to Hell from AC/DC).
And here is what I got so far as reworking the JSON response of my request :
export const GET_JSON = async function (url) {
try {
const res = await Promise.race([
fetch(url),
timeout(CONSTANTS.TIMEOUT_SEC),
]);
const data = await res.json();
if (!res.ok) throw new Error(`${data.message} (${res.status})`);
return data;
} catch (err) {
throw err;
}
};
export const loadTrackDetail = async function (id) {
try {
const trackData = await GET_JSON(
encodeURI(
`${CONSTANTS.API_URL}${id}?inc=genres+artists+ratings+releases&fmt=json`
)
);
details.trackDetails = {
trackTitle: trackData.title,
trackID: trackData.id,
trackLength: trackData.length ?? "No duration provided",
trackArtists: trackData["artist-credit"].length
? trackData["artist-credit"]
: "No information on artists",
trackReleases: trackData["releases"].length
? trackData["releases"]
: "No information on releases",
trackGenres: trackData["genres"].length
? trackData["genres"]
: "No information on genres",
trackRating: trackData.rating.value ?? "No rating yet",
};
console.log(details.trackDetails);
} catch (err) {
throw err;
}
Now this isn't half bad, but the releases property for example is an array of objects (each one being a specific release on which the track is present) but for each of those releases, I want to "reduce" the object to its id and title only. The rest does not interest me. Moreover, I'd like to say that if, for example, the title of a release is similar to that of a previous one already present, the entire object is not added to the new array.
I've thought about doing a foreach function, but I just can't wrap my head around how to write it correctly, if it's actually possible at all, if I should use an array.map for example, or another iterative method.
If anyone has some nice way of doing this in pure JS (not Jquery !), efficient and clean, it'd be much appreciated !
Cheers

There are a few things that make this question a little difficult to answer, but I believe the below will get you pointed in the right direction.
You don't include the GET_JSON method, so your example isn't complete and can't be used immediately to iterate on.
In the example you bring, there isn't a name property on the objects contained in the releases array. I substituted name with title below to demonstrate the approach.
You state
Moreover, I'd like to say that if, for example, the name of a release
is similar to that of a previous one already present, the entire
object is not added to the new array.
But you don't define what you consider that would make releases similar.
Given the above, as stated, I assumed you meant title when you said name and I also assumed that what would constitute a similar release would be one with the same name/title.
Assuming those assumptions are correct, I just fetch to retrieve the results. The response has a json method on it that will convert the response to a JSON object. The I map each release to the smaller data set you are interested in(id, title) and then reduce that array to remove 'duplicate' releases.
fetch('https://musicbrainz.org/ws/2/recording/5935ec91-8124-42ff-937f-f31a20ffe58f?inc=genres+ratings+releases+artists&fmt=json')
.then(m => m.json())
.then(j => {
const reducedReleases = j.releases
.map(release => ({ id: release.id, name: release.title }))
.reduce(
(accumulator, currentValue, currentIndex, sourceArray) => {
if (!accumulator.find(a => a.name === currentValue.name)) {
accumulator.push(currentValue);
}
return accumulator;
},
[]);
console.log(reducedReleases);
});

const releasesReduced = []
const titleNotExist = (title) => {
return releasesReduced.every(release => {
if(release.title === title) return false;
return true
})
}
trackData["releases"].forEach(release => {
if (titleNotExist(release.title))
releasesReduced.push({id: release.id, title: release.title})
})
console.log(releasesReduced)

The array details.trackDetails.trackReleases has a path to an id and name from different objects. If you meant: ["release-events"]=>["area"]["id"]and["area"]["name"]` then see the demo below.
Demo uses flatMap() on each level of path to extract "release-events" then "area" to return an array of objects
[{name: area.name, id: area.id}, {name: area.name, id: area.id},...]
Then runs the array of pairs into a for...of loop and sets each unique name with id into a ES6 Map. Then it returns the Map as an object.
{name: id, name: id, ...}
To review this functioning, go to this Plunker
const releaseEvents = (details.trackDetails.trackReleases) => {
let trackClone = JSON.parse(JSON.stringify(objArr));
let areas = trackClone.flatMap(obj => {
if (obj["release-events"]) {
let countries = obj["release-events"].flatMap(o => {
if (o["area"]) {
let area = {};
area.name = o["area"]["name"];
area.id = o["area"]["id"];
return [area];
} else {
return [];
}
});
return countries;
} else {
return [];
}
});
let eventAreas = new Map();
for (let area of areas) {
if (!eventAreas.has(area.name)) {
eventAreas.set(area.name, area.id);
}
}
return Object.fromEntries([...eventAreas]);
};
console.log(releaseEvents(releases));

How to get Firebase Node children

I'm pretty new to firebase on JavaScript so it's most likely a silly and possibly a duplicate question but I'm having trouble trying to get these two children from each "Task". I tried getting access of the task's properties with ["Task"] and ["Status"] but it's showing undefined for me.
var members = projectRef.child("Data").child("Members").orderByKey().startAt(firebase.auth().currentUser.uid).once('value').then(function(snapshot) // Gets all members with their data
{
var data = [];
var taskArray = [];
snapshot.forEach(function(childSnapshot) // Goes to each member
{
var childData = childSnapshot.val(); // Gets all of the member's children as an object from the childSnapshot
var childTasks = childData.Tasks;
if(childTasks)
{
console.log(childTasks);
for (task in childTasks)
{
console.log(task);
console.log(task["Status"]);
console.log(task["Task"]);
}
}
data.push(childData);
});
return data;
});
Results for childTasks
{
'-ME2IdeYYwHqWJ82ynob': { Status: 'Not completed', Task: 'flsdjlfksdjflksd' },
'-ME5YHsoMVxX5sTq5RVA': { Status: 'Not completed', Task: 'lfjsdlkfjslkdfjlksf' },
'-ME5YKh30-rs_Fq2QhBU': { Status: 'Not completed', Task: 'sflksdjfsklfjdksljsdlf' }
}
Results for the task, task["Status"], and task["Task"]
-ME2IdeYYwHqWJ82ynob
undefined
undefined
-ME5YHsoMVxX5sTq5RVA
undefined
undefined
-ME5YKh30-rs_Fq2QhBU
undefined
undefined

It looks like you're missing a forEach over the lowest level repeated node type.
projectRef.child("Data").child("Members").orderByKey().startAt(firebase.auth().currentUser.uid).once('value').then(function(snapshot) {
snapshot.forEach(function(memberSnapshot) {
var tasksSnapshot = memberSnapshot.child("Tasks");
tasksSnapshot.forEach(function(taskSnapshot) {
console.log(taskSnapshot.val());
console.log(taskSnapshot.child("Status").val());
})
});
});
Things I changed that I'd highly recommend:
Stick to navigating snapshots as long as possible, before calling val().
Give your snapshots meaningful names, representing the entities in your JSON.
In general though, it seems you initial query can be replaced by a direct lookup, since it seems you want to look up the member by their UID and it's unique. By looking up the user (and their Tasks node( directly, you can remove another forEach loop, making the code even easier to read:
var memberRef =
projectRef.child("Data").child("Members").child(firebase.auth().currentUser.uid);
memberRef.once('value').then(function(memberSnapshot) {
var tasksSnapshot = memberSnapshot.child("Tasks");
tasksSnapshot.forEach(function(taskSnapshot) {
console.log(taskSnapshot.val());
console.log(taskSnapshot.child("Status").val());
})
});
And since you only use the Tasks child node, you can simplify this even further to:
var memberRef =
projectRef.child("Data").child("Members").child(firebase.auth().currentUser.uid);
var tasksRef = memberRef.child("Tasks");
tasksRef.once('value').then(function(tasksSnapshot) {
tasksSnapshot.forEach(function(taskSnapshot) {
console.log(taskSnapshot.val());
console.log(taskSnapshot.child("Status").val());
})
});

How to query MongoDB inside for loop without skipping results?

My database has three collections, challenges, users and entries. Challenges have fields like title, description and challenge id. Entries are the completed challenges and contain fields like user id, challenge id and image. I want to join the data in entry collection to it's corresponding challenge so I could have a document containing challenge title, description, challenge id and image.
I am trying to query the challenges collection based on an array of id's gotten from entries collection and then adding the entry query result as a new field to the document.
I have implemented a for loop, which allows me to query with a different id each time. I would like to add the result of the query to an array, but sometimes it skips the results and only some of the queries are present in the resulting array. For example, when I send the API call for the first time, the server returns 2 JSON-objects in an array, but the next time it only returns one. I think there is something wrong with the synchronization of the for loop and the queries. How can I make it return the correct documents every time? Also, is there a better way to join the two collections together without a for loop?
I've tried countless of different ways to finish the for loop without skipping any queries or returning the finished array too early,but have failed to do so. This current implementation works on the first API call, but on the next one it fails. I am using MongoDB (and MERN stack) and I have a REST API where I send calls from my React front end.
exports.getDoneChallenges = [check("userId").isLength({ min: 24 }),
function(req, res) {
var myPromise = () =>
new Promise((resolve, reject) => {
// Find all challenges the user has completed.
Entry.find({ userId: req.params.id }, { _id: 0 })
.sort({ challengeId: -1 })
.exec()
.then(result => {
// Check if the user hasn't completed any challenges.
if (!result) {
console.log("Zero completed challenges.");
res
.status(401)
.json({ message: "No completed challenges found." });
} else {
// Save the completed challenge's identifiers in an array.
var ids = new Array();
for (var i = 0; i < result.length; i++) {
// Cast identifiers to ObjectID
ids.push(ObjectID(result[i].challengeId));
}
// Array of completed challenges + images relating to each.
var challenge_arr = new Array();
for (let i = 0; i < result.length; i++) {
// Match the corresponding challenge id's from entries to
challenges and add image as a new field.
Challenge.aggregate([
{ $match: { challengeId: ids[i] } },
{ $addFields: { image: result[i] } }
])
.exec()
.then(challenge => {
/* Create a new object, which has the needed fields for
the response.*/
var challenge_obj = new Object();
challenge_obj.title = challenge[0].title;
challenge_obj.challengeId = challenge[0].challengeId;
challenge_obj.description = challenge[0].description;
challenge_obj.date = challenge[0].image.date;
challenge_obj.img = challenge[0].image.img;
// Save the challenges into the challenge array.
challenge_arr.push(challenge_obj);
console.log(i)
/* If the loop is in the last round, return the filled
array.*/
if (i == result.length - 1) {
// Return the filled array.
return challenge_arr;
}
})
.then(challenge_arr => {
// Check that the array isn't undefined.
if (typeof challenge_arr !== "undefined") {
// Resolve the promise.
resolve(challenge_arr);
}
});
}
}
});
});
// Call promise function and send a response after resolving it.
myPromise().then(data => {
res.status(200).json({ data: data });
});
}
];
var EntrySchema = new Schema({
challengeId: ObjectId,
userId: ObjectId,
date: { type: Date, default: Date.now},
img: { data: Buffer, contentType: String}
})
var ChallengeSchema = new Schema({
challengeId: mongoose.SchemaTypes.ObjectId,
title: String,
description: String,
date: {type: Date}
})
I have two entries in the Entries collection, which have the same challenge id's as two of the challenges in challenge collection. I query the challenge collection with the id's of the entries, and I am supposed to get 2 documents which have the corresponding entry field added. Sometimes I get the documents correctly, but most of the time it only returns some of them. For example, from 4 expected documents, it returns {chall 1, null, chall 2, chall 3}.

Promise.all can help you orchestrate multiple promises rather than using a for loop. The way you currently have it, you end up calling resolve when the last loop is done with its work, and not when every loop is done with its work.
It might look something like:
const promises = result.map((image, i) =>
Challenge.aggregate([
{ $match: { challengeId: ids[i] } },
{ $addFields: { image } }
]).exec());
Promise.all(promises)
.then((promise_results) => ...);
Using async/await can generally make code like this much simpler to write and understand:
for (const result of results) {
const challenge = await Challenge.aggregate(...
}

Will Mongo handle my service?

I've been using MongoDB with node.js and mongoose library. I decided to start using MongoDB because I found everywhere that it is the best solution for node.js applications.
Although the response times of my API are good, I'm unsure that MongoDB will handle it when scaling it.
I've noticed that most of my queries aren't enough to get all the data I need, so I rely on creating several queries and using some javascript map/reduce functions (that is what I'm afraid of).
Look at this example:
User
.find({
idol : true
})
.sort({
'metas.followers' : -1
})
.select('-password -__v -posts -email')
.skip(offset)
.limit(30)
.exec(function(err, retData)
{
promisedIdols = retData.map(function(idol)
{
return idol.withStatistics(Post, Follow, req.user);
});
idols = [];
if(promisedIdols.length == 0)
{
callback();
}
for(var i=0; i<promisedIdols.length; i++)
{
promisedIdols[i].delegate(function(result)
{
idols.push(result);
if(idols.length == promisedIdols.length)
{
callback();
}
});
}
});
I've used a map to gather an array of promises that will be resolved after running the following code:
var obj = this.toObject();
var deferred = new Promise();
Post
.find({ idol : obj._id })
.lean()
.exec(function(err, posts)
{
var postViews = 0;
var postLikes = 0;
var postShares = 0;
posts.reduce(function(prev, next)
{
postViews += next.views.length;
postLikes += next.likes.length;
postShares += next.shares.length;
}, 0);
obj.metas.postViews = postViews;
obj.metas.postLikes = postLikes;
obj.metas.postShares = postShares;
obj.metas.postCount = posts.length;
Subscription
.count({ idol : obj._id }, function(err, count)
{
obj.metas.subscribers = count;
deferred.deliver(obj);
});
});
that uses a reduce function.
I can't see this code working well on big scale. Maybe should I restructure my database? Maybe should I change my database system? Maybe I'm using MongoDB wrongly?
Experts?
Thanks.

Mongo can handle a lot, if you setup a good data model. There are a few things to keep in mind when you want to scale.
Try to avoid normalizing the data much and split it into different collections.
Data duplication is (sometimes, when used wisely) your friend, it will help you make simpler queries, populate right away. Yeah, that may mean that when you're updating data, you'll have to update in two places, but Mongo is ok with a lot of writes if you do it asynchronously (promises or not).
To your specific query, I don't see the full data model, but maybe you can use aggregation framework. That pipeline is native (C++, as opossed to mapReduce JavaScript) and will work really really fast.
Something like:
db.post.aggregate(
// First $match to reduce the dataset
{
$match: {idol : obj._id}
},
// then group and aggregate your data
{
$group: {
_id: '$idol', // group by that idol thing
postViews: {$sum: '$postViews'},
postLikes: {$sum: '$postLikes'}
},
},
// Then use project to arrange the result the way you like it
{
$project: {
_id: false, //or true if you need it
metas: {
postViews: '$postViews'
},
likeCountOfPosts: '$postLikes', // that's how you'd rename
whatIsIt: {$literal: 'a great post'}
}
}
);
You can also do a lot of conditional, groupings, sortings, winding and unwinding, mixing and shuffling the pipeline.
It's much much faster then Mongo mapReduce.

Why is hashtagseen[] empty after I call the addposthashtags function?

I am trying to add hashtags in the post's hashtag[] array as a object with a num:1 variable to the users hashtagseen[] array if it is not already in it else add 1 the num if the hashtag is already in the hashtagseen[] array. How do I fix my code? Here is the code, thanks in advanced.
edit: I think I am not finding post.hashtag with this.hashtag and that is why it will not go to else. Just a guess.
The user object
Accounts.createUser({
username: username,
password: password,
email: email,
profile: {
hashtagsl:[],
}
});
collections/post.js
var post = _.extend(_.pick(postAttributes, 'title', 'posttext','hashtags'), {
userId: user._id,
username: user.username,
submitted: new Date().getTime(),
commentsCount: 0,
upvoters: [], votes: 0,
});
calling it
Meteor.call('addposthashtags',this.hashtags,Meteor.user().profile.hashtagsl);
lib/usershash
Meteor.methods({
addposthashtags: function (hashtags,hashtagsl) {
//supposed to make hashtagseen a array with the names from the hashtagsl object in it
var hashtagseen = _.pluck(hashtagsl, 'name');
//supposed to run once per each hashtag in the posts array.
for (var a = 0; a < hashtags.length; a++) {
//supposed set hashtagnumber to the number indexOf spits out.
var hashnumber=hashtagseen.indexOf(hashtags[a]);
//supposed to check if the current hashtag[a] === a idem in the hashtagseen.
if(hashnumber===-1){
var newhashtag = this.hashtags[a];
//supposed to make the object with a name = to the current hashtags
Meteor.users.update({"_id": this.userId},{"$push":{"profile.hashtagsl": {name: newhashtag, num: 1}}})
} else {
var hashi = hashtagseen[hashnumber];
//supposed to ad one to the num variable within the current object in hashtagsl
Meteor.users.update({"_id": this.userId, "profile.hashtagsl.name":hashi},{"$inc":{"profile.hashtagsl.num":1}});
}
}
}
});

Your addposthashtags function is full of issues. You also haven't provided a "schema" for hashtag objects.
addposthashtags: function () {
for (a = 0; a < this.hashtag.length; a++) {
// Issue1: You're querying out the user for every iteration of the loop!?
for (i = 0; i < Meteor.user().profile.hashtagseen.length; i++) {
// Issue2: You're comparing two _objects_ with ===
// Issue3: Even if you use EJSON.equals - the `num` property wont match
// Issue4: You're querying out the user again?
if (this.hashtag[a] === Meteor.user().profile.hashtagseen[i]) {
// Issue5 no `var` statement for hashtagseeni?
// Issue6 You're querying out the user again??
hashtagseeni = Meteor.user().profile.hashtagseen[i];
//Issue7 undefined hashtagsli?
//Issue8 Calling multiple methods for the one action (eg in a loop) is a waste of resources.
Meteor.call('addseen', hashtagsli);
} else {
//Issue9 no `var` statement for newhashtag?
newhashtag = this.hashtag[a];
newhashtag.num = 1;
//Issue8b Calling multiple methods for the one action (eg in a loop) is a waste of resources.
Meteor.call('updateUser', newhashtag, function (err, result) {
if (err)
console.log(err);
});
}
}
}
}
Also, the method has similiar issues:
addseen: function (hashtagseeni) {
// Issue10: var `profile` is undefined
// Issue11: should use `this.userId`
// Issue12: hashtagseeni wouldn't match profile.hashtagseen due to "num" field.
Meteor.users.update({"_id": Meteor.userId, "profile.hashtagseen": profile.hashtagseeni}, {"$inc":{"profile.hashtagseen.$.num":1}});
}
New issues with your new set of code:
Meteor.methods({
addposthashtags: function (hashtags,hashtagsl) {
//Issue1 `hashtag` is undefined, guessing you mean `hashtags`
//Issue2 no `var` for a
for (a = 0; a < hashtag.length; a++) {
//Issue3 no `var` for i
//Issue4 Why are you looping through both?
// don't you just want to check if hashtag[a] is in hashtagsl?
for (i = 0; i < hashtagsl.length; i++) {
if (hashtags[a] === hashtagsl[i].name) {
var hashi = hashtagsl[i].name;
//supposed to ad one to the num variable within the current object in hashtagsl.
// Issue5: This query wont do what you think. Test until you've got it right.
Meteor.users.update({"_id": Meteor.userId, 'profile.hashtagsl':hashi}, {"$inc":{"num":1}});
} else {
// Issue6 `this.hashtag` isn't defined. guessing you mean `hashtags[a]`
var newhashtag = this.hashtag[a];
// Issue7 superfluous statement
var newhashtagnum = num = 1;
// Issue8 Obvious syntax errors
// Perhaps try Meteor.users.update({"_id": this.userId},{"$push":{"profile.hashtagsl": {name: newhashtag, num: 1}}})
Meteor.users.update({"_id": Meteor.userId, 'profile'},{"$addToSet":{"hashtagsl"[newhashtag]=newhashtagnum}})
};
};
};
};
});

I'd suggest you trying the following
1) Assuming that after newhashtag=hashtag[a] you get a JSON object in newhashtag variable, try replacing newhashtag:{num:1}; with newhashtag.num = 1 - this will add the num variable to the object and set the value.
1.a) For debugging purposes try adding some console.log(JSON.stringify(newhashtag)); after each of the two lines where you're setting and changing the newhashtag variable - this way you'll know exactly what you're trying to add to the mongoDB document.
2) The update to increment the views also doesn't seem to me that will work. Couple of things to note here - $set:{'profile.hashtagseen[i]':num++} - MongoDB won't be able to identify the 'i' in 'profile.hashtagseen[i]' and 'num++' is not how increments are done in Mongo.
I'd suggest you look into the $inc and to the positional update documentation of MongoDB.
Your final increment update statement will look something like
Meteor.users.update({"_id": Meteor.userId, "profile.hashtagseen": profile.hashtagseen[i]}, {"$inc":{"profile.hashtagseen.$.num":1}});

I see that executing addposthashtags is in the client, and you must to pay attention because this function will execute in minimongo and doesn't work all operations. First you try execute this operation under mongo if it's work you must to create one function inside the folder server.
Add text of the documentation of Minimongo
In this release, Minimongo has some limitations:
$pull in modifiers can only accept certain kinds of selectors.
findAndModify, aggregate functions, and map/reduce aren't supported.
All of these will be addressed in a future release. For full Minimongo
release notes, see packages/minimongo/NOTES in the repository.
Minimongo doesn't currently have indexes. It's rare for this to be an
issue, since it's unusual for a client to have enough data that an
index is worthwhile.
You try create one method on the server, with the same operation.
Server:
Meteor.methods({
updateUser: function (newhashtag) {
Meteor.users.update(this.userId,
{
$addToSet: {'profile.$.hashtagseen': newhashtag}
});
}
});
Client:
Meteor.call('updateUser',newhashtag,function(err,result){
if (err)
console.log(err);// there you can print the erro if there are
});
Minimongo doesn't support alls operation, for test you can to execute in the console for testing the method if supported. After that you can to execute the operation under mongo directly, that clears your doubts.