We Keep Coding

Javascript : Filter a JSON object from API call to only get the infos I need

I'm trying to create a small project to work on API calls. I have created an async that recovers infos about a track using the MusicBrainz API. You can check the result of the request by clicking there : https://musicbrainz.org/ws/2/recording/5935ec91-8124-42ff-937f-f31a20ffe58f?inc=genres+ratings+releases+artists&fmt=json (I chose Highway to Hell from AC/DC).
And here is what I got so far as reworking the JSON response of my request :
export const GET_JSON = async function (url) {
try {
const res = await Promise.race([
fetch(url),
timeout(CONSTANTS.TIMEOUT_SEC),
]);
const data = await res.json();
if (!res.ok) throw new Error(`${data.message} (${res.status})`);
return data;
} catch (err) {
throw err;
}
};
export const loadTrackDetail = async function (id) {
try {
const trackData = await GET_JSON(
encodeURI(
`${CONSTANTS.API_URL}${id}?inc=genres+artists+ratings+releases&fmt=json`
)
);
details.trackDetails = {
trackTitle: trackData.title,
trackID: trackData.id,
trackLength: trackData.length ?? "No duration provided",
trackArtists: trackData["artist-credit"].length
? trackData["artist-credit"]
: "No information on artists",
trackReleases: trackData["releases"].length
? trackData["releases"]
: "No information on releases",
trackGenres: trackData["genres"].length
? trackData["genres"]
: "No information on genres",
trackRating: trackData.rating.value ?? "No rating yet",
};
console.log(details.trackDetails);
} catch (err) {
throw err;
}
Now this isn't half bad, but the releases property for example is an array of objects (each one being a specific release on which the track is present) but for each of those releases, I want to "reduce" the object to its id and title only. The rest does not interest me. Moreover, I'd like to say that if, for example, the title of a release is similar to that of a previous one already present, the entire object is not added to the new array.
I've thought about doing a foreach function, but I just can't wrap my head around how to write it correctly, if it's actually possible at all, if I should use an array.map for example, or another iterative method.
If anyone has some nice way of doing this in pure JS (not Jquery !), efficient and clean, it'd be much appreciated !
Cheers

There are a few things that make this question a little difficult to answer, but I believe the below will get you pointed in the right direction.
You don't include the GET_JSON method, so your example isn't complete and can't be used immediately to iterate on.
In the example you bring, there isn't a name property on the objects contained in the releases array. I substituted name with title below to demonstrate the approach.
You state
Moreover, I'd like to say that if, for example, the name of a release
is similar to that of a previous one already present, the entire
object is not added to the new array.
But you don't define what you consider that would make releases similar.
Given the above, as stated, I assumed you meant title when you said name and I also assumed that what would constitute a similar release would be one with the same name/title.
Assuming those assumptions are correct, I just fetch to retrieve the results. The response has a json method on it that will convert the response to a JSON object. The I map each release to the smaller data set you are interested in(id, title) and then reduce that array to remove 'duplicate' releases.
fetch('https://musicbrainz.org/ws/2/recording/5935ec91-8124-42ff-937f-f31a20ffe58f?inc=genres+ratings+releases+artists&fmt=json')
.then(m => m.json())
.then(j => {
const reducedReleases = j.releases
.map(release => ({ id: release.id, name: release.title }))
.reduce(
(accumulator, currentValue, currentIndex, sourceArray) => {
if (!accumulator.find(a => a.name === currentValue.name)) {
accumulator.push(currentValue);
}
return accumulator;
},
[]);
console.log(reducedReleases);
});

const releasesReduced = []
const titleNotExist = (title) => {
return releasesReduced.every(release => {
if(release.title === title) return false;
return true
})
}
trackData["releases"].forEach(release => {
if (titleNotExist(release.title))
releasesReduced.push({id: release.id, title: release.title})
})
console.log(releasesReduced)

The array details.trackDetails.trackReleases has a path to an id and name from different objects. If you meant: ["release-events"]=>["area"]["id"]and["area"]["name"]` then see the demo below.
Demo uses flatMap() on each level of path to extract "release-events" then "area" to return an array of objects
[{name: area.name, id: area.id}, {name: area.name, id: area.id},...]
Then runs the array of pairs into a for...of loop and sets each unique name with id into a ES6 Map. Then it returns the Map as an object.
{name: id, name: id, ...}
To review this functioning, go to this Plunker
const releaseEvents = (details.trackDetails.trackReleases) => {
let trackClone = JSON.parse(JSON.stringify(objArr));
let areas = trackClone.flatMap(obj => {
if (obj["release-events"]) {
let countries = obj["release-events"].flatMap(o => {
if (o["area"]) {
let area = {};
area.name = o["area"]["name"];
area.id = o["area"]["id"];
return [area];
} else {
return [];
}
});
return countries;
} else {
return [];
}
});
let eventAreas = new Map();
for (let area of areas) {
if (!eventAreas.has(area.name)) {
eventAreas.set(area.name, area.id);
}
}
return Object.fromEntries([...eventAreas]);
};
console.log(releaseEvents(releases));

Firebase Firestore - Async/Await Not Waiting To Get Data Before Moving On?

I'm new to the "async/await" aspect of JS and I'm trying to learn how it works.
The error I'm getting is Line 10 of the following code. I have created a firestore database and am trying to listen for and get a certain document from the Collection 'rooms'. I am trying to get the data from the doc 'joiner' and use that data to update the innerHTML of other elements.
// References and Variables
const db = firebase.firestore();
const roomRef = await db.collection('rooms');
const remoteNameDOM = document.getElementById('remoteName');
const chatNameDOM = document.getElementById('title');
let remoteUser;
// Snapshot Listener
roomRef.onSnapshot(snapshot => {
snapshot.docChanges().forEach(async change => {
if (roomId != null){
if (role == "creator"){
const usersInfo = await roomRef.doc(roomId).collection('userInfo');
usersInfo.doc('joiner').get().then(async (doc) => {
remoteUser = await doc.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
}
}
})
})
})
However, I am getting the error:
Uncaught (in promise) TypeError: Cannot read property 'joinerName' of undefined
Similarly if I change the lines 10-12 to:
remoteUser = await doc.data();
remoteNameDOM.innerHTML = `${remoteUser.joinerName} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser.joinerName}`;
I get the same error.
My current understanding is that await will wait for the line/function to finish before moving forward, and so remoteUser shouldn't be null before trying to call it. I will mention that sometimes the code works fine, and the DOM elements are updated and there are no console errors.
My questions: Am I thinking about async/await calls incorrectly? Is this not how I should be getting documents from Firestore? And most importantly, why does it seem to work only sometimes?
Edit: Here are screenshots of the Firestore database as requested by #Dharmaraj. I appreciate the advice.

You are mixing the use of async/await and then(), which is not recommended. I propose below a solution based on Promise.all() which helps understanding the different arrays that are involved in the code. You can adapt it with async/await and a for-of loop as #Dharmaraj proposed.
roomRef.onSnapshot((snapshot) => {
// snapshot.docChanges() Returns an array of the documents changes since the last snapshot.
// you may check the type of the change. I guess you maybe don’t want to treat deletions
const promises = [];
snapshot.docChanges().forEach(docChange => {
// No need to use a roomId, you get the doc via docChange.doc
// see https://firebase.google.com/docs/reference/js/firebase.firestore.DocumentChange
if (role == "creator") { // It is not clear from where you get the value of role...
const joinerRef = docChange.doc.collection('userInfo').doc('joiner');
promises.push(joinerRef.get());
}
});
Promise.all(promises)
.then(docSnapshotArray => {
// docSnapshotArray is an Array of all the docSnapshots
// corresponding to all the joiner docs corresponding to all
// the rooms that changed when the listener was triggered
docSnapshotArray.forEach(docSnapshot => {
remoteUser = docSnapshot.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
});
});
However, what is not clear to me is how you differentiate the different elements of the "first" snapshot (i.e. roomRef.onSnapshot((snapshot) => {...}))). If several rooms change, the snapshot.docChanges() Array will contain several changes and, at the end, you will overwrite the remoteNameDOM and chatNameDOM elements in the last loop.
Or you know upfront that this "first" snapshot will ALWAYS contain a single doc (because of the architecture of your app) and then you could simplify the code by just treating the first and unique element as follows:
roomRef.onSnapshot((snapshot) => {
const roomDoc = snapshot.docChanges()[0];
// ...
});

There are few mistakes in this:
db.collection() does not return a promise and hence await is not necessary there
forEach ignores promises so you can't actually use await inside of forEach. for-of is preferred in that case.
Please try the following code:
const db = firebase.firestore();
const roomRef = db.collection('rooms');
const remoteNameDOM = document.getElementById('remoteName');
const chatNameDOM = document.getElementById('title');
let remoteUser;
// Snapshot Listener
roomRef.onSnapshot(async (snapshot) => {
for (const change of snapshot.docChanges()) {
if (roomId != null){
if (role == "creator"){
const usersInfo = roomRef.doc(roomId).collection('userInfo').doc("joiner");
usersInfo.doc('joiner').get().then(async (doc) => {
remoteUser = doc.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
}
}
}
})

Multiple REST API calls in succession returns undefined

I'm trying to query some JIRA issues using a the jira-connector package.
I'm running into issues with the data returned not being defined until after everything else is executed in my code. I'm not sure if this is some issue with concurrency, but I can't for the life of me figure out where and how I'm messing up.
If inside the getJiraTimeEstimations function only call the getJiraTimeEstimate once it works just fine and I get access to the data to use further down in the program. It is when I'm trying to do it inside a map or foreach where I iterate over the Array.from(dataFromMaconomy.keys()) array that I seem to run into issues.
My understanding is that adding .then().catch() in the getJiraTimeEstimate function should be enough to stop it from continuing to run before all the calls are finished? Or am I misunderstanding how asynchronous calls work in Node and JS?
I've also tried converting it to an async getJiraTimeEstimations and adding an await infront of the search. But it doesn't seem to work either.
I am not populating the dataFromMaconomy array as I'm debugging. Which is what I was trying to do with the log statement. The log statement just prints undefined right now. But if I only call it with a single item from the rks array then it works fine.
function getJiraTimeEstimate(taskNumber, jiraClient) {
jiraClient.search.search({
jql: `id = ${taskNumber}`,
}).then((res) => res.issues[0].fields.timeoriginalestimate).catch((err) => err);
}
function getJiraTimeEstimations(dataFromMaconomy) {
const settings = JSON.parse(fs.readFileSync(path.join(__dirname, 'konfig.json'), 'utf8'));
const privateKeyData = fs.readFileSync(path.join(__dirname, settings.jira.consumerPrivateKeyFile), 'utf8');
const jira = new JiraClient({
host: settings.jira.server,
strictSSL: false, // Error: unable to verify the first certificate
rejectUnauthorized: false,
oauth: {
consumer_key: settings.jira.consumerKey,
private_key: privateKeyData,
token: settings.jira.accessToken,
token_secret: settings.jira.accessTokenSecret,
},
});
console.log('getting time estimations from Jira');
const dataFromMaconomyWithJira = [];
const rks = Array.from(dataFromMaconomy.keys());
rks.map((rk) => console.log(getJiraTimeEstimate(rk, jira)));
return dataFromMaconomyWithJira;
}
function generateData(){
const dataWithJira = getJiraTimeEstimations(convertedData);
// More functions where I use the data from getJiraTimeEstimations
// This gets run before all of the getJiraTimeEstimations have finished getting the data.
}

Giving your clarification in the comment, the getJiraTimeEstimate() function does not return anything. Try:
function getJiraTimeEstimate(taskNumber, jiraClient) {
return jiraClient.search.search({
jql: `id = ${taskNumber}`,
}).then((res) => res.issues[0].fields.timeoriginalestimate).catch((err) => err);
}
Also, you mentioned trying async / await but without luck. The async version of it would be:
async function getJiraTimeEstimate(taskNumber, jiraClient) {
try {
const res = await jiraClient.search.search({
jql: `id = ${taskNumber}`,
});
return res.issues[0].fields.timeoriginalestimate;
} catch (e) {
return e;
}
}

Puppeteer - testing different selectors for page

I'm trying to have my puppeteer script iterate through selectors.
The reason being - depending on what I'm querying through my script, I can get slightly different elements on the page.
Essentially I have a page.evaluate method that does the scraping like this
while (currentPage <= pagesToScrape) {
let newProducts = await page.evaluate(({identified}) => {
let results = [];
let items = document.querySelectorAll(
identified
);
console.log(items)
items.forEach((item) => {
var prod, price;
if (identified == selectors[0]) {
prod = item.querySelector("div>div>div>div>div>a>h3").innerText;
price = item.querySelector("div>div>div>div>div>div>span>span")
.innerText;
} else {
prod = item.querySelector("div>a>h4").innerText;
price = item.querySelector("div>div>div>div>span>span").innerText;
}
results.push({
Product: prod !== "" ? prod : "",
Price: price !== "" ? price : "",
});
});
console.log("results");
console.log(results.length);
return results;
});
product_GSH = product_GSH.concat(newProducts);
if (currentPage < pagesToScrape) {
console.log(identified)
await Promise.all([
await page.click(buttonSelector),
await page.waitForSelector(identified),
]);
}
Now before the script starts, I need to ensure I have the correct selector.
const selectors = ['div[class = "sh-dlr__list-result"',"div[class = 'sh-dgr__content'"]
//works
const chooseSelector = await page.waitForFunction((selectors) => {
for (const selector of selectors) {
if (document.querySelector(selector) !== null) {
return selector;
}
}
return false;
}, {}, selectors);
const identified = await chooseSelector.jsonValue();
console.log(identified)
The issue I'm having is, from within the page.evaluate, I can run the identifier easily and find the correct one to use. But I need to have it parsed at the end of the query again to scrape the next page. When I try to re-assign the variable name to the correct identifier inside the page.evaluate, it doesn't parse it.
When i run this, the code runs, but I cannot change the selector inside the promise at the bottom with page.waitfor (so it works with some pages but when it's the wrong page I can't alternate the selector being chosen). this is the full code fyi.
product_GSH = product_GSH.concat(newProducts);
if (currentPage < pagesToScrape) {
await Promise.all([
await page.click(buttonSelector),
page.waitForNavigation()
]);
}
currentPage++;
}
browser.close();
return res.send(product_GSH);
} catch (e) {
return res.send(e);
}
});
});
I'm thinking one way to solve this issue is to look at the promise.all
function and replace it with something slightly different.
Thanks for helping with this issue!
Last question if you can help - How do i make sure when I choose say 5 pages, and there are only 3 pages of results, that it sends the 3 pages. What I'm finding is that if i say there's more pages it doesn't send any response.
Ideally, I'm trying to have this code be able to iterate through different selectors. I've tried a bunch of different methods, and CORS errors and more aside, very lost. It would be good to get some sort of definite error from puppeteer as well!
Appreciate the help :)

You have to use page.waitForNavigation along with page.click(buttonSelector) promises. Also, to use Promise.all, you have to pass it actual promises and not resolved promises like you're doing:
if (currentPage < pagesToScrape) {
await Promise.all([
page.click(buttonSelector),
page.waitForNavigation()
]);
}
You can simplify selectors, for example
div[class = "sh-dlr__list-result"
can be
div.sh-dlr__list-result
The selector
const buttonSelector = "a[id='pnnext']>span[style='display:block;margin-left:53px']";
is wrong; you should never rely on style to query a selector; that can easily be dynamic changed; instead you can define it like this
const buttonSelector = "a#pnnext";
After we make these changes we'll get the proper results, for example it will output:
product_GSH.length 100
product_GSH [...]
UPDATE
If you want to handle results with less than pagesToScrape pages, then you have to look for buttonSelector before you perform a click on it like this:
if (currentPage < pagesToScrape && await page.$(buttonSelector) !== null) {
await Promise.all([
page.click(buttonSelector),
page.waitForNavigation()
]);
}

Promise.all looks like the place to solve this. I'm not the best with promise functions though

How to print promised data to a webpage?

I am trying to bring an array of strings from a database to a dropdown menu on a website I have created. I have everything working properly except for the final transfer of the data from the retrieval method to the website. Right now the data is in the form of a Promise, and I cannot for the life of me figure out how to get it to print out on my webpage. right now I'm just sending it to localhost:3000, I'm not at the point where I'm putting it into the dropdown yet. How would I do this ?
I've found very very little on this issue online and thus have been mainly just trying hack fixes that haven't really worked (tacking on the resolve() method, all() method). both of those resulted in syntax errors. All Var names/SQL queries have been changed btw. My latest attempt is below:
//code that sends the names to the webpage
app.get('/formfetch', function(req, res) {
const data = async() => {
let rawDat = await dbFormFetch.getNames();
return rawDat;
}
}
const hNs = data();
hNs.then((names) => {
if (names === null) {
res.end("Error: Names list came through as null.");
} else if (names.length > 0) {
resolve(names);
for (var i = 0; i < names.length; i++) {
res.end(names[i]);
}
res.status('200');
}
})
.catch((err) => {
res.status('404').json(err)
console.log("conversion of promise failed")
})
});
//the getNames() method (in a different file)
async function getNames() {
console.log("trying to get Names");
let query = `select NAME from NAMESTAB`;
console.log("query: " + query);
const binds = {};
const result = await database.simpleExecute(query, binds);
var results = [];
console.log("for loop in formfetch.js: ");
for (var i = 0; i < result.rows.length; i++) {
results[i] = i + ": " + result.rows[i].NAME+ ' \n';
}
return results;
}
The res.send method from the app.get function prints out "Made it to the Web server:" on my localhost. I checked the console, and I didn't see anything hidden in the html or something like that.
**Note: all of the data that should be in the promise is in the code (I can print it to console at any point in the code), but when I put it on the website it won't print. **

so big surprise here, I was doing it all wrong. Lesson of the day: read up on Promises and how they work before running and gunning your way through some async code. It's not as intuitive as you would hope.
// I only had made changes to the first of the two methods.
app.get('/formfetch', function(req, res) {
async function data() {
let rawDat = await dbFormFetch.getNames();
return rawDat;
}
data().then((Names) => {
if (Names === undefined) {
res.end("Error: Names list came through as null.");
} else if (Names.length > 0) {
res.setHeader('Content-Type', 'application/json');
res.status(200).json({ "names": Names });
}
})
.catch((err) => {
res.status('404').send("name retrieval failed in server.js module")
console.log(err)
console.log("conversion of promise failed")
})
});
when you use res.end() , it sets the header status and renders it immutable after calling this method, so it was the wrong thing to use. Instead of this, I used the setHeader() method to tell the website what kind of information I'm sending it, and then filled in the content by chaining the .json() method to the status() response I sent. I've never worked with promises before and I'm fairly new to NodeJS so this was a bit of a learning curve, but hopefully this helps people who are where I was yesterday. if you're new to promises, see this article and this article before you try to use this coding tool. you'll save yourself hours of debugging and error tracing.