I’m trying to read the sentiment of multiple Reddit posts. I’ve got the idea to work using 6 API calls but I think we can refactor it to 2 calls.
The wall I’m hitting - is it possible to loop through multiple APIs (one loop for each Reddit post we’re scrapping), print the results, and then add them into a single variable?
The last part is where I’m stuck. After looping through the API, I get separate outputs for each loop and I don’t know how to add them into a single variable…
Here’s a simple version of what the code looks like:
import React, { useState, useEffect } from 'react';
function App() {
const [testRedditComments, setTestRedditComments] = useState([]);
const URLs = [
'https://www.reddit.com/r/SEO/comments/tepprk/is_ahrefs_worth_it/',
'https://www.reddit.com/r/juststart/comments/jvs0d1/is_ahrefs_worth_it_with_these_features/',
];
useEffect(() => {
URLs.forEach((URL) => {
fetch(URL + '.json').then((res) => {
res.json().then((data) => {
if (data != null) setTestRedditComments(data[1].data.children);
});
});
});
}, []);
//This below finds the reddit comments and puts them into an array
const testCommentsArr = testRedditComments.map(
(comments) => comments.data.body
);
//This below takes the reddit comments and terns them into a string.
const testCommentsArrToString = testCommentsArr.join(' ');
console.log(testCommentsArrToString);
I've tried multiple approaches to adding the strings together but I've sunk a bunch of time. Anyone know how this works. Or is there a simpler way to accomplish this?
Thanks for your time and if you need any clarification let me know.
-Josh
const URLs = [
"https://www.reddit.com/r/SEO/comments/tepprk/is_ahrefs_worth_it/",
"https://www.reddit.com/r/juststart/comments/jvs0d1/is_ahrefs_worth_it_with_these_features/",
];
Promise.all(
URLs.map(async (url) => {
const resp = await fetch(url + ".json");
return resp.json();
})
).then((res) => console.log(res));
I have used Promise.all and got the response and attached a react sandbox with the same.
Based on your requirements, you can use state value or you can prepare your api response before setting it to state.
Related
i have to get some data from a complex objects json (in my case is: valor_custo_fornecedor from the detalhe object, how can i do this ?
i already tried with axios like this:
const [valorCusto, setValorCusto] = useState([]);
useEffect(() => {
axios.get('MY API LINK').then((res) => {
let valor = res.detalhe.valor_custo_fornecedor;
setValorCusto(res.valorCusto.valor_custo_fornecedor);
});
});
It is hard to tell without seeing the complete json what key/value pairs exist in the data, but from what I can see you are passing a value that does not seem to exist in the data to setValorCusto, namely: res.valorCusto.valor_custo_fornecedor
I am guessing what you want to pass to setValorCusto is the valor variable that you have defined above? However the res.data.detalheis as others have replied an array, which you would either have to iterate through or specify an index for.
Another detail, not relating to the question in itself, is that I would add a dependency array to the effect, so that this api request is not called on every component re-render. So the end result would look something like this:
const [valorCusto, setValorCusto] = useState([]);
useEffect(() => {
axios.get('MY API LINK').then((res) => {
const valor = res.data.detalhe.map((detalheItem) => {
return detalheItem.valor_custo_fornecedor;
});
setValorCusto(valor);
});
}, []);
Notice the empty array as the second parameter in the useEffect call. This means that the effect will only be run on the initial rendering of the component.
Then you can access the value by simply using valorCusto wherever you need it in the component.
After setting response in setValorCusto ; you can loop over you state variable and get the desired value
Like the other answers state. What you are returning from your API is an array of objects. Currently in your code, that array lives inside the response object. So in this snippet below:
axios.get('MY API LINK').then((res) => {
let valor = res.detalhe.valor_custo_fornecedor;
You need to specify from which item in the array you would like to get the detalhe object from (like res[0].detalhe).
One way you could do if you like to use everything from the array, is to store the entire array in your useState. Afterwards, you could loop over the array and do something with the objects inside of it.
PS. If you're not entirely sure what the state holds. You could log it to the page by doing something like <div>{JSON.stringify(valorCusto)}</div>
export default function App() {
const [valorCusto, setValorCusto] = useState(null);
useEffect(() => {
const getData = async () => {
const res = await axios.get("API");
const data = await res.data;
setValorCusto(data);
};
getData();
}, []);
return (
<main>
<div>
{valorCusto.map((item) => {
<div>{item.id}</div>;
})}
</div>
</main>
);
}
You can convert json like this:
useEffect(() => {
axios.get('MY API LINK').then((res) => {
let obj = JSON.parse(res.detalhe.valor_custo_fornecedor)
});
});
I am trying to map an array of id and fetch relevant data for that id from an external API in React.
Here is my code:
const getDetails = async () => {
ids.map(async(id) => {
const details = await axios.get(`url/${id}`);
setIdDetails(details);
});
};
getDetails();
And trying to display those details in a dropdown menu like this:
{idDetails.map((detail) => {
<MenuItem value={detail}>{detail.data.title}</MenuItem>
})}
I get the error that idDetails.map is not a function. Also, I tried pushing to the array instead of setting state, no luck either. I have this feeling that I am doing the whole data fetching thing wrong. What is the best practice to handle a situation like this?
As mentioned by #SurjeetBhadauriya, use the spread syntax setIdDetails([...idDetails, details]);.
This will push the buffer after spreading it as single items, which will allow you to map them in your jsx.
Just use setIdDetails([...idDetails, details]); to push the new data into the array.
const [idDetails, setIdDetails] = useState([]);
const getDetails = async () => {
ids.map(async(id) => {
const details = await axios.get(`url/${id}`);
setIdDetails([...idDetails, details]);
});
};
getDetails();
Also, make this change to avoid data specific error
{(idDetails || []).map((detail) => {
<MenuItem value={detail}>{detail?.data?.title}</MenuItem>
})}
Note: I am hoping that you are getting details as an object here const details = await axios.get(url/${id});
I'm using React to build a web app. At one point I have a list of ids, and I want to use those to retrieve a list of items from a database, get a list of metrics from each one, and then push those metrics to an array. My code so far is:
useEffect(() => {
const newMetrics = [];
currentItems.forEach((item) => {
const url = `items/listmetrics/${item.id}`;
Client.getData(url).then(async (metrics) => {
let promises = metrics.map((metricId: string) => {
// Get metric info
const urlMetric = `metrics/${metricId}`;
return Client.getData(urlMetric);
});
await Promise.all(promises).then((metrics: Array<any>) => {
metrics.forEach((metric: MetricModel) => {
const metricItem = {
id: metric.id,
metricName: metric.name
};
newMetrics.push(metricItem);
}
});
});
});
setMetrics(newMetrics);
});
}, [currentItems]);
where "metrics" is a state variable, set by setMetrics.
This appears to work ok, but when I try to access the resulting metrics array, it seems to be in the wrong format. If I try to read the value of metrics[0], it says it's undefined (although I know there are several items in metrics). Looking at it in the console, metrics looks like this:
However, normally the console shows arrays like this (this is a different variable, I'm just showing how it's listed with (2) [{...},{...}], whereas the one I've created shows as []):
I'm not confident with using Promise.all, so I suspect that that's where I've gone wrong, but I don't know how to fix it.
I'm new to the "async/await" aspect of JS and I'm trying to learn how it works.
The error I'm getting is Line 10 of the following code. I have created a firestore database and am trying to listen for and get a certain document from the Collection 'rooms'. I am trying to get the data from the doc 'joiner' and use that data to update the innerHTML of other elements.
// References and Variables
const db = firebase.firestore();
const roomRef = await db.collection('rooms');
const remoteNameDOM = document.getElementById('remoteName');
const chatNameDOM = document.getElementById('title');
let remoteUser;
// Snapshot Listener
roomRef.onSnapshot(snapshot => {
snapshot.docChanges().forEach(async change => {
if (roomId != null){
if (role == "creator"){
const usersInfo = await roomRef.doc(roomId).collection('userInfo');
usersInfo.doc('joiner').get().then(async (doc) => {
remoteUser = await doc.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
}
}
})
})
})
However, I am getting the error:
Uncaught (in promise) TypeError: Cannot read property 'joinerName' of undefined
Similarly if I change the lines 10-12 to:
remoteUser = await doc.data();
remoteNameDOM.innerHTML = `${remoteUser.joinerName} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser.joinerName}`;
I get the same error.
My current understanding is that await will wait for the line/function to finish before moving forward, and so remoteUser shouldn't be null before trying to call it. I will mention that sometimes the code works fine, and the DOM elements are updated and there are no console errors.
My questions: Am I thinking about async/await calls incorrectly? Is this not how I should be getting documents from Firestore? And most importantly, why does it seem to work only sometimes?
Edit: Here are screenshots of the Firestore database as requested by #Dharmaraj. I appreciate the advice.
You are mixing the use of async/await and then(), which is not recommended. I propose below a solution based on Promise.all() which helps understanding the different arrays that are involved in the code. You can adapt it with async/await and a for-of loop as #Dharmaraj proposed.
roomRef.onSnapshot((snapshot) => {
// snapshot.docChanges() Returns an array of the documents changes since the last snapshot.
// you may check the type of the change. I guess you maybe don’t want to treat deletions
const promises = [];
snapshot.docChanges().forEach(docChange => {
// No need to use a roomId, you get the doc via docChange.doc
// see https://firebase.google.com/docs/reference/js/firebase.firestore.DocumentChange
if (role == "creator") { // It is not clear from where you get the value of role...
const joinerRef = docChange.doc.collection('userInfo').doc('joiner');
promises.push(joinerRef.get());
}
});
Promise.all(promises)
.then(docSnapshotArray => {
// docSnapshotArray is an Array of all the docSnapshots
// corresponding to all the joiner docs corresponding to all
// the rooms that changed when the listener was triggered
docSnapshotArray.forEach(docSnapshot => {
remoteUser = docSnapshot.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
});
});
However, what is not clear to me is how you differentiate the different elements of the "first" snapshot (i.e. roomRef.onSnapshot((snapshot) => {...}))). If several rooms change, the snapshot.docChanges() Array will contain several changes and, at the end, you will overwrite the remoteNameDOM and chatNameDOM elements in the last loop.
Or you know upfront that this "first" snapshot will ALWAYS contain a single doc (because of the architecture of your app) and then you could simplify the code by just treating the first and unique element as follows:
roomRef.onSnapshot((snapshot) => {
const roomDoc = snapshot.docChanges()[0];
// ...
});
There are few mistakes in this:
db.collection() does not return a promise and hence await is not necessary there
forEach ignores promises so you can't actually use await inside of forEach. for-of is preferred in that case.
Please try the following code:
const db = firebase.firestore();
const roomRef = db.collection('rooms');
const remoteNameDOM = document.getElementById('remoteName');
const chatNameDOM = document.getElementById('title');
let remoteUser;
// Snapshot Listener
roomRef.onSnapshot(async (snapshot) => {
for (const change of snapshot.docChanges()) {
if (roomId != null){
if (role == "creator"){
const usersInfo = roomRef.doc(roomId).collection('userInfo').doc("joiner");
usersInfo.doc('joiner').get().then(async (doc) => {
remoteUser = doc.data().joinerName;
remoteNameDOM.innerHTML = `${remoteUser} (Other)`;
chatNameDOM.innerHTML = `Chatting with ${remoteUser}`;
})
}
}
}
})
I'm gonna try to explain this as clearly as I can, but it's very confusing to me so bear with me.
For this project, I'm using Node.js with the modules Axios and Cheerio.
I am trying to fetch HTML data from a webshop (similar to Amazon/eBay), and store the product information in a dictionary. I managed to store most things (title, price, image), but the product description is on a different page. To do a request to this page, I'm using the URL I got from the first request, so they are nested.
This first part is done with the following request:
let request = axios.get(url)
.then(res => {
// This gets the HTML for every product
getProducts(res.data);
console.log("Got products in HTML");
})
.then(res => {
// This parses the product HTML into a dictionary of product items
parseProducts(productsHTML);
console.log("Generated dictionary with all the products");
})
.then(res => {
// This loops through the products to fetch and add the description
updateProducts(products);
})
.catch(e => {
console.log(e);
})
I'll also provide the way I'm creating product objects, as it might clarify the function where I think the problem occurs.
function parseProducts(html) {
for (item in productsHTML) {
// Store the data from the first request
const $ = cheerio.load(productsHTML[item]);
let product = {};
let mpUrl = $("a").attr("href");
product["title"] = $("a").attr("title");
product["mpUrl"] = mpUrl;
product["imgUrl"] = $("img").attr("src");
let priceText = $("span.subtext").text().split("\xa0")[1].replace(",", ".");
product["price"] = parseFloat(priceText);
products.push(product);
}
}
The problem resides in the updateProducts function. If I console.log the dictionary afterwards, the description is not added. I think this is because the console will log before the description gets added. This is the update function:
function updateProducts(prodDict) {
for (i in prodDict) {
let request2 = axios.get(prodDict[i]["mpUrl"])
.then(res => {
const $ = cheerio.load(res.data);
description = $("div.description p").text();
prodDict[i]["descr"] = description;
// If I console.log the product here, the description is included
})
}
// If I console.log the product here, the description is NOT included
}
I don't know what to try anymore, I guess it can be solved with something like async/await or putting timeouts on the code. Can someone please help me with updating the products properly, and adding the product descriptions? Thank you SO much in advance.
To refactor this with async/await one would do:
async function fetchAndUpdateProducts() => {
try {
const response = await axios.get(url);
getProducts(response.data);
console.log("Got products in HTML");
parseProducts(productsHTML);
console.log("Generated dictionary with all the products");
await updateProducts(products);
} catch(e) {
console.log(e);
}
}
fetchAndUpdateProducts().then(() => console.log('Done'));
and
async function updateProducts(prodDict) {
for (i in prodDict) {
const response = await axios.get(prodDict[i]["mpUrl"]);
const $ = cheerio.load(response.data);
description = $("div.description p").text();
prodDict[i]["descr"] = description;
}
}
This will not proceed to conclude the call to fetchAndUpdateProducts unless the promise returned by updateProducts has been resolved.