We Keep Coding

Javacript for loop wait for function to finish? [duplicate]

This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 1 year ago.
I'm making a program that consists of three different functions:
downloadPDF: download a PDF from the web
getPDF: read and parse the pdf
getDaata: loop through getPDF
Problem I'm having is that the third function(getData) that has a for of loop that runs getPDF, it seems as if it doesn't let getPDF finish before trying to console.log the result that getPDF returns.
Here are the three functions:
async function downloadPDF(pdfURL, outputFilename) {
let pdfBuffer = await request.get({uri: pdfURL, encoding: null});
console.log("Writing downloaded PDF file to " + outputFilename + "...");
fs.writeFileSync(outputFilename, pdfBuffer);
}
async function getPDF(query, siteName, templateUrl, charToReplace) {
const currentWeek = currentWeekNumber().toString();
await downloadPDF(templateUrl.replace(charToReplace, currentWeek), "temp/pdf.pdf");
var resultsArray = []
let dataBuffer = fs.readFileSync("temp/pdf.pdf");
pdf(dataBuffer).then(function(data) {
pdfContent = data.text;
const splittedArray = pdfContent.split("\n");
const parsedArray = splittedArray.map((item, index) => {
if(item.includes(query)) {
resultsArray.push({result: item, caseId: splittedArray[index-1].split(',', 1)[0], site: siteName});
}
}).filter(value => value);
return(resultsArray);
});
fs.unlinkSync("temp/pdf.pdf"); //deletes the downloaded file
}
async function getData(query, desiredSites) {
var resultsArray = []
for (const value of desiredSites) {
let result = await getPDF(query, sitesList.sites[value].name, sitesList.sites[value].templateUrl, sitesList.sites[value].charToReplace);
console.log(result)
}
}
getData("test", ['a', 'b']);
In the bottom function(getData), the console.log results in undefined
I'm guessing this has something to do with the promises. Any ideas? Thanks a lot!

In getPDF, you should chain all your async functions with await instead of .then or vice versa.
You can mix await with .then but this would be not easy to chain them with linear codes. The reason people use await because they want to make the codes look linear and easy to maintain.
async function downloadPDF(pdfURL, outputFilename) {
let pdfBuffer = await request.get({ uri: pdfURL, encoding: null });
console.log("Writing downloaded PDF file to " + outputFilename + "...");
fs.writeFileSync(outputFilename, pdfBuffer);
}
async function getPDF(query, siteName, templateUrl, charToReplace) {
const currentWeek = currentWeekNumber().toString();
await downloadPDF(
templateUrl.replace(charToReplace, currentWeek),
"temp/pdf.pdf"
);
var resultsArray = [];
let dataBuffer = fs.readFileSync("temp/pdf.pdf");
const data = await pdf(dataBuffer);
pdfContent = data.text;
const splittedArray = pdfContent.split("\n");
const resultsArray = splittedArray
.filter(item => item.includes(query))
.map(item => ({
result: item,
caseId: splittedArray[index - 1].split(",", 1)[0],
site: siteName,
}));
fs.unlinkSync("temp/pdf.pdf"); //deletes the downloaded file
return resultsArray;
}
async function getData(query, desiredSites) {
for (const value of desiredSites) {
let result = await getPDF(
query,
sitesList.sites[value].name,
sitesList.sites[value].templateUrl,
sitesList.sites[value].charToReplace
);
console.log(result);
}
}
getData("test", ["a", "b"])
.then(() => console.log("done"))
.catch(console.log);

How to call web service endpoints one after other?

I want to implement a function in JavaScript which calls a series of web service endpoint and checks for a value in the response of the API call.
I need to achieve it in a way that the first endpoint page is called first then the there would be a filter method to filter out the specific object from the response. If the object is found, this process should break and the object must be returned. However if the object is not found in the first endpoint, then the second endpoint must be called and the same process is repeated until the object is found.
The Web service endpoint that I am working on is:
https://jsonmock.hackerrank.com/api/countries?page=1
This API returns a list of country data. Here the value of page query varies from 1 to 25. I need to call the endpoint and check for a specific country from 1 to 25 until the country object is found.
I tried achieving this using JavaScript Promise and Fetch API and couldn't think of a way to call the APIs one after the other.
I am really looking forward for your answer. Thank you in advance.

You can use async and await for this:
async function findCountry(country) {
for (let page = 1; page < 26; page++) {
console.log("page = " + page); // for debugging only
let response = await fetch("https://jsonmock.hackerrank.com/api/countries?page=" + page);
let {data} = await response.json();
let obj = data.find(obj => obj.name == country);
if (obj) return obj;
}
}
let country = "Belgium";
findCountry(country).then(obj => {
if (obj) {
console.log("The capital of " + country + " is " + obj.capital);
} else {
console.log("Could not find " + country);
}
});
If you know that the data is sorted by country name, then you could reduce the average number of requests by using a binary search.

Here's a way that you can do it.
const url = 'https://jsonmock.hackerrank.com/api/countries'
const fetchFromApi = async (countryName, page) => {
const res = await fetch(`${url}?page=${page}`)
return await res.json()
}
const getCountryFromResults = (countryName, data) => {
const country = countryName.toLowerCase()
return data.find(({name}) => name.toLowerCase() === country)
}
const findCountry = async (countryName) => {
let page = 1;
let totalPages = 1;
while(page <= totalPages) {
const res = await fetchFromApi(countryName, page);
if(totalPages < res.total_pages) {
totalPages = res.total_pages
}
const country = getCountryFromResults(countryName, res.data)
if(country){
return country
}
page = page + 1
}
}
( async () => {
console.log(await findCountry("Afghanistan"))
console.log(await findCountry("Argentina"))
}
)()

How to get items value json

I am getting JSON from a api GET call. I am trying to get a value of a item which I think is a array. I am trying to console log the low price of the json.
I tried to cycle through it like a array like open.openDate.btcusd[5] and so on.
//JSON DATA FROM API
btcusd":{
"high":"9206.36",
"low":"8804.57",
"volume":"1291.122483",
"last":"8989.64",
"bid":"8987.88",
"ask":"8998.24"
//Call
coin.getOpen()
.then(data=>{
coin.ui(data);
});
//Function
async getOpen(){
const openres = await
fetch(`https://api.lakebtc.com/api_v2/ticker`);
const openBtc = await openres.json();
return {
openDate : openBtc
}
}
//New Function to console.log
ui(open){
console.log(open.openDate.btcusd); //I want the low value
}

do call this function like
async function getOpen(){
const openres = await fetch(`https://api.lakebtc.com/api_v2/ticker`);
const openBtc = await openres.json();
// console.log(openBtc.btcusd.low);
return {
openDate : openBtc
}
}
// function call
getOpen()
.then(res => console.log(res.openDate.btcusd.low))
.catch(err => console.error(err))

Didnt completely get your requirement here, but the response here is an object not an array.
We can convert to array and print the low values for different entries like :
async function getOpen(){
const openres = await fetch(`https://api.lakebtc.com/api_v2/ticker`);
const openBtc = await openres.json();
return {
openDate : openBtc
}
};
getOpen().then(data=>{
ui(data);
});
function ui(obj){
var arr = Object.entries(obj.openDate);
var lowValues = arr.map(d => console.log(d[0] + " value of low is " + d[1].low));
}

Code snippet executes after function is finished in Firebase Cloud Functions

I have a bit of a problem using firebase cloud functions. The code bellow is a function that writes to the Firestore DB an object that contains 2 arrays. After deploying the function, the idf_words array is populated, and the idf_weight is empty.
I tried placing a few log messages in the for loop and found the the query.get() executes after the function ends. Is there a way to make firestore wait until the query.get() finishes?
export const updateCakeAndPastriesIDF = functions.firestore.document("TF/tf/Cake_and_Pastries/{itemCategory}")
.onUpdate((change, context) => {
const itemBefore = change.before.data();
const itemAfter = change.after.data();
if (itemAfter['tf_tf_score'] === itemBefore['tf_tf_score']){
console.log('This TF score of the words in this item has not changed');
return null;
} else {
console.log('This TF score of the words in this item has changed');
const tfWords:string[] = itemAfter['tf_unique_words'];
const tfItemUid:string = itemAfter['tf_item_uid'];
const idfWords:string[] = [];
const idfWeight: number[] = [];
const db = admin.firestore().collection('TF').doc('tf').collection('Cake_and_Pastries');
tfWords.forEach(function (tfword) {
idfWords.push(tfword);
const query = db.where("tf_unique_words", "array-contains", tfword);
query.get().then(function (itemDoc) {
if (!itemDoc.empty){
const numberOfDocs = itemDoc.size;
console.log("For item: "+tfItemUid+", there are "+numberOfDocs+"Documents");
admin.firestore().collection('Number_of_Items')
.doc('Cake_and_Pastries')
.get()
.then(function (numberDoc){
const numberOfCakesAndPastries = numberDoc.data()['number_of_items_in_category'];
const idfOfWord = (Math.log(numberOfDocs/numberOfCakesAndPastries)+1);
idfWeight.push(idfOfWord);
console.log("Word IDF: "+idfOfWord);
console.log(idfWeight);
})
}else {
console.log("No such document!");
}
})
});
console.log("IDF weight array outside of loop: "+idfWeight);
admin.firestore()
.collection('IDF')
.doc('idf')
.collection('Cake_and_Pastries')
.doc(tfItemUid).set({
idf_item_uid: tfItemUid,
idf_words: idfWords,
idf_weight: idfWeight
});
}
});

Batch get DocumentReferences?

I'm trying to improve a firestore get function, I have something like:
return admin.firestore().collection("submissions").get().then(
async (x) => {
var toRet: any = [];
for (var i = 0; i < 10; i++) {
try {
var hasMedia = x.docs[i].data()['mediaRef'];
if (hasMedia != null) {
var docData = (await x.docs[i].data()) as MediaSubmission;
let submission: MediaSubmission = new MediaSubmission();
submission.author = x.docs[i].data()['author'];
submission.description = x.docs[i].data()['description'];
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
submission.media = mediaRef.data() as MediaData;
toRet.push(submission);
}
}
catch (e) {
console.log("ERROR GETTIGN MEDIA: " + e);
}
}
return res.status(200).send(toRet);
});
The first get is fine but the performance is worst on the line:
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
I think this is because the call is not batched.
Would it be possible to do a batch get on an array of mediaRefs to improve performance?
Essentially I have a collection of documents which have foreign references stored by a string pointing to the path in a separate collection and getting those references has been proven to be slow.

What about this? I did some refactoring to use more await/async code, hopefully my comments are helpful.
The main idea is to use Promise.all and await all the mediaRefs retrieval
async function test(req, res) {
// get all docs
const { docs } = await admin
.firestore()
.collection('submissions')
.get();
// get data property only of docs with mediaRef
const datas = await Promise.all(
docs.map(doc => doc.data()).filter(data => data.mediaRef),
);
// get all media in one batch - this is the important change
const mediaRefs = await Promise.all(
datas.map(({ mediaRef }) =>
admin
.firestore()
.doc(mediaRef)
.get(),
),
);
// create return object
const toRet = datas.map((data: MediaSubmission, i) => {
const submission = new MediaSubmission();
submission.author = data.author;
submission.description = data.description;
submission.media = mediaRefs[i].data() as MediaData;
return submission;
});
return res.status(200).send(toRet);
}