We Keep Coding

Fetch API prints promise in incorrect order

I am trying to fetch the HTML script of two webpages using their URLs. This is my code:
const links = ["url1" : "https://.......", "url2" : "https://......"];
var responses = [];
for(const [key,value] of Object.entries(links)){
let resp = fetch('https://api.codetabs.com/v1/proxy?quest='+value)
responses.push(resp);
}
Promise.all(responses)
.then( htmlfiles =>{
htmlfiles.forEach(file=>{
file.text().then(function(data){
gethtmldata(data);
})
})
})
In my function gethtmldata, I am parsing this data in HTML format:
function gethtmldata(html_data){
var parser = new DOMParser();
var htmldoc = parser.parseFromString(html_data, "text/html");
console.log(htmldoc); //shows data of url2, then url1
}
To my utter surprise, the data of url2 gets printed first, then url1. Why?
It should show the html data of url1 then url2. How do I fix this?

The iterations of your for loop aren't paused when you do file.text().then(function(data){...}. Instead, your loop fires off multiple .text() calls which will complete sometime in the future, with no guaranteed order on which ones will complete first.
You should .push() a Promise that resolves to your .text() data instead when you create resp:
const links = {"url1" : "https://.......", "url2" : "https://......"};
const urls = Object.values(links);
const responses = [];
for(const value of urls){
const txtPromise = fetch('https://api.codetabs.com/v1/proxy?quest='+value).then(resp => resp.text());
responses.push(txtPromise);
}
Promise.all(responses)
.then(htmlData => {
htmlData.forEach(data=>{
gethtmldata(data);
});
});
You can refactor the above by using .map() and async/await like so:
async function fetchHTMLData(urls) {
const promises = urls.map(async url => {
const resp = await fetch('https://api.codetabs.com/v1/proxy?quest='+url);
return resp.text();
});
return Primise.all(promises);
};
async function processHTMLData() {
const links = {"url1" : "https://.......", "url2" : "https://......"};
const urls = Object.values(links);
const htmlArr = await fetchHTMLData(urls);
htmlArr.forEach(htmlStr => {
gethtmldata(htmlStr);
});
}

How to make a chained fetch and show results as each step becomes available?

I'm fetching article list data from API and use/fetch Unsplash API to get relative images according to each fetched article title.
This is my code:
let url = 'http://127.0.0.1:8000/api';
async function getData(url) {
const res = await fetch(url);
const objects = await res.json();
await Promise.all(objects.map(async (object) => {
const res = await fetch('https://api.unsplash.com/search/photos?client_id=XXX&content_filter=high&per_page=1&query=' + object.title);
const image = await res.json();
object.image_url = image.results[0].urls.small
object.image_alt = image.results[0].alt_description
}));
}
let articles_1 = getData(url + '/articles/index/1/');
let articles_2 = getData(url + '/articles/index/2/');
let articles_3 = getData(url + '/articles/index/3/');
I am showing three different categories at once on the same page. That's why I call that function three times.
Question:
When this function kicks, results are shown after both article data and images are fetched. But I want to show article data first when it's been fetched and then images when they get fetched in order to shorten the user waiting time. How can I achieve it wether with Svelte reactive declaration or plain Javascript?

You would want to seperate the two functions, so they can be called in sequence.
const endpoint = 'http://127.0.0.1:8000/api';
const getArticles = async (url) => {
return fetch(url).res.json();
};
const renderArticles = async (articles) => {
// render article set and return it as a DOM object
return articlesDOM;
};
const getImageForArticle = async (articleNode) => {
const res = await = fetch('https://api.unsplash.com/search/photos?client_id=XXX&content_filter=high&per_page=1&query=' + object.title);
const img = new Image();
img.src = res.results[0].urls.small;
img.alt = res.results[0].alt_description;
return {img, articleNode};
};
const renderImage = async (stuff) => {
const {img, articleNode} = stuff;
// inject your img into your article
};
// now call in sequence
getArticles(endpoint+'/articles/index/1/').then(renderArticles).then(articleNodes => {
const promises = articleNodes.map(articleNode => {
return getImageForArticle(articleNode).then(renderImage);
});
return Promise.all(promises);
});

While I'm not completely sure what you're trying to do (I don't have a minimal working example), here's my best attempt at it:
var url = 'http://127.0.0.1:8000/api';
async function getData(url) {
var data = fetch(url)
.then(data => data.json())
await data.then(data => ArticleFunc(data))
await data.then(function(data) {
data.map(function(object) {
fetch('https://api.unsplash.com/search/photos?client_id=XXX&content_filter=high&per_page=1&query=' + object.title)
.then(data => data.json())
object.image_url = image.results[0].urls.small
object.image_alt = image.results[0].alt_description
ImageFunc(object)
})
})
}
function ArticleFunc(data){
//display article
}
function ImageFunc(data){
//display image
}
getData(url + '/articles/index/1/');
getData(url + '/articles/index/2/');
getData(url + '/articles/index/3/');
Note that this is to be treated as pseudocode, as again, it is untested due to the absense of a minimal working example.

skip iteration in for loop

I have a problem with if statement in for loop.
I am getting list of files from directory filesSAS loop through them and each one converting from csv to JSON. after that i check if output has id in their object if yes copy file (copyFile(dirSas, dirOut, filename) if id is present add Date and save as CSV.
problem is that in frist iteration it copy the file but it is also executing saveCSV function which overrides my result. What i want to achive is to if id is not present copy file and that it for this iteration, go for another iteration. I was trying to put saveCSV inside for loop with no luck
EDIT: when my for loop hit object with no id i want to copy file. when id is present i want to add date to it and save as csv
let noId = [{
user:"Mark",
job:"Job"
}]
let withId = [{
id:1,
user:"Mark",
job:"Job"
}]
output
let withId = [{
id:1,
user:"Mark",
job:"Job"
date: 12-09-2019
}]
const saveNewFile = async (filesSAS, dirSas, dirOut, dirArchive) => {
filesSAS.forEach(async filename => {
const newData = await csv().fromFile(`${dirSas.path}/${filename}`);
for await (const iterator of object) {
if (iterator.Id === null || iterator.Id === undefined) {
await copyFile(dirSas, dirOut, filename);
}
rec.Date = moment(Date.now()).format("DD-MMM-YYYY");
}
await saveCSV(newData, `${dirOut.path}/${filename}`, "output");
});
};
regards

JavaScript does have the continue statement, which jumps over one iteration of a loop:
JavaScript Break and Continue
However, I think the fact that your loop is asynchronous makes it hard to restart the loop in a sequential fashion. You can always just use an if statement to not do the remainder of the loop.

What you want to do can be achieved with a boolean that we update in case we don't find an id
const saveNewFile = async (filesSAS, dirSas, dirOut, dirArchive) => {
filesSAS.forEach(async (filename, index) => {
const newData = await csv().fromFile(`${dirSas.path}/${filename}`);
for await (const iterator of object) {
if (iterator.Id === null || iterator.Id === undefined) {
await copyFile(dirSas, dirOut, filename);
}
else {
rec.Date = moment(Date.now()).format("DD-MMM-YYYY");
await saveCSV(newData, `${dirOut.path}/${filename}`, "output");
}
}
});
};

Hey so I mange to do it with refactor
const saveNewFile = async (filesSAS, dirSas, dirOut, dirArchive) => {
filesSAS.forEach(async filename => {
const newData = await csv().fromFile(`${dirSas.path}/${filename}`);
if (!newData[0].hasOwnProperty(pkCol)) {
copyFile(dirSas, dirOut, filename);
} else {
for await (const rec of newData) {
rec.DateCreated = moment(Date.now()).format("DD-MMM-YYYY");
}
await saveCSV(newData, `${dirOut.path}/${filename}`, "output");
}
});
};

How to get items value json

I am getting JSON from a api GET call. I am trying to get a value of a item which I think is a array. I am trying to console log the low price of the json.
I tried to cycle through it like a array like open.openDate.btcusd[5] and so on.
//JSON DATA FROM API
btcusd":{
"high":"9206.36",
"low":"8804.57",
"volume":"1291.122483",
"last":"8989.64",
"bid":"8987.88",
"ask":"8998.24"
//Call
coin.getOpen()
.then(data=>{
coin.ui(data);
});
//Function
async getOpen(){
const openres = await
fetch(`https://api.lakebtc.com/api_v2/ticker`);
const openBtc = await openres.json();
return {
openDate : openBtc
}
}
//New Function to console.log
ui(open){
console.log(open.openDate.btcusd); //I want the low value
}

do call this function like
async function getOpen(){
const openres = await fetch(`https://api.lakebtc.com/api_v2/ticker`);
const openBtc = await openres.json();
// console.log(openBtc.btcusd.low);
return {
openDate : openBtc
}
}
// function call
getOpen()
.then(res => console.log(res.openDate.btcusd.low))
.catch(err => console.error(err))

Didnt completely get your requirement here, but the response here is an object not an array.
We can convert to array and print the low values for different entries like :
async function getOpen(){
const openres = await fetch(`https://api.lakebtc.com/api_v2/ticker`);
const openBtc = await openres.json();
return {
openDate : openBtc
}
};
getOpen().then(data=>{
ui(data);
});
function ui(obj){
var arr = Object.entries(obj.openDate);
var lowValues = arr.map(d => console.log(d[0] + " value of low is " + d[1].low));
}

Batch get DocumentReferences?

I'm trying to improve a firestore get function, I have something like:
return admin.firestore().collection("submissions").get().then(
async (x) => {
var toRet: any = [];
for (var i = 0; i < 10; i++) {
try {
var hasMedia = x.docs[i].data()['mediaRef'];
if (hasMedia != null) {
var docData = (await x.docs[i].data()) as MediaSubmission;
let submission: MediaSubmission = new MediaSubmission();
submission.author = x.docs[i].data()['author'];
submission.description = x.docs[i].data()['description'];
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
submission.media = mediaRef.data() as MediaData;
toRet.push(submission);
}
}
catch (e) {
console.log("ERROR GETTIGN MEDIA: " + e);
}
}
return res.status(200).send(toRet);
});
The first get is fine but the performance is worst on the line:
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
I think this is because the call is not batched.
Would it be possible to do a batch get on an array of mediaRefs to improve performance?
Essentially I have a collection of documents which have foreign references stored by a string pointing to the path in a separate collection and getting those references has been proven to be slow.

What about this? I did some refactoring to use more await/async code, hopefully my comments are helpful.
The main idea is to use Promise.all and await all the mediaRefs retrieval
async function test(req, res) {
// get all docs
const { docs } = await admin
.firestore()
.collection('submissions')
.get();
// get data property only of docs with mediaRef
const datas = await Promise.all(
docs.map(doc => doc.data()).filter(data => data.mediaRef),
);
// get all media in one batch - this is the important change
const mediaRefs = await Promise.all(
datas.map(({ mediaRef }) =>
admin
.firestore()
.doc(mediaRef)
.get(),
),
);
// create return object
const toRet = datas.map((data: MediaSubmission, i) => {
const submission = new MediaSubmission();
submission.author = data.author;
submission.description = data.description;
submission.media = mediaRefs[i].data() as MediaData;
return submission;
});
return res.status(200).send(toRet);
}