We Keep Coding

Promise.all fetch continue executing after throwing error?

I am trying to fetch JSON data from the WordPress Developer Reference site. I need to search a keyword without knowing if it's a function, class, hook, or method, which is part of the url I need to fetch. So I'm using Promise.all to cycle through all possible urls. It works if the response.status <= 299, throwing the error immediately, and if the response is ok, then it continues to .then. Fine, but occasionally it will return an ok status if the JSON exists and only returns an empty array. So I need to check if the JSON data is an empty array, which I can't seem to do in the first part. I can only check in the second part as far as I know. And if it throws the error it doesn't continue trying the other urls. Any suggestions?
var keyword = 'AtomParser';
const refs = ['function', 'hook', 'class', 'method'];
// Store the promises
let promises = [];
// Cycle through each type until we find one we're looking for
for (let t = 0; t < refs.length; t++) {
const url =
'https://developer.wordpress.org/wp-json/wp/v2/wp-parser-' +
refs[t] +
'?search=' +
keyword;
// console.log(url);
promises.push(fetch(url));
}
Promise.all(promises)
.then(function(response) {
console.log(response[0]);
// Get the status
console.log('Status code: ' + response[0].status);
if (response[0].status <= 299) {
// The API call was successful!
return response[0].json();
} else {
throw new Error('Broken link status code: ' + response[0].status);
}
})
.then(function(data) {
// This is the HTML from our response as a text string
console.log(data);
// Make sure we have data
if (data.length == 0) {
throw new Error('Empty Array');
}
// ref
const reference = data[0];
// Only continue if not null or empty
if (reference !== null && reference !== undefined && data.length > 0) {
// Success
// Return what I want from the reference
}
})
.catch(function handleError(error) {
console.log('Error' + error);
});
Is there some way to get the JSON data in the first part so I can check if it's in an array while I'm checking the response status?

I would recommend encapsulating the success / failure logic for individual requests, then you can determine all the resolved and rejected responses based on the result of that encapsulation.
For example
const checkKeyword = async (ref, keyword) => {
const params = new URLSearchParams({ search: keyword });
const res = await fetch(
`https://developer.wordpress.org/wp-json/wp/v2/wp-parser-${encodeURIComponent(
ref
)}?${params}`
);
if (!res.ok) {
throw new Error(`${res.status}: ${await res.text()}`);
}
const data = await res.json();
if (data.length === 0) {
throw new Error(`Empty results for '${ref}'`);
}
return { ref, data };
};
Now you can use something like Promise.any() or Promise.allSettled() to find the first successful request or all successful requests, respectively
const keyword = "AtomParser";
const refs = ["function", "hook", "class", "method"];
const promises = refs.map((ref) => checkKeyword(ref, keyword));
// First success
Promise.any(promises)
.then(({ ref, data }) => {
console.log(ref, data);
})
.catch(console.error);
// All successes
Promise.allSettled(promises)
.then((responses) =>
responses.reduce(
(arr, { status, value }) =>
status === "fulfilled" ? [...arr, value] : arr,
[]
)
)
.then((results) => {
// results has all the successful responses
});

For whatever reason I couldn't get Phil's answer to work, so I ended up doing the following which works fine for me. (This is for a discord bot in case you're wondering what the other stuff is all about).
var keyword = 'AtomParser';
const refs = ['function', 'hook', 'class', 'method'];
// Store the successful result or error
let final: any[] = [];
let finalError = '';
// Cycle through each type until we find one we're looking for
for (let t = 0; t < refs.length; t++) {
const url =
'https://developer.wordpress.org/wp-json/wp/v2/wp-parser-' +
refs[t] +
'?search=' +
keyword;
console.log(url);
// Try to fetch it
await fetch(url)
.then(function (response) {
console.log(response);
// Get the status
console.log('Status code: ' + response.status);
if (response.status > 299) {
finalError = '`' + refs[t] + '` does not exist.';
throw new Error(finalError);
} else {
// The API call was successful!
return response.json();
}
})
.then(function (data) {
// This is the HTML from our response as a text string
console.log(data);
// Make sure we have data
if (data.length == 0) {
finalError = "Sorry, I couldn't find `" + keyword + '`';
throw new Error(finalError);
}
// Only continue if not null or empty
if (data[0] !== null && data[0] !== undefined && data.length > 0) {
for (let d = 0; d < data.length; d++) {
// Add it to the final array
final.push(data[d]);
}
}
})
.catch(function handleError(error) {
console.log(error);
});
}
if (final.length > 0) {
for (let f = 0; f < final.length; f++) {
// ref
const reference = final[f];
// Get the link
const link = reference.link;
// Get the title
var title = reference.title.rendered;
title = excerpt.replace('>', '>');
// Get the excerpt
var excerpt = reference.excerpt.rendered;
excerpt = excerpt.replace('<p>', '');
excerpt = excerpt.replace('</p>', '');
excerpt = excerpt.replace('<b>', '**');
excerpt = excerpt.replace('</b>', '**');
console.log(excerpt);
message.reply(
new discord.Embed({
title: `${title}`,
url: link,
description: `${excerpt}\n\n`,
footer: {
text: `WordPress Developer Code Reference\nhttps://developer.wordpress.org/`,
},
})
);
}
} else if (finalError != '') {
message.reply(finalError);
} else {
message.reply('Something went wrong...');
}

wp module
#Phil's answer puts you on the right track but I want to expand on some of his ideas. Use of URLSearchParamas is great but you can improve by using the high-level URL API and forego encodeURIComponent and constructing search params manually. Notice I'm putting this code in its own wp module so I can separate concerns more easily. We don't want all of this code leaking into your main program.
// wp.js
import { fetch } from "whatwg-fetch" // or your chosen implementation
const baseURL = "https://developer.wordpress.org"
async function search1(path, query) {
const u = new URL(path, baseURL)
u.searchParams.set("search", query)
const result = await fetch(u)
if (!result.ok) throw Error(`Search failed (${result.status}): ${u}`)
return result.json()
}
search1 searches one path, but we can write search to search all the necessary paths. I don't think there's any reason to get fancy with each path here, so just write them out -
// wp.js (continued)
function search(query) {
const endpoints = [
"/wp-json/wp/v2/wp-parser-function",
"/wp-json/wp/v2/wp-parser-hook",
"/wp-json/wp/v2/wp-parser-class",
"/wp-json/wp/v2/wp-parser-method"
]
return Promise
.all(endpoints.map(e => search1(e, query)))
.then(results => results.flat())
}
export { search }
main module
Notice we only exported search as search1 is internal to the wp module. Let's see how we can use it in our main module now -
// main.js
import { search } from "./wp.js"
for (const result of await search("database"))
if(result.guid.rendered)
console.log(`${result.title.rendered}\n${result.guid.rendered}\n`)
In this example, we first search for "database" -
wp_should_replace_insecure_home_url()
https://developer.wordpress.org/reference/functions/wp_should_replace_insecure_home_url/
wp_delete_signup_on_user_delete()
https://developer.wordpress.org/reference/functions/wp_delete_signup_on_user_delete/
get_post_datetime()
https://developer.wordpress.org/reference/functions/get_post_datetime/
wp_ajax_health_check_get_sizes()
https://developer.wordpress.org/reference/functions/wp_ajax_health_check_get_sizes/
wp_should_replace_insecure_home_url
https://developer.wordpress.org/reference/hooks/wp_should_replace_insecure_home_url/
comments_pre_query
https://developer.wordpress.org/reference/hooks/comments_pre_query/
users_pre_query
https://developer.wordpress.org/reference/hooks/users_pre_query/
WP_Object_Cache
http://developer.wordpress.org/reference/classes/wp_object_cache/
wpdb
http://developer.wordpress.org/reference/classes/wpdb/
WP_REST_Menu_Items_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_menu_items_controller/prepare_item_for_database/
WP_REST_Global_Styles_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_global_styles_controller/prepare_item_for_database/
WP_REST_Menus_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_menus_controller/prepare_item_for_database/
WP_REST_Templates_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_templates_controller/prepare_item_for_database/
WP_REST_Application_Passwords_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_application_passwords_controller/prepare_item_for_database/
wpdb::db_server_info()
https://developer.wordpress.org/reference/classes/wpdb/db_server_info/
WP_REST_Attachments_Controller::insert_attachment()
https://developer.wordpress.org/reference/classes/wp_rest_attachments_controller/insert_attachment/
WP_Debug_Data::get_database_size()
https://developer.wordpress.org/reference/classes/wp_debug_data/get_database_size/
WP_REST_Meta_Fields::update_multi_meta_value()
https://developer.wordpress.org/method/wp_rest_meta_fields/update_multi_meta_value/
another search example
Now let's search for "image" -
for (const result of await search("image"))
if(result.guid.rendered)
console.log(`${result.title.rendered}\n${result.guid.rendered}\n`)
get_adjacent_image_link()
https://developer.wordpress.org/reference/functions/get_adjacent_image_link/
get_next_image_link()
https://developer.wordpress.org/reference/functions/get_next_image_link/
get_previous_image_link()
https://developer.wordpress.org/reference/functions/get_previous_image_link/
wp_robots_max_image_preview_large()
https://developer.wordpress.org/reference/functions/wp_robots_max_image_preview_large/
wp_getimagesize()
https://developer.wordpress.org/reference/functions/wp_getimagesize/
is_gd_image()
https://developer.wordpress.org/reference/functions/is_gd_image/
wp_show_heic_upload_error()
https://developer.wordpress.org/reference/functions/wp_show_heic_upload_error/
wp_image_src_get_dimensions()
https://developer.wordpress.org/reference/functions/wp_image_src_get_dimensions/
wp_image_file_matches_image_meta()
https://developer.wordpress.org/reference/functions/wp_image_file_matches_image_meta/
_wp_check_existing_file_names()
https://developer.wordpress.org/reference/functions/_wp_check_existing_file_names/
edit_custom_thumbnail_sizes
https://developer.wordpress.org/reference/hooks/edit_custom_thumbnail_sizes/
get_header_image_tag_attributes
https://developer.wordpress.org/reference/hooks/get_header_image_tag_attributes/
image_editor_output_format
https://developer.wordpress.org/reference/hooks/image_editor_output_format/
wp_image_src_get_dimensions
https://developer.wordpress.org/reference/hooks/wp_image_src_get_dimensions/
wp_get_attachment_image
https://developer.wordpress.org/reference/hooks/wp_get_attachment_image/
image_sideload_extensions
https://developer.wordpress.org/reference/hooks/image_sideload_extensions/
wp_edited_image_metadata
https://developer.wordpress.org/reference/hooks/wp_edited_image_metadata/
wp_img_tag_add_loading_attr
https://developer.wordpress.org/reference/hooks/wp_img_tag_add_loading_attr/
wp_image_file_matches_image_meta
https://developer.wordpress.org/reference/hooks/wp_image_file_matches_image_meta/
get_custom_logo_image_attributes
https://developer.wordpress.org/reference/hooks/get_custom_logo_image_attributes/
Custom_Image_Header
http://developer.wordpress.org/reference/classes/custom_image_header/
WP_Image_Editor_Imagick
http://developer.wordpress.org/reference/classes/wp_image_editor_imagick/
WP_Embed
http://developer.wordpress.org/reference/classes/wp_embed/
WP_Image_Editor
http://developer.wordpress.org/reference/classes/wp_image_editor/
WP_Customize_Background_Image_Setting
http://developer.wordpress.org/reference/classes/wp_customize_background_image_setting/
WP_Customize_Header_Image_Setting
http://developer.wordpress.org/reference/classes/wp_customize_header_image_setting/
WP_Image_Editor_GD
http://developer.wordpress.org/reference/classes/wp_image_editor_gd/
WP_Customize_Header_Image_Control
http://developer.wordpress.org/reference/classes/wp_customize_header_image_control/
WP_REST_Server::add_image_to_index()
https://developer.wordpress.org/reference/classes/wp_rest_server/add_image_to_index/
WP_REST_URL_Details_Controller::get_image()
https://developer.wordpress.org/reference/classes/wp_rest_url_details_controller/get_image/
WP_Image_Editor::get_default_quality()
https://developer.wordpress.org/reference/classes/wp_image_editor/get_default_quality/
WP_Theme_JSON::get_blocks_metadata()
https://developer.wordpress.org/reference/classes/wp_theme_json/get_blocks_metadata/
WP_Image_Editor_Imagick::pdf_load_source()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/pdf_load_source/
WP_Image_Editor_Imagick::write_image()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/write_image/
WP_Image_Editor_Imagick::maybe_exif_rotate()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/maybe_exif_rotate/
WP_Image_Editor_Imagick::make_subsize()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/make_subsize/
WP_Image_Editor_GD::make_subsize()
https://developer.wordpress.org/reference/classes/wp_image_editor_gd/make_subsize/
empty search result
Searching for "zzz" will yield no results -
for (const result of await search("zzz"))
if(result.guid.rendered)
console.log(`${result.title.rendered}\n${result.guid.rendered}\n`)
<empty result>

How to assemble the JavaScript async fetch results in the "original order"

The good thing about async is, it speeds up the fetch, and we get whatever is available first instead of server to respond.
But this messes up the ORDER of the requests which in this case is important for me.
I wanted to scrape different parts of a continuous story, so the order is a must.
Here's what I did:
async function getSingle(url) {
await fetch(url).then(function (response) {
return response.text();
}).then(function (html) {
var output = document.querySelector('.output');
output.innerHTML=html;
text += "\n" + document.querySelector('.post-body').innerText;
});
}
It gets everything perfectly, but when I call it for multiple URLs, it returns them unordered.
[Beginner here, so please pardon if it's something really trivial.]

You function the same with (I recommended use async/await completely)
async function getSingle(url) {
const response = await fetch(url);
const html = await response.text();
const output = document.querySelector('.output');
output.innerHTML = html;
text += "\n" + document.querySelector('.post-body').innerText; // "text" ???
}
To keep the order, you can call the function in order by a for-loop, the request will be called one by one:
const urls = ['url1', 'url2'];
for (const url of urls) {
await getSingle(url);
}
If you want to call the requests in parallel, let's use Promise.all. But you have to refactor your function - it will returns html string:
async function getSingle(url) {
const response = await fetch(url);
const html = await response.text();
return html
}
const htmls = await Promise.all(urls.map(url => getSingle(url));
for (const html of htmls) {
const output = document.querySelector('.output');
output.innerHTML = html;
text += "\n" + document.querySelector('.post-body').innerText; // "text" ???
}

You could try to keep track of the order they were sent, then when you receive the response, try and add your html to them dom in the same order. something like this (I didn't test it):
let order = [];
let index = 0;
let completed_index=0;
async function getSingle(url) {
let i=index++;
order[i] = {index:i}
await fetch(url).then(function (response) {
order[i] = response.text();
}).then(function (html) {
while(order[completed_index]!==undefined{
completed_index++;
var output = document.querySelector('.output');
output.innerHTML=order[completed_index];
text += "\n" + document.querySelector('.post-body').innerText;
}
});
}
you might want to clear the array afterwards

How to call web service endpoints one after other?

I want to implement a function in JavaScript which calls a series of web service endpoint and checks for a value in the response of the API call.
I need to achieve it in a way that the first endpoint page is called first then the there would be a filter method to filter out the specific object from the response. If the object is found, this process should break and the object must be returned. However if the object is not found in the first endpoint, then the second endpoint must be called and the same process is repeated until the object is found.
The Web service endpoint that I am working on is:
https://jsonmock.hackerrank.com/api/countries?page=1
This API returns a list of country data. Here the value of page query varies from 1 to 25. I need to call the endpoint and check for a specific country from 1 to 25 until the country object is found.
I tried achieving this using JavaScript Promise and Fetch API and couldn't think of a way to call the APIs one after the other.
I am really looking forward for your answer. Thank you in advance.

You can use async and await for this:
async function findCountry(country) {
for (let page = 1; page < 26; page++) {
console.log("page = " + page); // for debugging only
let response = await fetch("https://jsonmock.hackerrank.com/api/countries?page=" + page);
let {data} = await response.json();
let obj = data.find(obj => obj.name == country);
if (obj) return obj;
}
}
let country = "Belgium";
findCountry(country).then(obj => {
if (obj) {
console.log("The capital of " + country + " is " + obj.capital);
} else {
console.log("Could not find " + country);
}
});
If you know that the data is sorted by country name, then you could reduce the average number of requests by using a binary search.

Here's a way that you can do it.
const url = 'https://jsonmock.hackerrank.com/api/countries'
const fetchFromApi = async (countryName, page) => {
const res = await fetch(`${url}?page=${page}`)
return await res.json()
}
const getCountryFromResults = (countryName, data) => {
const country = countryName.toLowerCase()
return data.find(({name}) => name.toLowerCase() === country)
}
const findCountry = async (countryName) => {
let page = 1;
let totalPages = 1;
while(page <= totalPages) {
const res = await fetchFromApi(countryName, page);
if(totalPages < res.total_pages) {
totalPages = res.total_pages
}
const country = getCountryFromResults(countryName, res.data)
if(country){
return country
}
page = page + 1
}
}
( async () => {
console.log(await findCountry("Afghanistan"))
console.log(await findCountry("Argentina"))
}
)()

Trying to figure out why a thrown error is not caught

Here's the complete code I'm trying to run. I will, however, censor the url of the website for personal reasons. I am trying to scrape titles from a very slow website which occasionally sends error status codes in the 4xx range because that, so to handle that I throw an error then retry fetching the same pages after a couple of seconds. The problem is that this error is never caught by the "catch" block. Any idea what I am doing wrong?
const URL = "https://webpage.com/page=";
const SELECTOR = ".post-title.entry-title>a";
const MAX_CONCURRENT_FETCH = 5;
const NB_OF_PAGES = 125;
const ERROR_WAIT_TIME = 20000;
const titles = [];
const parser = new DOMParser();
function fetchPages() {
helper(1, MAX_CONCURRENT_FETCH);
function helper(first, last) {
const requests = [];
console.log("\n" + "*".repeat(40));
for (let i = first; i <= last; i++) {
requests.push(fetch(URL + i));
console.log(`Fetching page: ${i}`);
}
console.log("*".repeat(40) + "\n");
try {
Promise.all(requests).then(responses => {
responses.forEach(async(response, i) => {
/* if no code errors, parse the page, extract the titles then add them to the "titles" array */
if (response.ok) {
const htmlPage = await response.text();
console.log("\n" + "*".repeat(40));
console.log(`Extracting titles from page: ${first+i}`);
console.log("*".repeat(40) + "\n");
const htmlObject = parser.parseFromString(htmlPage, "text/html");
htmlObject.querySelectorAll(SELECTOR).forEach(node => {
titles.push(node.textContent);
console.log(`Title: ${node.textContent}`);
});
}
/* code error, get out of the forEach method by throwing an error */
else {
console.log("*".repeat(40) + "\n");
console.log("Throwing error...");
throw {
response: response
}
}
});
/* keep fetching until the last page */
if (last + 1 < NB_OF_PAGES)
helper(last + 1, last + MAX_CONCURRENT_FETCH);
/* once all pages have been fetched, show the result on screen */
else showResult();
});
}
/* catch the error that was throw inside the forEach method, show the status code error and how long before the pages will be fetched again */
catch (err) {
console.log("Error captured...");
console.log(`Status Code: ${err.response.status}, retrying in ${ERROR_WAIT_TIME/1000} seconds.`);
console.log("*".repeat(40) + "\n");
setTimeout(() => helper(first, last), ERROR_WAIT_TIME);
}
}
}
function showResult() {
const uniqueTitles = [...new Set(titles)].sort();
const titlesUl = document.createElement("ul");
uniqueTitles.forEach(title => {
const titleLi = document.createElement("li");
titleLi.textContent = title;
titlesUl.appendChild(titleLi);
});
document.body.innerHTML = "";
document.body.appendChild(titlesUl);
}
fetchPages();

The try...catch statement contains Promises.all(). This is a non-blocking asynchronous function. The try...catch only applies to initializing the Promises.all() function, it does not catch resolved Promises.
Also note that you are sending all fetch requests in parallel, this may overload the server somewhat. It would be more polite to fire requests one by one using async await.

Javascript won't wait for firebase data to load

Im retrieving some data from firebase but the code won't wait for the data, it just keep running and some time later it retrieves the data. Im trying with await async but still... Whats wrong? Thanks!
var interlocutor = this.getRandomInterlocutor();
FriendlyChat.prototype.getRandomInterlocutor = async function(){
var numberOfUsers = await this.getNumberOfUsersRef();
console.log('numberOfUsers = ' + numberOfUsers);
var randomIndex = Math.floor(Math.random() * numberOfUsers);
console.log('randomIndex = ' + randomIndex);
var ref = await firebase.database().ref('companies/' + this.company + '/users/');
ref.limitToFirst(randomIndex).limitToLast(1).once('value').then(snapshot =>
{
var user = snapshot.val();
console.log('getRandomInterlocutor = ' + user);
});
}
FriendlyChat.prototype.getNumberOfUsersRef = async function(){
var numberOfUsersRef = await firebase.database().ref('companies/' + this.company + '/countregs');
var numberOfUsers;
numberOfUsersRef.on('value', function(snapshot) {
console.log(snapshot.val());
numberOfUsers = snapshot.val();
return numberOfUsers;
}, function (errorObject) {
console.log("The read failed: " + errorObject.code);
});
}

There are several things wrong here:
You are await-ing on the ref call, which is not async.
You are using on('value') which emits a stream of values, not a single read.
You are using callback-style code when async/await should be using Promises.
Here's a fixed version of your second function to demonstrate better practices:
FriendlyChat.prototype.getNumberOfUsersRef = async function(){
var numberOfUsersRef = firebase.database()
.ref('companies')
.child(this.company)
.child('countregs');
try {
var snapshot = await numberOfUsersRef.once('value');
return snapshot.val();
} catch (errorObject) {
console.log('The read failed:', errorObject.stack);
}
}