We Keep Coding

how find which element cause error in promis.map and log it in console, or skip it by code?

I using following code, but some URL from URL List, get 500 Error because of structure of the page.
I get the error exactly on .map((htmlOnePage, index) => line when some URLs page not valid, and the flow control of program goes at Catch Part. How I can find which URL is invalid?
const requestPromise = require('request-promise');
const Promise = require('bluebird');
const cheerio = require('cheerio');
for (var i = 1; i <= 250; i++) {
p = "https://mywebsite.com/" + i.toString()
urls[i - 1] = p
}
Promise.map(urls, requestPromise)
.map((htmlOnePage, index) => {
const $ = cheerio.load(htmlOnePage);
$('.txtSearch1').each(function() {
var h = "";
h = $(this).text()
h = h.replace(/(\r\n|\n|\r)/gm, "")
html44.push(h)
})
shareTuple[urls[index]] = html44;
html44 = []
fs.writeFileSync("data.json", JSON.stringify(shareTuple))
}, {
concurrency: 1
})
.then()
.catch((e) => console.log('We encountered an error' + e));
in other word how I can show which URL get into catch?

You should add try catch phrases inside all iterating functions to pin point the problem. and do the logging for every one based on op that they are catching.
For example i would wrap it like this:
try {
const $ = cheerio.load(htmlOnePage);
$('.txtSearch1').each(function () {
try {
var h="";
h=$(this).text()
h= h.replace(/(\r\n|\n|\r)/gm, "")
html44.push (h)
}catch (error) {
console.log('Error in getting p text');
console.log(error);
}
} catch (error) {
console.log('Error in loading'+ htmlOnePage);
console.log(error);
}
You can break down the error more by looking up through error object values to find the problem if you want to manualy remove it.
The other way to programaticaly remove it is to try doing the request for the page after creating it and also wrapping it in try catch. If it throws an exception you can add in catch to remove it from the list.
That text before console log of error is just so you can see where it broke.

Use a wrapper around requestPromise that catches the error. Note, the return undefined is not really needed. It's just for clarification, that in case of an error nothing is returned.
const requestPromise = require('request-promise');
....
const noThrowRequest = async (url) => {
try {
return await requestPromise(url);
} catch (e) {
return undefined;
}
}
Or if you prefer .then().catch() you can do it as follows
const noThrowRequest = (url) => {
return requestPromise(url)
.then(result => { return result; })
.catch(e => { return undefined; });
}
And then use that wrapper instead of requestPromise and check whether the current result valid or not. I don't know what you want to do in case of an invalid result, so I just return from the callback without any further ado. Adapt that if necessary.
Promise.map(urls, noThrowRequest)
.map((htmlOnePage, index) => {
if (htmlOnePage === undefined) {
console.log(`there was an error at index ${index}`);
return; //stop callback for erronous indexes.
}
...
}

Undefined errors using Node.js, Mongoose, and Discord.js [Cannot read property of undefined]

I've been scouring similar problems but haven't seem to have found a solution that quite works on my end. So I'm working on a Discord bot that takes data from a MongoDB database and displays said data in the form of a discord embedded message using Mongoose. For the most part, everything is working fine, however one little section of my code is giving me trouble.
So I need to import an array of both all available users and the "time" data of each of those users. Here is the block of code I use to import said data:
for (i = 0;i < totalObj; i++){
timeArray[i] = await getData('time', i);
userArray[i] = await getData('user', i);
}
Now this for loop references a function I made called getData which obtains the data from MongoDB by this method:
async function getData(field, value){
var data;
await stats.find({}, function(err, result){
if(err){
result.send(err);
}else{
data = result[value];
}
});
if(field == "user"){
return data.user;
}else if (field == "time"){
return data.time;
}else{
return 0;
}
So that for loop is where my errors currently lie. When I try to run this code and display my data through a discord message, I get this error and the message does not get sent:
(node:13936) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'time' of undefined
Now the strange thing is, this error does not happen every time. If I continue calling the command that triggers this code from my discord server, it's almost like a 50/50 shot if the command actually shows the message or instead gives this error. It is very inconsistent.
This error is confounding me, as the undefined part does not make sense to me. The objects that are being searched for in the mongoDB collection are definitely defined, and the for loop never exceeds the number of objects present. My only conclusion is that I'm doing something wrong with my asynchronous function design. I have tried altering code to use the getData function less often, or to not use awaits or asynchronous design at all, however this leaves my final discord message with several undefined variables and an eventual crash.
If anyone has any advice or suggestions, that would be very much appreciated. Just for reference, here is the full function that receives the data, sorts it, and prepares a string to be displayed on the discord server (though the error only seems to occur in the first for loop):
async function buildString(){
var string = "";
var totalObj;
var timeArray = [];
var userArray = [];
var stopSort = false;
await stats.find({}, function(err, result){
if(err){
result.send(err);
}else{
totalObj = result.length;
}
});
for (i = 0;i < totalObj; i++){
timeArray[i] = await getData('time', i);
userArray[i] = await getData('user', i);
}
while(!stopSort){
var keepSorting = false;
for(i = 0; i < totalObj ; i++){
var target = await convertTime(timeArray[i]);
for(j = i + 1 ; j < totalObj ; j++){
var comparison = await convertTime(timeArray[j]);
if(target > comparison){
//Switch target time with comparison time so that the lower time is up front
var temp = timeArray[i];
timeArray[i] = timeArray[j];
timeArray[j] = temp;
//Then switch the users around so that the user always corresponds with their time
var userTemp = userArray[i];
userArray[i] = userArray[j];
userArray[j] = userTemp;
//The loop will continue if even a single switch is made
keepSorting = true;
}
}
}
if(!keepSorting){
stopSort = true;
}
}
//String building starts here
var placeArray = [':first_place: **1st', ':second_place: **2nd', ':third_place: **3rd', '**4th', '**5th', '**6th', '**7th', '**8th', '**9th', '**10th'];
for(i = 0; i < totalObj; i++){
string = await string.concat(placeArray[i] + ": " + userArray[i] + "** - " + timeArray[i] + " \n\n");
console.log('butt');
}
console.log("This String:" + string);
return string;
}

I think problem is you are trying to await function with callback, it will not work => access to data.time may run before data = result[value]. If you need await callback, you can use custom Promise (or use util.promisify, more info here)
Promise:
function findStats(options) {
return new Promise((resolve, reject) => {
return stats.find(options, function (err, result) {
if (err) {
return reject(err)
}
return resolve(result)
})
})
}
utils.promisify
const util = require('util');
const findStats = util.promisify(stats.find);
Now you can use await in your function
async function getData(field, value) {
try {
const result = await findStats({})
const data = result.value
if (field === 'user') {
return data.user
}
if (field === 'time') {
return data.time
}
return 0
} catch (error) {
// here process error the way you like
// or remove try-catch block and sanitize error in your wrap function
}
}

How to call web service endpoints one after other?

I want to implement a function in JavaScript which calls a series of web service endpoint and checks for a value in the response of the API call.
I need to achieve it in a way that the first endpoint page is called first then the there would be a filter method to filter out the specific object from the response. If the object is found, this process should break and the object must be returned. However if the object is not found in the first endpoint, then the second endpoint must be called and the same process is repeated until the object is found.
The Web service endpoint that I am working on is:
https://jsonmock.hackerrank.com/api/countries?page=1
This API returns a list of country data. Here the value of page query varies from 1 to 25. I need to call the endpoint and check for a specific country from 1 to 25 until the country object is found.
I tried achieving this using JavaScript Promise and Fetch API and couldn't think of a way to call the APIs one after the other.
I am really looking forward for your answer. Thank you in advance.

You can use async and await for this:
async function findCountry(country) {
for (let page = 1; page < 26; page++) {
console.log("page = " + page); // for debugging only
let response = await fetch("https://jsonmock.hackerrank.com/api/countries?page=" + page);
let {data} = await response.json();
let obj = data.find(obj => obj.name == country);
if (obj) return obj;
}
}
let country = "Belgium";
findCountry(country).then(obj => {
if (obj) {
console.log("The capital of " + country + " is " + obj.capital);
} else {
console.log("Could not find " + country);
}
});
If you know that the data is sorted by country name, then you could reduce the average number of requests by using a binary search.

Here's a way that you can do it.
const url = 'https://jsonmock.hackerrank.com/api/countries'
const fetchFromApi = async (countryName, page) => {
const res = await fetch(`${url}?page=${page}`)
return await res.json()
}
const getCountryFromResults = (countryName, data) => {
const country = countryName.toLowerCase()
return data.find(({name}) => name.toLowerCase() === country)
}
const findCountry = async (countryName) => {
let page = 1;
let totalPages = 1;
while(page <= totalPages) {
const res = await fetchFromApi(countryName, page);
if(totalPages < res.total_pages) {
totalPages = res.total_pages
}
const country = getCountryFromResults(countryName, res.data)
if(country){
return country
}
page = page + 1
}
}
( async () => {
console.log(await findCountry("Afghanistan"))
console.log(await findCountry("Argentina"))
}
)()

Cannot read property 'length' of undefined...Although the object is defined

First I have the following code section:
export default class Recipe {
constructor(recID) {
this.RecipeID = recID;
}
async GetRecipe() {
try {
let Result = await Axios(
`https://forkify-api.herokuapp.com/api/get?rId=${this.RecipeID}`
);
this.Tilte = Result.data.recipe.title;
this.Author = Result.data.recipe.publisher;
this.Image = Result.data.recipe.image_url;
this.Url = Result.data.recipe.source_url;
this.Ingredients = Result.data.recipe.ingredients;
this.PublisherUrl = Result.data.recipe.publisher_url;
this.Rank = Result.data.recipe.social_rank;
} catch (error) {
alert(error);
}
}
CalculateTime() {
try {
this.Time = Math.ceil(this.Ingredients.length / 3) * 15; // error is here
} catch (error) {
console.log(this.RecipeID + ": Length Error->"+error);
}
}
}
Then I call the above code in a separate file like:
import Recipe from "./Recipe";
const RecipeController = async () => {
const ID = window.location.hash.replace("#", "");
if (ID) {
AppState.Recipe = new Recipe(ID);
try {
await AppState.Recipe.GetRecipe();
AppState.Recipe.CalculateTime();
console.log(AppState.Recipe);
} catch (error) {
alert(error);
}
}
};
Now as shown in the following image, that I do get the response of the request & promised is resolved plus there are elements in the 'ingredients' array but sometimes I still get the error "cannot read property 'length' of undefined" when I call the 'CalculateTime()' although the array is now defined and sometimes I don't get any error & it works perfectly fine.Why this random behavior? Even the IDs in the JSON response & the error I logged match i.e. 47746.

This is one reason why having too many try/catches can obscure the causes of errors, making debugging difficult. The problem can be reduced to the following:
class Recipe {
constructor(recID) {
this.RecipeID = recID;
}
async GetRecipe() {
let Result = await fetch(
`https://forkify-api.herokuapp.com/api/get?rId=47746`
).then(res => res.json());
console.log(Result); // <----- look at this log
this.Tilte = Result.data.recipe.title;
// on the above line, the error is thrown
// Cannot read property 'recipe' of undefined
}
}
const r = new Recipe();
r.GetRecipe();
See the log: your Result object does not have a .data property, so referencing Result.data.recipe throws an error. Try Result.recipe instead:
class Recipe {
constructor(recID) {
this.RecipeID = recID;
}
async GetRecipe() {
let Result = await fetch(
`https://forkify-api.herokuapp.com/api/get?rId=47746`
).then(res => res.json());
const { recipe } = Result;
this.Tilte = recipe.title;
this.Author = recipe.publisher;
this.Image = recipe.image_url;
this.Url = recipe.source_url;
this.Ingredients = recipe.ingredients;
this.PublisherUrl = recipe.publisher_url;
this.Rank = recipe.social_rank;
}
CalculateTime() {
this.Time = Math.ceil(this.Ingredients.length / 3) * 15; // error is here
console.log('got time', this.Time);
}
}
(async () => {
const r = new Recipe();
await r.GetRecipe();
r.CalculateTime();
})();
Unless you can actually handle an error at a particular point, it's usually good to allow the error to percolate upwards to the caller, so that the caller can see that there was an error, and handle it if it can. Consider changing your original code so that RecipeController (and only RecipeController) can see errors and deal with them - you can remove the try/catches from the Recipe.

Are you sure some of the responses are not missing ingredients? And calculateTime is always called after getRecipe?
I would add a condition or fallback to prevent errors, as in.
this.Time = Math.ceil((this.Ingredients || []).length / 3) * 15;

I would speculate that this "random" behavior can be related to asynchronous code. You need to ensure that the class has Ingredients in place before you calculate. I have a feeling that you should try changing your syntax to a Promise handling using .then() and .catch(), especially since you already use try/catch in your code. This approach will ensure proper resolution of Promise on axios request, and will eliminate "randomness", because you will have a better control over different stages of Promise processing.
let Result = await Axios(
`https://forkify-api.herokuapp.com/api/get?rId=${this.RecipeID}`
)
.then((data) => {
this.Tilte = data.data.recipe.title;
this.Author = data.data.recipe.publisher;
this.Image = data.data.recipe.image_url;
this.Url = data.data.recipe.source_url;
this.Ingredients = data.data.recipe.ingredients;
this.PublisherUrl = data.data.recipe.publisher_url;
this.Rank = data.data.recipe.social_rank;
this.Ingerdients = data.data.recipe.ingredient;
}
.catch((err) => {
console.log(err);
return null;
});

Trying to figure out why a thrown error is not caught

Here's the complete code I'm trying to run. I will, however, censor the url of the website for personal reasons. I am trying to scrape titles from a very slow website which occasionally sends error status codes in the 4xx range because that, so to handle that I throw an error then retry fetching the same pages after a couple of seconds. The problem is that this error is never caught by the "catch" block. Any idea what I am doing wrong?
const URL = "https://webpage.com/page=";
const SELECTOR = ".post-title.entry-title>a";
const MAX_CONCURRENT_FETCH = 5;
const NB_OF_PAGES = 125;
const ERROR_WAIT_TIME = 20000;
const titles = [];
const parser = new DOMParser();
function fetchPages() {
helper(1, MAX_CONCURRENT_FETCH);
function helper(first, last) {
const requests = [];
console.log("\n" + "*".repeat(40));
for (let i = first; i <= last; i++) {
requests.push(fetch(URL + i));
console.log(`Fetching page: ${i}`);
}
console.log("*".repeat(40) + "\n");
try {
Promise.all(requests).then(responses => {
responses.forEach(async(response, i) => {
/* if no code errors, parse the page, extract the titles then add them to the "titles" array */
if (response.ok) {
const htmlPage = await response.text();
console.log("\n" + "*".repeat(40));
console.log(`Extracting titles from page: ${first+i}`);
console.log("*".repeat(40) + "\n");
const htmlObject = parser.parseFromString(htmlPage, "text/html");
htmlObject.querySelectorAll(SELECTOR).forEach(node => {
titles.push(node.textContent);
console.log(`Title: ${node.textContent}`);
});
}
/* code error, get out of the forEach method by throwing an error */
else {
console.log("*".repeat(40) + "\n");
console.log("Throwing error...");
throw {
response: response
}
}
});
/* keep fetching until the last page */
if (last + 1 < NB_OF_PAGES)
helper(last + 1, last + MAX_CONCURRENT_FETCH);
/* once all pages have been fetched, show the result on screen */
else showResult();
});
}
/* catch the error that was throw inside the forEach method, show the status code error and how long before the pages will be fetched again */
catch (err) {
console.log("Error captured...");
console.log(`Status Code: ${err.response.status}, retrying in ${ERROR_WAIT_TIME/1000} seconds.`);
console.log("*".repeat(40) + "\n");
setTimeout(() => helper(first, last), ERROR_WAIT_TIME);
}
}
}
function showResult() {
const uniqueTitles = [...new Set(titles)].sort();
const titlesUl = document.createElement("ul");
uniqueTitles.forEach(title => {
const titleLi = document.createElement("li");
titleLi.textContent = title;
titlesUl.appendChild(titleLi);
});
document.body.innerHTML = "";
document.body.appendChild(titlesUl);
}
fetchPages();

The try...catch statement contains Promises.all(). This is a non-blocking asynchronous function. The try...catch only applies to initializing the Promises.all() function, it does not catch resolved Promises.
Also note that you are sending all fetch requests in parallel, this may overload the server somewhat. It would be more polite to fire requests one by one using async await.