First I have the following code section:
export default class Recipe {
constructor(recID) {
this.RecipeID = recID;
}
async GetRecipe() {
try {
let Result = await Axios(
`https://forkify-api.herokuapp.com/api/get?rId=${this.RecipeID}`
);
this.Tilte = Result.data.recipe.title;
this.Author = Result.data.recipe.publisher;
this.Image = Result.data.recipe.image_url;
this.Url = Result.data.recipe.source_url;
this.Ingredients = Result.data.recipe.ingredients;
this.PublisherUrl = Result.data.recipe.publisher_url;
this.Rank = Result.data.recipe.social_rank;
} catch (error) {
alert(error);
}
}
CalculateTime() {
try {
this.Time = Math.ceil(this.Ingredients.length / 3) * 15; // error is here
} catch (error) {
console.log(this.RecipeID + ": Length Error->"+error);
}
}
}
Then I call the above code in a separate file like:
import Recipe from "./Recipe";
const RecipeController = async () => {
const ID = window.location.hash.replace("#", "");
if (ID) {
AppState.Recipe = new Recipe(ID);
try {
await AppState.Recipe.GetRecipe();
AppState.Recipe.CalculateTime();
console.log(AppState.Recipe);
} catch (error) {
alert(error);
}
}
};
Now as shown in the following image, that I do get the response of the request & promised is resolved plus there are elements in the 'ingredients' array but sometimes I still get the error "cannot read property 'length' of undefined" when I call the 'CalculateTime()' although the array is now defined and sometimes I don't get any error & it works perfectly fine.Why this random behavior? Even the IDs in the JSON response & the error I logged match i.e. 47746.
This is one reason why having too many try/catches can obscure the causes of errors, making debugging difficult. The problem can be reduced to the following:
class Recipe {
constructor(recID) {
this.RecipeID = recID;
}
async GetRecipe() {
let Result = await fetch(
`https://forkify-api.herokuapp.com/api/get?rId=47746`
).then(res => res.json());
console.log(Result); // <----- look at this log
this.Tilte = Result.data.recipe.title;
// on the above line, the error is thrown
// Cannot read property 'recipe' of undefined
}
}
const r = new Recipe();
r.GetRecipe();
See the log: your Result object does not have a .data property, so referencing Result.data.recipe throws an error. Try Result.recipe instead:
class Recipe {
constructor(recID) {
this.RecipeID = recID;
}
async GetRecipe() {
let Result = await fetch(
`https://forkify-api.herokuapp.com/api/get?rId=47746`
).then(res => res.json());
const { recipe } = Result;
this.Tilte = recipe.title;
this.Author = recipe.publisher;
this.Image = recipe.image_url;
this.Url = recipe.source_url;
this.Ingredients = recipe.ingredients;
this.PublisherUrl = recipe.publisher_url;
this.Rank = recipe.social_rank;
}
CalculateTime() {
this.Time = Math.ceil(this.Ingredients.length / 3) * 15; // error is here
console.log('got time', this.Time);
}
}
(async () => {
const r = new Recipe();
await r.GetRecipe();
r.CalculateTime();
})();
Unless you can actually handle an error at a particular point, it's usually good to allow the error to percolate upwards to the caller, so that the caller can see that there was an error, and handle it if it can. Consider changing your original code so that RecipeController (and only RecipeController) can see errors and deal with them - you can remove the try/catches from the Recipe.
Are you sure some of the responses are not missing ingredients? And calculateTime is always called after getRecipe?
I would add a condition or fallback to prevent errors, as in.
this.Time = Math.ceil((this.Ingredients || []).length / 3) * 15;
I would speculate that this "random" behavior can be related to asynchronous code. You need to ensure that the class has Ingredients in place before you calculate. I have a feeling that you should try changing your syntax to a Promise handling using .then() and .catch(), especially since you already use try/catch in your code. This approach will ensure proper resolution of Promise on axios request, and will eliminate "randomness", because you will have a better control over different stages of Promise processing.
let Result = await Axios(
`https://forkify-api.herokuapp.com/api/get?rId=${this.RecipeID}`
)
.then((data) => {
this.Tilte = data.data.recipe.title;
this.Author = data.data.recipe.publisher;
this.Image = data.data.recipe.image_url;
this.Url = data.data.recipe.source_url;
this.Ingredients = data.data.recipe.ingredients;
this.PublisherUrl = data.data.recipe.publisher_url;
this.Rank = data.data.recipe.social_rank;
this.Ingerdients = data.data.recipe.ingredient;
}
.catch((err) => {
console.log(err);
return null;
});
Related
I using following code, but some URL from URL List, get 500 Error because of structure of the page.
I get the error exactly on .map((htmlOnePage, index) => line when some URLs page not valid, and the flow control of program goes at Catch Part. How I can find which URL is invalid?
const requestPromise = require('request-promise');
const Promise = require('bluebird');
const cheerio = require('cheerio');
for (var i = 1; i <= 250; i++) {
p = "https://mywebsite.com/" + i.toString()
urls[i - 1] = p
}
Promise.map(urls, requestPromise)
.map((htmlOnePage, index) => {
const $ = cheerio.load(htmlOnePage);
$('.txtSearch1').each(function() {
var h = "";
h = $(this).text()
h = h.replace(/(\r\n|\n|\r)/gm, "")
html44.push(h)
})
shareTuple[urls[index]] = html44;
html44 = []
fs.writeFileSync("data.json", JSON.stringify(shareTuple))
}, {
concurrency: 1
})
.then()
.catch((e) => console.log('We encountered an error' + e));
in other word how I can show which URL get into catch?
You should add try catch phrases inside all iterating functions to pin point the problem. and do the logging for every one based on op that they are catching.
For example i would wrap it like this:
try {
const $ = cheerio.load(htmlOnePage);
$('.txtSearch1').each(function () {
try {
var h="";
h=$(this).text()
h= h.replace(/(\r\n|\n|\r)/gm, "")
html44.push (h)
}catch (error) {
console.log('Error in getting p text');
console.log(error);
}
} catch (error) {
console.log('Error in loading'+ htmlOnePage);
console.log(error);
}
You can break down the error more by looking up through error object values to find the problem if you want to manualy remove it.
The other way to programaticaly remove it is to try doing the request for the page after creating it and also wrapping it in try catch. If it throws an exception you can add in catch to remove it from the list.
That text before console log of error is just so you can see where it broke.
Use a wrapper around requestPromise that catches the error. Note, the return undefined is not really needed. It's just for clarification, that in case of an error nothing is returned.
const requestPromise = require('request-promise');
....
const noThrowRequest = async (url) => {
try {
return await requestPromise(url);
} catch (e) {
return undefined;
}
}
Or if you prefer .then().catch() you can do it as follows
const noThrowRequest = (url) => {
return requestPromise(url)
.then(result => { return result; })
.catch(e => { return undefined; });
}
And then use that wrapper instead of requestPromise and check whether the current result valid or not. I don't know what you want to do in case of an invalid result, so I just return from the callback without any further ado. Adapt that if necessary.
Promise.map(urls, noThrowRequest)
.map((htmlOnePage, index) => {
if (htmlOnePage === undefined) {
console.log(`there was an error at index ${index}`);
return; //stop callback for erronous indexes.
}
...
}
I am trying to fetch JSON data from the WordPress Developer Reference site. I need to search a keyword without knowing if it's a function, class, hook, or method, which is part of the url I need to fetch. So I'm using Promise.all to cycle through all possible urls. It works if the response.status <= 299, throwing the error immediately, and if the response is ok, then it continues to .then. Fine, but occasionally it will return an ok status if the JSON exists and only returns an empty array. So I need to check if the JSON data is an empty array, which I can't seem to do in the first part. I can only check in the second part as far as I know. And if it throws the error it doesn't continue trying the other urls. Any suggestions?
var keyword = 'AtomParser';
const refs = ['function', 'hook', 'class', 'method'];
// Store the promises
let promises = [];
// Cycle through each type until we find one we're looking for
for (let t = 0; t < refs.length; t++) {
const url =
'https://developer.wordpress.org/wp-json/wp/v2/wp-parser-' +
refs[t] +
'?search=' +
keyword;
// console.log(url);
promises.push(fetch(url));
}
Promise.all(promises)
.then(function(response) {
console.log(response[0]);
// Get the status
console.log('Status code: ' + response[0].status);
if (response[0].status <= 299) {
// The API call was successful!
return response[0].json();
} else {
throw new Error('Broken link status code: ' + response[0].status);
}
})
.then(function(data) {
// This is the HTML from our response as a text string
console.log(data);
// Make sure we have data
if (data.length == 0) {
throw new Error('Empty Array');
}
// ref
const reference = data[0];
// Only continue if not null or empty
if (reference !== null && reference !== undefined && data.length > 0) {
// Success
// Return what I want from the reference
}
})
.catch(function handleError(error) {
console.log('Error' + error);
});
Is there some way to get the JSON data in the first part so I can check if it's in an array while I'm checking the response status?
I would recommend encapsulating the success / failure logic for individual requests, then you can determine all the resolved and rejected responses based on the result of that encapsulation.
For example
const checkKeyword = async (ref, keyword) => {
const params = new URLSearchParams({ search: keyword });
const res = await fetch(
`https://developer.wordpress.org/wp-json/wp/v2/wp-parser-${encodeURIComponent(
ref
)}?${params}`
);
if (!res.ok) {
throw new Error(`${res.status}: ${await res.text()}`);
}
const data = await res.json();
if (data.length === 0) {
throw new Error(`Empty results for '${ref}'`);
}
return { ref, data };
};
Now you can use something like Promise.any() or Promise.allSettled() to find the first successful request or all successful requests, respectively
const keyword = "AtomParser";
const refs = ["function", "hook", "class", "method"];
const promises = refs.map((ref) => checkKeyword(ref, keyword));
// First success
Promise.any(promises)
.then(({ ref, data }) => {
console.log(ref, data);
})
.catch(console.error);
// All successes
Promise.allSettled(promises)
.then((responses) =>
responses.reduce(
(arr, { status, value }) =>
status === "fulfilled" ? [...arr, value] : arr,
[]
)
)
.then((results) => {
// results has all the successful responses
});
For whatever reason I couldn't get Phil's answer to work, so I ended up doing the following which works fine for me. (This is for a discord bot in case you're wondering what the other stuff is all about).
var keyword = 'AtomParser';
const refs = ['function', 'hook', 'class', 'method'];
// Store the successful result or error
let final: any[] = [];
let finalError = '';
// Cycle through each type until we find one we're looking for
for (let t = 0; t < refs.length; t++) {
const url =
'https://developer.wordpress.org/wp-json/wp/v2/wp-parser-' +
refs[t] +
'?search=' +
keyword;
console.log(url);
// Try to fetch it
await fetch(url)
.then(function (response) {
console.log(response);
// Get the status
console.log('Status code: ' + response.status);
if (response.status > 299) {
finalError = '`' + refs[t] + '` does not exist.';
throw new Error(finalError);
} else {
// The API call was successful!
return response.json();
}
})
.then(function (data) {
// This is the HTML from our response as a text string
console.log(data);
// Make sure we have data
if (data.length == 0) {
finalError = "Sorry, I couldn't find `" + keyword + '`';
throw new Error(finalError);
}
// Only continue if not null or empty
if (data[0] !== null && data[0] !== undefined && data.length > 0) {
for (let d = 0; d < data.length; d++) {
// Add it to the final array
final.push(data[d]);
}
}
})
.catch(function handleError(error) {
console.log(error);
});
}
if (final.length > 0) {
for (let f = 0; f < final.length; f++) {
// ref
const reference = final[f];
// Get the link
const link = reference.link;
// Get the title
var title = reference.title.rendered;
title = excerpt.replace('>', '>');
// Get the excerpt
var excerpt = reference.excerpt.rendered;
excerpt = excerpt.replace('<p>', '');
excerpt = excerpt.replace('</p>', '');
excerpt = excerpt.replace('<b>', '**');
excerpt = excerpt.replace('</b>', '**');
console.log(excerpt);
message.reply(
new discord.Embed({
title: `${title}`,
url: link,
description: `${excerpt}\n\n`,
footer: {
text: `WordPress Developer Code Reference\nhttps://developer.wordpress.org/`,
},
})
);
}
} else if (finalError != '') {
message.reply(finalError);
} else {
message.reply('Something went wrong...');
}
wp module
#Phil's answer puts you on the right track but I want to expand on some of his ideas. Use of URLSearchParamas is great but you can improve by using the high-level URL API and forego encodeURIComponent and constructing search params manually. Notice I'm putting this code in its own wp module so I can separate concerns more easily. We don't want all of this code leaking into your main program.
// wp.js
import { fetch } from "whatwg-fetch" // or your chosen implementation
const baseURL = "https://developer.wordpress.org"
async function search1(path, query) {
const u = new URL(path, baseURL)
u.searchParams.set("search", query)
const result = await fetch(u)
if (!result.ok) throw Error(`Search failed (${result.status}): ${u}`)
return result.json()
}
search1 searches one path, but we can write search to search all the necessary paths. I don't think there's any reason to get fancy with each path here, so just write them out -
// wp.js (continued)
function search(query) {
const endpoints = [
"/wp-json/wp/v2/wp-parser-function",
"/wp-json/wp/v2/wp-parser-hook",
"/wp-json/wp/v2/wp-parser-class",
"/wp-json/wp/v2/wp-parser-method"
]
return Promise
.all(endpoints.map(e => search1(e, query)))
.then(results => results.flat())
}
export { search }
main module
Notice we only exported search as search1 is internal to the wp module. Let's see how we can use it in our main module now -
// main.js
import { search } from "./wp.js"
for (const result of await search("database"))
if(result.guid.rendered)
console.log(`${result.title.rendered}\n${result.guid.rendered}\n`)
In this example, we first search for "database" -
wp_should_replace_insecure_home_url()
https://developer.wordpress.org/reference/functions/wp_should_replace_insecure_home_url/
wp_delete_signup_on_user_delete()
https://developer.wordpress.org/reference/functions/wp_delete_signup_on_user_delete/
get_post_datetime()
https://developer.wordpress.org/reference/functions/get_post_datetime/
wp_ajax_health_check_get_sizes()
https://developer.wordpress.org/reference/functions/wp_ajax_health_check_get_sizes/
wp_should_replace_insecure_home_url
https://developer.wordpress.org/reference/hooks/wp_should_replace_insecure_home_url/
comments_pre_query
https://developer.wordpress.org/reference/hooks/comments_pre_query/
users_pre_query
https://developer.wordpress.org/reference/hooks/users_pre_query/
WP_Object_Cache
http://developer.wordpress.org/reference/classes/wp_object_cache/
wpdb
http://developer.wordpress.org/reference/classes/wpdb/
WP_REST_Menu_Items_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_menu_items_controller/prepare_item_for_database/
WP_REST_Global_Styles_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_global_styles_controller/prepare_item_for_database/
WP_REST_Menus_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_menus_controller/prepare_item_for_database/
WP_REST_Templates_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_templates_controller/prepare_item_for_database/
WP_REST_Application_Passwords_Controller::prepare_item_for_database()
https://developer.wordpress.org/reference/classes/wp_rest_application_passwords_controller/prepare_item_for_database/
wpdb::db_server_info()
https://developer.wordpress.org/reference/classes/wpdb/db_server_info/
WP_REST_Attachments_Controller::insert_attachment()
https://developer.wordpress.org/reference/classes/wp_rest_attachments_controller/insert_attachment/
WP_Debug_Data::get_database_size()
https://developer.wordpress.org/reference/classes/wp_debug_data/get_database_size/
WP_REST_Meta_Fields::update_multi_meta_value()
https://developer.wordpress.org/method/wp_rest_meta_fields/update_multi_meta_value/
another search example
Now let's search for "image" -
for (const result of await search("image"))
if(result.guid.rendered)
console.log(`${result.title.rendered}\n${result.guid.rendered}\n`)
get_adjacent_image_link()
https://developer.wordpress.org/reference/functions/get_adjacent_image_link/
get_next_image_link()
https://developer.wordpress.org/reference/functions/get_next_image_link/
get_previous_image_link()
https://developer.wordpress.org/reference/functions/get_previous_image_link/
wp_robots_max_image_preview_large()
https://developer.wordpress.org/reference/functions/wp_robots_max_image_preview_large/
wp_getimagesize()
https://developer.wordpress.org/reference/functions/wp_getimagesize/
is_gd_image()
https://developer.wordpress.org/reference/functions/is_gd_image/
wp_show_heic_upload_error()
https://developer.wordpress.org/reference/functions/wp_show_heic_upload_error/
wp_image_src_get_dimensions()
https://developer.wordpress.org/reference/functions/wp_image_src_get_dimensions/
wp_image_file_matches_image_meta()
https://developer.wordpress.org/reference/functions/wp_image_file_matches_image_meta/
_wp_check_existing_file_names()
https://developer.wordpress.org/reference/functions/_wp_check_existing_file_names/
edit_custom_thumbnail_sizes
https://developer.wordpress.org/reference/hooks/edit_custom_thumbnail_sizes/
get_header_image_tag_attributes
https://developer.wordpress.org/reference/hooks/get_header_image_tag_attributes/
image_editor_output_format
https://developer.wordpress.org/reference/hooks/image_editor_output_format/
wp_image_src_get_dimensions
https://developer.wordpress.org/reference/hooks/wp_image_src_get_dimensions/
wp_get_attachment_image
https://developer.wordpress.org/reference/hooks/wp_get_attachment_image/
image_sideload_extensions
https://developer.wordpress.org/reference/hooks/image_sideload_extensions/
wp_edited_image_metadata
https://developer.wordpress.org/reference/hooks/wp_edited_image_metadata/
wp_img_tag_add_loading_attr
https://developer.wordpress.org/reference/hooks/wp_img_tag_add_loading_attr/
wp_image_file_matches_image_meta
https://developer.wordpress.org/reference/hooks/wp_image_file_matches_image_meta/
get_custom_logo_image_attributes
https://developer.wordpress.org/reference/hooks/get_custom_logo_image_attributes/
Custom_Image_Header
http://developer.wordpress.org/reference/classes/custom_image_header/
WP_Image_Editor_Imagick
http://developer.wordpress.org/reference/classes/wp_image_editor_imagick/
WP_Embed
http://developer.wordpress.org/reference/classes/wp_embed/
WP_Image_Editor
http://developer.wordpress.org/reference/classes/wp_image_editor/
WP_Customize_Background_Image_Setting
http://developer.wordpress.org/reference/classes/wp_customize_background_image_setting/
WP_Customize_Header_Image_Setting
http://developer.wordpress.org/reference/classes/wp_customize_header_image_setting/
WP_Image_Editor_GD
http://developer.wordpress.org/reference/classes/wp_image_editor_gd/
WP_Customize_Header_Image_Control
http://developer.wordpress.org/reference/classes/wp_customize_header_image_control/
WP_REST_Server::add_image_to_index()
https://developer.wordpress.org/reference/classes/wp_rest_server/add_image_to_index/
WP_REST_URL_Details_Controller::get_image()
https://developer.wordpress.org/reference/classes/wp_rest_url_details_controller/get_image/
WP_Image_Editor::get_default_quality()
https://developer.wordpress.org/reference/classes/wp_image_editor/get_default_quality/
WP_Theme_JSON::get_blocks_metadata()
https://developer.wordpress.org/reference/classes/wp_theme_json/get_blocks_metadata/
WP_Image_Editor_Imagick::pdf_load_source()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/pdf_load_source/
WP_Image_Editor_Imagick::write_image()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/write_image/
WP_Image_Editor_Imagick::maybe_exif_rotate()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/maybe_exif_rotate/
WP_Image_Editor_Imagick::make_subsize()
https://developer.wordpress.org/reference/classes/wp_image_editor_imagick/make_subsize/
WP_Image_Editor_GD::make_subsize()
https://developer.wordpress.org/reference/classes/wp_image_editor_gd/make_subsize/
empty search result
Searching for "zzz" will yield no results -
for (const result of await search("zzz"))
if(result.guid.rendered)
console.log(`${result.title.rendered}\n${result.guid.rendered}\n`)
<empty result>
Here's the complete code I'm trying to run. I will, however, censor the url of the website for personal reasons. I am trying to scrape titles from a very slow website which occasionally sends error status codes in the 4xx range because that, so to handle that I throw an error then retry fetching the same pages after a couple of seconds. The problem is that this error is never caught by the "catch" block. Any idea what I am doing wrong?
const URL = "https://webpage.com/page=";
const SELECTOR = ".post-title.entry-title>a";
const MAX_CONCURRENT_FETCH = 5;
const NB_OF_PAGES = 125;
const ERROR_WAIT_TIME = 20000;
const titles = [];
const parser = new DOMParser();
function fetchPages() {
helper(1, MAX_CONCURRENT_FETCH);
function helper(first, last) {
const requests = [];
console.log("\n" + "*".repeat(40));
for (let i = first; i <= last; i++) {
requests.push(fetch(URL + i));
console.log(`Fetching page: ${i}`);
}
console.log("*".repeat(40) + "\n");
try {
Promise.all(requests).then(responses => {
responses.forEach(async(response, i) => {
/* if no code errors, parse the page, extract the titles then add them to the "titles" array */
if (response.ok) {
const htmlPage = await response.text();
console.log("\n" + "*".repeat(40));
console.log(`Extracting titles from page: ${first+i}`);
console.log("*".repeat(40) + "\n");
const htmlObject = parser.parseFromString(htmlPage, "text/html");
htmlObject.querySelectorAll(SELECTOR).forEach(node => {
titles.push(node.textContent);
console.log(`Title: ${node.textContent}`);
});
}
/* code error, get out of the forEach method by throwing an error */
else {
console.log("*".repeat(40) + "\n");
console.log("Throwing error...");
throw {
response: response
}
}
});
/* keep fetching until the last page */
if (last + 1 < NB_OF_PAGES)
helper(last + 1, last + MAX_CONCURRENT_FETCH);
/* once all pages have been fetched, show the result on screen */
else showResult();
});
}
/* catch the error that was throw inside the forEach method, show the status code error and how long before the pages will be fetched again */
catch (err) {
console.log("Error captured...");
console.log(`Status Code: ${err.response.status}, retrying in ${ERROR_WAIT_TIME/1000} seconds.`);
console.log("*".repeat(40) + "\n");
setTimeout(() => helper(first, last), ERROR_WAIT_TIME);
}
}
}
function showResult() {
const uniqueTitles = [...new Set(titles)].sort();
const titlesUl = document.createElement("ul");
uniqueTitles.forEach(title => {
const titleLi = document.createElement("li");
titleLi.textContent = title;
titlesUl.appendChild(titleLi);
});
document.body.innerHTML = "";
document.body.appendChild(titlesUl);
}
fetchPages();
The try...catch statement contains Promises.all(). This is a non-blocking asynchronous function. The try...catch only applies to initializing the Promises.all() function, it does not catch resolved Promises.
Also note that you are sending all fetch requests in parallel, this may overload the server somewhat. It would be more polite to fire requests one by one using async await.
I have posted a question in this post earlier. It finally shows some output after changing synchronous functions into asynchronous; however, it turns out that in the output array, some of the elements are null. I am not sure why?
the main code looks like this:
router.post('/form1', async (req, res, next)=>{
try{
const emdate = new Date(req.body.emdate);
const address = req.body.address;
const stationDataCursor = stationData.filteredData(instantData, emdate);
stationDataCursor.toArray().then(async (docArr)=>{
return await Promise.all(docArr.map(async function(val, ind){
try {
const feature = await calcDSV.calcDSV(val);
return feature
} catch (error) {
console.log(ind);
console.log("Error happened in data array", error);
}
}));
}).then((doc)=>{
res.json(doc)
})
} catch(error){
res.status(400).send(error)
}
})
I tried add Promise.resolve(featureJSON) in the return of calcDSV(val)
async function calcDSV(featuresJSON){
// featuresJSON
const SVscore = [];
const tuEval = featuresJSON.features.properties.TU90; // array
const obArr = featuresJSON.features.properties.OB; // array
const periodObj = await getPeriods(tuEval);// get period position
const paramObj = await getParams(periodObj, obArr); // get parameters
const periodDate = await getPeriodDate(featuresJSON, periodObj);
const removeTime = periodDate.beginDate.map(x=>x.split('T')[0]);
let hourly = paramObj.hourCounts;
let avgTemps = paramObj.avgTemps;
for(let i = 0;i<hourly.length; i++){
let score = await assignScore(avgTemps[i], hourly[i]);
SVscore.push(score);
}
// output sv score for date
const aggreScore = await accumScore(removeTime, SVscore);
aggreScore.DSVdate = aggreScore.Date.map(x=>new Date(x));
featuresJSON.features.properties.periodSV = SVscore;
featuresJSON.features.properties.Periods = periodDate;
featuresJSON.features.properties.DSVscore = aggreScore;
return Promise.resolve(featuresJSON);
}
module.exports.calcDSV = calcDSV;
The error message:
2
Error happened in data array ReferenceError: error is not defined
at getParams (/media/swaggyp1985/HDD4T/OSU_Projects_2017-2018/App_Projects/wallin_model/dev/dev_js/wallin-server/src/utils/periodFunc.js:145:9)
at Object.calcDSV (/media/swaggyp1985/HDD4T/OSU_Projects_2017-2018/App_Projects/wallin_model/dev/dev_js/wallin-server/src/utils/wallinLogic.js:842:32)
at processTicksAndRejections (internal/process/task_queues.js:86:5)
6
Error happened in data array ReferenceError: error is not defined
at getParams (/media/swaggyp1985/HDD4T/OSU_Projects_2017-2018/App_Projects/wallin_model/dev/dev_js/wallin-server/src/utils/periodFunc.js:145:9)
at Object.calcDSV (/media/swaggyp1985/HDD4T/OSU_Projects_2017-2018/App_Projects/wallin_model/dev/dev_js/wallin-server/src/utils/wallinLogic.js:842:32)
at processTicksAndRejections (internal/process/task_queues.js:86:5)
...
I expect every element would be resolved.
Thanks to the comments, I was able to find out the source: the throw error() in other functions I defined. It should be throw Error().
I'm trying to improve a firestore get function, I have something like:
return admin.firestore().collection("submissions").get().then(
async (x) => {
var toRet: any = [];
for (var i = 0; i < 10; i++) {
try {
var hasMedia = x.docs[i].data()['mediaRef'];
if (hasMedia != null) {
var docData = (await x.docs[i].data()) as MediaSubmission;
let submission: MediaSubmission = new MediaSubmission();
submission.author = x.docs[i].data()['author'];
submission.description = x.docs[i].data()['description'];
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
submission.media = mediaRef.data() as MediaData;
toRet.push(submission);
}
}
catch (e) {
console.log("ERROR GETTIGN MEDIA: " + e);
}
}
return res.status(200).send(toRet);
});
The first get is fine but the performance is worst on the line:
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
I think this is because the call is not batched.
Would it be possible to do a batch get on an array of mediaRefs to improve performance?
Essentially I have a collection of documents which have foreign references stored by a string pointing to the path in a separate collection and getting those references has been proven to be slow.
What about this? I did some refactoring to use more await/async code, hopefully my comments are helpful.
The main idea is to use Promise.all and await all the mediaRefs retrieval
async function test(req, res) {
// get all docs
const { docs } = await admin
.firestore()
.collection('submissions')
.get();
// get data property only of docs with mediaRef
const datas = await Promise.all(
docs.map(doc => doc.data()).filter(data => data.mediaRef),
);
// get all media in one batch - this is the important change
const mediaRefs = await Promise.all(
datas.map(({ mediaRef }) =>
admin
.firestore()
.doc(mediaRef)
.get(),
),
);
// create return object
const toRet = datas.map((data: MediaSubmission, i) => {
const submission = new MediaSubmission();
submission.author = data.author;
submission.description = data.description;
submission.media = mediaRefs[i].data() as MediaData;
return submission;
});
return res.status(200).send(toRet);
}