We Keep Coding

Return Response When First request failed And Try In Second Request

I try to explain the problem.in App.js I have Function getUser .when call this function.in first request get 401 error . For this reason in axios.interceptors.response I receive error 401.At this time, I receive a token and repeat my request again.And it is done successfully.But not return response in Function getUser.
I have hook for authentication and send request.
import React from "react";
import axios from "axios";
const API_URL = "http://127.0.0.1:4000/api/";
function useJWT() {
axios.interceptors.request.use(
(request) => {
request.headers.common["Accept"] = "application/json";
console.log("request Send ");
return request;
},
(error) => {
return Promise.reject(error);
}
);
axios.interceptors.response.use(
(response) => {
console.log("answer = ", response);
return response;
},
(error) => {
if (error?.response?.status) {
switch (error.response.status) {
case 401:
refreshToken().then((responseTwo) => {
return
sendPostRequest(
error.response.config.url
.split("/")
.findLast((item) => true)
.toString(),
error.response.config.data
);
});
break;
case 500:
// Actions for Error 500
throw error;
default:
console.error("from hook interceptor => ", error);
throw error;
}
} else {
// Occurs for axios error.message = 'Network Error'
throw error;
}
}
);
const refreshToken = () => {
const token = localStorage.getItem("refresh");
return axios
.post(API_URL + "token", {
token,
})
.then((response) => {
if (response.data.access) {
localStorage.setItem("access", response.data.access);
}
if (response.data.refresh) {
localStorage.setItem("refresh", response.data.refresh);
}
return response.data;
});
};
function login(email, password) {
return axios
.post(API_URL + "login", {
email,
password,
})
.then((response) => {
if (response.data.access) {
localStorage.setItem("access", response.data.access);
}
if (response.data.refresh) {
localStorage.setItem("refresh", response.data.refresh);
}
return response.data;
});
}
const sendPostRequest = (url, data) => {
console.log(300);
const token = localStorage.getItem("access");
axios.defaults.headers.common["jwt"] = token;
return axios.post(API_URL + url, {
data,
});
};
const logout = () => {
const token = localStorage.getItem("refresh");
return axios
.delete(API_URL + "logout", {
token,
})
.then((response) => {
localStorage.removeItem("access");
localStorage.removeItem("refresh");
});
};
return {
login,
logout,
refreshToken,
sendPostRequest,
};
}
export default useJWT;
In App.js ,I want to repeat the same request again if a 401 error is issued when I read the user information.
The request is successfully repeated but does not return the value.
When first request fail response is return equals null . and in catch when receive 401 error i am send second request but not return response.
I send request below code .
const getUser = () => {
console.log(12);
return sendPostRequest("user");
};
useEffect(() => {
let token = localStorage.getItem("access");
console.log("token = ", token);
if (token != null) {
//Here I have done simulation for 401 error
localStorage.setItem("access", "");
getUser()
.then((response) => {
console.log("response 1= ", response);
})
.catch((exception) => {
console.log("exception = ", exception.toString());
})
.then((response) => {
console.log("response 2= ", response);
});
} else {
navigate("/login");
}
}, []);
Best regards.

I didn't fully understand what exactly you want to do here.
But if you are looking to retry when 401 happens, you could use axios-retry to do it for you.
I'll pass the basics, but you can look more into what this does.
// First you need to create an axios instance
const axiosClient = axios.create({
baseURL: 'API_URL',
// not needed
timeout: 30000
});
// Then you need to add this to the axiosRetry lib
axiosRetry(axiosClient, {
retries: 3,
// Doesn't need to be this, it can be a number in ms
retryDelay: axiosRetry.exponentialDelay,
retryCondition: (error) => {
// You could do this way or try to implement your own
return error.response.status > 400
// something like this works too.
// error.response.status === 401 || error.response.status >= 500;
}
});
Just like in your code, we need to use interceptors if you want to avoid breaking your page, otherwise you can use try catch to catch any errors that may happen in a request.
// It could be something like this, like I said, it's not really needed.
axiosClient.interceptors.response.use(
(success) => success,
(err) => err
);
And finally, you could use the axiosClient directly since it now has your API_URL, calling it like this axiosClient.post('/user').
More or less that's it, you should just debug this code and see what is causing the return value to be null.
I would change these then/catch to be an async/await function, it would be more readable making your debugging easier.
axios-retry example if you didn't understand my explanation.

I find anwser for this question.
When error 401 occurs then create new Promise
I Wrote this code.
case 401:
return new Promise((resolve, reject) => {
refreshToken().then((responseTwo) => {
resolve(
sendPostRequest(
error.response.config.url
.split("/")
.findLast((item) => true)
.toString(),
error.response.config.data
)
);
});
});

AWS Lambda retries despite returning value

I have found out that my and my colleagues lambda doesn't return data we anticipate.
After what I have already found, we used deprecated invokeAsync which returnes only statusCode.
I have upgraded aws-sdk to the newest version and change the invocation code to use invoke with
InvocationType: 'RequestResponse'
as that should give us the returned value.
Lambdas code itself looks good, its an async function without arguments.
Im not using the callback here (3rd argument of handler), as we are doing some async stuff and it would require a small refactor to not use async/await, also i have read that just returning value is ok as well.
Earlier lambda was returning array, but after some investigation and googling I have changed it to
return {
statusCode: 200,
body: JSON.stringify(result)
}
where result is the mentioned array, as this was recurring pattern in search results.
Overall result is in our service where we do the invocation, we get timeout, lambda logs that it is returning value, but then retries itself again and again.
Dumping code
invocation in our service:
const lambda = new AWS.Lambda({ httpOptions: { timeout: 600000 } })
const results = await new Promise((resolve, reject) => {
lambda.invoke(
{
FunctionName: `lambda-name`,
InvocationType: 'RequestResponse',
Payload: '""'
},
(err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
}
)
})
lambda handler
import { parallelLimit } from 'async'
const PARALLEL_FETCHERS_NUMBER = 2
export async function runTasksInParallel (
tasks,
limit,
) {
return new Promise(
(
resolve,
reject,
) => {
parallelLimit(tasks, limit, (error, results) => {
if (error === null) {
resolve(results)
} else {
reject(error)
}
})
},
)
}
async function connectToDB(logger) {
const MONGO_URL = process.env.MONGO_URL || 'mongodb://localhost:27017/'
let db
logger.debug('Connecting to mongo')
try {
db = await MongoClient.connect(
MONGO_URL,
{
sslValidate: false,
}
)
} catch (error) {
logger.error('Could not connect to Mongo', { error })
throw error
}
logger.debug('Connected')
return db
}
async function fetchAndStore(
list,
sources,
logger
) {
const results = []
const tasks = sources.map((source) => async (callback) => {
const { fetchAdapter, storageAdapter } = source
try {
const { entries, timestamp } = await fetchAdapter.fetchLatest(list)
await storageAdapter.storeMany(timestamp, entries)
results.push(['fetch.success', 1, [ `fetcher:${fetchAdapter.name}` ] ])
} catch (error) {
const errorMessage = `Failed to fetch and store from adapter ${fetchAdapter.name}`
const { message, name, stack } = error
logger.error(errorMessage, { error: { message, name, stack } })
results.push(['fetch.error', 1, [ `fetcher:${fetchAdapter.name}` ] ])
}
callback && callback(null)
})
await runTasksInParallel(tasks, PARALLEL_FETCHERS_NUMBER)
return results
}
export async function handle() {
const logger = createLambdaLogger() // just a wrapper for console.log to match interface in service
logger.debug('Starting fetching data')
const db: Db = await connectToDB(logger)
const primaryCollection = db.collection(PRIMARY_COLLECTION)
const itemsColletion = db.collection(ITEMS_COLLECTION)
const secondaryCollection = db.collection(SECONDARY_PRICING_COLLECTION)
const primaryStorageAdapter = new StorageAdapter(
Promise.resolve(primaryCollection),
logger
)
const itemsStorageAdapter = new ItemsStorageAdapter(
Promise.resolve(itemsColletion),
logger
)
const secondaryStorageAdapter = new StorageAdapter(
Promise.resolve(secondaryCollection),
logger
)
const primaryFetchAdapter = new PrimaryFetchAdapter(getFetcherCredentials('PRIMARY'), logger)
const secondaryFetchAdapter = new SecondaryFetchAdapter(getFetcherCredentials('SECONDARY'), logger)
const sources = [
{ fetchAdapter: primaryFetchAdapter, storageAdapter: primaryStorageAdapter },
{ fetchAdapter: secondaryFetchAdapter, storageAdapter: secondaryStorageAdapter },
]
try {
const list = await itemsStorageAdapter.getItems()
logger.debug(`Total items to fetch ${list.length}`)
const result = await fetchAndStore(list, sources, logger)
logger.debug('Returning: ', { result })
return {
statusCode: 200,
body: JSON.stringify(result)
}
} catch (error) {
const errorMessage = 'failed to do task'
const { message, name, stack } = error
logger.error(errorMessage, { error: { message, name, stack } })
return {
statusCode: 500,
body: JSON.stringify(new Error(errorMessage))
}
} finally {
await db.close()
}
}
Edit:
I have changed the way i'm invoking the lambda to this below. Basically I tried to listen for every possible event that can say me something. Result is that i get a error event, with response message being Converting circular structure to JSON. Thing is I don't really know about which structure this is, as the value I'm returning is a two elements array of 3 elements (strings and number) arrays.
const lambda = new AWS.Lambda({
httpOptions: { timeout: 660000 },
maxRetries: 0
})
const request = lambda.invoke(
{
FunctionName: `${config.get('env')}-credit-rubric-pricing-fetching-lambda`,
InvocationType: 'RequestResponse',
Payload: '""'
},
(err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
}
)
const results = await request
.on('send', () => {
logger.debug('Request sent to lambda')
})
.on('retry', response => {
logger.debug('Retrying.', { response })
})
.on('extractError', response => {
logger.debug('ExtractError', { response })
})
.on('extractData', response => {
logger.debug('ExtractData', { response })
})
.on('error', response => {
logger.debug('error', { response })
})
.on('succes', response => {
logger.debug('success', { response })
})
.on('complete', response => {
logger.debug('complete', { response })
})
.on('httpHeaders', (statusCode, headers, response, statusMessage) => {
logger.debug('httpHeaders', { statusCode, headers, response, statusMessage })
})
.on('httpData', (chunk, response) => {
logger.debug('httpData', { response, chunk })
})
.on('httpError', (error, response) => {
logger.debug('httpError', { response, error })
})
.on('httpDone', response => {
logger.debug('httpDone', { response })
})
.promise()

"Converting circular structure to JSON" error despite functions being executed properly

I'm getting the following error:
TypeError: Converting circular structure to JSON
at JSON.stringify (<anonymous>)
at stringify (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/response.js:1123:12)
at ServerResponse.json (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/response.js:260:14)
at cors (/Users/landing-page-backend/functions/zohoCrmHook.js:45:43)
at process._tickCallback (internal/process/next_tick.js:68:7)
for the HTTP response in my createLead function, despite the fact that the function is executed properly and does what it's supposed to do (which is to create an entry in my CRM).
I've pointed out in the following code where the error occurs:
const axios = require('axios');
const functions = require('firebase-functions');
const cors = require('cors')({ origin: true })
const clientId = functions.config().zoho.client_id;
const clientSecret = functions.config().zoho.client_secret;
const refreshToken = functions.config().zoho.refresh_token;
const baseURL = 'https://accounts.zoho.com';
module.exports = (req, res) => {
cors(req, res, async () => {
const newLead = {
'data': [
{
'Email': String(req.body.email),
'Last_Name': String(req.body.lastName),
'First_Name': String(req.body.firstName),
}
],
'trigger': [
'approval',
'workflow',
'blueprint'
]
};
const { data } = await getAccessToken();
const accessToken = data.access_token;
const leads = await getLeads(accessToken);
const result = checkLeads(leads.data.data, newLead.data[0].Email);
if (result.length < 1) {
try {
return res.json(await createLead(accessToken, newLead)); // this is where the error occurs
} catch (e) {
console.log(e);
}
} else {
return res.json({ message: 'Lead already in CRM' })
}
})
}
function getAccessToken () {
const url = `https://accounts.zoho.com/oauth/v2/token?refresh_token=${refreshToken}&client_id=${clientId}&client_secret=${clientSecret}&grant_type=refresh_token`;
return new Promise((resolve, reject) => {
axios.post(url)
.then((response) => {
return resolve(response);
})
.catch(e => console.log("getAccessToken error", e))
});
}
function getLeads(token) {
const url = 'https://www.zohoapis.com/crm/v2/Leads';
return new Promise((resolve, reject) => {
axios.get(url, {
headers: {
'Authorization': `Zoho-oauthtoken ${token}`
}
})
.then((response) => {
return resolve(response);
})
.catch(e => console.log("getLeads error", e))
})
}
function createLead(token, lead) {
const url = 'https://www.zohoapis.com/crm/v2/Leads';
return new Promise((resolve, reject) => {
const data = JSON.stringify(lead);
axios.post(url, data, {
headers: {
'Authorization': `Zoho-oauthtoken ${token}`
}
})
.then((response) => {
console.log("response in createLead", response)
return resolve(response);
})
.catch(e => reject(e))
})
}
function checkLeads(leads, currentLead) {
return leads.filter(lead => lead.Email === currentLead)
}
Console.logging all the parameters indicate that the they are not the issue. The configuration of the Zoho API is, also, not the issue considering the fact that the entries are being properly made into the CRM. My assumption is that it would have to do with the HTTP response in the JSON format.

You're trying to convert a promise to JSON, not going to work. Your createLead function returns a promise, not a JSON. The promise is the 'circular object'.

How do I return the contents of this function to the client side?

I have a function that I call using
fetch(http://localhost:8888/.netlify/functions/token-hider?
stateName=' +stateName)
on my client side.
the token-hider function looks like this:
const qs = require("qs");
const fetch = require("node-fetch");
var alertEndpoint = "";
var parkEndpoint = "";
var parksWithAlerts = "";
exports.handler = async function getURLS(event, context, callback)
{
// Get env var values defined in our Netlify site UI
const {api_key, alert_api_url, park_api_url} = process.env;
var stateName =event.queryStringParameters.stateName;
alertEndpoint = `${alert_api_url}${stateName}${api_key}`;
parkEndpoint = `${park_api_url}${stateName}${api_key}`;
getData();
async function getData(alertsArea, alertHeader) {
const [getAlertData, getParkData] = await
Promise.all([fetch(alertEndpoint), fetch(parkEndpoint)] );
var alertResults = await getAlertData.json();
var parkResults= await getParkData.json();
var alertData = alertResults.data;
var parkData = parkResults.data;
parksWithAlerts = parkData.map(park => {
park.alertData = alertData.filter(alert => alert.parkCode ===
park.parkCode);
return park
});
console.log(parksWithAlerts);
}
console.log(callback);
};
how could I return the contents of parksWithAlerts back to the client side after this function is finished?

Try to learn more about callback functions in Javascript.
It is right there in your code, the callback that you are printing is actually suppose to be called after you have executed your code and you can do like this callback(parksWithAlerts);.
While calling the function getURLS you will provide the function which is suppose to get called with args.
Examples : https://www.geeksforgeeks.org/javascript-callbacks/

Here is an example with error handling and returning a response type of JSON
token-hider
import fetch from "node-fetch";
// Get env var values defined in our Netlify site UI
const {api_key, alert_api_url, park_api_url} = process.env;
async function getJson(response) {
return await response.json();
}
const alertEndpoint = stateName => {
return new Promise(function(resolve, reject) {
fetch(`${alert_api_url}${stateName}${api_key}`)
.then(response => {
if (!response.ok) { // NOT res.status >= 200 && res.status < 300
return reject({ statusCode: response.status, body: response.statusText });
}
return resolve(getJson(response))
})
.catch(err => {
console.log('alertEndpoint invocation error:', err); // output to netlify function log
reject({ statusCode: 500, body: err.message });
})
});
}
const parkEndpoint = stateName => {
return new Promise(function(resolve, reject) {
fetch(`${park_api_url}${stateName}${api_key}`)
.then(response => {
if (!response.ok) { // NOT res.status >= 200 && res.status < 300
return reject({ statusCode: response.status, body: response.statusText });
}
return resolve(getJson(response))
})
.catch(err => {
console.log('parkEndpoint invocation error:', err); // output to netlify function log
reject({ statusCode: 500, body: err.message });
})
})
}
exports.handler = function(event, context) {
const stateName = event.queryStringParameters.stateName;
return Promise.all([alertEndpoint(stateName), parkEndpoint(stateName)])
.then(values => {
const [alertData, parkData] = values;
const parksWithAlerts = parkData.map(park => {
park.alertData = alertData.filter(alert => alert.parkCode === park.parkCode);
return park;
});
return {
statusCode: 200,
headers: { 'content-type': 'application/json' },
body: JSON.stringify(parksWithAlerts)
};
})
.catch(error => {
return error;
});
};
NOTE: If you are trying to hide the token, make sure to not deploy this from a public repository on Netlify.
Also, this code has not been tested 100%, so there may be some things to resolve. The response layout and structure is something I use in a few of my lambda functions on Netlify.

Recursive loop with promises crawler NodeJS

I'm trying to use Recursive Loop and Promises to Scrape a website.
But it fails.. It make the request only for the first page and at the second the program stops giving to me unhandled promise rejection warning
I have this three JS files:
scrapeAll.js (is the recursive loop that calls scrapePage.js)
scrapePage.js
scrapeComponents.js
scrapeAll.js:
var indexPage = 0;
scrapePage(indexPage).then((json)=>{
console.log(JSON.stringify(json, null, 4));
if(indexPage === Number.MAX_SAFE_INTEGER){
console.log("MAX SAFE INTEGER");
return;
}
save(json);
indexpage++;
scrapePage(indexPage);
}).catch((data)=>{
console.log(data);
if(indexPage === Number.MAX_SAFE_INTEGER){
console.log("MAX SAFE INTEGER");
return;
}
indexPage++;
scrapePage(indexPage);
});
ScrapePage.JS
let makeRequestCounter = 0;
function scrapePage(number) {
return new Promise((resolve, reject) => {
let url = URL + number;
let options = {
url: url,
headers: {
Host: SITE,
Connection: "keep-alive",
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Language": "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7",
"Cache-Control": "max-age=0",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
"Cookie": restoreCookieToString()
}
};
makeRequest(options).then((jsonData) => {
resolve(jsonData);
}).catch((error) => {
//REQUEST_LIMIT_EXCEEDED
if (error === CONSTANTS.REQUEST_LIMIT_EXCEEDED) {
reject(CONSTANTS.REQUEST_LIMIT_EXCEEDED);
}
//ALREADY_EXIST
else if (error === CONSTANTS.ALREADY_EXIST) {
reject(CONSTANTS.ALREADY_EXIST);
}
else if (error === 404) {
reject("no data found at this page");
}
//error can beeconnrefused or econnreset
else if (error.code !== undefined) {
//econnrefused
if (error.code === CONSTANTS.ECONNREFUSED) {
reject("WRONG_URL", url);
}
//econnreset
else if (error.code === CONSTANTS.ECONNRESET) {
console.log("\neconnreset error\n");
makeRequest(options);
}
}
}
);
});
}
function makeRequest(options) {
return new Promise((resolve, reject) => {
let json = {
category: [],
imgs: [],
title: "",
description: "",
url: ""
};
if (makeRequestCounter === CONSTANTS.REQUEST_LIMIT) {
reject(CONSTANTS.REQUEST_LIMIT_EXCEEDED);
}
makeRequestCounter++;
console.log("request to: ", options.url);
request(options, function (error, response, html) {
if (error) {
//error: possible econnreset econnrefused
reject(error);
} else {
if (response.statusCode === 200) {
cookieSave(response.headers);
//---------- check if in db the url is already saved -------------//
check(response.request.uri.href, (err) => {
if (!err) {
reject(CONSTANTS.ALREADY_EXIST);
}
});
//----------finish checking, is new -------------------//
//GETTING TITLE
title(html, json_recipe).then((json) => {
//GETTING category
category(html, json).then((json) => {
//GETTING images
imgs(html, json).then((json) => {
description(html, json).then((json) => {
json.url = response.request.uri.href;
resolve(json);
//description error
}).catch((error) => {
console.log(error);
});
//images error
}).catch((error) => {
console.log(error);
});
//category error
}).catch((error) => {
console.log(error);
});
//title error
}
).catch((error) => {
console.log(error);
});
}
//no data in this page
if (response.statusCode === 404) {
reject(response.statusCode);
}
}
});
});
}
scrapeComponents.js
...
function description(html, json) {
return new Promise((resolve, reject) => {
const $ = cheerio.load(html);
let description = $('.submitter__description').text().trim();
json.description = JSON.parse(description);
resolve(json);
});
}
...
error:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): no data found at this page
The program make the first request and return correctly at the scrapeAll.js that correctly the scrapePage(indexPage = 1).
The second time my program do exactly same as first time but when is time to return to the scrapeAll.js ( reject("no data found at this page"); in ScrapePage.js ) the program ends with the error.
Both the pages are without data but program fails also with good pages saving only the first.
I think that I made a big mistake with promises.
Thank you very much guys.

Your call for scrapPage function is running only once and you are not calling it iteratively. You might have to call it in iteration using a function. Update your scrapeAll.js:
function callScrapPage() {
var indexPage = 0;
while (indexPage < Number.MAX_SAFE_INTEGER) {
scrapePage(indexPage).then((json) => {
console.log(JSON.stringify(json, null, 4));
save(json);
indexpage++;
}
}
}

The problem is that one or more of your calls to scrapePage(indexPage) in scrapeAll.js are failing. You cannot recursively call a promise the way you might with other code, so you need a .then and .catch on the additional calls as well. Adding a .catch to the other calls will enable you to see the true source of failure.
scrapePage(indexPage)
.then((json)=>{
console.log(JSON.stringify(json, null, 4));
if(indexPage === Number.MAX_SAFE_INTEGER){
console.log("MAX SAFE INTEGER");
return;
}
save(json);
indexpage++;
scrapePage(indexPage).catch(e => console.log(e));
})
.catch((data)=>{
console.log(data);
if(indexPage === Number.MAX_SAFE_INTEGER){
console.log("MAX SAFE INTEGER");
return;
}
indexPage++;
scrapePage(indexPage).catch(e => console.log(e));
});