AWS Lambda retries despite returning value - javascript
I have found out that my and my colleagues lambda doesn't return data we anticipate.
After what I have already found, we used deprecated invokeAsync which returnes only statusCode.
I have upgraded aws-sdk to the newest version and change the invocation code to use invoke with
InvocationType: 'RequestResponse'
as that should give us the returned value.
Lambdas code itself looks good, its an async function without arguments.
Im not using the callback here (3rd argument of handler), as we are doing some async stuff and it would require a small refactor to not use async/await, also i have read that just returning value is ok as well.
Earlier lambda was returning array, but after some investigation and googling I have changed it to
return {
statusCode: 200,
body: JSON.stringify(result)
}
where result is the mentioned array, as this was recurring pattern in search results.
Overall result is in our service where we do the invocation, we get timeout, lambda logs that it is returning value, but then retries itself again and again.
Dumping code
invocation in our service:
const lambda = new AWS.Lambda({ httpOptions: { timeout: 600000 } })
const results = await new Promise((resolve, reject) => {
lambda.invoke(
{
FunctionName: `lambda-name`,
InvocationType: 'RequestResponse',
Payload: '""'
},
(err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
}
)
})
lambda handler
import { parallelLimit } from 'async'
const PARALLEL_FETCHERS_NUMBER = 2
export async function runTasksInParallel (
tasks,
limit,
) {
return new Promise(
(
resolve,
reject,
) => {
parallelLimit(tasks, limit, (error, results) => {
if (error === null) {
resolve(results)
} else {
reject(error)
}
})
},
)
}
async function connectToDB(logger) {
const MONGO_URL = process.env.MONGO_URL || 'mongodb://localhost:27017/'
let db
logger.debug('Connecting to mongo')
try {
db = await MongoClient.connect(
MONGO_URL,
{
sslValidate: false,
}
)
} catch (error) {
logger.error('Could not connect to Mongo', { error })
throw error
}
logger.debug('Connected')
return db
}
async function fetchAndStore(
list,
sources,
logger
) {
const results = []
const tasks = sources.map((source) => async (callback) => {
const { fetchAdapter, storageAdapter } = source
try {
const { entries, timestamp } = await fetchAdapter.fetchLatest(list)
await storageAdapter.storeMany(timestamp, entries)
results.push(['fetch.success', 1, [ `fetcher:${fetchAdapter.name}` ] ])
} catch (error) {
const errorMessage = `Failed to fetch and store from adapter ${fetchAdapter.name}`
const { message, name, stack } = error
logger.error(errorMessage, { error: { message, name, stack } })
results.push(['fetch.error', 1, [ `fetcher:${fetchAdapter.name}` ] ])
}
callback && callback(null)
})
await runTasksInParallel(tasks, PARALLEL_FETCHERS_NUMBER)
return results
}
export async function handle() {
const logger = createLambdaLogger() // just a wrapper for console.log to match interface in service
logger.debug('Starting fetching data')
const db: Db = await connectToDB(logger)
const primaryCollection = db.collection(PRIMARY_COLLECTION)
const itemsColletion = db.collection(ITEMS_COLLECTION)
const secondaryCollection = db.collection(SECONDARY_PRICING_COLLECTION)
const primaryStorageAdapter = new StorageAdapter(
Promise.resolve(primaryCollection),
logger
)
const itemsStorageAdapter = new ItemsStorageAdapter(
Promise.resolve(itemsColletion),
logger
)
const secondaryStorageAdapter = new StorageAdapter(
Promise.resolve(secondaryCollection),
logger
)
const primaryFetchAdapter = new PrimaryFetchAdapter(getFetcherCredentials('PRIMARY'), logger)
const secondaryFetchAdapter = new SecondaryFetchAdapter(getFetcherCredentials('SECONDARY'), logger)
const sources = [
{ fetchAdapter: primaryFetchAdapter, storageAdapter: primaryStorageAdapter },
{ fetchAdapter: secondaryFetchAdapter, storageAdapter: secondaryStorageAdapter },
]
try {
const list = await itemsStorageAdapter.getItems()
logger.debug(`Total items to fetch ${list.length}`)
const result = await fetchAndStore(list, sources, logger)
logger.debug('Returning: ', { result })
return {
statusCode: 200,
body: JSON.stringify(result)
}
} catch (error) {
const errorMessage = 'failed to do task'
const { message, name, stack } = error
logger.error(errorMessage, { error: { message, name, stack } })
return {
statusCode: 500,
body: JSON.stringify(new Error(errorMessage))
}
} finally {
await db.close()
}
}
Edit:
I have changed the way i'm invoking the lambda to this below. Basically I tried to listen for every possible event that can say me something. Result is that i get a error event, with response message being Converting circular structure to JSON. Thing is I don't really know about which structure this is, as the value I'm returning is a two elements array of 3 elements (strings and number) arrays.
const lambda = new AWS.Lambda({
httpOptions: { timeout: 660000 },
maxRetries: 0
})
const request = lambda.invoke(
{
FunctionName: `${config.get('env')}-credit-rubric-pricing-fetching-lambda`,
InvocationType: 'RequestResponse',
Payload: '""'
},
(err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
}
)
const results = await request
.on('send', () => {
logger.debug('Request sent to lambda')
})
.on('retry', response => {
logger.debug('Retrying.', { response })
})
.on('extractError', response => {
logger.debug('ExtractError', { response })
})
.on('extractData', response => {
logger.debug('ExtractData', { response })
})
.on('error', response => {
logger.debug('error', { response })
})
.on('succes', response => {
logger.debug('success', { response })
})
.on('complete', response => {
logger.debug('complete', { response })
})
.on('httpHeaders', (statusCode, headers, response, statusMessage) => {
logger.debug('httpHeaders', { statusCode, headers, response, statusMessage })
})
.on('httpData', (chunk, response) => {
logger.debug('httpData', { response, chunk })
})
.on('httpError', (error, response) => {
logger.debug('httpError', { response, error })
})
.on('httpDone', response => {
logger.debug('httpDone', { response })
})
.promise()
Related
Error: function uses multiple asynchronous interfaces: callback and promise
Error: function uses multiple asynchronous interfaces: callback and promise to use the callback interface: do not return a promise to use the promise interface: remove the last argument to the function I'm trying to write a cucumber test to one of my GET node API, and keep getting the above, looked at few GitHub and stack-overflow posts, and could not understand the issue, below are my test method details. App.ts async getSsoId(refId: any): Promise<string> { let ssoId = ''; const secrets = await this.vaultService.getClientSecrets(); this.decrypt(refId, secrets.encryption_secret, secrets.encryption_Id).then((value: any) => { ssoId = value; }); return ssoId; } api.get('/status', async (req, res) => { let id; const statusCode = 200; try { id = await this.getId(String('123456')); } catch (err: any) { throw new ApiError('Error fetching id'); } try { const item = await dbService.getItem(id); if (item) { statusCode = 201; } else { statusCode = 202; } } catch (err: any) { throw new ApiError( 'The API encountered an error while attempting to communicate with the DB service.' ); } res.status(statusCode).send(statusCode); }); Step Definition: Given('a valid customer', function () {}); When("I call the get-id api", { timeout: 2 * 5000 }, async function (val) { util.callGetIdAPI().then((response) => { this.setApiResponseStatus(response.status); this.setResponseBody(JSON.stringify(response.body)); }); }); Then("the apiResponseStatus should be <apiResponseStatus>", function (status) { assert.strictEqual(status, this.apiResponseStatus); }); Then("the responseBody should be {string}", function (responseBody) { assert.equal(responseBody, this.responseBody); }); Util Function callGetIdAPI = async () => { const headers = { 'Content-Type': 'application/json;v=1', Accept: 'application/json;v=1' } const client = await getClient('url'); const options = { method: 'GET', headers: headers, version: 3 }; let response; try { response = await client.get('/status', options); return { status: response.statusCode, body: response.body }; } catch(error) { return { status: error.statusCode, body: { error: { id: error.id, message: error.message } } } } }; I'm new to this and trying to understand how multiple Premisses and Callbacks works in parallel, any thoughts or inputs on what possibly cause the error, or am I missing anything ??
"Converting circular structure to JSON" error despite functions being executed properly
I'm getting the following error: TypeError: Converting circular structure to JSON at JSON.stringify (<anonymous>) at stringify (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/response.js:1123:12) at ServerResponse.json (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/response.js:260:14) at cors (/Users/landing-page-backend/functions/zohoCrmHook.js:45:43) at process._tickCallback (internal/process/next_tick.js:68:7) for the HTTP response in my createLead function, despite the fact that the function is executed properly and does what it's supposed to do (which is to create an entry in my CRM). I've pointed out in the following code where the error occurs: const axios = require('axios'); const functions = require('firebase-functions'); const cors = require('cors')({ origin: true }) const clientId = functions.config().zoho.client_id; const clientSecret = functions.config().zoho.client_secret; const refreshToken = functions.config().zoho.refresh_token; const baseURL = 'https://accounts.zoho.com'; module.exports = (req, res) => { cors(req, res, async () => { const newLead = { 'data': [ { 'Email': String(req.body.email), 'Last_Name': String(req.body.lastName), 'First_Name': String(req.body.firstName), } ], 'trigger': [ 'approval', 'workflow', 'blueprint' ] }; const { data } = await getAccessToken(); const accessToken = data.access_token; const leads = await getLeads(accessToken); const result = checkLeads(leads.data.data, newLead.data[0].Email); if (result.length < 1) { try { return res.json(await createLead(accessToken, newLead)); // this is where the error occurs } catch (e) { console.log(e); } } else { return res.json({ message: 'Lead already in CRM' }) } }) } function getAccessToken () { const url = `https://accounts.zoho.com/oauth/v2/token?refresh_token=${refreshToken}&client_id=${clientId}&client_secret=${clientSecret}&grant_type=refresh_token`; return new Promise((resolve, reject) => { axios.post(url) .then((response) => { return resolve(response); }) .catch(e => console.log("getAccessToken error", e)) }); } function getLeads(token) { const url = 'https://www.zohoapis.com/crm/v2/Leads'; return new Promise((resolve, reject) => { axios.get(url, { headers: { 'Authorization': `Zoho-oauthtoken ${token}` } }) .then((response) => { return resolve(response); }) .catch(e => console.log("getLeads error", e)) }) } function createLead(token, lead) { const url = 'https://www.zohoapis.com/crm/v2/Leads'; return new Promise((resolve, reject) => { const data = JSON.stringify(lead); axios.post(url, data, { headers: { 'Authorization': `Zoho-oauthtoken ${token}` } }) .then((response) => { console.log("response in createLead", response) return resolve(response); }) .catch(e => reject(e)) }) } function checkLeads(leads, currentLead) { return leads.filter(lead => lead.Email === currentLead) } Console.logging all the parameters indicate that the they are not the issue. The configuration of the Zoho API is, also, not the issue considering the fact that the entries are being properly made into the CRM. My assumption is that it would have to do with the HTTP response in the JSON format.
You're trying to convert a promise to JSON, not going to work. Your createLead function returns a promise, not a JSON. The promise is the 'circular object'.
My javascript Async Await api request returns a strange object
I'm making an API request with Frisbee: const Frisbee = require('frisbee') const api = new Frisbee({ baseURI: 'http://192.168.1.8:4000', headers: { Accept: 'application/json', 'Content-Type': 'application/json' } }) export const handleSubmit = async (values): void => { Toast.show('Uploading Product', { duration: 3000, position: 30, shadow: true, animation: true, hideOnPress: true, delay: 0 }) try { const response = api.post('/products', { body: encodeAddProductAction(values) }) if (response.err) throw response.err console.log(response) } catch (err) { console.error(err) } } export const encodeAddProductAction = (values: any) => { const submitPayload = Object.assign({}, values) Object.keys(submitPayload).forEach((key) => { if (key != 'Categories') { submitPayload[key] = encodeURIComponent( JSON.stringify(submitPayload[key]) ) } else { // values[key] = JSON.stringify(values[key]) submitPayload[key] = submitPayload[key].join(',') } }) return submitPayload } It console logs this: Why is my API response body buried in some strange fields like _55? My response is:
You are using async but you are not awaiting response, that's why you get Promise returned. try { const response = await api.post('/products', { body: encodeAddProductAction(values) }) if (response.err) throw response.err console.log(response) } catch (err) { console.error(err) } Async by default returns a Promise.
How to run a Node script
I need to be able to run a node script to delete an object from an external API. So I should be able to run this command: node server.js Customer55555 And it should delete the object. I have called to the API by using Axios. const axios = require("axios"); const API = "http://dummy.restapiexample.com/api/v1/employees"; function getAllEmployees() { axios .get("http://dummy.restapiexample.com/api/v1/employees") .then(response => { // console.log(response.data); console.log(response.status); function filterEmployee() { const employeeData = response.data; employeeData.filter(employee => { console.log(employee); }); // console.log(employeeData); } filterEmployee(); }) .catch(error => { console.log(error); }); } function deleteEmployee() { axios({ method: "DELETE", url: "http://dummy.restapiexample.com/api/v1/delete/36720", headers: { "Content-Type": "application/json" } }) .then( // Observe the data keyword this time. Very important // payload is the request body // Do something console.log("user deleted") ) .catch(function(error) { // handle error console.log(error); }); } // getAllEmployees(); deleteEmployee(); I am able to get an individual object, but I need to figure out how to delete it by running the command above.
You can do something like this: const axios = require("axios") const API = "http://dummy.restapiexample.com/api/v1/employees" async function getAllEmployees(filter = null) { try { const response = await axios.get("http://dummy.restapiexample.com/api/v1/employees") console.log(response.status) let employeeData = response.data if (filter) { // return only employees whose name contains filter.name employeeData = employeeData.filter(({ employee_name }) => { return employee_name.toLowerCase().indexOf(filter.name.toLowerCase()) >= 0 }) } return employeeData } catch(error) { console.error(error) return [] } } async function deleteEmployee({ id }) { if (!id) { throw new Error('You should pass a parameter') } try { const response = await axios({ method: "DELETE", url: `http://dummy.restapiexample.com/api/v1/delete/${id}`, headers: { "Content-Type": "application/json" } }) console.log("user deleted " + id) } catch(error) { // handle error console.error(error) } } async function main(params) { const employees = await getAllEmployees({ name: params[0] || '' }) // Returns a promise to wait all delete promises return Promise.all(employess.map(employee => deleteEmployee(employee))) } // process.argv contains console parameters. (https://stackoverflow.com/questions/4351521/how-do-i-pass-command-line-arguments-to-a-node-js-program) main(process.argv.slice(2)).then(() => { // returns 0 (Success) (https://stackoverflow.com/questions/5266152/how-to-exit-in-node-js) process.exit(0) }).catch(() => { // returns 1 (error) process.exit(1) }) You should adapt this sample to get proper filtering and error reporting.
Facebook graph API: return object from GraphRequest callback
I have a function that successfully makes requests, but am having trouble translating it to the React-Redux data flow because I can't figure out how to return the object that is received in the callback. What I currently have doesn't work, but I am confident it makes the request successfully because I can see it when I console.log it. import FBSDK, { AccessToken, GraphRequest, GraphRequestManager } from 'react-native-fbsdk' export const callback = (error, result) => { if (error) { console.log(error); return error } else { console.log(result); return result; } } export const graphRequestFor = fields => { AccessToken.getCurrentAccessToken() .then(token => { console.log(token); const request = new GraphRequest( '/me', { accessToken: token.accessToken, parameters: { fields: { string: fields } } }, callback ) new GraphRequestManager().addRequest(request).start() }) } export const login = () => { graphRequestFor('first_name, last_name') .then(results => { return results }) //undefined }
Not an answer to my direct question, but a solution to my problem is to make the api call directly inside the action and dispatching the object from the callback import FBSDK, { AccessToken, GraphRequest, GraphRequestManager } from 'react-native-fbsdk' export const login = params => dispatch => { AccessToken.getCurrentAccessToken() .then(token => { const request = new GraphRequest( '/me', { accessToken: token.accessToken, parameters: { fields: { string: params } } }, (error, result) => { if (error) { console.log(error); return dispatch(receiveCurrentUser(error)) } else { return dispatch(receiveCurrentUser(result)) } } ) new GraphRequestManager().addRequest(request).start() }) }