We Keep Coding

Can't get req.body with client fetch() and Node.js

I have simple app.js for Node.js under localhost:3000
app.js:
let http = require('http');
http.createServer((req, res) => {
res.writeHead(200);
let response;
if(~req.url.indexOf('post')) {
response = req.body.content;
} else {
response = '<script src="http://localhost/fetch.js"></script>';
}
res.end(response);
}).listen(3000);
The file fetch.js is placed on my another local server and is successfully enqueued to the page
fetch.js:
read('http://localhost:3000/?post').then((response) => {
console.log(response);
});
async function read(url) {
const response = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json;charset=utf-8',
},
body: JSON.stringify({
content: 'Text'
})
});
return response.text();
}
So I render HTML with fetch.js which then send POST request to the same page, but with a query ?post
However, when I run node app.js I get the error
Can not read property 'content' of undefined
So I don't get req.body
Why and how to resolve?

i think you are missing parser for your http server, there is no body because you actually didn't parse the body.
assemble the chunks like below then parse it as the header sais.
this is my work for myself
private parseBody(req: IncomingMessage) {
return new Promise((resolve, reject) => {
const chunks: any[] = []
req.on("data", (chunk) => {
chunks.push(chunk)
})
req.on("end", () => {
const data = Buffer.concat(chunks)
switch (req.headers["content-type"]) {
case "application/json":
resolve(this.parseJson(data.toString()))
break
case "application/x-www-form-urlencoded":
resolve(this.parseUrlEncoded(data.toString()))
break
default:
resolve({})
}
})
})
http server is very abstract and doesn't support anything basicly, i suggest using express or fastify.
working example: https://frontendguruji.com/blog/how-to-parse-post-request-in-node-js-without-expressjs-body-parser/
update
this is the class im using
http.resolver.ts
private parseJson(data: string) {
return JSON.parse(data)
}
private parseUrlEncoded(data: string) {
const parsedData = new URLSearchParams(data)
const dataObj: any = {}
for (var pair of parsedData.entries()) {
dataObj[pair[0]] = pair[1]
}
return dataObj
}
private parseBody(req: IncomingMessage) {
return new Promise((resolve, reject) => {
const chunks: any[] = []
req.on("data", (chunk) => {
chunks.push(chunk)
})
req.on("end", () => {
const data = Buffer.concat(chunks)
switch (req.headers["content-type"]) {
case "application/json":
resolve(this.parseJson(data.toString()))
break
case "application/x-www-form-urlencoded":
resolve(this.parseUrlEncoded(data.toString()))
break
default:
resolve(parse(req.url ?? "/", true).query)
}
})
})
}
you may use await behind the parseBody function after

Ok, thanks to #Fide. The link he posted has the answer:
let http = require('http');
http.createServer((req, res) => {
if(~req.url.indexOf('post')) {
let body;
req.on('data', function(data) {
body = data;
})
req.on('end', function() {
res.writeHead(200);
res.end(body);
})
} else {
res.writeHead(200);
res.end('<script src="http://localhost/fetch.js"></script>');
}
}).listen(3000);

AWS Lambda retries despite returning value

I have found out that my and my colleagues lambda doesn't return data we anticipate.
After what I have already found, we used deprecated invokeAsync which returnes only statusCode.
I have upgraded aws-sdk to the newest version and change the invocation code to use invoke with
InvocationType: 'RequestResponse'
as that should give us the returned value.
Lambdas code itself looks good, its an async function without arguments.
Im not using the callback here (3rd argument of handler), as we are doing some async stuff and it would require a small refactor to not use async/await, also i have read that just returning value is ok as well.
Earlier lambda was returning array, but after some investigation and googling I have changed it to
return {
statusCode: 200,
body: JSON.stringify(result)
}
where result is the mentioned array, as this was recurring pattern in search results.
Overall result is in our service where we do the invocation, we get timeout, lambda logs that it is returning value, but then retries itself again and again.
Dumping code
invocation in our service:
const lambda = new AWS.Lambda({ httpOptions: { timeout: 600000 } })
const results = await new Promise((resolve, reject) => {
lambda.invoke(
{
FunctionName: `lambda-name`,
InvocationType: 'RequestResponse',
Payload: '""'
},
(err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
}
)
})
lambda handler
import { parallelLimit } from 'async'
const PARALLEL_FETCHERS_NUMBER = 2
export async function runTasksInParallel (
tasks,
limit,
) {
return new Promise(
(
resolve,
reject,
) => {
parallelLimit(tasks, limit, (error, results) => {
if (error === null) {
resolve(results)
} else {
reject(error)
}
})
},
)
}
async function connectToDB(logger) {
const MONGO_URL = process.env.MONGO_URL || 'mongodb://localhost:27017/'
let db
logger.debug('Connecting to mongo')
try {
db = await MongoClient.connect(
MONGO_URL,
{
sslValidate: false,
}
)
} catch (error) {
logger.error('Could not connect to Mongo', { error })
throw error
}
logger.debug('Connected')
return db
}
async function fetchAndStore(
list,
sources,
logger
) {
const results = []
const tasks = sources.map((source) => async (callback) => {
const { fetchAdapter, storageAdapter } = source
try {
const { entries, timestamp } = await fetchAdapter.fetchLatest(list)
await storageAdapter.storeMany(timestamp, entries)
results.push(['fetch.success', 1, [ `fetcher:${fetchAdapter.name}` ] ])
} catch (error) {
const errorMessage = `Failed to fetch and store from adapter ${fetchAdapter.name}`
const { message, name, stack } = error
logger.error(errorMessage, { error: { message, name, stack } })
results.push(['fetch.error', 1, [ `fetcher:${fetchAdapter.name}` ] ])
}
callback && callback(null)
})
await runTasksInParallel(tasks, PARALLEL_FETCHERS_NUMBER)
return results
}
export async function handle() {
const logger = createLambdaLogger() // just a wrapper for console.log to match interface in service
logger.debug('Starting fetching data')
const db: Db = await connectToDB(logger)
const primaryCollection = db.collection(PRIMARY_COLLECTION)
const itemsColletion = db.collection(ITEMS_COLLECTION)
const secondaryCollection = db.collection(SECONDARY_PRICING_COLLECTION)
const primaryStorageAdapter = new StorageAdapter(
Promise.resolve(primaryCollection),
logger
)
const itemsStorageAdapter = new ItemsStorageAdapter(
Promise.resolve(itemsColletion),
logger
)
const secondaryStorageAdapter = new StorageAdapter(
Promise.resolve(secondaryCollection),
logger
)
const primaryFetchAdapter = new PrimaryFetchAdapter(getFetcherCredentials('PRIMARY'), logger)
const secondaryFetchAdapter = new SecondaryFetchAdapter(getFetcherCredentials('SECONDARY'), logger)
const sources = [
{ fetchAdapter: primaryFetchAdapter, storageAdapter: primaryStorageAdapter },
{ fetchAdapter: secondaryFetchAdapter, storageAdapter: secondaryStorageAdapter },
]
try {
const list = await itemsStorageAdapter.getItems()
logger.debug(`Total items to fetch ${list.length}`)
const result = await fetchAndStore(list, sources, logger)
logger.debug('Returning: ', { result })
return {
statusCode: 200,
body: JSON.stringify(result)
}
} catch (error) {
const errorMessage = 'failed to do task'
const { message, name, stack } = error
logger.error(errorMessage, { error: { message, name, stack } })
return {
statusCode: 500,
body: JSON.stringify(new Error(errorMessage))
}
} finally {
await db.close()
}
}
Edit:
I have changed the way i'm invoking the lambda to this below. Basically I tried to listen for every possible event that can say me something. Result is that i get a error event, with response message being Converting circular structure to JSON. Thing is I don't really know about which structure this is, as the value I'm returning is a two elements array of 3 elements (strings and number) arrays.
const lambda = new AWS.Lambda({
httpOptions: { timeout: 660000 },
maxRetries: 0
})
const request = lambda.invoke(
{
FunctionName: `${config.get('env')}-credit-rubric-pricing-fetching-lambda`,
InvocationType: 'RequestResponse',
Payload: '""'
},
(err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
}
)
const results = await request
.on('send', () => {
logger.debug('Request sent to lambda')
})
.on('retry', response => {
logger.debug('Retrying.', { response })
})
.on('extractError', response => {
logger.debug('ExtractError', { response })
})
.on('extractData', response => {
logger.debug('ExtractData', { response })
})
.on('error', response => {
logger.debug('error', { response })
})
.on('succes', response => {
logger.debug('success', { response })
})
.on('complete', response => {
logger.debug('complete', { response })
})
.on('httpHeaders', (statusCode, headers, response, statusMessage) => {
logger.debug('httpHeaders', { statusCode, headers, response, statusMessage })
})
.on('httpData', (chunk, response) => {
logger.debug('httpData', { response, chunk })
})
.on('httpError', (error, response) => {
logger.debug('httpError', { response, error })
})
.on('httpDone', response => {
logger.debug('httpDone', { response })
})
.promise()

Only one of many Firebase Functions being invoked

I have 2 Firebase functions that I want to execute when there is an Http request, one function (createEmailList) to save data in the Firebase database, the other (zohoCrmHook) to to save in a 3rd party CRM called Zoho.
When the functions are deployed to Firebase, the functions log shows that both are properly deployed. However, when the Http request is made from the frontend, the log shows that only one of the functions (createEmailList) is being executed.
As the log shows, the first function createEmailList is being executed and the data shows up in the Firebase database with no problem. However, The second function zohoCrmHook is not even being executed.
index.js
const functions = require('firebase-functions');
const admin = require("firebase-admin")
const serviceAccount = require("./service_account.json");
const createEmailList = require('./createEmailList')
// zoho
const zohoCrmHook = require('./zohoCrmHook')
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://landing-page.firebaseio.com"
})
exports.zohoCrmHook = functions.https.onRequest(zohoCrmHook)
exports.createEmailList = functions.https.onRequest(createEmailList)
createEmailList.js
const admin = require('firebase-admin')
const cors = require('cors')({ origin: true })
module.exports = (req, res) => {
cors(req, res, () => {
if (!req.body.email) {
return res.status(422).send({ error: 'Bad Input'})
}
const email = String(req.body.email)
const firstName = String(req.body.firstName)
const lastName = String(req.body.lastName)
const data = {
email,
firstName,
lastName
}
const db = admin.firestore()
const docRef = db.collection('users')
.doc(email)
.set(data, { merge: false })
.catch(err => res.status(422).send({ error: err }))
res.status(204).end();
})
}
zohoCrmHook.js
const axios = require('axios');
const functions = require('firebase-functions');
// zoho
const clientId = functions.config().zoho.client_id;
const clientSecret = functions.config().zoho.client_secret;
const refreshToken = functions.config().zoho.refresh_token;
const baseURL = 'https://accounts.zoho.com';
module.exports = async (req, res) => {
const newLead = {
'data': [
{
'Email': req.body.email,
'Last_Name': req.body.lastName,
'First_Name': req.body.firstName,
}
],
'trigger': [
'approval',
'workflow',
'blueprint'
]
};
const { data } = await getAccessToken();
const accessToken = data.access_token;
const leads = await getLeads(accessToken);
const result = checkLeads(leads.data.data, newLead.data[0].Email);
if (result.length < 1) {
try {
return res.json(await createLead(accessToken, newLead));
}
catch (e) {
console.log(e);
}
}
else res.json({ message: 'Lead already in CRM' })
}
function getAccessToken () {
const url = `https://accounts.zoho.com/oauth/v2/token?refresh_token=${refreshToken}&client_id=${clientId}&client_secret=${clientSecret}&grant_type=refresh_token`;
return new Promise((resolve, reject) => {
axios.post(url)
.then((response) => {
return resolve(response);
})
.catch(e => console.log(e))
});
}
function getLeads(token) {
const url = 'https://www.zohoapis.com/crm/v2/Leads';
return new Promise((resolve, reject) => {
axios.get(url, {
headers: {
'Authorization': `Zoho-oauthtoken ${token}`
}
})
.then((response) => {
return resolve(response);
})
.catch(e => console.log(e))
})
}
function createLead(token, lead) {
const url = 'https://www.zohoapis.com/crm/v2/Leads';
return new Promise((resolve, reject) => {
const data = JSON.stringify(lead);
axios.post(url, data, {
headers: {
'Authorization': `Zoho-oauthtoken ${token}`
}
})
.then((response) => {
console.log(response.data)
return resolve(response);
})
.catch(e => reject(e))
})
}
function checkLeads(leads, currentLead) {
return leads.filter(lead => lead.Email === currentLead)
}

Since you're exporting two functions.https.onRequest declarations, you'll end up with two Cloud Functions, each with their own URL/endpoint. So if that's what you need, you'll need to configure two web hooks that call these functions.
From reading your question however, it sounds more like you want a single Cloud Function that does two things, in which case you should only have one functions.https.onRequest declaration that then calls two regular JavaScript functions (for example).
So something more like:
exports.myWebHook = functions.https.onRequest(function(req, res) {
zohoCrmHook(...);
createEmailList(...);
})
You'll need to figure out what to pass into the two function calls here, as you can't pass the request and response along.
Alternatively you can call the two Cloud Functions from here, but that typically just drives up your cost with little benefit.

My javascript Async Await api request returns a strange object

I'm making an API request with Frisbee:
const Frisbee = require('frisbee')
const api = new Frisbee({
baseURI: 'http://192.168.1.8:4000',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json'
}
})
export const handleSubmit = async (values): void => {
Toast.show('Uploading Product', {
duration: 3000,
position: 30,
shadow: true,
animation: true,
hideOnPress: true,
delay: 0
})
try {
const response = api.post('/products', {
body: encodeAddProductAction(values)
})
if (response.err) throw response.err
console.log(response)
} catch (err) {
console.error(err)
}
}
export const encodeAddProductAction = (values: any) => {
const submitPayload = Object.assign({}, values)
Object.keys(submitPayload).forEach((key) => {
if (key != 'Categories') {
submitPayload[key] = encodeURIComponent(
JSON.stringify(submitPayload[key])
)
} else {
// values[key] = JSON.stringify(values[key])
submitPayload[key] = submitPayload[key].join(',')
}
})
return submitPayload
}
It console logs this:
Why is my API response body buried in some strange fields like _55?
My response is:

You are using async but you are not awaiting response, that's why you get Promise returned.
try {
const response = await api.post('/products', {
body: encodeAddProductAction(values)
})
if (response.err) throw response.err
console.log(response)
} catch (err) {
console.error(err)
}
Async by default returns a Promise.

How do I return the contents of this function to the client side?

I have a function that I call using
fetch(http://localhost:8888/.netlify/functions/token-hider?
stateName=' +stateName)
on my client side.
the token-hider function looks like this:
const qs = require("qs");
const fetch = require("node-fetch");
var alertEndpoint = "";
var parkEndpoint = "";
var parksWithAlerts = "";
exports.handler = async function getURLS(event, context, callback)
{
// Get env var values defined in our Netlify site UI
const {api_key, alert_api_url, park_api_url} = process.env;
var stateName =event.queryStringParameters.stateName;
alertEndpoint = `${alert_api_url}${stateName}${api_key}`;
parkEndpoint = `${park_api_url}${stateName}${api_key}`;
getData();
async function getData(alertsArea, alertHeader) {
const [getAlertData, getParkData] = await
Promise.all([fetch(alertEndpoint), fetch(parkEndpoint)] );
var alertResults = await getAlertData.json();
var parkResults= await getParkData.json();
var alertData = alertResults.data;
var parkData = parkResults.data;
parksWithAlerts = parkData.map(park => {
park.alertData = alertData.filter(alert => alert.parkCode ===
park.parkCode);
return park
});
console.log(parksWithAlerts);
}
console.log(callback);
};
how could I return the contents of parksWithAlerts back to the client side after this function is finished?

Try to learn more about callback functions in Javascript.
It is right there in your code, the callback that you are printing is actually suppose to be called after you have executed your code and you can do like this callback(parksWithAlerts);.
While calling the function getURLS you will provide the function which is suppose to get called with args.
Examples : https://www.geeksforgeeks.org/javascript-callbacks/

Here is an example with error handling and returning a response type of JSON
token-hider
import fetch from "node-fetch";
// Get env var values defined in our Netlify site UI
const {api_key, alert_api_url, park_api_url} = process.env;
async function getJson(response) {
return await response.json();
}
const alertEndpoint = stateName => {
return new Promise(function(resolve, reject) {
fetch(`${alert_api_url}${stateName}${api_key}`)
.then(response => {
if (!response.ok) { // NOT res.status >= 200 && res.status < 300
return reject({ statusCode: response.status, body: response.statusText });
}
return resolve(getJson(response))
})
.catch(err => {
console.log('alertEndpoint invocation error:', err); // output to netlify function log
reject({ statusCode: 500, body: err.message });
})
});
}
const parkEndpoint = stateName => {
return new Promise(function(resolve, reject) {
fetch(`${park_api_url}${stateName}${api_key}`)
.then(response => {
if (!response.ok) { // NOT res.status >= 200 && res.status < 300
return reject({ statusCode: response.status, body: response.statusText });
}
return resolve(getJson(response))
})
.catch(err => {
console.log('parkEndpoint invocation error:', err); // output to netlify function log
reject({ statusCode: 500, body: err.message });
})
})
}
exports.handler = function(event, context) {
const stateName = event.queryStringParameters.stateName;
return Promise.all([alertEndpoint(stateName), parkEndpoint(stateName)])
.then(values => {
const [alertData, parkData] = values;
const parksWithAlerts = parkData.map(park => {
park.alertData = alertData.filter(alert => alert.parkCode === park.parkCode);
return park;
});
return {
statusCode: 200,
headers: { 'content-type': 'application/json' },
body: JSON.stringify(parksWithAlerts)
};
})
.catch(error => {
return error;
});
};
NOTE: If you are trying to hide the token, make sure to not deploy this from a public repository on Netlify.
Also, this code has not been tested 100%, so there may be some things to resolve. The response layout and structure is something I use in a few of my lambda functions on Netlify.