We Keep Coding

Promise is opening too many HTTP Connections (Example)

I have a very simple code that calls an external API
const promises = [];
for (const elem of data) {
const carId = Number(elem.carId);
promises.push(getCarData(carId));
}
console.log('Waiting for all promises to finish');
const result = await Promise.allSettled(promises);
export const getCarData = async (carId: number) => {
return new Promise(async (resolve, reject) => {
const car = await API.getCar(carId);
if (!car) {
return reject(new Error('Could not find car'));
}
return resolve(car);
});
}
-- API --
import axios from 'axios';
import https from 'https';
const axiosInstance = axios.create({
httpsAgent: new https.Agent({
rejectUnauthorized: false
})
});
static getCar = async(carId: number) => {
try {
const response = await axiosInstance.get(`${baseUrl}/v1/car/${carId}.json`);
const car = response.data;
return car;
} catch (err) {
// something else
}
}
There are like 10.000 requests being made in the promise array for each element in data. It starts working well but after some time I start getting errors in some of the promises
"connect EMFILE IP:443 - Local (undefined:undefined)"
I checked and it seems that it's related to the OS not wanting to open more sockets. This is hosted in AWS obviously, so what can I do to slow down the requests and not open so many connections at the same time?

Promise.allSettled is part of the concurrent methos.
You could write your own promise handle or find some package.
simple example assuming each request takes the same time:
//run multiple promises sequentialy.
async function runSeq(ps:Promise[]){
for(const p of ps){
await p;
}
}
//fake 20 requests.
const ps =new Array(20).fill(Promise.resolve(0)).map((_v,i)=>i);
//split into an array with 3 elements.
d3=a.length/3;
const pss = [a.splice(0,Math.ceil(d3)),a.splice(0,Math.ceil(d3)),a]
//execute 3 concurrent promises.
await Promise.all(pss.map(ps=>await runSeq(ps)))

Cannot render and map POST request array promise

I have an API called getQuote and a component called QuoteCard. Inside QuoteCard I'm trying to render an array of users that liked a quote. The API works fine, I have tested it, and the code below for getting the users works fine too.
const Post = async (url, body) => {
let res = await fetch(url, {
method: "POST",
headers: {
"Content-Type": "application/json",
"accept": "*/*"
},
body: JSON.stringify(body)
}).then(r => r.json());
return res;
}
const getAllLikes = async () => {
let users = await Post('api/getQuote', {
id: "639e3aff914d4c4f65418a1b"
})
return users
}
console.log(getAllLikes())
The result is working as expected :
However, when trying to map this promise result array to render it onto the page is where I have problems. I try to render like this:
<div>
{getAllLikes().map((user) => (
<p>{user}</p>
))}
</div>
However, I get an error that states:
getAllLikes(...).map is not a function
I don't understand why this is happening. Why can't I map the array? Is it because it's a promise or something?
And if anyone needs to see the getQuote API, here it is:
//Look ma I wrote an API by myself! :D
import clientPromise from "../../lib/mongodb";
const ObjectId = require('mongodb').ObjectId;
import nc from "next-connect";
const app = nc()
app.post(async function getQuote(req, res) {
const client = await clientPromise;
const db = client.db("the-quotes-place");
try {
let quote = await db.collection('quotes').findOne({
_id: new ObjectId(req.body.id)
})
res.status(200).json(JSON.parse(JSON.stringify(quote.likes.by)));
} catch (e) {
res.status(500).json({
message: "Error getting quote",
success: false
})
console.error(e);
}
})
export default app
Thanks for any help!

It is due to the fact that getAllLikes is an async function and thus it returns promise which does not have a map function.
You can either save it in a state variable before using await Or chain it with .then.
Minimal reproducible example which works
const getAllLikes = async () => {
return ['a', 'b']
}
getAllLikes().then((r) => r.map((g) => { console.log(g) }))
Edit: The above code won't work if directly used with jsx since the return of getAllLikes will still be a promise. Solution would be to save it in a state variable and then using it.

I am from Angular and I believe we call pipe on Observables (or Promises). Map can then be called inside the pipe function
observable$ = getAllLikes().pipe(map( user => <p>{user}</p>))
If there is no pipe, I can only think of manually subscribing (which is not a good practice)
sub$ = getAllLikes().subscribe( user => <p>{user}</p>)
// unsub from sub$ appropriately
// We do this from ngOnDestroy in angular
ngOnDestroy() {
this.sub$?.unsubscribe()
}

How do I call Airtable rest API from inside an AWS Lambda?

I am trying to call my rest api endpoint in AIRTABLE from inside an AWS Lambda with no success. I get no errors, no outputs.
If I call the same code using node - it works.
I am able to use Axios in my code.
Pure airtable code (works)
var Airtable = require('airtable');
var base = new Airtable({apiKey: 'keyoMYSECRETKEY'}).base('Mybaseid');
base('MyBase').select({maxRecords: 3,view: "MyView"}).eachPage(function page(records, fetchNextPage) {
// This function (`page`) will get called for each page of records.
records.forEach(function(record) {
console.log('Retrieved',JSON.stringify(record.get('Session Information')));
});
fetchNextPage();
}, function done(err) {
if (err) { console.error(err); return; }
});
If I put it inside a Lambda handler - I get nothing.
const axios = require('axios')
const url = 'https://checkip.amazonaws.com/';
var Airtable = require('airtable');
var base = new Airtable({apiKey: 'keySECRETKEY'}).base('MYBASEID');
let response;
exports.lambdaHandler = async (event, context) => {
try {
base('MyBase').select({maxRecords: 3,view: "MyView"}).eachPage(function page(records, fetchNextPage) {
records.forEach(function(record) { //HERE - NOTHING HAPPENS
console.log('Retrieved',JSON.stringify(record.get('Session Information')));
});
fetchNextPage();
}, function done(err) {
if (err) { console.error(err); return; }
});
const ret = await axios(url); //THIS WORKS
response = {
'statusCode': 200,
'body': JSON.stringify({
message: 'hello world - boo',
location: ret.data.trim()
})
}
} catch (err) {
console.log(err);
return err;
}
return response
};
What am I missing so I can call Airtable API from inside an AWS Lambda?

It seems that your lambda terminates before the API call execution your trying to perform.
I believe this will be solved using a synchronous lambda or with a correct usage of promises with await calls.

Best way to troubleshoot this is to go back to the basics.
See if you can at least get a meaningful console log by wrapping a simpler fetch request into a lambda handler:
const baseId = 'exampleAppId123';
const tableName = 'Table 1';
const api_key = 'keyExample123';
const url = `https://api.airtable.com/v0/${baseId}/${tableName}?api_key=${api_key}`;
exports.lambdaHandler = async () => {
const res = await fetch(url)
.then(res => res.json())
.then(data=>console.log(data))
.then(() => {
//do more stuff
})
}
Then report back if you can't. Or better yet, report back either way as that's bound to help more people in the future.
Worst case? The above code still doesn't do anything. If that happens, I suggest going with #Shoty's first instinct and turning this code into a synchronous fetch request by removing the async/await syntax and returning chained thenables. Not that blocking behavior of this sort is acceptable from a UX perspective, but it should at least help with debugging.

Why is my asynchronous input undefined in useEffect?

I'm writing a React application that fetches image data from a server for an array of URLs. I am storing the camera images as large strings that are placed into the image's src attribute. I am using useReducer to store my dictionary of camera objects.
I am having a couple of problems getting the reducer to work, and one of them has to do with some confusion I'm having with asynchronous values and why the async function returns correct output but the completion handler (.then()) receives undefined as a result.
Here is the code for useEffect() and the asynchronous fetching function.
useEffect()
//Why is cameras undefined?
useEffect(() => {
if (phase === 0) {
let cameras = {}
getCameraInformation().then((cameras) => {
debugger;
dispatch({
type: 'loadedCameraInformation',
payload: {cameras: cameras}
});
}).finally(() => setPhase(1))
}
});
My function signature and variables:
export default function Main() {
const [state, dispatch] = useReducer(cameraReducer, initialState);
let [phase, setPhase] = useState(0);
My function for getCameraInformation:
This returns a dictionary full of correct information!
async function getCameraInformation() {
//returns a json with the following: url, cam_name, cam_pass, cam_user, channel, chunk, group, path, port,
// uptime, username.
let cam_json = await axios
.get(getCamerasURL, { headers: { auth: get_cookie("token") } })
.then(response => {
let tempCameraArray = response.data.body;
let tempCameraDictionary = {};
for (var camera in tempCameraArray) {
tempCameraDictionary[tempCameraArray[camera].sid] = {
cameraInformation: tempCameraArray[camera],
cameraImage: null
};
}
return tempCameraDictionary;
})
.catch(error => console.log(error));
}

Your async function getCameraInformation doesn't have a return statement, so its promise will not resolve any value. There is a return in the then callback, but that's a different function entirely.
You are also using await and then() on the same promise, which isn't ideal. Use one or the other, because it's very easy to get confused when you mix and match here.
You already have an async, so don't use then at all in side that function.
async function getCameraInformation() {
//returns a json with the following: url, cam_name, cam_pass, cam_user, channel, chunk, group, path, port,
// uptime, username.
let response = await axios.get(getCamerasURL, { headers: { auth: get_cookie('token') } })
let tempCameraArray = response.data.body
let tempCameraDictionary = {}
for (var camera in tempCameraArray) {
tempCameraDictionary[tempCameraArray[camera].sid] = {
cameraInformation: tempCameraArray[camera],
cameraImage: null,
}
}
return tempCameraDictionary
}

How to avoid sending multiple duplicate AJAX requests in axios

Is it possible to automatically throttle all requests going to a particular list of endpoints using axios? Perhaps using axios interceptor?
Currently I throttle the user action that sends the axios request, but the problem with that is that I have to write this everywhere I have a user action that results in some AJAX request. Like this
const throttledDismissNotification = throttle(dismissNotification, 1000)
const dismiss = (event: any) => {
throttledDismissNotification();
};
render() {
return (
<Button onClick={dismiss}>Dismiss Notification</Button>
)
}
This results in a lot of clutter and I was wondering if this could be automated.
Something like:
if(request.url in listOfEndpointsToThrottle && request.params in cacheOfPreviousRequestsToThisEndpoint) {
StopRequest();
}
Obviously this is pseudocode but you get the idea.

Perhaps you could try to use the Cancellation feature that axios provides.
With it, you can ensure that you don't have any two (or more, depending on your implementation) similar requests in a pending state.
Below, you will find a small simplified example of how to ensure that only the latest request is processed. You can adjust it a bit to make it function like a pool of requests
import axios, { CancelToken } from 'axios';
const pendingRequests = {};
const makeCancellable = (headers, requestId) => {
if (!requestId) {
return headers;
}
if (pendingRequests[requestId]) {
// cancel an existing request
pendingRequests[requestId].cancel();
}
const source = CancelToken.source();
const newHeaders = {
...headers,
cancelToken: source.token
};
pendingRequests[requestId] = source;
return newHeaders;
};
const request = ({
url,
method = 'GET',
headers,
id
}) => {
const requestConfig = {
url,
method,
headers: makeCancellable(headers || {}, id)
};
return axios.request(requestConfig)
.then((res) => {
delete pendingRequests[id];
return ({ data: res.data });
})
.catch((error) => {
delete pendingRequests[id];
if (axios.isCancel(error)) {
console.log(`A request to url ${url} was cancelled`); // cancelled
} else {
return handleReject(error);
}
});
};
export default request;

It's quite easy to throttle an axios request itself. The real headache is how to handle the promises that are returned from nullified requests. What is considered sane behavior when dealing with promises that are returned from a nullified axios request? Should they stay pending forever?
I don't see any perfect solution to this problem. But then I come to a solution that is kind of cheating:
What if we don't throttle the axios call, instead we throttle the actual XMLHttpRequest?
This makes things way easier, because it avoids the promise problem, and it's easier to implement. The idea is to implement a cache for recent requests, and if a new request matches a recent one, you just pull the result from cache and skip the XMLHttpRequest.
Because of the way axios interceptors work, the following snippet can be used to skip a certain XHR call conditionally:
// This should be the *last* request interceptor to add
axios.interceptors.request.use(function (config) {
/* check the cache, if hit, then intentionally throw
* this will cause the XHR call to be skipped
* but the error is still handled by response interceptor
* we can then recover from error to the cached response
**/
if (requestCache.isCached(config)) {
const skipXHRError = new Error('skip')
skipXHRError.isSkipXHR = true
skipXHRError.request = config
throw skipXHRError
} else {
/* if not cached yet
* check if request should be throttled
* then open up the cache to wait for a response
**/
if (requestCache.shouldThrottle(config)) {
requestCache.waitForResponse(config)
}
return config;
}
});
// This should be the *first* response interceptor to add
axios.interceptors.response.use(function (response) {
requestCache.setCachedResponse(response.config, response)
return response;
}, function (error) {
/* recover from error back to normality
* but this time we use an cached response result
**/
if (error.isSkipXHR) {
return requestCache.getCachedResponse(error.request)
}
return Promise.reject(error);
});

I have a similar problem, thru my research it seems to lack a good solution. All I saw were some ad hoc solutions so I open an issue for axios, hoping someone can answer my question https://github.com/axios/axios/issues/2118
I also find this article Throttling Axios requests but I did not try the solution he suggested.
And I have a discussion related to this My implementation of debounce axios request left the promise in pending state forever, is there a better way?

I finish one, #hackape thank you for you answer, the code is as follows:
const pendings = {}
const caches = {}
const cacheUtils = {
getUniqueUrl: function (config) {
// you can set the rule based on your own requirement
return config.url + '&' + config.method
},
isCached: function (config) {
let uniqueUrl = this.getUniqueUrl(config)
return caches[uniqueUrl] !== undefined
},
isPending: function (config) {
let uniqueUrl = this.getUniqueUrl(config)
if (!pendings[uniqueUrl]) {
pendings[uniqueUrl] = [config]
return false
} else {
console.log(`cache url: ${uniqueUrl}`)
pendings[uniqueUrl].push(config)
return true
}
},
setCachedResponse: function (config, response) {
let uniqueUrl = this.getUniqueUrl(config)
caches[uniqueUrl] = response
if (pendings[uniqueUrl]) {
pendings[uniqueUrl].forEach(configItem => {
configItem.isFinished = true
})
}
},
getError: function(config) {
const skipXHRError = new Error('skip')
skipXHRError.isSkipXHR = true
skipXHRError.requestConfig = config
return skipXHRError
},
getCachedResponse: function (config) {
let uniqueUrl = this.getUniqueUrl(config)
return caches[uniqueUrl]
}
}
// This should be the *last* request interceptor to add
axios.interceptors.request.use(function (config) {
// to avoid careless bug, only the request that explicitly declares *canCache* parameter can use cache
if (config.canCache) {
if (cacheUtils.isCached(config)) {
let error = cacheUtils.getError(config)
throw error
}
if (cacheUtils.isPending(config)) {
return new Promise((resolve, reject) => {
let interval = setInterval(() => {
if(config.isFinished) {
clearInterval(interval)
let error = cacheUtils.getError(config)
reject(error)
}
}, 200)
});
} else {
// the head of cacheable requests queue, get the response by http request
return config
}
} else {
return config
}
});