I'm trying to cover some code in a unit test (specifically, the code inside the catch block) and I believe it's failing due to memory leak errors probably due to timers that are not properly mocked.
Here's an example of the main code:
const delay = new Promise(resolve => setTimeout(() => resolve(), 1000);
const readCustomerStatus = async (retry = 2) => {
let content = '';
try{
await delay;
content = await fs.readFileSync(CUSTOMER_PATH,'utf8');
}
catch(error){
while(retry > 0){
readCustomerStatus(retry - 1);
}
}
return content;
}
And here's the unit test code:
test('Should return error when readFileSync fails'),async()=>{
fs.readFileSync = jest.fn()
.mockRejectedValueOnce(new Error('readFileSync failed'))
.mockRejectedValueOnce(new Error('readFileSync failed'));
const customerStatusResult = await readCustomerStatus();
expect(customerStatusResult).toThrow(‘readFileSync failed’);
}
When I remove the recursive call from the main code, the unit test works. It looks like somehow the delay promise might be somehow affecting the test. I have also tried adding jest.useFakeTimers() in the beforeAll function but that doesn't seem to be helping either.
It looks like your code has multiple issues, most of them related to the way you are (not) awaiting Promises.
When you create a Promise this way, it will start executing right away, and will be resolved only once:
// Timer starts right away, which I believe it is not what you wanted
const delay = new Promise(resolve => setTimeout(() => resolve(), 1000);
So this is the behavior:
await delay // Awaits `delay` Promise to be resolved.
// Depending on when it started, might be less than 1 second
await delay // This second invocation will be already resolved. 0 seconds delay.
Same is happening to your recursive version. You are awaiting that same delay Promise multiple times (on each recursion), so you are not actually pausing for 1 second on each iteration.
Here you have another issue:
catch(error){
while(retry > 0){
readCustomerStatus(retry - 1); // issue: readCustomerStatus is async
}
}
In this line you are invoking an async function without await, so you are just creating a new unhandled Promise, and completing the execution of the first one with success (because you trapped the error and ignored it).
You also shouldn't use a while loop in your retries, because that will cause a never ending loop, since retry variable is local to each call. Replacing by an if will fix it.
Finally, you are missing to raise the original error if all retries failed (will show how to fix below).
I wouldn't recommend using recursion to handle retries, I think it makes the code less readable and also there is some unnecessary stack allocation. However, for educational purposes, let's fix this code:
const delay = (ms=1000)=>new Promise(resolve=>setTimeout(resolve, ms));
const readCustomerStatus = async (retry = 2) => {
let content = '';
try {
// Notice we invoked function here to create a new Promise on each iteration
await delay();
content = await fs.readFileSync(CUSTOMER_PATH,'utf8');
} catch(error) {
if (retry > 0) {
// it is important to use await here and assign the result to `content`
content = await readCustomerStatus(retry - 1);
} else {
// This is our base case, so we just raise whatever error we found
throw error;
}
}
return content;
}
Related
I am calling two APIs with two different functions and having a setTimeout for both, is it possible to handle their timeouts in such a way that they finish in 10s instead of 15s.
function delay(s){
return new Promise(resolve => setTimeout(resolve,s));
}
async function getXml(){
let ans;
await delay(10000)
const {data} = await axios.get('https://gist.githubusercontent.com/SwayamShah97/a3619c5828ac8ed8085c4ae295a855d9/raw/e4e372552e042bd8bd9e8ab87da93eb030114f86/people.xml');
xml2js.parseString(data, (err, result) => {
if(err) {
throw err;
}
ans = result
});
return ans;
}
async function getPeople(){
await delay(5000)
const { data } = await axios.get('https://gist.githubusercontent.com/SwayamShah97/0f2cb53ddfae54eceea083d4aa8d0d65/raw/d7d89c672057cf7d33e10e558e001f33a10868b2/people.json');
return data; // this will be the array of people objects
}
Is there a way to run this code only in 10s, so that both the APIs get called in 10s time period
Forcing either one or both axios.get() functions to complete below some time limit (other than failing on a timeout), isn't doable, since you don't control the transport.
One thing you can do is force one or more functions to complete at or after some time threshold, with something like this...
function padToTime(promise, interval) {
// delay returns a promise that resolves after an interval
const delay = interval => new Promise(resolve => setTimeout(resolve, interval));
// caller can provide a singular or an array of promises, avoiding the extra .all
let promises = Array.isArray(promise) ? promise : [promise];
return Promise.all([...promises, delay(interval)])
.then(results => results.slice(0, -1));
}
EDIT good idea from #VLAZ to append an extra promise that forces minimal time (then slice its result off later).
The caller can say:
async function getXml(){
// op function, no calls to "delay"
}
async function getPeople(){
// op function, no calls to "delay"
}
// run these both together, have them take *no fewer than* 10s
padToTime([getXml(),getPeople()], 10000).then(results => {
// we'll get here in no fewer than 10sec with the promise all results
})
I want to test my logic and ensure that I don't exceed a limit of X requests every X seconds. Using mocha#^9.1.3, chai#^4.3.4 and sinon#^12.0.1, latest versions as of this comment.
I've read answers to very similar problems to mine, tried applying the knowledge gained but to no avail:
https://stackoverflow.com/a/52196951 — great explanation as to the why, however I must've misunderstood something
https://stackoverflow.com/a/55448441 — similar to 1.
https://stackoverflow.com/a/43048888, https://stackoverflow.com/a/60629795 — adding clock.tickAsync inside the wait utility function would not let me step through the code
I have a wait utility function looking like this:
const wait = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
and my code to reproduce my issue in my test file looks like (please do read comments in code):
import sinon from 'sinon';
import got from 'got';
describe('sinon usefaketimers promise-timeouts', () => {
let clock: sinon.SinonFakeTimers;
beforeEach(() => {
clock = sinon.useFakeTimers({ toFake: ['setTimeout'] });
// i've also tried this
// clock = sinon.useFakeTimers();
});
afterEach(() => {
clock.restore();
});
it('steps through all promise-timeouts manually', async () => {
const main = async () => {
console.log('starting');
await wait(1000);
console.log('waited 1s');
await wait(1000);
console.log('waited 2s');
// switching out `got()` below to `await Promise.resolve()`
// finishes the test successfully
await got('https://jsonplaceholder.typicode.com/todos/1').json(); // <-- switch to `await Promise.resolve()` makes it work
console.log('request done');
await wait(1000);
console.log('waited 3s');
};
const promise = main();
await clock.tickAsync(1000); // I have tried adding different combinations
await clock.tickAsync(1000); // of `await Promise.resolve()` after the tick(s)
await clock.tickAsync(1000);
return promise;
});
});
and the resulting output is, (it hangs after "request done", missing the last line "waited 3s"):
starting
waited 1s
waited 2s
request done
Note, I've tried switching out got to superagent and node-fetch without any success. I'm not entirely sure why they would be any different from Promise.resolve as they are all promises returned and awaited on.
Really hope to clear up whatever I didn't get from reading above questions (and so many other threads) from yesterday and todays debugging.
I have an API which is limited regarding how many requests per minute (50/minute) I can send to any endpoint provided by that API.
In the following code-section, I filter the objects orders with an URL as property, every object with an URL that provides data should be stored in successfullResponses in my app.component.ts.
Promise.all(
orders.map(order => this.api.getURL(order.resource_url).catch(() => null))
).then(responses => {
const successfulResponses = responses.filter(response => response != null)
for(let data of successfulResponses) {
// some other requests should be sent with data here
}
});
There are more than 50 orders to check, but I just can check maximum 50 orders at once, so I try to handle it in my service. I set the first date when the first request is sent. After that I compare the dates of the new request with the first one. If the difference is over 60, I set the current date to the new one and set maxReq again to 50. If it is under 60, I check if there are requests left, if yes I send the request and if not I just wait one minute :
sleep(ms){
return new Promise(resolve => setTimeout(resolve, ms));
}
async getURL(){
if(!this.date){
let date = new Date();
this.date = date;
}
if((new Date().getSeconds() - this.date.getSeconds() > 60 )){
this.maxReq = 50;
this.date = new Date();
return this.http.get(url, this.httpOptions).toPromise();
} else {
if(this.maxReq > 0){
this.maxReq -= 1;
return this.http.get(url, this.httpOptions).toPromise();
} else{
console.log("wait");
await this.sleep(60*1000);
this.maxReq = 50;
this.date = new Date();
return this.http.get(url, this.httpOptions).toPromise();
}
}
}
However the code in app.component.tsis not waiting for the function getURL() and executes further code with requests which leads to the problem that I send ´too many requests too quickly´.
What can I do about that?
I had a similar problem while trying to use promises with multiple async functions. It's an easy thing to forget, but in order to make them all wait, you have to use await on the root line that calls the function in question.
I'm not entirely certain, but my presumption is that your await this.sleep(60*1000); line is indeed waiting for a timeout to occur, but whilst it is doing this, the code that called getURL() is executing the rest of its lines, because it did not have an await (or equivalent, like .then) before getURL().
The way I discovered this in my case was by using a good debugging tool (I used Chrome DevTools's own debugging features). I advise you do the same, adding breakpoints everywhere, and see where your code is going with each line.
Here is a short, rough example to show what I mean:
// This code increments a number from 1 to 2 to 3 and returns it each time after a delay of 1 second.
async function loop() {
for (i = 1; i <= 3; i++) {
console.log('Input start');
/* The following waits for result of aSync before continuing.
Without 'await', it would execute the last line
of this function whilst aSync's own 'await'
waited for its result.
--- This is where I think your code goes wrong. --- */
await aSync(i);
console.log('Input end');
}
}
async function aSync(num) {
console.log('Return start');
/* The following waits for the 1-second delay before continuing.
Without 'await', it would return a pending promise immediately
each time. */
let result = await new Promise(
// I'm not using arrow functions to show what it's doing more clearly.
function(rs, rj) {
setTimeout(
function() {
/* For those who didn't know, the following passes the number
into the 'resolved' ('rs') parameter of the promise's executor
function. Without doing this, the promise would never be fulfilled. */
rs(num);
}, 1000
)
}
);
console.log(result);
console.log('Return end');
}
loop();
Given the code samples below, is there any difference in behavior, and, if so, what are those differences?
return await promise
async function delay1Second() {
return (await delay(1000));
}
return promise
async function delay1Second() {
return delay(1000);
}
As I understand it, the first would have error-handling within the async function, and errors would bubble out of the async function's Promise. However, the second would require one less tick. Is this correct?
This snippet is just a common function to return a Promise for reference.
function delay(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
Most of the time, there is no observable difference between return and return await. Both versions of delay1Second have the exact same observable behavior (but depending on the implementation, the return await version might use slightly more memory because an intermediate Promise object might be created).
However, as #PitaJ pointed out, there is one case where there is a difference: if the return or return await is nested in a try-catch block. Consider this example
async function rejectionWithReturnAwait () {
try {
return await Promise.reject(new Error())
} catch (e) {
return 'Saved!'
}
}
async function rejectionWithReturn () {
try {
return Promise.reject(new Error())
} catch (e) {
return 'Saved!'
}
}
In the first version, the async function awaits the rejected promise before returning its result, which causes the rejection to be turned into an exception and the catch clause to be reached; the function will thus return a promise resolving to the string "Saved!".
The second version of the function, however, does return the rejected promise directly without awaiting it within the async function, which means that the catch case is not called and the caller gets the rejection instead.
As other answers mentioned, there is likely a slight performance benefit when letting the promise bubble up by returning it directly — simply because you don’t have to await the result first and then wrap it with another promise again. However, no one has talked about tail call optimization yet.
Tail call optimization, or “proper tail calls”, is a technique that the interpreter uses to optimize the call stack. Currently, not many runtimes support it yet — even though it’s technically part of the ES6 Standard — but it’s possible support might be added in the future, so you can prepare for that by writing good code in the present.
In a nutshell, TCO (or PTC) optimizes the call stack by not opening a new frame for a function that is directly returned by another function. Instead, it reuses the same frame.
async function delay1Second() {
return delay(1000);
}
Since delay() is directly returned by delay1Second(), runtimes supporting PTC will first open a frame for delay1Second() (the outer function), but then instead of opening another frame for delay() (the inner function), it will just reuse the same frame that was opened for the outer function. This optimizes the stack because it can prevent a stack overflow (hehe) with very large recursive functions, e.g., fibonacci(5e+25). Essentially it becomes a loop, which is much faster.
PTC is only enabled when the inner function is directly returned. It’s not used when the result of the function is altered before it is returned, for example, if you had return (delay(1000) || null), or return await delay(1000).
But like I said, most runtimes and browsers don’t support PTC yet, so it probably doesn’t make a huge difference now, but it couldn’t hurt to future-proof your code.
Read more in this question: Node.js: Are there optimizations for tail calls in async functions?
Noticeable difference: Promise rejection gets handled at different places
return somePromise will pass somePromise to the call site, and await somePromise to settle at call site (if there is any). Therefore, if somePromise is rejected, it will not be handled by the local catch block, but the call site's catch block.
async function foo () {
try {
return Promise.reject();
} catch (e) {
console.log('IN');
}
}
(async function main () {
try {
let a = await foo();
} catch (e) {
console.log('OUT');
}
})();
// 'OUT'
return await somePromise will first await somePromise to settle locally. Therefore, the value or Exception will first be handled locally. => Local catch block will be executed if somePromise is rejected.
async function foo () {
try {
return await Promise.reject();
} catch (e) {
console.log('IN');
}
}
(async function main () {
try {
let a = await foo();
} catch (e) {
console.log('OUT');
}
})();
// 'IN'
Reason: return await Promise awaits both locally and outside, return Promise awaits only outside
Detailed Steps:
return Promise
async function delay1Second() {
return delay(1000);
}
call delay1Second();
const result = await delay1Second();
Inside delay1Second(), function delay(1000) returns a promise immediately with [[PromiseStatus]]: 'pending. Let's call it delayPromise.
async function delay1Second() {
return delayPromise;
// delayPromise.[[PromiseStatus]]: 'pending'
// delayPromise.[[PromiseValue]]: undefined
}
Async functions will wrap their return value inside Promise.resolve()(Source). Because delay1Second is an async function, we have:
const result = await Promise.resolve(delayPromise);
// delayPromise.[[PromiseStatus]]: 'pending'
// delayPromise.[[PromiseValue]]: undefined
Promise.resolve(delayPromise) returns delayPromise without doing anything because the input is already a promise (see MDN Promise.resolve):
const result = await delayPromise;
// delayPromise.[[PromiseStatus]]: 'pending'
// delayPromise.[[PromiseValue]]: undefined
await waits until the delayPromise is settled.
IF delayPromise is fulfilled with PromiseValue=1:
const result = 1;
ELSE is delayPromise is rejected:
// jump to catch block if there is any
return await Promise
async function delay1Second() {
return await delay(1000);
}
call delay1Second();
const result = await delay1Second();
Inside delay1Second(), function delay(1000) returns a promise immediately with [[PromiseStatus]]: 'pending. Let's call it delayPromise.
async function delay1Second() {
return await delayPromise;
// delayPromise.[[PromiseStatus]]: 'pending'
// delayPromise.[[PromiseValue]]: undefined
}
Local await will wait until delayPromise gets settled.
Case 1: delayPromise is fulfilled with PromiseValue=1:
async function delay1Second() {
return 1;
}
const result = await Promise.resolve(1); // let's call it "newPromise"
const result = await newPromise;
// newPromise.[[PromiseStatus]]: 'resolved'
// newPromise.[[PromiseValue]]: 1
const result = 1;
Case 2: delayPromise is rejected:
// jump to catch block inside `delay1Second` if there is any
// let's say a value -1 is returned in the end
const result = await Promise.resolve(-1); // call it newPromise
const result = await newPromise;
// newPromise.[[PromiseStatus]]: 'resolved'
// newPromise.[[PromiseValue]]: -1
const result = -1;
Glossary:
Settle: Promise.[[PromiseStatus]] changes from pending to resolved or rejected
This is a hard question to answer, because it depends in practice on how your transpiler (probably babel) actually renders async/await. The things that are clear regardless:
Both implementations should behave the same, though the first implementation may have one less Promise in the chain.
Especially if you drop the unnecessary await, the second version would not require any extra code from the transpiler, while the first one does.
So from a code performance and debugging perspective, the second version is preferable, though only very slightly so, while the first version has a slight legibility benefit, in that it clearly indicates that it returns a promise.
In our project, we decided to always use 'return await'.
The argument is that "the risk of forgetting to add the 'await' when later on a try-catch block is put around the return expression justifies having the redundant 'await' now."
Here is a typescript example that you can run and convince yourself that you need that "return await"
async function test() {
try {
return await throwErr(); // this is correct
// return throwErr(); // this will prevent inner catch to ever to be reached
}
catch (err) {
console.log("inner catch is reached")
return
}
}
const throwErr = async () => {
throw("Fake error")
}
void test().then(() => {
console.log("done")
}).catch(e => {
console.log("outer catch is reached")
});
here i leave some code practical for you can undertand it the diferrence
let x = async function () {
return new Promise((res, rej) => {
setTimeout(async function () {
console.log("finished 1");
return await new Promise((resolve, reject) => { // delete the return and you will see the difference
setTimeout(function () {
resolve("woo2");
console.log("finished 2");
}, 5000);
});
res("woo1");
}, 3000);
});
};
(async function () {
var counter = 0;
const a = setInterval(function () { // counter for every second, this is just to see the precision and understand the code
if (counter == 7) {
clearInterval(a);
}
console.log(counter);
counter = counter + 1;
}, 1000);
console.time("time1");
console.log("hello i starting first of all");
await x();
console.log("more code...");
console.timeEnd("time1");
})();
the function "x" just is a function async than it have other fucn
if will delete the return it print "more code..."
the variable x is just an asynchronous function that in turn has another asynchronous function, in the main of the code we invoke a wait to call the function of the variable x, when it completes it follows the sequence of the code, that would be normal for "async / await ", but inside the x function there is another asynchronous function, and this returns a promise or returns a" promise "it will stay inside the x function, forgetting the main code, that is, it will not print the" console.log ("more code .. "), on the other hand if we put" await "it will wait for every function that completes and finally follows the normal sequence of the main code.
below the "console.log (" finished 1 "delete the" return ", you will see the behavior.
My code in order to create multiple placements in an online service that has a 60 write constraint per minute:
placementsToAdd.forEach((placement, index) => {
setTimeout(() => {
options.url = `https://api.company.com/placement?publisher_id=${existingPub ? existingPub : placementsJson[0].PublisherId}&site_id=${placement.siteId}`
options.body = `{"placement":{"name":"${placement.placement}"}}`
request(options, callback);
},1000 * (index + 1))
})
It works this way but I am concerned about the wait time if there are a list of placements of 2000 or 3000 at one time, the wait time might be excessively long.
Is there a better way to refactor this code in order to get my requests built one per second no matter what? Without that "* (index + 1)," it seems to keep trying to build all at once hitting the wall after 60.
I've tried to use promises and async await (which is new to me) but it doesn't seem to change the behavior.
Thanks!
As requested, show how I've tried to use promises with this code:
async function createThePlacements() {
let promise = new Promise((resolve, reject) => {
for (let i = 0; i < placementsToAdd.length; i++) {
setTimeout(() => {
options.url = `https://api.company.com/placement?publisher_id=${existingPub ? existingPub : placementsJson[0].PublisherId}&site_id=${placementsToAdd[i].siteId}`
options.body = `{"placement":{"name":"${placementsToAdd[i].placement}"}}`
request(options, callback);
},1000)
}
});
let result = await promise; // pause till the promise resolves
console.log('result - ', result);
}
createThePlacements();
So, bit of a disclaimer - as mentioned, I've never used Async Await before so reading up to try to understand how it works as well. This seems to be the syntax but my result doesn't seem to be anything at the moment but the code also continues to do what it's supposed to do, just trying to make all the calls in my test of 300 all at once.
Also, of note, i have a resolve inside the callback of the request call. It resolves so even the next parts of my app finish up all the way to the end. That's why I don't have a reject or resolve here.
How do you run a setTimeout once a second inside a forEach loop?
The most straightforward approach would be:
const wait = ms => new Promise(resolve => setTimeout(resolve, ms));
for (const placement of placementsToAdd) {
const options = {...};
request(options, callback);
await wait(1000);
}
await works predictably inside plain for-loops, not inside forEach.
I haven't touched your callback but it would need to handle errors. More refactoring is possible.
The most significant improvement here, I think, is that we're not pushing requests ahead of time. This way we retain control, and should needs change or anything go haywire, we can break out of the loop without spamming the server for another minute.
The best option would be to have a request method that returns a Promise.
Then you could rewrite your code like this.
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
function requestPlacement(placement) {
const options = {...};
return request(options);
}
async function requestAllPlacements(placements) {
for(let i = 0; i < placements.length; i+=60) {
if (i > 0) {
// wait 1 minute
await(sleep(60000));
}
await Promise.all(
placements
.slice(i, 60)
.map(requestPlacement);
);
}
}