We Keep Coding

Retry getting a file after waiting a set timeframe in Node

I am trying to download a report that is generated daily on the first request to the report's endpoint.
When the report is being created, the endpoint returns a HTTP 202.
I have the following code to handle some errors and redirects, as well as trying to "sleep" for 60 seconds before continuing try the endpoint again. Unfortunately, the second console log to tell me the download completed is never called, though the file does indeed download successfully and the filestream closes.
// Main function
run()
async function run() {
await getReport()
await processReport()
}
async function getReport() {
console.log(`Downloading ${reportFileName}`)
await downloadFile(url, reportFileName)
console.log(`Downloaded ${reportFileName} successfully.`) // This is never called?
}
async function downloadFile (url, targetFile) {
return await new Promise((resolve, reject) => {
https.get(url, async response => {
const code = response.statusCode ?? 0
if (code >= 400) {
return reject(new Error(response.statusMessage))
}
// handle redirects
if (code > 300 && code < 400 && !!response.headers.location) {
resolve(downloadFile(response.headers.location, targetFile))
return
}
// handle file creation pending
if (code == 202) {
console.log(`Report: ${reportFileName} is still being generated, trying again in ${timeToSleepMs/1000} seconds...`)
await sleep(timeToSleepMs)
resolve(downloadFile(url, targetFile))
return
}
// make download directory regardless of if it exists
fs.mkdirSync(outputPath, { recursive: true }, (err) => {
if (error) throw error;
});
// save the file to disk
const fileWriter = fs
.createWriteStream(`${outputPath}/${targetFile}`)
.on('finish', () => {
resolve({})
})
response.pipe(fileWriter)
}).on('error', error => {
reject(error)
})
})
}
Finally my sleep function:
let timeToSleepMs = (60 * 1000)
function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
I'm pretty sure this has to do with some sort of async issue because that always seems to be my issue with Node, but I'm not sure how to handle it. I just want to fetch a file and download it locally, retrying if I get a HTTP 202. If there's a better way, please let me know!
tl;dr - How do I properly handle waiting for a HTTP 202 response to turn into a HTTP 200 when the file is generated, then continue executing code after the file is downloaded?

await downloadFile(url, reportFileName) invokes a recursive function and waits for the promise from the outermost invocation to resolve. But if the function calls itself recursively in
await sleep(timeToSleepMs)
return downloadFile(url, targetFile)
this outermost promise is never resolved.
Replace the two lines above with
await sleep(timeToSleepMs)
resolve(downloadFile(url, targetFile))
return
then the resolution of the outermost promise is the resolution of the recursively invoked second-outermost promise, and so on.

Heiko already identified the problem: when you return from the callback, you never resolve the promise. To avoid such mistakes in general, it is recommended not to mix the promisification of a function and the business logic. Do use async/await only in the latter, do not pass async functions as callbacks. In your case, that would be
function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
function httpsGet(url) {
return new Promise((resolve, reject) => {
https.get(url, resolve).on('error', reject);
});
}
function writeFile(path, readableStream) {
return new Promise((resolve, reject) => {
const fileWriter = fs
.createWriteStream(path)
.on('finish', resolve)
.on('error', reject);
readableStream.pipe(fileWriter);
});
}
Then you can easily write a straightforward function without any callbacks:
async function downloadFile (url, targetFile) {
const response = httpsGet(url);
const code = response.statusCode ?? 0
if (code >= 400) {
throw new Error(response.statusMessage);
}
// handle redirects
if (code > 300 && code < 400 && !!response.headers.location) {
return downloadFile(response.headers.location, targetFile)
}
// handle file creation pending
if (code == 202) {
console.log(`Report: ${reportFileName} is still being generated, trying again in ${timeToSleepMs/1000} seconds...`)
await sleep(timeToSleepMs)
return downloadFile(url, targetFile)
}
// make download directory regardless of if it exists
fs.mkdirSync(outputPath, { recursive: true });
// save the file to disk
await writeFile(`${outputPath}/${targetFile}`, response);
}

Make Function Wait Until First Function is Completed to Run

I have this function in my React App. It calls a few other functions. Everything works except, I need the .then(() => this.getHCAid()) to complete before .then(() => this.addDocsList()) runs. I have not been able to make that happens. I'm sure it's simple, I just don't know how.
createHCA() {
fetch(API_URL + `/hca/create`, {
method: "PUT",
body: JSON.stringify({
client: this.state.client,
short: this.state.short,
}),
headers: { "Content-Type": "application/json" },
})
.then((res) => {
if (!res.ok) {
throw new Error();
}
return res.json();
})
.then((data) => console.log(data))
.catch((err) => console.log(err))
.then(() => this.getHCAid()) <--- Need this to complete
.then(() => this.addDocsList()) <--- Before this runs
.then(() => this.getAllHCAs());
this.setState({ showHide: false});
}

I'd encourage you to do a couple of things here...
Use async/await rather than a myraid of .then promises memoize
these function calls and use a useEffect (or multiple) with
dependencies to control when side-effects should be triggered. Convert to functional component to achieve this result.
"showHide" is a very confusing term for future developers, be kind to
your future self and future devs and make a term like "show" or
"isHidden"
If you really want to continue this pattern (chained .then statements), put your .catch at the very bottom to handle any issues/errors. This is where async/await is worth the time to quickly learn.
You only want functions doing one main thing, and this is to createHCA, let your other code trigger the functions. As you scale you will be thankful you did.
async function createHCA() {
let data
try {
data = await fetch(API_URL + `/hca/create`, {
method: "PUT",
body: JSON.stringify({
client: this.state.client,
short: this.state.short,
}),
headers: { "Content-Type": "application/json" },
})
} catch e => {
throw new Error(e.message)
return
}
this.setState({ showHide: false});
return data?.json()
}

It sounds like there are two issues going on here.
The first is that your functions sound like database queries or updates which will be asynchronous so you can't rely on the next step in the cycle to have access to any data returned if it hasn't been told to wait.
The solution you tried to come up with, setting state in one of the then methods, won't work either because it too batches up queries and processes them asynchronously, and the next step won't have access to that data either.
So, ideally what you should probably do is use a promise to return the id from getHCAid, and then pass that into addDocsList which also returns a promise, etc.
Here's a simplified example using async/await.
getHCAid() {
return new Promise((res, rej) => {
// Do DB process, get an id
res(id);
});
}
async getData() {
const response = await fetch(API_URL);
const data = await response.json();
const id = await this.getHCAid();
const list = await this.addDocsList(id);
const hcas = await this.getAllHCAs();
this.setState({ showHide: false });
}

How do I correctly test asynchronous stream error event handling with mocha/sinon/chai?

I am testing an async method that returns some data from a web request using the native https.request() method in NodeJS. I am using mocha, chai, and sinon with the relevant extensions for each.
The method I'm testing essentially wraps the boilerplate https.request() code provided in the NodeJS docs in a Promise and resolves when the response 'end' event is received or rejects if the request 'error' event is received. The bits relevant to discussion look like this:
async fetch(...) {
// setup
return new Promise((resolve, reject) => {
const req = https.request(url, opts, (res) => {
// handle response events
});
req.on('error', (e) => {
logger.error(e); // <-- this is what i want to verify
reject(e);
});
req.end();
});
}
As indicated in the comment, what I want to test is that if the request error event is emitted, the error gets logged correctly. This is what I'm currently doing to achieve this:
it('should log request error events', async () => {
const sut = new MyService();
const err = new Error('boom');
const req = new EventEmitter();
req.end = sinon.fake();
const res = new EventEmitter();
sinon.stub(logger, 'error');
sinon.stub(https, 'request').callsFake((url, opt, cb) => {
cb(res);
return req;
});
try {
const response = sut.fetch(...);
req.emit('error', err);
await response;
} catch() {}
logger.error.should.have.been.calledOnceWith(err);
});
This feels like a hack, but I can't figure out how to do this correctly using the normal patterns. The main problem is I need to emit the error event after the method is called but before the promise is fulfilled, and I can't see how to do that if I am returning the promise as you normally would with mocha.

I should have thought of this, but #Bergi's comment about using setTimeout() in the fake gave me an idea and I now have this working with the preferred syntax:
it('should log request error events', () => {
const sut = new MyService();
const err = new Error('boom');
const req = new EventEmitter();
req.end = sinon.fake();
const res = new EventEmitter();
sinon.stub(logger, 'error');
sinon.stub(https, 'request').callsFake((url, opt, cb) => {
cb(res);
setTimeout(() => { req.emit('error', err); });
return req;
});
return sut.fetch(...).should.eventually.be.rejectedWith(err)
.then(() => {
logger.error.should.have.been.calledOnceWith(err);
});
});
I don't like adding any delays in unit tests unless I'm specifically testing delayed functionality, so I used setTimeout() with 0 delay just to push the emit call to the end of the message queue. By moving it to the fake I was able to just use promise chaining to test the call to the logger method.

can async with fetch poll until a condition is met? (survive rejection)

Using fetch API and async/await, is it possible to continue polling indefinitely, regardless of availability of a URL? I anticipate that a URL might become available eventually, so I want to keep trying until a condition is met. Tried to come up with a minimum viable code sample and I'm not sure I pulled it off:
// this is just a placeholder. It will eventually be a function
// that evaluates something real.
// Assume validContinue gets updated elsewhere.
function shouldContinue() {
return validContinue;
}
async function wonderPoll(someUrl) {
// just a delay mechanism
function wait(ms = 1000) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
// the actual individual poll
async function pollingFunction(url) {
const response = await fetch(url, {
cache: 'no-store'
});
if (response.ok) {
return response;
} else {
Promise.reject(response);
}
}
// allegedly keep polling until condition is met.
// But the rejected Promise is breaking out!
while (shouldContinue()) {
await wait();
result = await pollingFunction(someUrl);
}
// when the fetch hits a rejected state, we never get here!
console.log('done with the while loop, returning last successful result')
return result;
}
const sampleUrl = 'https://get.geojs.io/v1/ip/country.json?ip=8.8.8.8';
const sampleUrl2 = 'http://totallybroken_fo_sho';
// swap the URL to test
wonderPoll(sampleUrl)
.then((result) => {
console.log('got a result', result)
})
.catch((err) => {
console.log('got an error', err)
});
I see what's happening (I think). The parent call ultimately executes the polling function, which rejects on the Promise. The condition to continue is still theoretically met, but the rejection breaks out of the While loop and sends to rejection directly up. This propagates all the way up to the catch method of the original/initial Promise. It doesn't even hit any code that would have come after the While loop in the case of resolved Promises.
What I don't know is how to prevent that from happening. I think I don't understand the syntax for intercepting and resolving the promise. When I replace Promise.reject in the response parser with Promise.resolve(response), it still ends up rejecting up to the top.
If the URL I provide is valid, it will continue until the condition is no longer met.
Here's a fiddle: https://jsfiddle.net/gregpettit/qf495bjm/5/
To use the fiddle, the "stop" button simulates the condition being met, and I've provided two different URLs that have to be manually swapped (by passing someUrl or someUrl2) to test.
Expected results:
with good URL, continuous polling (will have to dig into network in dev tools) until condition is met (by pressing Stop!) and then the calling function's 'then' can show the result.
with bad URL, continuous polling until condition is met, and then calling function's 'catch' shows the error
Actual results:
positive test case is OK
negative test case goes directly to the catch

You can try…catch it to prevent breaking out of loop.
while (shouldContinue()) {
try {
await wait();
result = await pollingFunction(someUrl);
} catch (e) {}
}

Change the code in while loop to try/catch so you can catch the error
result can hold a value when there's no error, or a reason when there is an error
Once the loop is stopped, you either return the value, or throw with the reason
As below
async function wonderPoll(someUrl) {
// just a delay mechanism
function wait(ms = 1000) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
// the actual individual poll
async function pollingFunction(url) {
const response = await fetch(url, {
cache: 'no-store'
});
if (response.ok) {
return response;
} else {
Promise.reject(response);
}
}
// allegedly keep polling until condition is met. But the rejected Promise is breaking out!
while (shouldContinue()) {
try {
await wait();
const value = await pollingFunction(someUrl);
result = {value};
} catch (reason) {
result = {reason};
}
}
// when the fetch hits a rejected state, we never get here!
console.log('done with the while loop, returning last successful result')
if (result.reason) {
throw result.reason;
}
return result.value;
}
Running example https://jsfiddle.net/twkbo9pg/
the example includes status in the result, but that is unnecessary (I borrowed code from my Promise.allSettled polyfill and forgot to remove that property)

you might want to check out observable streams! If you're going to have a lot of data coming in over time, that's rxjs's whole thing.
There's actually a few ways to do this if this feels janky (it kinda does haha).
import { ajax } from "rxjs/ajax";
import { duration } from "moment-timezone"; // I copied this from some old code... whatever.
import { catchError, map, share, switchMap } from "rxjs/operators";
const baseUrl = "http://foo.bar"
const base = (method, headers = {}) => ({
method,
headers: {
Accept: "application/json",
...headers,
},
crossDomain: true,
withCredentials: true,
})
const ajaxGet = url => ajax({ ...base("GET"), url })
export const userEpic = timer(0, duration(5, "minutes").asMilliseconds()).pipe(
switchMap(() =>
ajaxGet(`${baseUrl}/users`).pipe(
map(({ response }) => getUsersSuccess(response)),
catchError(e => of(getUsersError(e))),
)
),
share()
)

Two things
} else {
Promise.reject(response);
}
should return that. It's working "by accident" right now.
} else {
return Promise.reject(response);
}
Secondly, result = await pollingFunction(someUrl); might want to add .catch to it:
result = await pollingFunction(someUrl).catch(_=>null); or whatever can be tested for in the enclosing while
But I think you can simplify the whole thing thus:
export async function wonderPoll(someUrl) {
while (shouldContinue()) {
await wait();
const response = await fetch(someUrl, { cache: 'no-store' });
if (response.ok)
return response;
}
return Promise.reject(); // only if !shouldContinue()
}

Node.JS recursive promise not resolving

i'm working with an API that allows me to sync data to a local DB. There is a syncReady API that I'm calling recursively until the sync batch is ready to start sending data. The recursion is working correctly and the .then callback is called, but the resolve function never resolves the response.
const request = require('request-promise');
const config = require('../Configs/config.json');
function Sync(){}
Sync.prototype.syncReady = function (token, batchID) {
return new Promise((res, rej) => {
config.headers.Get.authorization = `bearer ${token}`;
config.properties.SyncPrep.id = batchID;
request({url: config.url.SyncReady, method: config.Method.Get, headers: config.headers.Get, qs: config.properties.SyncPrep})
.then((response) => {
console.log(`The Response: ${response}`);
res(response);
}, (error) => {
console.log(error.statusCode);
if(error.statusCode === 497){
this.syncReady(token, batchID);
} else rej(error);
}
);
});
};
I get the 497 logged and the "The Response: {"pagesTotal";0}" response but the res(response) never sends the response down the chain. I've added a console.log message along the entire chain and none of the .then functions back down the chain are firing.
I hope I've explained this well enough :-). Any ideas why the promise isn't resolving?
Thanks!

First, you don't need to wrap something that returns a promise with a new Promise. Second, for your error case you don't resolve the promise if it is 497.
const request = require('request-promise');
const config = require('../Configs/config.json');
function Sync(){}
Sync.prototype.syncReady = function (token, batchID) {
config.headers.Get.authorization = `bearer ${token}`;
config.properties.SyncPrep.id = batchID;
return request({url: config.url.SyncReady, method: config.Method.Get, headers: config.headers.Get, qs: config.properties.SyncPrep})
.then((response) => {
console.log(`The Response: ${response}`);
return response;
})
.catch((error) => {
console.log(error.statusCode);
if(error.statusCode === 497){
return this.syncReady(token, batchID);
} else {
throw error;
}
})
);
};
Maybe something like the above will work for you instead. Maybe try the above instead. As a general rule of thumb, it's you almost always want to return a Promise.