We Keep Coding

JavaScript NodeJS How to use stream/promises with async functions?

I have a JS async function in Node. Say, it downloads a file from a URL and does something with it, eg. unzip it. I wrote it like this, it works, but eslint showed me there is a thick code smell: error Promise executor functions should not be async no-async-promise-executor. Async was required because of await fetch in body function.
I am not skilled enough with streams nor async/await to correct it by myself. I would like to get rid of the Promise and fully use async/await. The module stream/promises seems the way to go from Node-15 on as commented here how-to-use-es8-async-await-with-streams. How to use await pipeline(...) in this context? Maybe there's a better and shorter way?
Here's the function:
function doSomething(url) {
return new Promise(async (resolve, reject) => {
try {
const fileWriteStream = fs.createWriteStream(someFile, {
autoClose: true,
flags: 'w',
});
const res = await fetch(url);
const body = res.body;
body
.pipe(fileWriteStream)
.on('error', (err) => {
reject(err);
})
.on('finish', async () => {
await doWhatever();
resolve('DONE');
});
} catch (err) {
reject(err);
}
});
}

You could simply perform the await before getting to the executor:
async function doSomething(url) {
const fileWriteStream = fs.createWriteStream(someFile, { autoClose: true, flags: 'w' });
let { body } = await fetch(url);
body.pipe(fileWriteStream);
return new Promise((resolve, reject) => {
body.on('error', reject);
body.on('finish', resolve);
});
};
My advice in general is to remove as much code from within your promise executors as possible. In this case the Promise is only necessary to capture resolution/rejection.
Note that I've also removed doWhatever from within doSomething - this makes doSomething much more robust. You can simply do:
doSomething('http://example.com').then(doWhatever);
Lastly I recommend you set someFile as a parameter of doSomething instead of referencing it from some broader context!

To use the pipeline function you're looking for, it would be
const { pipeline } = require('stream/promises');
async function doSomething(url) {
const fileWriteStream = fs.createWriteStream(someFile, {
autoClose: true,
flags: 'w',
});
const res = await fetch(url);
await pipeline(res.body, fileWriteStream);
await doWhatever();
return 'DONE';
}

You can use the fs/promises in NodeJS and trim your code down to the following:
import { writeFile } from 'fs/promises'
async function doSomething(url) {
const res = await fetch(url);
if (!res.ok) throw new Error('Response not ok');
await writeFile(someFile, res.body, { encoding: 'utf-8'})
await doWhatever();
return 'DONE';
});
}

Testing file read wtih fs using JEST

I am trying to test the following function:
async function read () {
return new Promise(function(resolve, reject) {
fs.readFile("INVALID_PATH", (err, contents) => {
if (err) {
reject(new Error('ERROR'));
}
resolve(contents);
});
});
}
through the test:
try {
await read();
} catch (e) {
expect(e).toMatch('ERROR');
}
However, I can't catch the rejection on catch, it gives me timeout.
Any suggestions?
EDIT:
I managed to get it working by using a mock, this way:
try {
fs.readFile.mockReset()
fs.readFile.mockImplementation((path, options, cb) => {
cb('ERROR')
})
await readFile()
} catch (err) {
expect(err).toBe('ERROR')
}
However, I still don't understand why this is necessary...

Have you tried declaring the test function as async in Jest? await cannot be used outside async functions (or in top level code), you have to do something like this:
it('works with async/await', async () => {
expect.assertions(1);
const data = await user.getUserName(4);
expect(data).toEqual('Mark');
});
// async/await can also be used with `.resolves`.
it('works with async/await and resolves', async () => {
expect.assertions(1);
await expect(user.getUserName(5)).resolves.toEqual('Paul');
});
More info for testing async/await in jest https://jestjs.io/docs/en/tutorial-async.html

How to cover setInterval in the unit test case in javascript

Hi I am write a unit test case of this function. When I run this function from the unit test case then it covers all statements but setInterval complete lines are not covered.
Does anyone know how to cover it in javascript? I am using mocha.
const open = async function (config) {
...... set of lines..
let retryIn = setInterval(async () => {
try {
client = await connect(config);
clearInterval(retryIn);
return client;
} catch (err) {
//error
}
}, 20000);
};
I am simply calling it like this
it("###test", async () => {
open(config);
});
});

First of all, you should never use setInterval in the case where you want to retry a task that has failed. You should use setTimeout instead.
Besides that, you cannot return a value from a classical callback base function like setInterval or setTimeout. So in its current form, the promise returned when calling open will be resolved before any connection is made.
With await/async you can create a clean and simple setup for such a situation:
function wait(seconds) {
return new Promise((resolve, _) => setTimeout(resolve, seconds))
}
async function connect() {
throw new Error('failed')
}
async function open(config) {
let client;
while (client === undefined /* || retries > maxRetries*/ ) {
try {
client = await connect(config);
} catch (err) {
// if connection failed due to an error, wait n seconds and retry
console.log('failed wait 2 seconds for reconnect');
await wait(2000)
// ++retries
}
}
if (!client) {
throw new Error('connection failed due to max number of retries reached.')
}
return client
}
async function main() {
let connection = await open()
}
main().catch(err => console.log(err))
You can further extend this snippet by adding a retry limit. See the comments for a rough idea on how that can be achieved.
To test the above code, you would write:
it("###test", function() {
return open(config);
});

Someone posted an answer about fake timers and then deleted it , The answer was correct so I re-posted again.
You can use sinonjs to create fake timers
Fake timers are synchronous implementations of setTimeout and friends
that Sinon.JS can overwrite the global functions with to allow you to
more easily test code using them
But from your code, it seems you are trying to test async code, in mocha, this can be achieved like this
describe('somthing', () => {
it('the result is 2', async () => {
const x = await add(1, 1)
expect(x).to.equal(4);
});
});
With something closer to your code
async function open() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve('done')
}, 1000);
});
};
describe('somthing', () => {
it('###test', async () => {
const x = await open()
chai.expect(x).to.equal("done");
});
});

Just wrap to Promise
const open = async function (config) {
...... set of lines..
return new Promise((resolve, reject) => {
let retryIn = setInterval(async () => {
client = await connect(asConfigParam);
clearInterval(retryIn);
return client;
}, 20000);
return resolve(retryIn);
});
};
it("###test", async () => {
const result = await open(config);
console.log('result', result)
});

async function returns Promise { <pending> }?

I have the following async function:
async function readFile () {
let content = await new Promise((resolve, reject) => {
fs.readFile('./file.txt', function (err, content) {
if (err) {
return reject(err)
}
resolve(content)
})
})
console.log(content)
}
readFile()
This runs just fine. It outputs the file buffer to the console as expected. But now, if I try to instead return the value:
async function readFile () {
let content = await new Promise((resolve, reject) => {
fs.readFile('./file.txt', function (err, content) {
if (err) {
return reject(err)
}
resolve(content)
})
})
return content
}
console.log(readFile())
I now get:
Promise { <pending> }
Why is this? Why can you use a value inside that function but when you return it out of the function it's now a Promise?
How do you actually make use of this in a normal workflow? For example, lets say I wanted to check if a file exists, then read in the file, then update some database with the content, the synchronous pseudo code would look something like this:
if (fileExists(path)) {
buffer = readFile(path)
updateDatabase(buffer)
}
That workflow consists of 3 individual async operations. How would you do something like this with async/await? Is the key that you have to have your entire script wrapped in an async function?
async function doSomething () {
if (fileExists(path)) {
buffer = readFile(path)
updateDatabase(buffer)
}
}
(Keep in mind that is just pseudo-code but hopefully its gets my point across).

All async functions return a promise as was mentioned in the comments. You could therefore re-write your readFile function like this:
function readFile() {
return new Promise((resolve, reject) => {
fs.readFile('./file.txt', function (err, content) {
if (err) {
return reject(err)
}
resolve(content)
})
})
}
You would then consume the return value of readFile via await:
console.log(await readFile()) // will log your actual file contents.
The usual workflow with this paradigm is to break your async operations into separate functions that each return a promise, and then run them all inside a broader async function, much like you suggest, but with awaits and some error handling like so:
async function doSomething () {
try {
const fileCheck = await fileExists(path)
if (fileCheck) {
const buffer = await readFile(path)
await updateDatabase(buffer)
// Whatever else you want to do
}
} catch (err) {
// handle any rejected Promises here.
}
}

const serchContentXmlFile = async (path, content) => {
return new Promise((resolve, reject) => {
fs.readFile(path, function (err, data) {
if (err) {
return reject(err)
}
resolve(data.indexOf(content))
})
})
}
await serchContentXmlFile("category.xml",xmlUrl);

How to use Typescript Async/ await with promise in Node JS FS Module

How to use Typescript async / await function and return typescript default promises in node js FS module and call other function upon promise resolved.
Following is the code :
if (value) {
tempValue = value;
fs.writeFile(FILE_TOKEN, value, WriteTokenFileResult);
}
function WriteTokenFileResult(err: any, data: any) {
if (err) {
console.log(err);
return false;
}
TOKEN = tempValue;
ReadGist(); // other FS read File call
};

Since NodeJS 10.0.0 fsPromises module can be used to achieve this result.
import { promises as fsPromises } from 'fs';
await fsPromises.writeFile('file.txt', 'data')

For now I think there is no other way as to go with wrapper function. Something like this:
function WriteFile(fileName, data): Promise<void>
{
return new Promise<void>((resolve, reject) =>
{
fs.writeFile(fileName, data, (err) =>
{
if (err)
{
reject(err);
}
else
{
resolve();
}
});
});
}
async function Sample()
{
await WriteFile("someFile.txt", "someData");
console.log("WriteFile is finished");
}
There is some lengthy discussion here about promises in node.js: Every async function returns Promise

If you don't want to write the Promise wrappers yourself you can use async-file.
Using this your code could look something like the following...
(async function () {
//...
await fs.writeFile(FILE_TOKEN, value);
var data = await fs.readFile('gist');
// do something with your "gist" data here...
})();