I have a JS async function in Node. Say, it downloads a file from a URL and does something with it, eg. unzip it. I wrote it like this, it works, but eslint showed me there is a thick code smell: error Promise executor functions should not be async no-async-promise-executor. Async was required because of await fetch in body function.
I am not skilled enough with streams nor async/await to correct it by myself. I would like to get rid of the Promise and fully use async/await. The module stream/promises seems the way to go from Node-15 on as commented here how-to-use-es8-async-await-with-streams. How to use await pipeline(...) in this context? Maybe there's a better and shorter way?
Here's the function:
function doSomething(url) {
return new Promise(async (resolve, reject) => {
try {
const fileWriteStream = fs.createWriteStream(someFile, {
autoClose: true,
flags: 'w',
});
const res = await fetch(url);
const body = res.body;
body
.pipe(fileWriteStream)
.on('error', (err) => {
reject(err);
})
.on('finish', async () => {
await doWhatever();
resolve('DONE');
});
} catch (err) {
reject(err);
}
});
}
You could simply perform the await before getting to the executor:
async function doSomething(url) {
const fileWriteStream = fs.createWriteStream(someFile, { autoClose: true, flags: 'w' });
let { body } = await fetch(url);
body.pipe(fileWriteStream);
return new Promise((resolve, reject) => {
body.on('error', reject);
body.on('finish', resolve);
});
};
My advice in general is to remove as much code from within your promise executors as possible. In this case the Promise is only necessary to capture resolution/rejection.
Note that I've also removed doWhatever from within doSomething - this makes doSomething much more robust. You can simply do:
doSomething('http://example.com').then(doWhatever);
Lastly I recommend you set someFile as a parameter of doSomething instead of referencing it from some broader context!
To use the pipeline function you're looking for, it would be
const { pipeline } = require('stream/promises');
async function doSomething(url) {
const fileWriteStream = fs.createWriteStream(someFile, {
autoClose: true,
flags: 'w',
});
const res = await fetch(url);
await pipeline(res.body, fileWriteStream);
await doWhatever();
return 'DONE';
}
You can use the fs/promises in NodeJS and trim your code down to the following:
import { writeFile } from 'fs/promises'
async function doSomething(url) {
const res = await fetch(url);
if (!res.ok) throw new Error('Response not ok');
await writeFile(someFile, res.body, { encoding: 'utf-8'})
await doWhatever();
return 'DONE';
});
}
Related
I'm using the async.eachLimit function to control the maximum number of operations at a time.
const { eachLimit } = require("async");
function myFunction() {
return new Promise(async (resolve, reject) => {
eachLimit((await getAsyncArray), 500, (item, callback) => {
// do other things that use native promises.
}, (error) => {
if (error) return reject(error);
// resolve here passing the next value.
});
});
}
As you can see, I can't declare the myFunction function as async because I don't have access to the value inside the second callback of the eachLimit function.
You're effectively using promises inside the promise constructor executor function, so this the Promise constructor anti-pattern.
Your code is a good example of the main risk: not propagating all errors safely. Read why there.
In addition, the use of async/await can make the same traps even more surprising. Compare:
let p = new Promise(resolve => {
""(); // TypeError
resolve();
});
(async () => {
await p;
})().catch(e => console.log("Caught: " + e)); // Catches it.
with a naive (wrong) async equivalent:
let p = new Promise(async resolve => {
""(); // TypeError
resolve();
});
(async () => {
await p;
})().catch(e => console.log("Caught: " + e)); // Doesn't catch it!
Look in your browser's web console for the last one.
The first one works because any immediate exception in a Promise constructor executor function conveniently rejects the newly constructed promise (but inside any .then you're on your own).
The second one doesn't work because any immediate exception in an async function rejects the implicit promise returned by the async function itself.
Since the return value of a promise constructor executor function is unused, that's bad news!
Your code
There's no reason you can't define myFunction as async:
async function myFunction() {
let array = await getAsyncArray();
return new Promise((resolve, reject) => {
eachLimit(array, 500, (item, callback) => {
// do other things that use native promises.
}, error => {
if (error) return reject(error);
// resolve here passing the next value.
});
});
}
Though why use outdated concurrency control libraries when you have await?
I agree with the answers given above and still, sometimes it's neater to have async inside your promise, especially if you want to chain several operations returning promises and avoid the then().then() hell. I would consider using something like this in that situation:
const operation1 = Promise.resolve(5)
const operation2 = Promise.resolve(15)
const publishResult = () => Promise.reject(`Can't publish`)
let p = new Promise((resolve, reject) => {
(async () => {
try {
const op1 = await operation1;
const op2 = await operation2;
if (op2 == null) {
throw new Error('Validation error');
}
const res = op1 + op2;
const result = await publishResult(res);
resolve(result)
} catch (err) {
reject(err)
}
})()
});
(async () => {
await p;
})().catch(e => console.log("Caught: " + e));
The function passed to Promise constructor is not async, so linters don't show errors.
All of the async functions can be called in sequential order using await.
Custom errors can be added to validate the results of async operations
The error is caught nicely eventually.
A drawback though is that you have to remember putting try/catch and attaching it to reject.
BELIEVING IN ANTI-PATTERNS IS AN ANTI-PATTERN
Throws within an async promise callback can easily be caught.
(async () => {
try {
await new Promise (async (FULFILL, BREAK) => {
try {
throw null;
}
catch (BALL) {
BREAK (BALL);
}
});
}
catch (BALL) {
console.log ("(A) BALL CAUGHT", BALL);
throw BALL;
}
}) ().
catch (BALL => {
console.log ("(B) BALL CAUGHT", BALL);
});
or even more simply,
(async () => {
await new Promise (async (FULFILL, BREAK) => {
try {
throw null;
}
catch (BALL) {
BREAK (BALL);
}
});
}) ().
catch (BALL => {
console.log ("(B) BALL CAUGHT", BALL);
});
I didn't realized it directly by reading the other answers, but what is important is to evaluate your async function to turn it into a Promise.
So if you define your async function using something like:
let f = async () => {
// ... You can use await, try/catch, throw syntax here (see answer of Vladyslav Zavalykhatko) ..
};
your turn it into a promise using:
let myPromise = f()
You can then manipulate is as a Promise, using for instance Promise.all([myPromise])...
Of course, you can turn it into a one liner using:
(async () => { code with await })()
static getPosts(){
return new Promise( (resolve, reject) =>{
try {
const res = axios.get(url);
const data = res.data;
resolve(
data.map(post => ({
...post,
createdAt: new Date(post.createdAt)
}))
)
} catch (err) {
reject(err);
}
})
}
remove await and async will solve this issue. because you have applied Promise object, that's enough.
I am trying to run this function but the it keeps returning undefined when I explicitly hardcode the return value.
const splitVideo = async (sid, video, part) => {
let framesLocation =`${process.cwd()}/${dirs.videoFrames}/${sid}_${part}`;
console.log(fs.existsSync(framesLocation));
if(!fs.existsSync(framesLocation)) {
console.log("making dir");
f.s.mkdirSync(framesLocation);
}
ffmpeg(video)
.on('end', () => {
return "done";
})
.on('error', (err) => {
throw err;
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
};
Please help this is very frustrating.
Your function does not return anything, thats why you are getting undefined. Wrap the ffmpeg call in new Promise(...) to be able to resolve its asynchronous result:
const splitVideo = async (sid, video, part) => {
// ...
return new Promise((resolve, reject) => {
ffmpeg(video)
.on('end', () => {
resolve("done");
})
.on('error', (err) => {
reject(err);
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
};
};
const ret = await splitVideo(...);
console.log(ret);
Also note you need to await this function to be able to read the result (or get the result in then handler).
You can only await promises.
const splitVideo = async (sid, video, part) => {
// ...
const val = await new Promise((resolve, reject) => {
ffmpeg(video)
.on('end', () => {
resolve("done");
})
.on('error', (err) => {
reject(err);
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
//... more code using the val maybe?
return val
};
};
The problem is that your ffmpeg(video) function inside your splitVideo async function is asynchronous. What's happening is that your splitVideo function is being called but it's returning undefined before your ffmpeg(video) function accomplishs. What can you do to resolve this?
You already defined your splitVideo function as an async function, this allows you to use the reserved word "await". But first let's encapsulate your ffmpeg(video) function inside a promise.
const splitVideo = async (sid, video, part) => {
//Your code
let promise = new Promise((resolve, reject) => {
ffmpeg(video)
.on('end', () => {
resolve("done");
})
.on('error', (err) => {
reject(err);
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
});
try{
return await promise;
}catch(err){
return err;
}
This should be enough for your splitVideo function, but it's importante to pay attention cause unhandled promises rejections are deprecated and in the future will terminate your node.js process. What you need to do is also add a .catch() statement to you splitVideoFunction, something like:
splitVideo.catch((err) => {//your code}).then((result) => {//Your code})
Or your can call splitVideo inside an another async function and use:
try {
await splitVideo(video);
}catch(err){
//Your code
}
If you have any doubt about async/await like I had, you may find this question useful.
Node.JS - Can`t get async throws with try/catch blocks
I hope it helps.
I'm trying to do async loop, where I do something and after it ends, I write to the console. It's look like that:
const http = require('http');
async function load(link)
{
try{
http.get(link, response => console.log(`File: ${link}`));
}catch(e){
console.log('error');
}
}
async function loop(arrayOfLinks)
{
for(let link of arrayOfLinks)
load(link);
}
module.exports = (arrayOfLinks) => {
(async () => {
await loop(arrayOfLinks);
console.log('Files: '+arrayOfLinks.length);
})();
}
But, what I have:
Files: 3
File: http://localhost:8000/1.jpg
File: http://localhost:8000/2.jpg
File: http://localhost:8000/3.jpg
And what I want:
File: http://localhost:8000/1.jpg
File: http://localhost:8000/2.jpg
File: http://localhost:8000/3.jpg
Files: 3
Questions:
Why await operator doesn't block the next step?
How can I solve this?
Thanks
You need to make sure load function returns Promise. http.get by itself is not going to return it so you want to wrap it (or use promise based HTTP library, like fetch, etc).
Simple promise wrapper could look like this:
async function load(link) {
try {
return new Promise((resolve, reject) => {
http.get(link, resolve).on('error', reject)
})
.then(response => console.log(`File: ${link}`))
} catch (e) {
console.log('error', e.message);
}
}
async function loop(arrayOfLinks) {
for (let link of arrayOfLinks)
await load(link);
}
You also need to use await load(link) in loop.
I'm using the async.eachLimit function to control the maximum number of operations at a time.
const { eachLimit } = require("async");
function myFunction() {
return new Promise(async (resolve, reject) => {
eachLimit((await getAsyncArray), 500, (item, callback) => {
// do other things that use native promises.
}, (error) => {
if (error) return reject(error);
// resolve here passing the next value.
});
});
}
As you can see, I can't declare the myFunction function as async because I don't have access to the value inside the second callback of the eachLimit function.
You're effectively using promises inside the promise constructor executor function, so this the Promise constructor anti-pattern.
Your code is a good example of the main risk: not propagating all errors safely. Read why there.
In addition, the use of async/await can make the same traps even more surprising. Compare:
let p = new Promise(resolve => {
""(); // TypeError
resolve();
});
(async () => {
await p;
})().catch(e => console.log("Caught: " + e)); // Catches it.
with a naive (wrong) async equivalent:
let p = new Promise(async resolve => {
""(); // TypeError
resolve();
});
(async () => {
await p;
})().catch(e => console.log("Caught: " + e)); // Doesn't catch it!
Look in your browser's web console for the last one.
The first one works because any immediate exception in a Promise constructor executor function conveniently rejects the newly constructed promise (but inside any .then you're on your own).
The second one doesn't work because any immediate exception in an async function rejects the implicit promise returned by the async function itself.
Since the return value of a promise constructor executor function is unused, that's bad news!
Your code
There's no reason you can't define myFunction as async:
async function myFunction() {
let array = await getAsyncArray();
return new Promise((resolve, reject) => {
eachLimit(array, 500, (item, callback) => {
// do other things that use native promises.
}, error => {
if (error) return reject(error);
// resolve here passing the next value.
});
});
}
Though why use outdated concurrency control libraries when you have await?
I agree with the answers given above and still, sometimes it's neater to have async inside your promise, especially if you want to chain several operations returning promises and avoid the then().then() hell. I would consider using something like this in that situation:
const operation1 = Promise.resolve(5)
const operation2 = Promise.resolve(15)
const publishResult = () => Promise.reject(`Can't publish`)
let p = new Promise((resolve, reject) => {
(async () => {
try {
const op1 = await operation1;
const op2 = await operation2;
if (op2 == null) {
throw new Error('Validation error');
}
const res = op1 + op2;
const result = await publishResult(res);
resolve(result)
} catch (err) {
reject(err)
}
})()
});
(async () => {
await p;
})().catch(e => console.log("Caught: " + e));
The function passed to Promise constructor is not async, so linters don't show errors.
All of the async functions can be called in sequential order using await.
Custom errors can be added to validate the results of async operations
The error is caught nicely eventually.
A drawback though is that you have to remember putting try/catch and attaching it to reject.
BELIEVING IN ANTI-PATTERNS IS AN ANTI-PATTERN
Throws within an async promise callback can easily be caught.
(async () => {
try {
await new Promise (async (FULFILL, BREAK) => {
try {
throw null;
}
catch (BALL) {
BREAK (BALL);
}
});
}
catch (BALL) {
console.log ("(A) BALL CAUGHT", BALL);
throw BALL;
}
}) ().
catch (BALL => {
console.log ("(B) BALL CAUGHT", BALL);
});
or even more simply,
(async () => {
await new Promise (async (FULFILL, BREAK) => {
try {
throw null;
}
catch (BALL) {
BREAK (BALL);
}
});
}) ().
catch (BALL => {
console.log ("(B) BALL CAUGHT", BALL);
});
I didn't realized it directly by reading the other answers, but what is important is to evaluate your async function to turn it into a Promise.
So if you define your async function using something like:
let f = async () => {
// ... You can use await, try/catch, throw syntax here (see answer of Vladyslav Zavalykhatko) ..
};
your turn it into a promise using:
let myPromise = f()
You can then manipulate is as a Promise, using for instance Promise.all([myPromise])...
Of course, you can turn it into a one liner using:
(async () => { code with await })()
static getPosts(){
return new Promise( (resolve, reject) =>{
try {
const res = axios.get(url);
const data = res.data;
resolve(
data.map(post => ({
...post,
createdAt: new Date(post.createdAt)
}))
)
} catch (err) {
reject(err);
}
})
}
remove await and async will solve this issue. because you have applied Promise object, that's enough.
How to use Typescript async / await function and return typescript default promises in node js FS module and call other function upon promise resolved.
Following is the code :
if (value) {
tempValue = value;
fs.writeFile(FILE_TOKEN, value, WriteTokenFileResult);
}
function WriteTokenFileResult(err: any, data: any) {
if (err) {
console.log(err);
return false;
}
TOKEN = tempValue;
ReadGist(); // other FS read File call
};
Since NodeJS 10.0.0 fsPromises module can be used to achieve this result.
import { promises as fsPromises } from 'fs';
await fsPromises.writeFile('file.txt', 'data')
For now I think there is no other way as to go with wrapper function. Something like this:
function WriteFile(fileName, data): Promise<void>
{
return new Promise<void>((resolve, reject) =>
{
fs.writeFile(fileName, data, (err) =>
{
if (err)
{
reject(err);
}
else
{
resolve();
}
});
});
}
async function Sample()
{
await WriteFile("someFile.txt", "someData");
console.log("WriteFile is finished");
}
There is some lengthy discussion here about promises in node.js: Every async function returns Promise
If you don't want to write the Promise wrappers yourself you can use async-file.
Using this your code could look something like the following...
(async function () {
//...
await fs.writeFile(FILE_TOKEN, value);
var data = await fs.readFile('gist');
// do something with your "gist" data here...
})();