This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 2 years ago.
I need to download all images and generate word document with them.
Using nodeJS and Meteor
WebApp.connectHandlers.use('/download', async function (req, res, next) {
// ...
const images = [];
await lines.forEach(async (line, k) => {
if (line.type && line.type === 'image') {
images.push({
id: line.id,
file: line.id + '.jpg',
});
download_image(line.imageUrl, line.id + '.jpg');
}
});
// ...
// Then I use images[] to insert them into a Word document.
});
const download_image = (url, image_path) =>
axios({
url,
responseType: 'stream',
}).then(
(response) =>
new Promise((resolve, reject) => {
response.data
.pipe(fs.createWriteStream(image_path))
.on('finish', () => resolve())
.on('error', (e) => reject(e));
})
);
The problem is images are not getting downloaded before I insert the into a Word document.
How to stop/await before images are finished to be downloaded? I am not so good with promises. What is missing her?
Thanks!
its common mistake to use .forEach (or similar array methods) with async function inside it. The async function just mean that it returns promise and the await works in a same way like chaining together promises with then. Therefore this line wait lines.forEach(async (line, k) => { will just create and return bunch of promises, but it will not wait for all the promises inside to finish.
WebApp.connectHandlers.use('/download', async function (req, res, next) {
// ...
const images = [];
const promises = [];
lines.forEach((line, k) => {
if (line.type && line.type === 'image') {
images.push({
id: line.id,
file: line.id + '.jpg',
});
promises.push(download_image(line.imageUrl, line.id + '.jpg'));
}
});
// here you get array with all the images downloaded
const downloadedImages = await Promise.all(promises);
// this line will be executed after you download all images
// ...
});
// This function would work same with or without the `async` keyword
// (because async function return promise - you are returning the promise.
// Async function allows to use await, but you are not using await in this function).
// However it is good practice to have all functions that returns promise
// marked as `async` so you know that you receive promise from it.
const download_image = async (url, image_path) =>
// Dont forget to return your promise otherwise you cannot await it
return axios({
url,
responseType: 'stream',
}).then(
(response) =>
new Promise((resolve, reject) => {
response.data
.pipe(fs.createWriteStream(image_path))
.on('finish', () => resolve())
.on('error', (e) => reject(e));
})
);
Related
I have an application that it has nodejs as backend and some scripts in Python
The problem is to make the 'PythonShell' (function to access the scripts) as a async function. I do not know why but it is not working.
I'll put the code from my router.js file and inside of it I put three 'console.log('steps')' to check the sequences.
It should be Step01 > Step02 > Step03, but as it is not working, It always prints Step01 > Step03 > Step02
Everything is working fine, except for this async problem! For me it should work as it is.
How can I edit my functions to execute first the 'goToscript/PythonShell' and then execute 'res.json(responseScript)'?
Thanks
router.put("/uploads/script-03", async (req, res) => {
let options = {
scriptPath: "scripts",
args: JSON.stringify(req.body)
};
const goToScript = async () => {
await PythonShell.run("script-01.py", options, (err, res) => {
if (err) {
}
if (res) {
responseScript = JSON.parse(res)
console.log('Step 02')
}
});
}
console.log('Step 01')
goToScript()
console.log('Step 03')
res.json(responseScript)
});
module.exports = router
A couple things:
1. Your goToScript is not actually async/returning a Promise
From what I can tell, PythonShell doesn't support async, only callbacks, so you can rewrite your gotToScript like so:
const goToScript = () => {
return new Promise((resolve, reject) => {
PythonShell.run("script-01.py", options, (err, res) => {
if (err) {
reject(err)
}
if (res) {
responseScript = JSON.parse(res)
console.log('Step 02')
resolve(responseScript)
}
})
})
}
const scriptResult = await goToScript()
This code will work like a regular async function, where the promise will resolve to the parsed JSON, and reject with the error if it meets one.
2. You are not awaiting your call to goToScript
When you want to make an async call that finishes in sequence with everything else, you need to await it. Take these two examples:
In this first chunk of code, waitFn waits before 100ms before logging "Step 2!":
const waitFn = () => {
return new Promise((resolve) => {
setTimeout(() => {
console.log('Step 2!')
resolve()
}, 100)
})
}
console.log('Step 1!')
waitFn()
console.log('Step 3!')
Because you do not await the result of the Promise, your code doesn't care that is has not finished, and will print:
Step 1!
Step 3!
Step 2!
Instead, however, if you await the result of the Promise returned in waitFn, it will execute in order:
const waitFn = () => {
return new Promise((resolve) => {
setTimeout(() => {
console.log('Step 2!')
resolve()
}, 100)
})
}
console.log('Step 1!')
await waitFn() // Finishes before moving on
console.log('Step 3!')
You can read a bunch more about Promises and async/await here :)
To be able to await function - this function needs to return a promise. In the npm page of your lib there is a description, about what PythonShell.run returns. It does not return a promise. So it is asynchronous, but not awaitable, it is callback based.
All you need to do - to promisify this function. Additionaly - you need to await the call to goToScript();.
router.put("/uploads/script-03", async (req, res) => {
let options = {
scriptPath: "scripts",
args: JSON.stringify(req.body)
};
const goToScript = async () => {
return new Promise((resolve, reject) => {
PythonShell.run("script-01.py", options, (err, res) => {
console.log("Step 02");
if (err) return reject(err);
return resolve(JSON.parse(res));
});
});
};
console.log("Step 01");
const responseScript = await goToScript();
console.log("Step 03");
res.json(responseScript);
});
module.exports = router;
How can I store the video title as a global variable?
ytdl.getBasicInfo(videoURL, function(err, info) {
console.log(info.title)
});
I've looked through other questions/answers here, but I seem to always print the title successfully to console, but no matter how I define a variable, it is undefined.
Thanks so much for your help!
EDIT:
This is current method I'm trying, as an example found from another user here:
var vidtitle;
function getTitleVideo (videoUrl){
return new Promise ((resolve, reject) => {
ytdl.getBasicInfo (videoUrl, (err, info) => {
resolve (info.title)
})
})
}
vidtitle = getTitleVideo(`https://www.youtube.com/watch?v=${youtube_video_id}`);
getBasicInfo can be used with a callback or a promise.
Using a callback:
ytdl.getBasicInfo(videoURL, (err, info) => {
// log any error
if (err) return console.error(err);
const {title} = info;
/*
* you must put all your code that uses the title here
* if you try to make a global variable it may be undefined when you
* run the rest of the synchronous code
*/
});
The const {title} = info; is a shorthand for const title = info.title;. You can read more about destructuring here.
Using a promise with then:
ytdl.getBasicInfo(videoURL)
.then(({title}) => {
// same as above you can only use the title here
})
// log any error
.catch(console.error);
({title}) => {...} is also destructuring.
Using a promise with await (can only be used in an async function):
(async () => {
try {
const {title} = await ytdl.getBasicInfo(videoURL);
// use title here
} catch (err) {
console.error(err);
}
})();
Explanation
getBasicInfo is an example of an asynchronous function. That is, it takes time for it to execute (it needs to fetch the video info online).
Before ES2015 (ES6), people used callbacks to work with async functions. (Some people also used promise libraries like bluebird, but I won’t go into that here.) The callback function would execute once the async function has completed or has an error.
For example, Node.js’ require('http').get still uses callbacks:
const {get} = require('http');
get('http://example.com', function (res) {
console.log(`The status code was ${res.statusCode}.`;
}).on('error', console.error);
In this example, the callback does not take an error and instead get returns a ClientRequest, which has an error event.
Another example using the deprecated request library (take from the readme):
const request = require('request');
request('http://www.google.com', function (error, response, body) {
console.error('error:', error); // Print the error if one occurred
console.log('statusCode:', response && response.statusCode); // Print the response status code if a response was received
console.log('body:', body); // Print the HTML for the Google homepage.
});
In ES2015, promises were added (as well as arrow functions). For example, you could create functions that return a promise like this:
// resolves after 500 ms
const doSomething = () => new Promise(resolve => setTimeout(resolve, 500));
// resolves with 'Hello, world!' after 500 ms
const getString = () =>
new Promise(resolve => setTimeout(resolve, 500, 'Hello, world!'));
// rejects with the error 'Error!' after 500 ms
const rejectingPromise = () =>
new Promise((resolve, reject) => setTimeout(reject, 500, new Error('Error!')))
// has a 50% chance to resolve and a 50% chance to reject
const mayReject = () => Math.random() > 0.5
? Promise.resolve('it worked!')
: Promise.reject(new Error("it didn't work"));
// using with then and catch
doSomething()
.then(() => {
console.log('done');
return getString();
})
.then(str => {
console.log(str);
return mayReject();
})
.then(str => {
console.log(str)
return rejectingPromise()
})
.catch(console.error);
You don’t have to reject with an Error but it’s good practice to do so.
In ES2016 (ES7), await and async functions were added. await is another way to "unwrap" a promise (pauses execution until the promise is resolved). await is only available in a function that has been marked as async.
// same stuff as before
const doSomething = () => new Promise(resolve => setTimeout(resolve, 500));
const getString = () => new Promise(resolve => setTimeout(resolve, 500, 'Hello, world!'));
const rejectingPromise = () => new Promise((resolve, reject) => setTimeout(reject, 500, new Error('Error!')));
const mayReject = () => Math.random() > 0.5 ? Promise.resolve('it worked!') : Promise.reject(new Error("it didn't work"));
// an async function
async function f() {}
// an async arrow function
const g = async () => {};
// an immediately invoked arrow function expression
// this does the same thing as the previous example with then/catch
(async () => {
try {
await doSomething();
console.log('done');
const str = await getString();
console.log(str);
// you don't have to assign it to a variable:
console.log(await mayReject());
await rejectingPromise();
} catch (err) {
console.error(err);
}
})();
I recommend reading MDN’s article on asynchronous JS.
This question already exists:
Transform forEach in Promise (without ASYC/AWAIT) [duplicate]
Closed 3 years ago.
I need to transform a forEach in promise. The code is legacy and I can't use async/await operators.
Promise.all(Object.entries(data).forEach(function (data) {
let [data1, data2] = data
let info;
consultData.getResponse(data1).then(result => info = result).then(function () {
return dataServices.find(info)
.then(function (result) {
// do things
})
.then(function (info) {
// do final things
})
})
})).then((result) => {
// do something when all things have intereted and finished
})
But output that Promise.all can't be used. If I try with Promise.resolve, the last is printed before all things have finished their processing.
How I can transform the forEach in a promise for i can use .then() after all iteration?
ASYNC/AWAIT DON'T WORK IN THIS CODE
As #Jonas Wilms indicated, you can use .map returning each promise and after that you can use Promise.all
const entries = Object.entries(data);
const arrayOfPromises = entries.map((item) => {
return new Promise((resolve, reject) => {
const [data1, data2] = item;
consultData.getResponse(data1).then((result) => {
return dataServices.find(info);
}).then((infoServiceResult) => {
return resolve(infoServiceResult);
}).catch((err) => {
return reject(err);
});
});
});
Promise.all(arrayOfPromises).then((data) => {
// data is an array of each infoServiceResult
console.log(data);
}).catch((err) => {
console.error(err);
});
This question already has answers here:
How do I convert an existing callback API to promises?
(24 answers)
Closed 4 years ago.
i am trying to create a function for rendering views in NodeJS that will be called using await later on.
const getTemplate = async (template, context) => {
app.render(template, { data: context }, (error, html) => {
return html;
});
}
const data = require('./data.json');
app.get('/pdf', async (req, res) => {
const html = await getTemplate('invoice', data);
res.send(html);
});
Right now it gives me an empty response and that's probably because it exits the getTemplate function before the render callback fires, but i don't know how to modify it in an elegant way.
You need to return a promise from getTemplate. Since you want to wait for the app.render to finish, I hope the following code does what you intend to do :)
// `async` is not really necessary as you are not using `await`, but you may provide it, if you want
const getTemplate = /* async */ (template, context) => {
return new Promise((resolve, reject) => {
app.render(template, { data: context }, (error, html) => {
if (error) return reject(error);
return resolve(html);
});
});
}
const data = require('./data.json');
app.get('/pdf', async (req, res) => {
const html = await getTemplate('invoice', data);
res.send(html);
});
I am using a package called Okrabyte to extract words from each image file in a folder. The result should be a new array containing the extracted text that I can use in other functions.
When I run this:
var fs = require("fs");
var okrabyte = require("okrabyte");
fs.readdir("imgs/", function(err, files){
files.map((file)=>{
okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), (err, data)=>{
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
console.log(word);
})
})
})
the console logs each word. To return an array with those words I've tried the following:
async function words() {
await fs.readdir("imgs/", function (err, files) {
return files.map(async (file) => {
await okrabyte.decodeBuffer(fs.readFileSync("imgs/" + file), async (err, data) => {
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
return word
})
})
})
}
var testing = await words();
console.log(testing);
This gives undefined I've tried turning everything into a promise, I've tried async-await, I've tried pushing each word into a new array and returning that array in closure but nothing works - what am I doing wrong??
If your map function is async then it's returning a promise, so your mapped array is in fact an array of promises. But you can then use a Promise.all to get the resolved values of that array.
Additionally, you're trying to await the call to fs.readdir and okrabyte.decodeBuffer, which both accept a callback and do not return a promise. So if you want to use a promise there you'll have to wrap them in a promise constructor manually.
Here's how I would do it:
async function words() {
// Wrap `fs` call into a promise, so we can await it:
const files = await new Promise((resolve, reject) => {
fs.readdir("imgs/", (err, files) => { err ? reject(err) : resolve(files); });
});
// Since map function returns a promise, we wrap into a Promise.all:
const mapped = await Promise.all(files.map((file) => {
// Wrap okrabyte.decodeBuffer into promise, and return it:
return new Promise((resolve, reject) => {
okrabyte.decodeBuffer(fs.readFileSync("imgs/" + file), (err, data) => {
if (err) return reject(err);
const splitWords = data.split(" ");
const word = splitWords[0].substr(1);
resolve(word);
})
})
}))
// Mapped is now an array containing each "word".
return mapped;
}
var testing = await words();
// Should now log your array of words correctly.
console.log(testing);
You should not be using async-await that way. That should be used when you are dealing with promises. The library okrabyte uses the concept of callbacks.
I suggest you follow this approach:
(1) Enclose the okrabyte.decodeBuffer part in a function that returns a promise that resolves in the callback.
(2) Use files.map to generate an array of promises calling the function you defined in (1)
(3) Use Promise.all to wait for all promises to execute and finish before moving on to dealing with all the words.
Walkthrough:
Part 1
const processWord = (file) => {
return new Promise((resolve, reject) => {
okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), (err, data)=>{
if (err) {
reject(err); // <--- reject the promise if there was an error
return;
}
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
resolve(word); // <--- resolve the promise with the word
})
});
}
You make a function that wraps the decoding part into a promise that eventually resolves with the word (or is rejected with an error).
Part 2
const promises = files.map((file)=>{
return processWord(file);
})
The above will generate an array of promises.
Part 3
fs.readdir("imgs/", function(err, files){
const promises = files.map((file)=>{
return processWord(file);
})
Promise.all(promises)
.then(responses => {
// responses holds a list of words
// You will get each word accessing responses[0], responses[1], responses[2], ...
console.log(responses);
})
.catch(error => {
console.log(error); // Deal with the error in some way
});
})
The above uses Promise.all to wait for all promises to resolve before going to the then() block, assuming no errors occurred.
You can further isolate the construct above in a method that will return a promise with a list of all the words, much in the same fashion that was done in the processWord function from Part 1. That way, you can finally use async-await if you wish, instead of handling things in the then() block:
const processEverything = () => {
return new Promise((resolve, reject) => {
fs.readdir("imgs/", function(err, files){
const promises = files.map((file)=>{
return processWord(file);
})
Promise.all(promises)
.then(responses => {
resolve(responses);
})
.catch(error => {
reject(error);
});
})
});
};
const words = await processEverything();
console.log(words);
You're returning word value to the enclosed function but not map() function.
Hope this code help you.
async function words() {
global.words = [];
await fs.readdir("imgs/", function(err, files){
return files.map( async(file)=>{
await okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), async (err, data)=>{
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
global.words.push(word);
})
})
})
}
var testing = await words();
testing = global.words;
console.log(testing);