I have an application that it has nodejs as backend and some scripts in Python
The problem is to make the 'PythonShell' (function to access the scripts) as a async function. I do not know why but it is not working.
I'll put the code from my router.js file and inside of it I put three 'console.log('steps')' to check the sequences.
It should be Step01 > Step02 > Step03, but as it is not working, It always prints Step01 > Step03 > Step02
Everything is working fine, except for this async problem! For me it should work as it is.
How can I edit my functions to execute first the 'goToscript/PythonShell' and then execute 'res.json(responseScript)'?
Thanks
router.put("/uploads/script-03", async (req, res) => {
let options = {
scriptPath: "scripts",
args: JSON.stringify(req.body)
};
const goToScript = async () => {
await PythonShell.run("script-01.py", options, (err, res) => {
if (err) {
}
if (res) {
responseScript = JSON.parse(res)
console.log('Step 02')
}
});
}
console.log('Step 01')
goToScript()
console.log('Step 03')
res.json(responseScript)
});
module.exports = router
A couple things:
1. Your goToScript is not actually async/returning a Promise
From what I can tell, PythonShell doesn't support async, only callbacks, so you can rewrite your gotToScript like so:
const goToScript = () => {
return new Promise((resolve, reject) => {
PythonShell.run("script-01.py", options, (err, res) => {
if (err) {
reject(err)
}
if (res) {
responseScript = JSON.parse(res)
console.log('Step 02')
resolve(responseScript)
}
})
})
}
const scriptResult = await goToScript()
This code will work like a regular async function, where the promise will resolve to the parsed JSON, and reject with the error if it meets one.
2. You are not awaiting your call to goToScript
When you want to make an async call that finishes in sequence with everything else, you need to await it. Take these two examples:
In this first chunk of code, waitFn waits before 100ms before logging "Step 2!":
const waitFn = () => {
return new Promise((resolve) => {
setTimeout(() => {
console.log('Step 2!')
resolve()
}, 100)
})
}
console.log('Step 1!')
waitFn()
console.log('Step 3!')
Because you do not await the result of the Promise, your code doesn't care that is has not finished, and will print:
Step 1!
Step 3!
Step 2!
Instead, however, if you await the result of the Promise returned in waitFn, it will execute in order:
const waitFn = () => {
return new Promise((resolve) => {
setTimeout(() => {
console.log('Step 2!')
resolve()
}, 100)
})
}
console.log('Step 1!')
await waitFn() // Finishes before moving on
console.log('Step 3!')
You can read a bunch more about Promises and async/await here :)
To be able to await function - this function needs to return a promise. In the npm page of your lib there is a description, about what PythonShell.run returns. It does not return a promise. So it is asynchronous, but not awaitable, it is callback based.
All you need to do - to promisify this function. Additionaly - you need to await the call to goToScript();.
router.put("/uploads/script-03", async (req, res) => {
let options = {
scriptPath: "scripts",
args: JSON.stringify(req.body)
};
const goToScript = async () => {
return new Promise((resolve, reject) => {
PythonShell.run("script-01.py", options, (err, res) => {
console.log("Step 02");
if (err) return reject(err);
return resolve(JSON.parse(res));
});
});
};
console.log("Step 01");
const responseScript = await goToScript();
console.log("Step 03");
res.json(responseScript);
});
module.exports = router;
Related
How can I store the video title as a global variable?
ytdl.getBasicInfo(videoURL, function(err, info) {
console.log(info.title)
});
I've looked through other questions/answers here, but I seem to always print the title successfully to console, but no matter how I define a variable, it is undefined.
Thanks so much for your help!
EDIT:
This is current method I'm trying, as an example found from another user here:
var vidtitle;
function getTitleVideo (videoUrl){
return new Promise ((resolve, reject) => {
ytdl.getBasicInfo (videoUrl, (err, info) => {
resolve (info.title)
})
})
}
vidtitle = getTitleVideo(`https://www.youtube.com/watch?v=${youtube_video_id}`);
getBasicInfo can be used with a callback or a promise.
Using a callback:
ytdl.getBasicInfo(videoURL, (err, info) => {
// log any error
if (err) return console.error(err);
const {title} = info;
/*
* you must put all your code that uses the title here
* if you try to make a global variable it may be undefined when you
* run the rest of the synchronous code
*/
});
The const {title} = info; is a shorthand for const title = info.title;. You can read more about destructuring here.
Using a promise with then:
ytdl.getBasicInfo(videoURL)
.then(({title}) => {
// same as above you can only use the title here
})
// log any error
.catch(console.error);
({title}) => {...} is also destructuring.
Using a promise with await (can only be used in an async function):
(async () => {
try {
const {title} = await ytdl.getBasicInfo(videoURL);
// use title here
} catch (err) {
console.error(err);
}
})();
Explanation
getBasicInfo is an example of an asynchronous function. That is, it takes time for it to execute (it needs to fetch the video info online).
Before ES2015 (ES6), people used callbacks to work with async functions. (Some people also used promise libraries like bluebird, but I won’t go into that here.) The callback function would execute once the async function has completed or has an error.
For example, Node.js’ require('http').get still uses callbacks:
const {get} = require('http');
get('http://example.com', function (res) {
console.log(`The status code was ${res.statusCode}.`;
}).on('error', console.error);
In this example, the callback does not take an error and instead get returns a ClientRequest, which has an error event.
Another example using the deprecated request library (take from the readme):
const request = require('request');
request('http://www.google.com', function (error, response, body) {
console.error('error:', error); // Print the error if one occurred
console.log('statusCode:', response && response.statusCode); // Print the response status code if a response was received
console.log('body:', body); // Print the HTML for the Google homepage.
});
In ES2015, promises were added (as well as arrow functions). For example, you could create functions that return a promise like this:
// resolves after 500 ms
const doSomething = () => new Promise(resolve => setTimeout(resolve, 500));
// resolves with 'Hello, world!' after 500 ms
const getString = () =>
new Promise(resolve => setTimeout(resolve, 500, 'Hello, world!'));
// rejects with the error 'Error!' after 500 ms
const rejectingPromise = () =>
new Promise((resolve, reject) => setTimeout(reject, 500, new Error('Error!')))
// has a 50% chance to resolve and a 50% chance to reject
const mayReject = () => Math.random() > 0.5
? Promise.resolve('it worked!')
: Promise.reject(new Error("it didn't work"));
// using with then and catch
doSomething()
.then(() => {
console.log('done');
return getString();
})
.then(str => {
console.log(str);
return mayReject();
})
.then(str => {
console.log(str)
return rejectingPromise()
})
.catch(console.error);
You don’t have to reject with an Error but it’s good practice to do so.
In ES2016 (ES7), await and async functions were added. await is another way to "unwrap" a promise (pauses execution until the promise is resolved). await is only available in a function that has been marked as async.
// same stuff as before
const doSomething = () => new Promise(resolve => setTimeout(resolve, 500));
const getString = () => new Promise(resolve => setTimeout(resolve, 500, 'Hello, world!'));
const rejectingPromise = () => new Promise((resolve, reject) => setTimeout(reject, 500, new Error('Error!')));
const mayReject = () => Math.random() > 0.5 ? Promise.resolve('it worked!') : Promise.reject(new Error("it didn't work"));
// an async function
async function f() {}
// an async arrow function
const g = async () => {};
// an immediately invoked arrow function expression
// this does the same thing as the previous example with then/catch
(async () => {
try {
await doSomething();
console.log('done');
const str = await getString();
console.log(str);
// you don't have to assign it to a variable:
console.log(await mayReject());
await rejectingPromise();
} catch (err) {
console.error(err);
}
})();
I recommend reading MDN’s article on asynchronous JS.
I have 3 functions which I need to run in order and one needs to finish before the other runs so I've done this:
var fs = require('fs');
async function create() {
fs.writeFile('newfile.txt', 'some text here', (err) => {
if (err) throw err;
console.log('File is created successfully.');
return ('File is created successfully.');
});
}
async function waitt() {
setTimeout(() => { return 'waited!' }, 10000);
}
async function read() {
fs.readFile('newfile.txt', {encoding: 'utf-8'}, (err,data) => {
if (!err) {
console.log('received data: ' + data);
return ('received data: ' + data);
} else {
console.log(err);
}
});
}
async function process() {
let [r1, r2, r3] = await Promise.all([create(), waitt(), read()]);
console.log(r1 + ' ' + r2 + ' ' + r3);
}
process();
So, process() runs create() which creates a file, then run waitt() which just pauses and finally read() shows the contents of the file.
The issue I'm having is that it's running in this order:
create()
read()
and then waitt()
instead of
create()
waitt()
read()
which is what I want.
How can I fix this?
This won't work:
async function waitt() {
setTimeout(() => { return 'waited!' }, 10000);
}
Because, you're return-ing from within the setTimeout callback, not the async-marked wait function.
To mix callback-style code and async/await you must first convert callback style code to use Promises, or use fs-extra which already provides fs functions returning promises.
Something like this:
function waitt() {
return new Promise((resolve) => {
setTimeout(() => {
resolve('waited...')
}, 10000)
})
}
The same applies to the rest of your functions.
Also note that if a function explicitly returns a Promise, it doesn't need to be marked as async to be awaited, since await essentially works with Promises.
Now for the order:
Promise.all doesn't guarantee order, for that you might be able to get away with a simple for..of, or just call the functions yourself.
function wait() {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('waited...')
resolve('foo')
}, 500)
})
}
// Assume this is your promisified read function.
function read() {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('read...')
resolve('bar')
}, 500)
})
}
// Assume this is your promisified create function.
function create() {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('created...')
resolve('baz')
}, 500)
})
}
;(async () => {
try {
await create()
await wait()
await read()
} catch (err) {
console.error(err)
}
})()
Your problem is that Promise.all does not guarantee the order of processing, just that all the promises in the list are processed.
Can you not just say:
await create();
await read();
await waitt();
I have 3 layer callbacks like this :
app.post('/', (req, res) => {
var filename = `outputs/${Date.now()}_output.json`;
let trainInput = req.files.trainInput;
let trainOutput = req.files.trainInput;
let testInput = req.files.trainInput;
//first
trainInput.mv(`inputs/${req.body.caseName}/train_input.csv`, function (err) {
if (err) return res.status(500).send(err);
//second
trainOutput.mv(`inputs/${req.body.caseName}/train_output.csv`, function (err) {
if (err) return res.status(500).send(err);
//third
testInput.mv(`inputs/${req.body.caseName}/test_input.csv`, function (err) {
if (err) return res.status(500).send(err);
res.send('success');
});
});
});
});
In this case, there are only 3 file uploads. In another case, I have more than 10 file uploads, and it makes 10 layer callbacks. I know it because of JavaScript asynchronous.
Is there any way, with this case, to make a beautiful code? This is because when it 10 layer callbacks, the code looks horizontally weird.
Thanks
You can use the following code to make you code look better and avoid callback hell
app.post('/', async (req, res) => {
var filename = `outputs/${Date.now()}_output.json`;
let trainInput = req.files.trainInput;
let trainOutput = req.files.trainInput;
let testInput = req.files.trainInput;
try {
var result1 = await trainInput.mv(`inputs/${req.body.caseName}/train_input.csv`);
var result2 = await trainInput.mv(`inputs/${req.body.caseName}/train_output.csv`);
var result2 = await testInput.mv(`inputs/${req.body.caseName}/test_input.csv`);
res.send('success');
}
catch (error) {
res.status(500).send(error);
}
});
You can make the functions return a Promise
I advice to make one function because you do the same thing 3 times. In this case I called the function 'save' but you can call it what ever you want. The first parameter is the file end the second the output filename.
function save(file, output) = return new Promise((resolve, reject) => {
file.mv(`inputs/${req.body.caseName}/${output}`, err =>
if (err) return reject(err)
resolve()
})
Promise.all([
save(req.files.trainInput, 'train_input.csv'),
save(req.files.trainInput, 'train_output.csv'),
save(req.files.trainInput, 'test_input.csv')
])
.then(_ => res.send(200))
.catch(err => res.send(400);
What version of Node you using? If async/await is available that cleans it up a bunch.
const moveCsv = (file, dest) => {
return new Promise((resolve, reject) => {
//third
file.mv(dest, function (err) {
if (err) reject(err);
resolve();
});
})
}
app.post('/', async(req, res) => {
try {
var filename = `outputs/${Date.now()}_output.json`;
const {
trainInput,
trainOutput,
testInput
} = req.files;
const prefix = `inputs/${req.body.caseName}`;
await moveCsv(trainInput, `${prefix}/train_input.csv`);
await moveCsv(trainOutput, `${prefix}/train_output.csv`);
await moveCsv(testInput, `${prefix}/test_input.csv`);
res.send('success');
} catch(err) {
res.status(500).send(err);
}
});
I'm also assuming here that your trainInput, trainOutput, testOutput weren't all meant to be req.files.trainInput.
Just be careful since the synchronous nature of the await calls are thread blocking. If that writer function takes ages you could also looking at putting those calls onto a worker thread. Won't really matter if your requests to that server endpoint are fast and non-frequent.
You can add RXJS to your project and use Observables.forkJoin()
Solution with Observables(assuming that trainInput.mv() returns Observable):
/* Without a selector */
var source = Rx.Observable.forkJoin(
trainInput.mv(`inputs/${req.body.caseName}/train_input.csv`),
trainInput.mv(`inputs/${req.body.caseName}/train_output.csv`),
trainInput.mv(`inputs/${req.body.caseName}/test_input.csv`)
);
var subscription = source.subscribe(
function (x) {
// On success callback
console.log('Success: %s', x);
},
function (err) {
// Error callback
console.log('Error');
},
function () {
// Completed - runs always
console.log('Completed');
});
// => Success: [result_1, result_2, result_3] or Error
// => Completed
I'm using through2 to generate multiple files from a Gulp stream. I'm using NodeJS 10.6.0 so thought I'd make full use of async/await, but am not fully understanding the mechanics yet. Currently the through2 done() callback is being fired before all files have been written.
Here's what I have (simplified) - note that I'm not returning the stream at the end as there is no need to.
async function createDirectory(pathDir) {
return new Promise((resolve, reject) => {
mkdirp(pathDir, (err) => {
if (err) reject(err);
else resolve();
});
});
}
async function writeFile(outputFilePath, outputFileContent) {
return new Promise((resolve, reject) => {
fs.writeFile(outputFilePath, outputFileContent, (err) => {
if (err) reject(err);
else resolve();
});
});
}
async function doWriteFile(outputFolderPath, outputFilePath, outputContent) {
await createDirectory(outputFolderPath);
await writeFile(outputFilePath, outputContent, outputContent);
}
async function doGenerateVariant(data, variantArr) {
for (const variant of variantArr) {
/* Do a load of stuff */
const variantOutputFolderPath = blah;
const variantOutputFilePath = blah;
const variantOutputContent = blah;
await doWriteFile(variantOutputFolderPath, variantOutputFilePath, variantOutputContent);
}
}
const generateVariant = () => {
return through.obj((file, enc, done) => {
const data = JSON.parse(file.contents.toString());
*/ Do a load of stuff */
const { variant } = data;
const variantArr = Object.values(variant);
doGenerateVariant(data, variantArr);
return done();
});
};
This doesn't work as done() gets returned before all files have been written. I'm guessing I'm missing a return or two but nothing I do seems to be working.
If I pass done() into doGenerateVariant and call it after doWriteFile everything works as expected but I know this isn't correct.
You need to wait for doGenerateVariant to do its job before calling done. Remember async function always returns a Promise. So you could do it this way
const generateVariant = () => {
return through.obj((file, enc, done) => {
const data = JSON.parse(file.contents.toString());
*/ Do a load of stuff */
const { variant } = data;
const variantArr = Object.values(variant);
doGenerateVariant(data, variantArr).then(() => done());
});
};
or using async/await
const generateVariant = () => {
return through.obj(async (file, enc, done) => {
const data = JSON.parse(file.contents.toString());
*/ Do a load of stuff */
const { variant } = data;
const variantArr = Object.values(variant);
await doGenerateVariant(data, variantArr);
done();
});
};
I am trying to run this function but the it keeps returning undefined when I explicitly hardcode the return value.
const splitVideo = async (sid, video, part) => {
let framesLocation =`${process.cwd()}/${dirs.videoFrames}/${sid}_${part}`;
console.log(fs.existsSync(framesLocation));
if(!fs.existsSync(framesLocation)) {
console.log("making dir");
f.s.mkdirSync(framesLocation);
}
ffmpeg(video)
.on('end', () => {
return "done";
})
.on('error', (err) => {
throw err;
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
};
Please help this is very frustrating.
Your function does not return anything, thats why you are getting undefined. Wrap the ffmpeg call in new Promise(...) to be able to resolve its asynchronous result:
const splitVideo = async (sid, video, part) => {
// ...
return new Promise((resolve, reject) => {
ffmpeg(video)
.on('end', () => {
resolve("done");
})
.on('error', (err) => {
reject(err);
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
};
};
const ret = await splitVideo(...);
console.log(ret);
Also note you need to await this function to be able to read the result (or get the result in then handler).
You can only await promises.
const splitVideo = async (sid, video, part) => {
// ...
const val = await new Promise((resolve, reject) => {
ffmpeg(video)
.on('end', () => {
resolve("done");
})
.on('error', (err) => {
reject(err);
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
//... more code using the val maybe?
return val
};
};
The problem is that your ffmpeg(video) function inside your splitVideo async function is asynchronous. What's happening is that your splitVideo function is being called but it's returning undefined before your ffmpeg(video) function accomplishs. What can you do to resolve this?
You already defined your splitVideo function as an async function, this allows you to use the reserved word "await". But first let's encapsulate your ffmpeg(video) function inside a promise.
const splitVideo = async (sid, video, part) => {
//Your code
let promise = new Promise((resolve, reject) => {
ffmpeg(video)
.on('end', () => {
resolve("done");
})
.on('error', (err) => {
reject(err);
})
.screenshots({
timestamps: [1,2],
filename: `${sid}_${part}/frame-%s.png`,
folder: `${process.cwd()}/${dirs.videoFrames}`
});
});
try{
return await promise;
}catch(err){
return err;
}
This should be enough for your splitVideo function, but it's importante to pay attention cause unhandled promises rejections are deprecated and in the future will terminate your node.js process. What you need to do is also add a .catch() statement to you splitVideoFunction, something like:
splitVideo.catch((err) => {//your code}).then((result) => {//Your code})
Or your can call splitVideo inside an another async function and use:
try {
await splitVideo(video);
}catch(err){
//Your code
}
If you have any doubt about async/await like I had, you may find this question useful.
Node.JS - Can`t get async throws with try/catch blocks
I hope it helps.