I have an array of video info from a YouTube playlist. I'm trying to retrieve the audio of each video using npm module youtube-mp3-downloader.
I'm want to iterate over the array asynchronously, in sequence.
The array I'm iterating has a bunch of objects like the following one:
{
snippet: {
title: 'Video Title'
},
contentDetails: {
videoId: 'video-Id'
}
}
I'm running the following loop and it successfully runs the 1st iteration (downloads the audio file and resolves the promise). Then it enters the 2nd iteration, it does console.log("started iteration " + i) but stops right there, before entering await getFromYT()
async function getAudio(list) {
for (let i = 0; i < list.length; i++) {
console.log("started iteration " + i)
await getFromYT(list[i]).then(message =>
console.log(message + " " + i)
).catch(error =>
console.log(error)
)
}
}
here is what I get in the console:
started iteration 0
Finished 0
started iteration 1
And here is the function that is being awaited:
function getFromYT(item) {
return new Promise((resolve, reject) => {
YD.download(item.contentDetails.videoId, item.snippet.title + '.mp3');
YD.on("error", function (error) {
reject(error);
});
YD.on("finished", function (err, data) {
resolve("Finished");
});
})
}
I have been trying to figure this out for a while now and haven't gotten anywhere.
Ideally I would like to know what's wrong with the code that I have. Why doesn't it continue the loop?
However, if you can suggest a straight forward way to achieve what I'm trying to do that would be helpful too.
You are mixing async/await style and promises. the function which loops through should look like this
async function getAudio(list) {
for (let i = 0; i < list.length; i++) {
try {
console.log("started iteration " + i)
const message = await getFromYT(list[i]);
console.log(message + " " + i);
} catch(err) {
console.error(err);
}
}
}
if the operations can be done in parallel, consider using Promise.all with .map
Here's how I'd do the same using a combination of async/await and Promise.
const getFromYT = (item) => new Promise((resolve, reject) => {
YD.download(item.contentDetails.videoId, item.snippet.title + '.mp3');
YD.on("error", (error) => {
reject(new Error(error));
});
YD.on("finished", (err, data) => {
resolve({ id: item.id, data});
});
});
const getAudio = async (list) => {
const promises = list.map((listItem) => {
return getFromYT(listItem)
.catch((err) => console.log(err));
});
let data = null;
try {
data = await Promise.all(promises);
} catch (err) {
console.log(err);
}
return data;
};
So, what I've done here is simply instantiated all the promises together in a loop using the map function.
Then, I execute all the promises parallely using Promise.all and await for all the promises to resolve parallely.
I ensure that the promise chain in getAudio doesn't break by catching rejections at the individual getFromYT promises itself.
You can get the data and the corresponding item without any mismatch as follows :-
const allData = await getAudio(list);
allData.forEach((datum) => {
const { id, data } = datum;
console.log(id);
console.log(data);
});
Hope this helps!
Edit 1: So as #bergi said, I can't return / throw from event-emitters, I need to promisify them. Here's how!
Related
I'm tryng to upgrade this code for a better maintenance, this code uploads two images to a server, i know it's possible to get rid of those .catch, by applying async await functions, and try catch blocks, but it's pretty confusing for me, any help will be apreciated.
this._uploadService.makeFileRequest(Global.url + "/upload-image1/" + response.product._id, [], this.filesToUpload1, 'image')
.then((result: Product) => {
this.filesToUpload1 = null;
this._uploadService.makeFileRequest(Global.url + "/upload-image/" + response.product._id, [], this.filesToUpload, 'image')
.then((result: Product) => {
this.filesToUpload = null;
setTimeout( () => this._router.navigate(['/detail', this.saveProduct._id]), 800 );
})
.catch(err => {
console.log(err);
this._router.navigate(['/detail', this.saveProduct._id]);
})
})
.catch(err => {
console.log(err);
this._router.navigate(['/detail', this.saveProduct._id]);
})
I suggest using a pen and paper to draw a block diagram for the logic involved, i.e. which api gets called first, with what kind of data, then which api comes afterwards; also include any logical conditionals through branching.
After that, you should attempt to write something like
const aggregateFunction = async() => {
try {
const someResponse = await callFirstApi(); // return response
await callSecondApi(someResponse); // use the response of the first api for the second api
if (someConditional) {
await callThirdApi(); // response not returned (i.e. when not required)
}
} catch (error) { // catch all errors from all await if they're not within another try-catch
console.log(error);
}
}
This pattern should eliminate all then and catch blocks. If you need more specific error handling for calling say a specific api, wrap function call inside another try-catch block, but everything should still be within the outer try-catch so that all errors will be caught regardless.
this._uploadService.makeFileRequest = function(){
return new Promise(resolve => {
// do logic of file request
resolve(true);
})
}
var waitForTime = function() {
return new Promise(resolve => {
setTimeout( () => {
this._router.navigate(['/detail', this.saveProduct._id]),
resolve(true)
}, 800 );
})
}
var f = async function(){
try {
await this._uploadService.makeFileRequest(Global.url + "/upload-image1/" + response.product._id, [], this.filesToUpload1, 'image');
await this.fileToUpload1 = null;
await this._uploadService.makeFileRequest(Global.url + "/upload-image/" + response.product._id, [], this.filesToUpload, 'image')
await this.fileToUpload = null;
await waitForTime();
}
catch(e) {
// error logic
}
}
if (this.filesToUpload1 && this.filesToUpload) {
f()
}
this might be another cleaner approach with async,await and promise
I want to return results from two database queries in one object.
function route(start, end) {
return new Promise((resolve, reject) => {
const queries = routeQuery(start, end);
var empty_obj = new Array();
for (i=0; i<queries.length; i++) {
query(queries[i], (err, res) => {
if (err) {
reject('query error', err);
console.log(err);
return;
} else {
empty_obj.push(res.rows);
}});
}
console.log(empty_obj);
resolve({coords: empty_obj});
});
}
This is my code right now, the queries are working fine but for some reason, pushing each result into an empty array does not work. When I console log that empty object, it stays empty. The goal is to resolve the promise with the generated object containing the two query results. I'm using node-postgres for the queries.
Output of res is an object:
{
command: 'SELECT',
rowCount: 18,
oid: null,
rows: [
{ ...
I suggest you turn your query function into a Promise so that you can use Promise.all:
// turn the callback-style asynchronous function into a `Promise`
function queryAsPromise(arg) {
return new Promise((resolve, reject) => {
query(arg, (err, res) => {
if (err) {
console.error(err);
reject(err);
return;
}
resolve(res);
});
});
}
Then, you could do the following in your route function:
function route(start, end) {
const queries = routeQuery(start, end);
// use `Promise.all` to resolve with
// an array of results from queries
return Promise.all(
queries.map(query => queryAsPromise(query))
)
// use `Array.reduce` w/ destructing assignment
// to combine results from queries into a single array
.then(results => results.reduce(
(acc, item) => [...acc, ...item.rows],
[]
))
// return an object with the `coords` property
// that contains the final array
.then(coords => {
return { coords };
});
}
route(1, 10)
.then(result => {
// { coords: [...] }
})
.catch(error => {
// handle errors appropriately
console.error(error);
});
References:
Promise.all - MDN
Array.reduce - MDN
Destructing assignment - MDN
Hope this helps.
The issue you currently face is due to the fact that:
resolve({coords: empty_obj});
Is not inside the callback. So the promise resolves before the query callbacks are called and the rows are pushed to empty_obj. You could move this into the query callback in the following manner:
empty_obj.push(res.rows); // already present
if (empty_obj.length == queries.length) resolve({coords: empty_obj});
This would resolve the promises when all rows are pushed, but leaves you with another issue. Callbacks might not be called in order. Meaning that the resulting order might not match the queries order.
The easiest way to solve this issue is to convert each individual callback to a promise. Then use Promise.all to wait until all promises are resolved. The resulting array will have the data in the same order.
function route(start, end)
const toPromise = queryText => new Promise((resolve, reject) => {
query(queryText, (error, response) => error ? reject(error) : resolve(response));
});
return Promise.all(routeQuery(start, end).map(toPromise))
.then(responses => ({coords: responses.map(response => response.rows)}))
.catch(error => {
console.error(error);
throw error;
});
}
I am building a web scraper to get all of user's submissions on codeforces.
I don't know much about async, await, promises.
I have used axios (promise based) to request codeforces and cheerio to parse HTML .
app.post("/", (req, res) => {
const usernameorhandle = req.body.userName;
getstatus(usernameorhandle).then ( ()=> {
var output = fs.createWriteStream(__dirname + '/Data/solutions.zip');
var archive = archiver('zip', {
zlib: { level: 9 } // Sets the compression level.
});
output.on('close', function() {
console.log(archive.pointer() + ' total bytes');
console.log('archiver has been finalized and the output file descriptor has closed.');
});
output.on('end', function() {
console.log('Data has been drained');
});
res.attachment(__dirname + "/Data/Problems", 'Codeforces-Solutions');
archive.pipe(res);
archive.directory(__dirname + "/Data/Problems", 'Codeforces-Solutions');
archive.finalize();
}) })
I am using to accept post request.
I am putting all the solutions into a folder and creating zip folder and then send to res.
Below is my getstatus function.
async function getstatus(handle){
return new Promise(async (resolve, reject)=> {
console.log("HELLLLLLLOOOOOOOO");
await axios.get("https://codeforces.com/api/user.status?handle=" + handle + "&from=1")
.then(response => {
if(response.data.status === 'OK'){
let results = response.data.result;
console.log("AAAAAAAAAAAAAAAAAAAAAAAa");
scrape(results).then( () =>{
console.log("DONE");
resolve();
})
.catch(err => console.log(err));
// resolve();
}
else console.log(submissions.comment);
})
})
}
I use scrape function to obtain HTML data and put to folder named Problems.
async function scrape (results){
console.log("inside scrape");
// console.log("HELLO");
return new Promise( async (resolve, reject) => {
await results.forEach(async (result)=> {
if(result.verdict === 'OK'){
await axios.get("https://codeforces.com/contest/" + result.contestId + "/submission/" + result.id)
.then(solutionPage => {
const $ = cheerio.load(solutionPage.data);
const path = "/home/srujan/Desktop/crawlerapp/Data/Problems/" + result.problem.name + ".cpp";
fs.writeFile(path, $('#program-source-text').text(), function(err){
if(err){
console.log(err);
}
else{
console.log("Saved file");
}
})
})
.catch( error => {
console.log("HTML PARSE ERROR" + error);
})
}
})
console.log("hey");
resolve();
})
The problem is I am getting
HELLLLLLLOOOOOOOO
AAAAAAAAAAAAAAAAAAAAAAAa
inside scrape
hey
DONE
saved file
saved file
...
Browser downloads after DONE and then files are saved.
I am new to js and don't know why I am getting this.
PS : I know this is very long question. I tried reading a lot about this. Didn't understand properly how to do that. I copy pasted some code which I didn't understand like how to zip a folder.
forEach(callback) executes callback. If callback returns a promise (ie, it's an async function), the promise won't be resolved before calling the callback on the next element of the array.
So, basically, you can't use async functions inside forEach... But you can use for-loops or Promise.all instead!
Also, fs.writeFile works with sync + callback, but there exists a fs.promise.writeFile that uses promises instead.
Here's a scrape function that should work better:
async function scrape(results) {
for (const result of results) {
if(result.verdict === 'OK') {
const solutionPage = await axios.get("https://codeforces.com/contest/" + result.contestId + "/submission/" + result.id);
const $ = cheerio.load(solutionPage.data);
const path = "/home/srujan/Desktop/crawlerapp/Data/Problems/" + result.problem.name + ".cpp";
try {
await fs.promises.writeFile(path, $('#program-source-text').text());
} catch(err) { console.log(err) }
}
}
}
The problem is to use result.forEach
Try to use a simple for(let i = 0; i < result.length; i++) without async.
If that doesn't work, try to return anything inside the then.
This is how I would construct getstatus function with await async
async function getstatus(handle) {
const response = await axios.get("https://codeforces.com/api/user.status?handle=" + handle + "&from=1")
if(response.data.status === 'OK') {
let results = response.data.result;
try {
await scrape(results);
console.log("DONE");
}
catch(error) {
}
}
}
and scrape function accordingly...
const fs = require('fs').promises;
async function scrape (results) {
results.forEach(async (result)=> {
if(result.verdict === 'OK') {
const solutionPage = await axios.get("https://codeforces.com/contest/" + result.contestId + "/submission/" + result.id)
const $ = cheerio.load(solutionPage.data);
const path = "/home/srujan/Desktop/crawlerapp/Data/Problems/" + result.problem.name + ".cpp";
try {
await fs.writeFile(path, $('#program-source-text').text())
console.log("Saved file");
}
catch(error) {
}
}
}
}
I'm using async await inside of for loop as below.
for (let i = 0; i < result.length; i += 1) {
try{
const client = await axios.get(
`${process.env.user}/client/${result[i].id}`
);
} catch(error){
console.log(error)
}
if (client.data.success === true) {
result[i].Name = rider.data.client.Name;
result[i].PhoneNumber = rider.data.client.Number;
}
}
But I want to make this using 'new Promise' and 'promiss.all' to make it asynclously.
But I don'k know how to make this correctly doing error handle well.
Could you recommend some advice for this? Thank you for reading it.
This can be a basic solution, i think
let playList = []
for (let i = 0; i < result.length; i += 1) {
playList.push(axios.get(
`${process.env.user}/client/${result[i].id}`
).then(res => {
if (res.data.success === true) {
result[i].Name = rider.data.client.Name;
result[i].PhoneNumber = rider.data.client.Number;
}
}).catch(ex => console.log(ex)));
}
await Promise.all(playList)
This can also be done by using a foreach loop.
The for/foreach loop can be simplified by using a map function.
Js map function is equivalent of c# select linq function.
The fat arrow in js map function is not bound to return a value unlike c# select inner function which must return a value.
await Promise.all(result.map(async r => {
let client;
try {
client = await axios.get(`${process.env.user}/client/${r.id}`);
} catch (error) {
console.log(error)
}
if (client.data.success === true) {
r.Name = rider.data.client.Name;
r.PhoneNumber = rider.data.client.Number;
}
}));
Try this
var promises = result.map(r => axios.get(`${process.env.user}/client/${r.id}`);
Promise.all(promises).then(function(values) {
console.log('All promises done');
});
The idea is that if you are awaiting something, that is promise, you can await it, or call it to get promise
Example:
function Foo()
{
return new Promise(...); // Promise of int for example
}
you can do
var p = Foo(); //you will get promise
Or
var v = await Foo(); // you will get int value when promise resolved
This is how you do it with async/await + Promise.all:
const myResult = await Promise.all(result.map(({ id }) => {
return axios.get(`${process.env.user}/client/${id}`);
}));
// deal with the result of those requests
const parsed = myResult.map(data => /* your code here */);
Here is an example using Array.map to call your function along with Promise.all. I wrapped the axios request in a function so if one of your request fails, it wont stop every other requests. If you don't mind stopping when you got an issue, look at others answers to your question.
function fakeRequest() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve({
data: {
success: true,
client: {
Name: 'Todd',
Number: 5,
},
},
});
}, 300);
});
}
(async() => {
const result = [{}, {}, {}];
await Promise.all(result.map(async(x, xi) => {
try {
const client = await fakeRequest();
if (client.data.success === true) {
result[xi].Name = client.data.client.Name;
result[xi].PhoneNumber = client.data.client.Number;
}
} catch (err) {
console.log(err)
}
}));
console.log(result);
})();
I'm trying to have a loop with some db calls, and once their all done ill send the result. - Using a promise, but if i have my promise after the callback it dosent work.
let notuser = [];
let promise = new Promise((resolve, reject) => {
users.forEach((x) => {
User.find({
/* query here */
}, function(err, results) {
if(err) throw err
if(results.length) {
notuser.push(x);
/* resolve(notuser) works here - but were not done yet*/
}
})
});
resolve(notuser); /*not giving me the array */
}).then((notuser) => {
return res.json(notuser)
})
how can i handle this ?
Below is a function called findManyUsers which does what you're looking for. Mongo find will return a promise to you, so just collect those promises in a loop and run them together with Promise.all(). So you can see it in action, I've added a mock User class with a promise-returning find method...
// User class pretends to be the mongo user. The find() method
// returns a promise to 'find" a user with a given id
class User {
static find(id) {
return new Promise(r => {
setTimeout(() => r({ id: `user-${id}` }), 500);
});
}
}
// return a promise to find all of the users with the given ids
async function findManyUsers(ids) {
let promises = ids.map(id => User.find(id));
return Promise.all(promises);
}
findManyUsers(['A', 'B', 'C']).then(result => console.log(result));
I suggest you take a look at async it's a great library for this sort of things and more, I really think you should get used to implement it.
I would solve your problem using the following
const async = require('async')
let notuser = [];
async.forEach(users, (user, callback)=>{
User.find({}, (err, results) => {
if (err) callback(err)
if(results.length) {
notUser.push(x)
callback(null)
}
})
}, (err) => {
err ? throw err : return(notuser)
})
However, if you don't want to use a 3rd party library, you are better off using promise.all and await for it to finish.
EDIT: Remember to install async using npm or yarn something similar to yarn add async -- npm install async
I used #danh solution for the basis of fixing in my scenario (so credit goes there), but thought my code may be relevant to someone else, looking to use standard mongoose without async. I want to gets a summary of how many reports for a certain status and return the last 5 for each, combined into one response.
const { Report } = require('../../models/report');
const Workspace = require('../../models/workspace');
// GET request to return page of items from users report
module.exports = (req, res, next) => {
const workspaceId = req.params.workspaceId || req.workspaceId;
let summary = [];
// returns a mongoose like promise
function addStatusSummary(status) {
let totalItems;
let $regex = `^${status}$`;
let query = {
$and: [{ workspace: workspaceId }, { status: { $regex, $options: 'i' } }],
};
return Report.find(query)
.countDocuments()
.then((numberOfItems) => {
totalItems = numberOfItems;
return Report.find(query)
.sort({ updatedAt: -1 })
.skip(0)
.limit(5);
})
.then((reports) => {
const items = reports.map((r) => r.displayForMember());
summary.push({
status,
items,
totalItems,
});
})
.catch((err) => {
if (!err.statusCode) {
err.statusCode = 500;
}
next(err);
});
}
Workspace.findById(workspaceId)
.then((workspace) => {
let promises = workspace.custom.statusList.map((status) =>
addStatusSummary(status)
);
return Promise.all(promises);
})
.then(() => {
res.status(200).json({
summary,
});
})
.catch((err) => {
if (!err.statusCode) {
err.statusCode = 500;
}
next(err);
});
};