Cannot get data needed due to Mongoose Queries/Promises - javascript

I am trying to create a Jeopardy like game. I have the topics for the game in a database. When the page is opened I want it to get 6 topics randomly from the database. I am putting them in an array which I will then pass to my ejs file. The problem is that when I go to pass the array to the ejs file, it is always empty. I understand that it is because of promises(actually queries) from Mongoose. My problem is that I cannot figure out how to handle this. I have read the Mongoose docs and searched everywhere but I can't figure it out.
I have tried using callbacks, but they usually just make the program hang and do nothing. I have tried using .then, but I must be using it wrong because it doesn't do what I want.
app.get("/", function(request, response){
//My array to put the topics into
var questionArr = [];
//I need to know the number of items in db for random
var getQuestions = Questions.countDocuments({}, function(err, count){
for(var i = 0; i < 6; i++){
!function(i){
var rand = math.randomInt(count);
//Here I get a random topic and add to array
//Seems to work and actually get it
Questions.findOne().skip(rand).exec(function(err, result){
questionArr.push(result);
});
}(i);
}
});
//I thought this would make it wait for the function to finish so
//that I could have my full array, but apparently not
getQuestions.then(() => {
//this runs before the other functions and give me a length of 0
console.log(questionArr.length);
response.render("jeopardy.ejs", {questions: questionArr});
});
});
I simply need to have the render run after it gets the information from the database. However, it still just runs with an empty array. Thanks for any help, I'm pretty new to async.

I see few issues with your code:
1) You're mixing promises and callbacks which makes things more complicated. The code doesn't work mainly because you're not awaiting for Questions.findOne() results.
2) There is no Math.randomInt
In order to make it work it has to be similar to below:
Questions.countDocuments()
.then((count) => {
return Promise.all([...new Array(6)].map(() => {
const rand = Math.floor(Math.random() * Math.floor(count));
return Questions.findOne().skip(rand).exec();
}))
})
.then((questionArr) => {
response.render("jeopardy.ejs", {questions: questionArr});
});
Best is to use async/await which will make it even more readable
app.get("/", async function(request, response){
const count = await Questions.countDocuments();
const questionArr = await Promise.all([...new Array(6)].map(() => {
const rand = Math.floor(Math.random() * Math.floor(count));
return Questions.findOne().skip(rand).exec();
}));
response.render("jeopardy.ejs", {questions: questionArr});
});
Also keep in mind that you better do proper error handling, but that's a subject of a separate post :)

Related

get length of all documents in collection mongoose

I am trying to get the count for how many documents I have in my collection. I have this, but it is not returning what I need it to, it is returning a whole bunch of unnecessary info I don't need for this simple task:
var estimatedDocumentCount = ServicesModel.countDocuments({});
console.log(estimatedDocumentCount)
It is returning the entire query, plus all its embedded parameters it seems like. how do I do this properly?
async function countDocuments() {
const count = await ServicesModel.countDocuments({});
return count;
};
const count = countDocuments();
It's probably because countDocuments is an asynchronous call and you are executing it synchronously.
Follow the syntax mentioned in Mongoose Docs which uses a callback function to get the count.
ServicesModel.countDocuments({}, function (err, count) {
console.log(count);
});

Refactoring Express code and cannot figure out variable scope

I hope this is not a silly question. I have a homepage route that loads up a lot of mongo databases, and I originally had it loop through the mongo databases and add them to an array that was rendered to a webpage. However, the databases have become more complicated and they need to be populated, so I can no longer use a loop to accomplish this, and needed to refactor to getting the databases individually. However, I seem to be having problems with the variable's scopes, as they are always returned as empty outside of the .find function.
My original code was this:
const collections = [User, Ticket, Client, Job, Transaction];
let endCollections = [];
for (let i = 0; i < collections.length; i++){
await collections[i].find({}, function(err, foundCollection){
if (err) {
console.log(err);
} else {
endCollections[i] = foundCollection;
}
});
}
res.render("dashboard", {transactions: endCollections[4], clients: endCollections[2], tickets: endCollections[1], jobs: endCollections[3]});
And this worked fine. But I need to populate the individual databases, so this was no longer useful. I rewrote it out to populate, but I am having problems changing the global variables inside of the functions. Here is the new code I am trying:
let transactions = [],
clients = [],
jobs = [],
tickets = [];
await Transaction.find({}).populate("job").populate("client").populate("deposited_by_user").exec(function(err, foundTransactions){
if(err){
console.log(err)
} else {
for (let i = 0; i < foundTransactions.length; i++){
foundTransactions[i]["transaction_info"]["new_amount"] = numberWithCommas(foundTransactions[i]["transaction_info"]["amount"]);
}
}
transactions = foundTransactions;
});
await Client.find({}).populate("transactions").populate("jobs").exec(function(err, foundClients){
if(err){
console.log(err)
}
clients = foundClients;
});
await Ticket.find({}).populate("created_by").populate("assigned_user").populate("completed_by_user").exec(function(err, foundTickets){
if(err){
console.log(err)
}
tickets = foundTickets;
});
await Job.find({}).populate("created_by").populate("client").populate("transactions").exec(function(err, foundJobs){
if(err){
console.log(err)
}
jobs = foundJobs;
});
res.render("dashboard", {transactions: transactions, clients: clients, tickets: tickets, jobs: jobs});
For example, if I console.log "jobs" right after the line jobs = foundJobs;, it will show the jobs array being populated. However, if I console.log "jobs" right before the res.render, it shows it as empty. Considering the global variable endCollections in my original code seemed to be changed within the functions before, I am unsure why my new code does not do the same as everything is returned empty. I know that somehow the scope of the variable is what is wrong here, but I cannot see how. Is there something obvious I am missing? Thanks.
Here now the answer so it is not buried in the post's comments.
After reading the docs, I think you should either use await with an empty exec() or use exec(callback).
What happens when you use both is that exec(callback) sees u passed a callback, it asynchronously executes your query and adds the callback to the promise.then of the query promise to be called once the query promise is settled. Then it immediately returns but it does not return the query promise since you passed a callback. The await is simply awaiting the normal (probably void/undefined) return of the function which is why removing it does not change anything.
After awaiting the return of the function, res.render executes and some time after that, the promise that had been created in the exec(callback) call settles and the callback you passed is executed.
So what is the appropriate way of fixing this? I would encourage you to read deeper into async/awai, promises, and the docs I linked above and find it out yourself before you read on, but since the solution is quite simple I'll leave it here.
// your variable declarations
try {
const foundTransactions = await Transaction.find({}).populate("job").populate("client").populate("deposited_by_user").exec();
// your for loop
transactions = foundTransactions;
// same for the other calls
tickets: tickets, jobs: jobs});
catch (e) {console.log(e);}
res.render("dashboard", {transactions: transactions, clients: clients,...

why my javascript its use to long time to run?

I'm working with Cloud Functions for Firebase, and I get a timeout with some of my functions. I'm pretty new with JavaScript. It looks like I need to put a for inside a promise, and I get some problems. The code actually get off from for too early, and I think he make this in a long time. Do you have some way to improve this code and make the code faster?
exports.firebaseFunctions = functions.database.ref("mess/{pushId}").onUpdate(event => {
//first i get event val and a object inside a firebase
const original = event.data.val();
const users = original.uids; // THIS ITS ALL USERS UIDS!!
// so fist i get all users uids and put inside a array
let usersUids = [];
for (let key in users) {
usersUids.push(users[key]);
}
// so now i gonna make a promise for use all this uids and get token's device
//and save them inside a another child in firebase!!
return new Promise((resolve) => {
let userTokens = [];
usersUids.forEach(element => {
admin.database().ref('users/' + element).child('token').once('value', snapShot => {
if (snapShot.val()) { // if token exist put him inside a array
userTokens.push(snapShot.val());
}
})
})
resolve({
userTokens
})
}) // now i make then here, from get userTokens and save in another child inside a firebase database
.then((res) => {
return admin.database().ref("USERS/TOKENS").push({
userTokens: res,
})
})
})
You are making network requests with firebase, so maybe that's why it's slow. You are making one request per user, so if you have 100 ids there, it might as well take a while.
But there's another problem that I notice, that is: you are just resolving to an empty list. To wait for several promises, create an array of promises, and then use Promise.all to create a promise that waits for all of them in parallel.
When you call resolve, you have already done the forEach, and you have started every promise, but they have not been added to the list yet. To make it better, chance it to a map and collect all the returned promises, and then return Promise.all.

Pg-promise inserts/transactions not working within async queue

I have found a lot of things related to the use of pg-promise and await/async but nothing that quite answers my issue with async (the node/npm package) and in particular the interaction between async.queue and pg-promise queries.
My issue: I need to make a few millions computations (matching score) asynchronously and commit their results in the same async process in a postgres db. My main process is a promise that first computes all of the possible distinct combinations of two records from a table and segments them in chunks of a thousand pairs at a time.
These chunks of a thousand pairs (i.e. [[0,1], [0,2], ... , [0, 1000]] is my array of chunks' first index' content) are fed to an instance of async.queue that performs first the computation of the matching score then the db recording.
The part that has had me scratching my head for hours is that the db committing doesn't work whether it is using insert statements or transactions. I know for sure the functions I use for the db part work since I've written manual tests using them.
My main code is as follows:
'use strict';
const promise = require('bluebird');
const initOptions = {
promiseLib: promise
};
const pgp = require('pg-promise')(initOptions);
const cn = {connexion parameters...};
const db = pgp(cn);
const async = require('async');
var mainPromise = (db, php, param) => {
return new Promise(resolve => {
//some code computing the chunksArray using param
...
var q = async.queue((chunk, done) => {
var scores = performScoresCalculations(chunk);
//scores is an array containing the 1000 scores for any chunk of a 1000 pairs
performDbCommitting(db, scores);
//commit those scores to the db using pg-promise
done();
}, 10);
q.drain = () => {
resolve(arr);
//admittedly not quite sure about that part, haven't used async.queue much so far
}
q.push(chunksArray);
)}.catch(err => {
console.error(err);
});
};
Now my scores array looks like this:
[{column1: 'value1_0', column2: 'value2_0', ..., columnN: 'valueN_0'}, ... , {column1: 'value1_999', column2: 'value2_999', column3: 'value3_999'}] with a thousand records in it.
My performDbCommitting function is as follows:
var performDbCommitting = (db, pgp, scores) => {
console.log('test1');
//displays 'test1', as expected
var query = pgp.helpers.insert(scores, ['column1', 'column2', 'column3'], 'myScoreTable');
console.log(query);
//display the full content of the query, as expected
db.any(query).then(data => {
console.log('test2');
//nothing is displayed
console.log(data);
//nothing is displayed
return;
}).catch(err => {
console.error(err);
});
}
So here is my problem:
when testing "manually" performDbCommitting works perfectly, I've even tried a version with transactions, same works flawlessly,
when used within async.queue everything in performDbCommitting seems to work until the db.any(query) call, as evidenced by the console.log displaying the info correctly until that point,
no error is thrown up, the computations over chunksArray keep on going by groups of 1000 as expected,
if I inspect any of the arrays (chunk, chunksArray, scores, etc) everything is as should be, the lengths are correct, their contents too.
pg-promise just doesn't seem to want to push my 1000 records at a time in the database when used with async.queue and that's where I'm stuck.
I have no trouble imagining the fault lies with me, it's about the first time I'm using async.queue, especially mixed with bluebird promising and pg-promise.
Thank you very much in advance for taking the time to read this and shed any light on this issue if you can.
I was experiencing this same issue on one of my machines in particular but none of the others.
What worked for me was updating pg-promise from version 10.5.0 to version 10.5.6 (via npm update pg-promise).
Your mainPromise doesn't wait for performDBCommitting to finish:
should be like:
//commit those scores to the db using pg-promise
performDbCommitting(db, scores).then(()=>{done();});
and performDBCommitting needs to return the promise too:
return db.any(query).then(data => {
console.log('test2');
//nothing is displayed
console.log(data);
//nothing is displayed
return null;
}).catch(err => {
console.error(err);
return null;
});

Duplicate Array Data Web Scraping

I can't seem to get the article duplicates out of my web scraper results, this is my code:
app.get("/scrape", function (req, res) {
request("https://www.nytimes.com/", function (error, response, html) {
// Load the HTML into cheerio and save it to a variable
// '$' becomes a shorthand for cheerio's selector commands, much like jQuery's '$'
var $ = cheerio.load(html);
var uniqueResults = [];
// With cheerio, find each p-tag with the "title" class
// (i: iterator. element: the current element)
$("div.collection").each(function (i, element) {
// An empty array to save the data that we'll scrape
var results = [];
// store scraped data in appropriate variables
results.link = $(element).find("a").attr("href");
results.title = $(element).find("a").text();
results.summary = $(element).find("p.summary").text().trim();
// Log the results once you've looped through each of the elements found with cheerio
db.Article.create(results)
.then(function (dbArticle) {
res.json(dbArticle);
}).catch(function (err) {
return res.json(err);
});
});
res.send("You scraped the data successfully.");
});
});
// Route for getting all Articles from the db
app.get("/articles", function (req, res) {
// Grab every document in the Articles collection
db.Article.find()
.then(function (dbArticle) {
res.json(dbArticle);
})
.catch(function (err) {
res.json(err);
});
});
Right now I am getting five copies of each article sent to the user. I have tried db.Article.distinct and various versions of this to filter the results down to only unique articles. Any tips?
In Short:
Switching the var results = [] from an Array to an Object var results = {} did the trick for me. Still haven't figured out the exact reason for the duplicate insertion of documents in database, will update as soon I find out.
Long Story:
You have multiple mistakes and points of improvement there in your code. I will try pointing them out:
Let's follow them first to make your code error free.
Mistakes
1. Although mongoose's model.create, new mongoose() does seem to work fine with Arrays but I haven't seen such a use before and it does not even look appropriate.
If you intend to create documents one after another then represent your documents using an object instead of an Array. Using an array is more mainstream when you intend to create multiple documents at once.
So switch -
var results = [];
to
var results = {};
2. Sending response headers after they are already sent will create for you an error. I don't know if you have already noticed it or not but its pretty much clear upfront as once the error is popped up the remaining documents won't get stored because of PromiseRejection Error if you haven't setup a try/catch block.
The block inside $("div.collection").each(function (i, element) runs asynchronously so your process control won't wait for each document to get processed, instead it would immediately execute res.send("You scraped the data successfully.");.
This will effectively terminate the Http connection between the client and the server and any further issue of response termination calls like res.json(dbArticle) or res.json(err) will throw an error.
So, just comment the res.json statements inside the .create's then and catch methods. This will although terminate the response even before the whole articles are saved in the DB but you need not to worry as your code would still work behind the scene saving articles in database for you (asynchronously).
If you want your response to be terminated only after you have successfully saved the data then change your middleware implementation to -
request('https://www.nytimes.com', (err, response, html) => {
var $ = cheerio.load(html);
var results = [];
$("div.collection").each(function (i, element) {
var ob = {};
ob.link = $(element).find("a").attr("href");
ob.title = $(element).find("a").text();
ob.summary = $(element).find("p.summary").text().trim();
results.push(ob);
});
db.Article.create(results)
.then(function (dbArticles) {
res.json(dbArticles);
}).catch(function (err) {
return res.json(err);
});
});
After making above changes and even after the first one, my version of your code ran fine. So if you want you can continue on with your current version, or you may try reading some points of improvement.
Points of Improvements
1. Era of callbacks is long gone:
Convert your implementation to utilise Promises as they are more maintainable and easier to reason about. Here are the things you can do -
Change request library from request to axios or any one which supports Promises by default.
2. Make effective use of mongoose methods for insertion. You can perform bulk inserts of multiple statements in just one query. You may find docs on creating documents in mongodb quite helpful.
3. Start using some frontend task automation library such as puppeteer or nightmare.js for data scraping related task. Trust me, they make life a hell lot easier than using cheerio or any other library for the same. Their docs are really good and well maintained so you won't have have hard time picking these up.

Categories