We Keep Coding

Async Javascript: Waiting for data to be processed in a for loop before proceeding to a new function

I'm having issues understanding how to work around Javascript's asynchronous behavior in a forEach loop. This issue is quite complex (sorry), but the idea of the loop is as followed:
Loop through every item in an array
Make an HTTP request from a provider script
I then need to multiply every element of the array by a constant
Assign the new array to an item in an object
After the loop, take all the arrays and add them together into one array
The data will be assigned to the indvCoinPortfolioChartData array
I'm looking for any flaws in my event loop. I believe the battle is making this task synchronous, making sure my data is assigned before aggregating data.
The issue
When I'm adding all the arrays together, ONE dataset isn't summed up (I think because it's still being processed after the function is called). There is no error, but it doesn't have all the coin data in the final aggregated array.
This is the issue I see in the aggregatePortfolioChartData function. It begins the for loop with only 2 items in the array, and then later shows 3. The third item was not processed until after the for loop started.
image of console log (logged from aggregatePortfolioChartData function)
debug log when aggregation is successful
var indivCoinPortfolioChartData = {'data': []};
for(var i = 0; i < this.storedCoins.Coins.length; i++)
{
let entry = this.storedCoins.Coins[i];
localThis._data.getChart(entry.symbol, true).subscribe(res => {localThis.generateCoinWatchlistGraph(res, entry);});
localThis._data.getChart(entry.symbol).subscribe(res => {
if(entry.holdings > 0)
{
let data = res['Data'].map((a) => (a.close * entry.holdings));
indivCoinPortfolioChartData.data.push({'coinData': data});
localThis.loadedCoinData(loader, indivCoinPortfolioChartData);
}
else
{
localThis.loadedCoinData(loader, indivCoinPortfolioChartData);
}
});
}
Loaded Coin Data
loadedCoinData(loader, indivCoinPortfolioChartData)
{
this.coinsWithData++;
if(this.coinsWithData === this.storedCoins.Coins.length - 1)
{
loader.dismiss();
this.aggregatePortfolioChartData(indivCoinPortfolioChartData);
}
}
aggregatePortfolioChartData
aggregatePortfolioChartData(indivCoinPortfolioChartData)
{
console.log(indivCoinPortfolioChartData);
var aggregatedPortfolioData = [];
if(indivCoinPortfolioChartData.data[0].coinData)
{
let dataProcessed = 0;
for(var i = 0; i < indivCoinPortfolioChartData.data[0].coinData.length; i++)
{
for(var j = 0; j< indivCoinPortfolioChartData.data.length; j++)
{
let data = indivCoinPortfolioChartData.data[j].coinData[i];
if(data)
{
aggregatedPortfolioData[i] = (aggregatedPortfolioData[i] ? aggregatedPortfolioData[i] : 0) + data;
dataProcessed++;
}
else
{
dataProcessed++;
}
}
if(dataProcessed === (indivCoinPortfolioChartData.data[0].coinData.length) * (indivCoinPortfolioChartData.data.length))
{
console.log(dataProcessed + " data points for portfolio chart");
this.displayPortfolioChart(aggregatedPortfolioData);
}
}
}
}
Thank you for helping me get through this irksome issue.

Call a Request function from outside the request

Im trying to make a webscraper(educational puposes), and I got really far, but this little issue is bugging me.
I made a request callback function, and im trying to get lines 75-78 to work. However to get this to work, I need PDF_LISTS and PDF_LINKS to initilaze to the right values.
I've already tried to make them global variables, and what not, for some reason that doesnt work. So my question is: How do I make a callback function that will call that for loop (75-78) and succesfully initilaze PDF_LISTS and PDF_LINKS to the correct values ?
(Dont worry I use this on educational content, with the prof's permission). First time posting here!
// URL_LINKS has the pdf links of the pages
PDF_LINKS = [];
// URL_LIST has the names of the pdf links
PDF_LIST = [];
function fillPDF(callback) {
request(url, function(err, res, body) {
$ = cheerio.load(body);
links = $('a'); //jquery get all hyperlinks
$(links).each(function(i, link) {
var value = $(link).attr('href');
// creates objects to hold the file
if (value.substring(value.length - 3, value.length) == "pdf") {
PDF_LINKS[i] = $(link).attr('href');
PDF_LIST[i] = $(link).text();
}
})
});
}
// must decleare fillPDF variable or else you wont initilze teh variables
fillPDF() {
//HERE I WANT PDF_LINKS and PDF_LIST to be intialized to 33.....
}
for (j = 0; j < PDF_LIST.length; j++) {
request(PDF_LINKS[j]).pipe(fs.createWriteStream(PDF_LIST[j]));
}

You may push your values into arrays using array's push method, avoiding array's element to be undefined.
You can put your final for loop into a function, and then use fillPDF();
You also need to call fillPDF's callback once the request is over.
PDF_LINKS = [];
PDF_LIST = [];
function fillPDF(callback) {
request(url, function(err, res, body) {
$ = cheerio.load(body);
links = $('a');
$(links).each(function(i, link) {
var value = $(link).attr('href');
if (value.slice(-3) == "pdf") {
PDF_LINKS.push(value);
PDF_LIST.push($(link).text());
}
})
callback();
});
}
function writePDF() {
for (j = 0; j < PDF_LIST.length; j++) {
request(PDF_LINKS[j]).pipe(fs.createWriteStream(PDF_LIST[j]));
}
}
fillPDF(writePDF);

Pouchdb promise or not a promise return formatting

I have situation where I am reading a database to find out if I need to images from the images database to send to the server. In the following code, I am reading the DB_WorkIssues database and checking each record to see if there is a field called IsAttachmentInserted. If so, I take values (which are the names of the images to get from the images database) and send them to the server.
I am having a problem figuring out how to set this up.
If there are no records with the IsAttachmentInserted field, then I would do nothing. So I would assume that I would return a null. If there is an IsAttachmentInsert value, then I would return a promise that is the .get of the images database and returns a document which is the array of images.
When I am done I do a submit.
I am not sure how to handle the .then after the images promise or null return.
function GetDirtyRecords() {
var startRows
var iTemp;
var insertedImages = "";
DB_WorkIssues.allDocs({ include_docs: true, descending: false }.then(function(response) {
data = response.rows;
startRows = data.length;
iTemp = 0
for (var i = 0; i < response.total_rows; i++) {
if (data[iTemp].doc.IsDeleted){
if(data[iTemp].doc.IsAttachmentInserted)
{
if(insertedImages == "")
insertedImages += ",";
insertedImages += data[iTemp].doc.IsAttachmentInserted;
}
iTemp++;
}
}
if(insertedImages != "")
{
return DB_TaskImages.get(id, {attachments: true});
}
else
{
return ;
}
)}.then(function(doc){
// do something with the attachments
$("form").submit();
})
}
I just need to find out how to handle the last part.
Thanks

How to wait to render page with express, while API grabs data?

I am trying to load data from the twitter api, getting user information and save that in a temporary array. That array will then be loaded on the page for viewing. The array is getting loaded by the API call, but it doesn't display.
I think I need to use an asynchronous thing like React or Angular, not sure. Would love some input!
function getUserIds (userId) {
T.get('statuses/retweeters/ids', { id: userId }, function (err, data, response) {
for(var i = 0; i < data.ids.length; i++){
ids.push(data.ids[i]);
}
getUserInfo();
});
}
function getUserInfo() {
for(var i = 0; i < ids.length; i++) {
T.get('users/lookup', { user_id: ids[i] }, function (err, data, response) {
names.push(data[0].screen_name);
pics.push(data[0].profile_image_url_https);
console.log(names);
});
}
res.render('display', {names: names, pics:pics});
}

The issue is that you are running ids.length async calls and those will finish some time in the future. You have to render your page only when they are all done. But, your for loop is synchronous so you are calling res.render() before any of them have finished. In addition, your T.get() calls may finish in any order (if that matters).
I would normally use promises for coordinating multiple asynchronous operations since it is a very, very good tool for that. But, if you aren't using promises, here's a simple technique to test when you have all your results back:
function getUserInfo() {
var names = [];
var pics = [];
for(var i = 0; i < ids.length; i++) {
T.get('users/lookup', { user_id: ids[i] }, function (err, data, response) {
if (err) {
// decide what to display if you get an API error
names.push("unknown due to API error");
} else {
names.push(data[0].screen_name);
pics.push(data[0].profile_image_url_https);
console.log(names);
}
if (names.length === ids.length) {
res.render('display', {names: names, pics:pics});
}
});
}
}
As I said above, this does not necessarily collect the results in order. If you need them in order, then you could do something like this:
function getUserInfo() {
var names = new Array(ids.length);
var pics = new Array(ids.length);
var doneCntr = 0;
ids.forEach(function(id, i) {
T.get('users/lookup', { user_id: id }, function (err, data, response) {
if (err) {
// decide what to display if you get an API error
names[i] = "unknown due to API error";
} else {
names[i] = data[0].screen_name;
pics[i] = data[0].profile_image_url_https;
}
++doneCntr;
if (doneCntr === ids.length) {
res.render('display', {names: names, pics: pics});
}
});
});
}
My preferred solution would to be to use Promise.all() and use a promisified version of T.get().

Grabbing and sending elements from MongoDB in Node.js server response

In my Express route, I am trying to return a list of elements that I am grabbing from MongoDB using Mongoose. I'm basically iterating through an array of items, and making MongoDB calls to get the parameter objects that each item has. However, I'm having trouble making sure that I get all the parameters before I send the response. I've tried using promises, other async library functions, etc, but none of them have seemed to work.
The current iteration of the code looks like this (I have tried a lot of different things):
exports.findAll = function(req, res){
Flow.find({}, function(err, items) {
console.log(items);
var payload = {}
var params = [];
for (var i=0; i < items.length; i++) {
var count2 = 0;
async.whilst(
function() {
return ((items[i]) && (count2 < items[i].params.length));
},
function(callback) {
Parameter.findById(items[i].params[count2], function(err, out) {
params.push(out);
count2++;
callback();
});
},
function(err) {
console.log(params);
var payload = {
"flows": items,
"params": params
};
res.send(payload);
console.log('success: flows found');
}
);
}
This code sends a payload with params not being completely full.
What would be a good way to deal with this? Honestly I just want these database calls to be synchronous, but I just can't figure out how to make this work.

This doesn't really seem necessary as you can actually use the $in operator with all the results from your first query:
Flow.find({},function(err,items) {
var ids = [];
// blocking? yes, but should be minor - do better if there are problems
for ( var i=0; i < items.length; i++ ) {
for ( var n=0; n < items[i].params.length; n++ ) {
ids.push( items[i].params[n] );
}
}
Parameter.find({ "_id": { "$in": ids } },function(err,params) {
res.send({ "flows": items, "params": params });
});
});
So there should be no reason to execute multiple queries inside an async loop, or loops as your code seems to be missing as the direct cause of the problem there.