We Keep Coding

Javascript code is running in a limited loop, need to refactor

I have a set of code that is working as intended, but my PR approver is telling me to refactor because it's running in a loop, and there should be a more efficient way to write the code. I'm still learning javascript, so often problems like this throw me for a loop (no pun intended), and I rely on the greater world of the internet for learning/help.
The code is meant to check a URL's domain against an array of 'valid' domains and return whether the URL's domain is valid (returns false) or invalid (returns true).
How do I go about refactoring this code so there is no loop in calling cleanupParams?
export const isValidLink = (hostname?: string, validDomains: string[] = []) => {
return validDomains.every((domain) => {
if (hostname && !hostname.startsWith("#")) {
return cleanupParams(hostname).indexOf(domain) < 0;
}
});
};
const cleanupParams = (url: string) => {
let domain = url;
try {
domain = new URL(url).hostname;
} catch {
domain = url;
}
return domain;
};

The method every will run the callback method for each item in the array. The hostname, though, is not affected by the array , and so, should not need to be cleaned up each time. You should do that, once, outside the every loop.
export const isValidLink = (hostname?: string, validDomains: string[] = []) => {
// early break if conditions are not met
const isValidHostname = hostname && !hostname.startsWith("#");
if (!isValidHostname) return false;
const cleanedUpHostname = cleanupParams(hostname);
return validDomains.every((domain) => {
return cleanedUpHostname.indexOf(domain) < 0;
});
};

async functions not executing in the correct order inside a map function

I have created an async function that will extra the data from the argument, create a Postgres query based on a data, then did some processing using the retrieved query data. Yet, when I call this function inside a map function, it seemed like it has looped through all the element to extra the data from the argument first before it proceed to the second and the third part, which lead to wrong computation on the second element and onwards(the first element is always correct). I am new to async function, can someone please take at the below code? Thanks!
async function testWeightedScore(test, examData) {
var grade = [];
const testID = examData[test.name];
console.log(testID);
var res = await DefaultPostgresPool().query(
//postgres query based on the score constant
);
var result = res.rows;
for (var i = 0; i < result.length; i++) {
const score = result[i].score;
var weightScore = score * 20;
//more computation
const mid = { "testID": testID, "score": weightScore, more values...};
grade.push(mid);
}
return grade;
}
(async () => {
const examSession = [{"name": "Sally"},{"name": "Bob"},{"name": "Steph"}]
const examData = {
"Sally": 384258,
"Bob": 718239,
"Steph": 349285,
};
var test = [];
examSession.map(async sesion => {
var result = await testWeightedScore(sesion,examData);
let counts = result.reduce((prev, curr) => {
let count = prev.get(curr.testID) || 0;
prev.set(curr.testID, curr.score + count);
return prev;
}, new Map());
let reducedObjArr = [...counts].map(([testID, score]) => {
return {testID, score}
})
console.info(reducedObjArr);
}
);
})();
// The console log printed out all the tokenID first(loop through all the element in examSession ), before it printed out reducedObjArr for each element

The async/await behaviour is that the code pause at await, and do something else (async) until the result of await is provided.
So your code will launch a testWeightedScore, leave at the postgresql query (second await) and in the meantime go to the other entries in your map, log the id, then leave again at the query level.
I didn't read your function in detail however so I am unsure if your function is properly isolated or the order and completion of each call is important.
If you want each test to be fully done one after the other and not in 'parallel', you should do a for loop instead of a map.

Using async/await within a read stream to batch upload to DynamoDB

A bit of a Node novice here...
I'm trying to write a function that pulls a CSV down from S3 and batch-writes the items to DynamoDB. DynamoDB has a limit of 25 in each batch so I need write the entries as I go. The problem I'm running into is that my await function to execute the DB write only fires at the .end(), rather than when I check.
I understand that I can't execute things like this, but I'm not sure how to fix it? I'm using Node12.
Thanks.
async function populateTable(
dataFile: bucketKey,
tableName: string
): Promise<void> {
const s3 = getS3Client();
const stream = s3.getObject(dataFile).createReadStream();
const BATCH_COUNT = 25; // Max size to write to DynamoDB
let counter = 0;
let datarows: any = [];
let datarow = {};
stream
.pipe(parse(DATA_HEADERS))
.on("data", async function(data: DataRow) {
counter++;
datarow = {
PutRequest: {
Item: data
}
};
datarows.push(datarow);
if (counter % BATCH_COUNT === 0) {
console.log("before batch write " + counter); // This fires!
await batchWriteToDynamo(datarows, tableName); // I want this function to fully execute before moving on
console.log("after batch write " + counter); // This does not
datarows = [];
}
})
.on("end", async function() {
await batchWriteToDynamo(datarows, tableName); // This fires!
});
}

I'd guess that these stream events aren't async compatible; you might have to resort to creating your own promise chain. You could potentially do that in the following manner:
let datarow = {};
let pr = Promise.resolve();
// ...
if (counter % BATCH_COUNT === 0) {
let scopedRows = datarows.slice(); // scoped shallow copy
pr = pr.then(()=> batchWriteToDynamo(scopedrows, tableName));
// ...
.on("end", async function() {
pr = pr.then(()=> batchWriteToDynamo(datarows, tableName));
This should make sure your batch writes happen one at a time and in the correct order. Note also the shallow copy of datarows during the data event. Pretty sure this is necessary since events and promises will be happening in an unpredictable order.
But in the end event it shouldn't be necessary since datarows shouldn't be changing any more at that point, I would guess.

asynchronous loop for in Javascript

I'm trying to iterate and print out in order an array in Javascript that contains the title of 2 events that I obtained from doing web scraping to a website but it prints out in disorder. I know Javascript is asynchronous but I'm new in this world of asynchronism. How can I implement the loop for to print the array in order and give customized info?
agent.add('...') is like console.log('...'). I'm doing a chatbot with DialogFlow and NodeJs 8 but that's not important at this moment. I used console.log() in the return just for debug.
I tried the next:
async function printEvent(event){
agent.add(event)
}
async function runLoop(eventsTitles){
for (let i = 0; i<eventsTitles.length; i++){
aux = await printEvent(eventsTitles[i])
}
}
But i got this error error Unexpected await inside a loop no-await-in-loop
async function showEvents(agent) {
const cheerio = require('cheerio');
const rp = require('request-promise');
const options = {
uri: 'https://www.utb.edu.co/eventos',
transform: function (body) {
return cheerio.load(body);
}
}
return rp(options)
.then($ => {
//** HERE START THE PROBLEM**
var eventsTitles = [] // array of event's titles
agent.add(`This mont we have these events available: \n`)
$('.product-title').each(function (i, elem) {
var event = $(this).text()
eventsTitles.push(event)
})
agent.add(`${eventsTitles}`) // The array prints out in order but if i iterate it, it prints out in disorder.
// *** IMPLEMENT LOOP FOR ***
agent.add(`To obtain more info click on this link https://www.utb.edu.co/eventos`)
return console.log(`Show available events`);
}).catch(err => {
agent.add(`${err}`)
return console.log(err)
})
}
I would like to always print out Event's title #1 and after Event's title #2. Something like this:
events titles.forEach((index,event) => {
agent.add(`${index}. ${event}`) // remember this is like console.log(`${index}. ${event}`)
})
Thanks for any help and explanation!

There no async case here but if you still face difficultly than use this loop
for (let index = 0; index < eventsTitles.length; index++) {
const element = eventsTitles[index];
agent.add(${index}. ${element})
}

How do I use the Request function in node.js within a for loop and have it return output in correct order?

I've been using request to iterate through several XML entries and return every article, date, and url to the console by using cheerio.js. The main issue is that the output will appear in a different order each time, as Request is an asynchronous function. I'm really inexperienced with javascript in general, and was wondering how I could retrieve consistent output (I've been reading about async and promises, I'm just unsure how to implement them).
Here is my code:
var count = 0;
for(var j = 0; j < arrNames.length; j++){
request('http://dblp.org/search/publ/api?q=' + arrNames[j], function(error, response, html){
if (!error && response.statusCode == 200){
var $ = cheerio.load(html, {xmlMode: true});
console.log($('query').text()+'\n');
$('title').each(function(i, element){
var title = $('title').eq(i).text();
var year = Number($('year').eq(i).text());
var url = $('ee').eq(i).text();
if (year >= arrTenures[count]){
console.log(title);
console.log(year);
console.log(url + '\n');
}
});
count++;
}
});
}

Though you've already found a solution, I thought I'd show you how you would do this using ES6 promises (a more modern approach for managing multiple asynchronous operations):
const rp = require('request-promise');
Promise.all(arrNames.map(item => {
return rp('http://dblp.org/search/publ/api?q=' + item).then(html => {
const $ = cheerio.load(html, {xmlMode: true});
return $('title').map(function(i, element){
const title = $(element).text();
const year = Number($('year').eq(i).text());
const url = $('ee').eq(i).text();
return {title, year, url};
}).get();
});
})).then(results => {
// results is an array of arrays, in order
console.log(results);
}).catch(err => {
console.log(err);
});
This offers several advantages:
Promise.all() puts the results in order for you.
rp() checks the 2xx status for you.
This can be chained/sequenced with other asynchronous operations more easily.
Error handling is simpler and low level errors are propagated out to the top level for you automatically.
This is throw-safe (if async errors are thrown, they are captured and propagated).

It looks like you're trying to capture the iteration number of each request, so use forEach and utilize its second parameter, which indicates the iteration index:
arrNames.forEach((q, requestIndex) => {
request('http://dblp.org/search/publ/api?q=' + q, (error, response, html) => {
if (error || response.statusCode == 200) return;
var $ = cheerio.load(html, {
xmlMode: true
});
console.log($('query').text() + '\n');
$('title').each(function(i, element) {
var title = $('title').eq(i).text();
var year = Number($('year').eq(i).text());
var url = $('ee').eq(i).text();
if (year >= arrTenures[requestIndex]) {
console.log(title);
console.log(year);
console.log(url + '\n');
}
});
});
});
As a side note, consistent indentation really improves code readability - you might consider a linter.

On your first try, you might have tried:
if (year >= arrTenures[j]) {
But noticed that didn't work. This is because of scoping issues
You can solve your problem by using an iterator like forEach(), or simply changing your for loop to use let:
for(let j = 0; j < arrNames.length; j++){
Now you can just use j in your check instead of count.
The real question though, is why are arrTenures and arrNames separate arrays? Their information clearly relates to each other, so relying on the array index to couple them seems like a bad idea. Instead, you should try and keep a single array of objects with all related information. For example:
[
{ name: 'some name', tenures: 2 },
{ name: 'another', tenures: 5 }
]