We Keep Coding

JS code after for loop runs before for loop finishes

I have a javascript application running on NodeJS. For context I am using the express framework as this is meant to be our backend. I have a chunk of code which is meant to get data from a database, filter it and then send it back to the client. Instead, the filtering happens AFTER the response is sent, meaning the client is getting incorrect data. The code is below.
let resultArray = [];
const bulkSearchPromise = new Promise((resolve, reject) => {
for (let index = 0; index < input.length; index++) {
collectionPool.query('SELECT * FROM users WHERE ' + type + ' = $1', [input[index]], (err2, result2) => { // Make a query for each input user is trying to search for
if (err2) console.log("Error in bulk search: " + err2);
else {
if (result2.rows.length > 0) { // If input user searched for was found
pool.query('UPDATE users SET usedsearches = usedsearches + 1 WHERE id = $1', [result.rows[0].id]); // Increments used searches
// The code below will filter useless key value pairs. For example if username: null then there is not a reason to send it back to the client
let filteredArray = [];
for (let index = 0; index < result2.rows.length; index++) {
let array = Object.entries(result2.rows[index]);
let filtered = array.filter(([key, value]) => value != null);
let filteredObject = Object.fromEntries(filtered);
filteredArray.push(filteredObject);
resultArray.push(filteredObject);
}
console.log("a"); // This should run first.
}
}
});
}
resolve("ok");
})
bulkSearchPromise.then((value) => {
console.log("b"); // This should run second
return res.json({
status: 'success',
content: resultArray
}); // resultArray should be populated after the filtering above. Instead it is empty.
})
When the endpoint is hit the output will always be
username
b
a
What I need is for the for loop to run first and then after resultArray is populated, return it back to the client.
I've tried wrapping this code into a promise, but that hasnt helped either as 'resolve("ok")' is still called before the for loop completes.

Your promise is resolving before collectionPool.query is done.
The for loop only runs after collectionPool.query is done but you are resolving the promise before it.
Note that collectionPool.query is async and the its callback will only run when the web-api is finished (if this concept is murky check this out http://latentflip.com/loupe)
Option 1:
Move resolve() inside the collectionPool.query (where the console.log("a");) call back and call resolve(filteredObject).
In addition you should reject(err2) when err2 is not null (not just console)
Option 2:
you can use Util.Promisify to transform collectionPool.query to promise base API which will save you the hustle of manually transform
const util = require('util');
util.promisify(collectionPool.query)(QUERY)
.then(queryRes => {
/* logic to refine queryRes to object */
return filteredObject;
})
in both options you can omit the resultArray from your code. if you resolve(filteredObject) or return filteredObject; in then() you will be able to access this data on the next then (the value in bulkSearchPromise.then((value)).

Why does async-await loop returns its result to origin array?

I am using a function fetchChain called with an array of objects, each object includes the url and other parameters to call another function fetchData to chain fetch-Calls. The async/await loop returns its results to the array I fed it with, like so (simplified):
fetchChain(array){
const loop = async (items) => {
for (let i = 0; i < items.length; i++) {
await fetchData(items[i])
}
}
return loop(array)
.then(()=>{ return array })
}
I can wait for all promises/results to return like so:
fetchChain(prepareAjax)
.then((res)=> console.log(res) )
.then(()=> console.log('done') )
So why does the loop returns its results to the array it got fed with? There is no specific return or then which returns the results to the origin and I can't wrap my head around about what happens here.
As requested, the fetchDacta-function:
fetchData(obj){
// some parameters get added, like so
obj.timeout = obj.timeout || 10000, [...];
const fetchRetry = async (url, options, n) => {
try {
return await fetch(url, options)
.then((response)=> return response.json());
} catch(err) {
if (n === 1) throw err;
return await sleep(obj.delay)
.then(()=> fetchRetry(url, options, n - 1));
}
};
return fetchRetry(url, {}, obj.retries);
}

I'm not sure if I understand the question correctly, but I think you're asking why the array argument in function fetchChain contains information assigned in fetchData.
For this you have to look at the difference of passing by reference vs passing by value.
In JavaScript Objects and Arrays are automatically passed by reference to a function; which means array points to the same memory as items, and when you modify items you are modifying array.
This is a simple example to illustrate passing by reference
let changeMessage = (o) => o.message = 'world';
let obj = { message: 'hello'};
changeMessage(obj);
console.log(obj.message);
// Output: world
You can avoid modifying the array by cloning it first

Add a resolved Promise to an object as a property

I'm developing a Dapp based on Ethereum and I got stuck with Promises.
In the for loop, the elements of the array have to be verified one by one. This happens at the validateRow() function, which returns a Promise at first. The Promise will be resolved to a number (0, when the element is valid; 1, 2 or 3, when it is not valid).
In the end, I would like to return a resultList[], which is an array of objects. Each object should have two properties:
row, containing the element (a string),
and result, which tells whether it is valid.
However, the resultList[] only contains the rows in the end, while the 'then' branch only holds the results ({"row":"","result":"0"}). I added the logs which are printed in the console as comments. Unfortunately, I can't figure out, how I could put the two together.
var resultList = [];
for (var i = 0; i < App.resultArray.length; i++) {
var promiseReturned = contractInstance.validateRow.call(App.resultId, App.resultArray[i]);
console.log(promiseReturned); //Promise {<pending>}
var rowObject = new Object();
console.log(App.resultArray[i]); //row1string
rowObject.row = App.resultArray[i];
promiseReturned.then(function(returnVal) {
console.log("the returnVal: " + returnVal); //the returnVal: 1
rowObject.result = returnVal;
console.log("the rowObject :" + JSON.stringify(rowObject)); //{"row":"","result":"0"}
return returnVal;
});
resultList.push(rowObject);
};
console.log(resultList); //[{"row":"row1string"},{"row": "row2string"}]
return resultList;

In Javascript, use forward slashes to denote comments, not backslashes, else you'll get syntax errors.
Use Promise.all to wait for all promises to be resolved before returning the object:
async function getResultList() {
const allPromises = App.resultArray.map((row) => (
contractInstance.validateRow.call(App.resultId, row)
.then(result => ({ result, row }))
));
const resultList = await Promise.all(allPromises);
return resultList; // This will return a Promise to the caller of getResultList
}
Note that you'll have to consume getResultList as a promise, since it doesn't run synchronously. eg
const resultList = await getResultList();

For completeness, CertainPerformance's answer but using async/await, and rewritten more concisely:
async function getResultList() {
return await Promise.all(
App.resultArray.map(async (row) => {
const result = await contractInstance.validateRow.call(App.resultId, row);
return {
row,
result,
};
})
);
}

How to use promises to wait for async API calls

I am creating an API that when GET, a series of calls to the News API are made, news article titles are extracted into a giant string, and that string is processed into an object to be delivered to a wordcloud on the front-end. So far, I've been able to use underscore's _.after and request-promise to make my app wait till all API calls have completed before calling processWordBank() which takes the giant string and cleans it up into an object. However, once processWordBank() is called, I don't understand where the flow of the program is. Ideally, processWordBank() returns obj to cloudObj in the router, so that the obj can be passed to res.json() and spit out as the response. I believe my use of _.after has put me in a weird situation, but it's the only way I've been able to get async calls to finish before proceeding to next desired action. Any suggestions?
(I've tried to leave out all unnecessary code but let me know if this is insufficient)
// includes...
var sourceString = ""
// router
export default ({ config }) => {
let news = Router()
news.get('/', function(req, res){
var cloudObj = getSources()
res.json({ cloudObj })
})
return news
}
// create list of words (sourceString) by pulling news data from various sources
function getSources() {
return getNewsApi()
}
// NEWS API
// GET top 10 news article titles from News API (news sources are determined by the values of newsApiSource array)
function getNewsApi() {
var finished = _.after(newsApiSource.length, processWordBank)
for(var i = 0; i < newsApiSource.length; i++) {
let options = {
uri: 'https://newsapi.org/v1/articles?source=' + newsApiSource[i] + '&sortBy=' + rank + '&apiKey=' + apiKey,
json: true
}
rp(options)
.then(function (res) {
let articles = res.articles // grab article objects from the response
let articleTitles = " " + _.pluck(articles, 'title') // extract title of each news article
sourceString += " " + articleTitles // add all titles to the word bank
finished() // this async task has finished
})
.catch(function (err) {
console.log(err)
})
}
}
// analyse word bank for patterns/trends
function processWordBank(){
var sourceArray = refineSource(sourceString)
sourceArray = combineCommon(sourceArray)
sourceArray = getWordFreq(sourceArray)
var obj = sortToObject(sourceArray[0], sourceArray[1])
console.log(obj)
return obj
}

A big issue in your asynchronous flow is that you use a shared variable sourceString to handle the results. When you have multiple calls to getNewsApi() your result is not predictable and will not always be the same, because there is no predefined order in which the asynchronous calls are executed. Not only that, but you never reset it, so all subsequent calls will also include the results of the previous calls. Avoid modifying shared variables in asynchronous calls and instead use the results directly.
I've been able to use underscore's _.after and request-promise to make my app wait till all API calls have completed before calling processWordBank()
Although it would possible to use _.after, this can be done very nicely with promises, and since you're already using promises for your requests, it's just a matter of collecting the results from them. So because you want to wait until all API calls are completed you can use Promise.all which returns a promise that resolves with an array of the values of all the promises, once all of them are fulfilled. Let's have a look at a very simple example to see how Promise.all works:
// Promise.resolve() creates a promise that is fulfilled with the given value
const p1 = Promise.resolve('a promise')
// A promise that completes after 1 second
const p2 = new Promise(resolve => setTimeout(() => resolve('after 1 second'), 1000))
const p3 = Promise.resolve('hello').then(s => s + ' world')
const promises = [p1, p2, p3]
console.log('Waiting for all promises')
Promise.all(promises).then(results => console.log('All promises finished', results))
console.log('Promise.all does not block execution')
Now we can modify getNewsApi() to use Promise.all. The array of promises that is given to Promise.all are all the API request you're doing in your loop. This will be created with Array.protoype.map. And also instead of creating a string out of the array returned from _.pluck, we can just use the array directly, so you don't need to parse the string back to an array at the end.
function getNewsApi() {
// Each element is a request promise
const apiCalls = newsApiSource.map(function (source) {
let options = {
uri: 'https://newsapi.org/v1/articles?source=' + source + '&sortBy=' + rank + '&apiKey=' + apiKey,
json: true
}
return rp(options)
.then(function (res) {
let articles = res.articles
let articleTitles = _.pluck(articles, 'title')
// The promise is fulfilled with the articleTitles
return articleTitles
})
.catch(function (err) {
console.log(err)
})
})
// Return the promise that is fulfilled with all request values
return Promise.all(apiCalls)
}
Then we need to use the values in the router. We know that the promise returned from getNewsApi() fulfils with an array of all the requests, which by themselves return an array of articles. That is a 2d array, but presumably you would want a 1d array with all the articles for your processWordBank() function, so we can flatten it first.
export default ({ config }) => {
let news = Router()
new.get('/', (req, res) => {
const cloudObj = getSources()
cloudObj.then(function (apiResponses) {
// Flatten the array
// From: [['source1article1', 'source1article2'], ['source2article1'], ...]
// To: ['source1article1', 'source1article2', 'source2article1', ...]
const articles = [].concat.apply([], apiResponses)
// Pass the articles as parameter
const processedArticles = processWordBank(articles)
// Respond with the processed object
res.json({ processedArticles })
})
})
}
And finally processWordBank() needs to be changed to use an input parameter instead of using the shared variable. refineSource is no longer needed, because you're already passing an array (unless you do some other modifications in it).
function processWordBank(articles) {
let sourceArray = combineCommon(articles)
sourceArray = getWordFreq(sourceArray)
var obj = sortToObject(sourceArray[0], sourceArray[1])
console.log(obj)
return obj
}
As a bonus the router and getNewsApi() can be cleaned up with some ES6 features (without the comments from the snippets above):
export default ({ config }) => {
const news = Router()
new.get('/', (req, res) => {
getSources().then(apiResponses => {
const articles = [].concat(...apiResponses)
const processedArticles = processWordBank(articles)
res.json({ processedArticles })
})
})
}
function getNewsApi() {
const apiCalls = newsApiSource.map(source => {
const options = {
uri: `https://newsapi.org/v1/articles?source=${source}&sortBy=${rank}&apiKey=${apiKey}`,
json: true
}
return rp(options)
.then(res => _.pluck(res.articles, 'title'))
.catch(err => console.log(err))
})
return Promise.all(apiCalls)
}

Am I using Promises well?

Probem
My problem is that I want my code to do the following:
Make an innitial request
Once I get that request's answer, I process it and make a batch of requests again
Once I am done with the batch, and have all its responses, I write a file
Once the file is done, I print a message
I got the first two steps right, but my program is not doing the last two as I expected.
Code
This code tries to exemplify what I am trying to achieve. With only promise and jsonfile it is a simple application that represents the architecture of my code in detail and it works out of the bat, provided you install both libraries.
let jsonfile = require("jsonfile");
let Promise = require("promise");
let write = Promise.denodeify(jsonfile.writeFile);
let writeOutput = function(filename, content) {
return write(filename, content, {spaces: 4});
};
//Returns a random number each time it is invoked
//after a random period of time between 1s and 6s
let requestSimulator = function() {
return new Promise((fulfil, reject) => {
let randomNum = Math.random();
let wait = Math.floor((Math.random() * 6000) + 2000);
setTimeout(() => fulfil(randomNum), wait, randomNum);
});
};
//Returns an array of rounded numbers
let roundNumbers = function(someNumbers) {
let numbersArr = [];
let tmpNum;
for (let number of someNumbers) {
tmpNum = Math.floor(number);
console.log("Rounded " + number + " to " + tmpNum);
numbersArr.push(tmpNum);
}
return numbersArr;
};
//Receives an array of rounded numbers, and for each number
//makes a new request.
//It then sums the response with the given number.
let sumNumbersBatch = function(numbersArr) {
let promisesArray = [];
for (let number of numbersArr) {
let promise = new Promise((fulfil, reject) => {
requestSimulator()
.then(result => {
let newNum = number + result;
console.log("Summing " + number + " with " + result + "resultint in " + newNum);
fulfil(newNum);
});
});
promisesArray.push(promise);
}
return new Promise.all(promisesArray);
};
//Starts the process
let getData = function() {
return new Promise((fulfil, reject) => {
requestSimulator()
.then(number => fulfil([number, number * 2, number * 3]));
});
};
console.log("Starting program");
getData()
.then(roundNumbers)
.then(sumNumbersBatch)
.then(newNumbers => writeOutput("testFile.txt", newNumbers))
.then(console.log("Program finished"))
.catch(console.log);
After running this, you can see that the output will be something like:
Starting program
Program finished
Rounded 0.20890058801647582 to 0
Rounded 0.41780117603295164 to 0
Rounded 0.6267017640494275 to 0
Summing 0 with 0.05537663551196226resultint in 0.05537663551196226
Summing 0 with 0.34853429001859215resultint in 0.34853429001859215
Summing 0 with 0.988336787994851resultint in 0.988336787994851
Which is wrong !!!!
Program finish should appear last, not second!
Questions:
So now I have doubts about my code:
Am I making a correct use of Promise.all ?
Am I promisifying the write function well?
Also, I am open to suggestions regarding code quality !!!!
Any help and explanation would be very appreciated.

I've rewritten the whole answer to match the modified code. My first attempt at an "answer" was no more than an extended comment of what seemed wrong with the first provided code; so nothing lost.
Most of your code is correct, this line is actually "wrong":
.then(console.log("Program finished"))
and confuses you because it calls console.log("Program finished") immediately, and return undefined, so that the then translates to .then(undefined).
it should be
.then(() => console.log("Program finished"))
And there should be no new in front of Promise.all()
Although a few things can be improved, especially your use of the Deferred antipattern. That's the manual creation of a Deferred Object when it is not necessary, when you're already dealing with promises at that place. Like this one:
//Starts the process
let getData = function() {
return new Promise((fulfil, reject) => {
requestSimulator()
.then(number => fulfil([number, number * 2, number * 3]));
});
};
better would be
//Starts the process
let getData = function() {
return requestSimulator().then(number => [number, number * 2, number * 3]);
};
whereas in requestSimulator you need to create a new Promise() in order to use Promises with setTimeout(). There it is appropriate.
let jsonfile = require("jsonfile");
let Promise = require("promise");
let write = Promise.denodeify(jsonfile.writeFile);
//OK, now this function has a purpose/additional value (formatting)
//and is not just forwarding the arguments
let writeOutput = function(filename, content) {
return write(filename, content, {spaces: 4});
};
//fine, a mock
let requestSimulator = function() {
return new Promise((fulfil, reject) => {
let randomNum = Math.random();
let wait = Math.floor((Math.random() * 6000) + 2000);
//there's no need/reason to pass `randomNum` to setTimeout as a third argument
setTimeout(() => fulfil(randomNum), wait);
});
};
//this can be shortened to, although it doesn't log anymore
let roundNumbers = function(someNumbers) {
return someNumbers.map(Math.floor);
};
//Receives an array of rounded numbers, and for each number
//makes a new request.
//It then sums the response with the given number.
let sumNumbersBatch = function(numbersArr) {
//this again can be achieved simpler by using `Array#map` instead of `for..of`
let promisesArray = numbersArr.map(number => {
return requestSimulator()
.then(result => result + number);
});
//no `new` here! Promise.all() is just a utility-function, no constructor.
return Promise.all(promisesArray);
};
//Starts the process
let getData = function() {
//removed the wrapping Promise.
return requestSimulator()
.then(number => [ number, number * 2, number * 3 ]);
};
console.log("Starting program");
getData()
.then(roundNumbers)
.then(sumNumbersBatch)
.then(newNumbers => writeOutput("testFile.txt", newNumbers))
//this executes `console.log()` immediately and passes the result (`undefined`) to `then()`
//.then(console.log("Program finished"))
//but `then()` needs a function:
.then(() => console.log("Program finished"))
//here you pass a reference to the function `console.log`, that's fine
.catch(console.log);

Am I making a correct use of Promise.all?
Yes and no, the way you use Promise.all is weird - you always provide an empty array. Promise.all expects as input an array of promises, it wait for all promises in the input to be resolved OR either of them to fail.
It returns a promise that is either resolved (if all input promises are ok) or rejected (if either fails). In your original case Promise.all is always resolved because the input list is empty
Is my file really being written after ALL the requests finished, or is it being written as they finish?
The method writeOutput in invoked after the Promise returned from makeBatchRequests is resolved, it is invoked with two arguments - fileName, which is undefined because it was never defined in your code and the second one is result - which is an Array whose members are resolved results of promisesArray, which is always empty. So technically, YES, the function is invoked after all the requests being finished, but no data would be written to a file (oh, actually, empty string "" which is [].toString() will be printed to file :] )
Am I promisifying the write function well?
YES, you're doing it right.
Other than that, try to rewrite your code, step by step, testing it when proceeding with every step and comparing the expected results with what you get.
As mentioned in the answers above, there're a lot of stuff to fix, good luck! :]

Well then, I think the problem might be on writeOutput(fileName, result), are you sure it returns a promise?
This is more of a suggestion than an actual answer, but try doing like so:
scrapy.getStanceMods()
.then(['www.google.com', 'www.reddit.com'])
.then(makeBatchRequest)
.then(result => {
return writeOutput(fileName, result))
.then(console.log("Completed."))
.catch(error=>console.log('will catch inner errors'))
})
.catch(error =>console.error(error));