We Keep Coding

Array of filtered axios results from paginated API is empty

In my code below I get an empty array on my console.log(response) but the console.log(filterdIds) inside the getIds function is showing my desired data. I think my resolve is not right.
Note that I run do..while once for testing. The API is paged. If the records are from yesterday it will keep going, if not then the do..while is stopped.
Can somebody point me to the right direction?
const axios = require("axios");
function getToken() {
// Get the token
}
function getIds(jwt) {
return new Promise((resolve) => {
let pageNumber = 1;
const filterdIds = [];
const config = {
//Config stuff
};
do {
axios(config)
.then((response) => {
response.forEach(element => {
//Some logic, if true then:
filterdIds.push(element.id);
console.log(filterdIds);
});
})
.catch(error => {
console.log(error);
});
} while (pageNumber != 1)
resolve(filterdIds);
});
}
getToken()
.then(token => {
return token;
})
.then(jwt => {
return getIds(jwt);
})
.then(response => {
console.log(response);
})
.catch(error => {
console.log(error);
});
I'm also not sure where to put the reject inside the getIds function because of the do..while.

The fundamental problem is that resolve(filterdIds); runs synchronously before the requests fire, so it's guaranteed to be empty.
Promise.all or Promise.allSettled can help if you know how many pages you want up front (or if you're using a chunk size to make multiple requests--more on that later). These methods run in parallel. Here's a runnable proof-of-concept example:
const pages = 10; // some page value you're using to run your loop
axios
.get("https://httpbin.org") // some initial request like getToken
.then(response => // response has the token, ignored for simplicity
Promise.all(
Array(pages).fill().map((_, i) => // make an array of request promisess
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${i + 1}`)
)
)
)
.then(responses => {
// perform your filter/reduce on the response data
const results = responses.flatMap(response =>
response.data
.filter(e => e.id % 2 === 0) // some silly filter
.map(({id, name}) => ({id, name}))
);
// use the results
console.log(results);
})
.catch(err => console.error(err))
;
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
The network tab shows the requests happening in parallel:
If the number of pages is unknown and you intend to fire requests one at a time until your API informs you of the end of the pages, a sequential loop is slow but can be used. Async/await is cleaner for this strategy:
(async () => {
// like getToken; should handle err
const tokenStub = await axios.get("https://httpbin.org");
const results = [];
// page += 10 to make the snippet run faster; you'd probably use page++
for (let page = 1;; page += 10) {
try {
const url = `https://jsonplaceholder.typicode.com/comments?postId=${page}`;
const response = await axios.get(url);
// check whatever condition your API sends to tell you no more pages
if (response.data.length === 0) {
break;
}
for (const comment of response.data) {
if (comment.id % 2 === 0) { // some silly filter
const {name, id} = comment;
results.push({name, id});
}
}
}
catch (err) { // hit the end of the pages or some other error
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
Here's the sequential request waterfall:
A task queue or chunked loop can be used if you want to increase parallelization. A chunked loop would combine the two techniques to request n records at a time and check each result in the chunk for the termination condition. Here's a simple example that strips out the filtering operation, which is sort of incidental to the asynchronous request issue and can be done synchronously after the responses arrive:
(async () => {
const results = [];
const chunk = 5;
for (let page = 1;; page += chunk) {
try {
const responses = await Promise.all(
Array(chunk).fill().map((_, i) =>
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${page + i}`)
)
);
for (const response of responses) {
for (const comment of response.data) {
const {name, id} = comment;
results.push({name, id});
}
}
// check end condition
if (responses.some(e => e.data.length === 0)) {
break;
}
}
catch (err) {
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
(above image is an except of the 100 requests, but the chunk size of 5 at once is visible)
Note that these snippets are proofs-of-concept and could stand to be less indiscriminate with catching errors, ensure all throws are caught, etc. When breaking it into sub-functions, make sure to .then and await all promises in the caller--don't try to turn it into synchronous code.
See also
How do I return the response from an asynchronous call? and Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference which explain why the array is empty.
What is the explicit promise construction antipattern and how do I avoid it?, which warns against adding a new Promise to help resolve code that already returns promises.

To take a step back and think about why you ran into this issue, we have to think about how synchronous and asynchronous javascript code works together. Your
synchronous getIds function is going to run to completion, stepping through each line until it gets to the end.
The axios function invocation is returning a Promise, which is an object that represents some future fulfillment or rejection value. That Promise isn't going to resolve until the next cycle of the event loop (at the earliest), and your code is telling it to do some stuff when that pending value is returned (which is the callback in the .then() method).
But your main getIds function isn't going to wait around... it invokes the axios function, gives the Promise that is returned something to do in the future, and keeps going, moving past the do/while loop and onto the resolve method which returns a value from the Promise you created at the beginning of the function... but the axios Promise hasn't resolved by that point and therefore filterIds hasn't been populated.
When you moved the resolve method for the promise you're creating into the callback that the axios resolved Promise will invoke, it started working because now your Promise waits for axios to resolve before resolving itself.
Hopefully that sheds some light on what you can do to get your multi-page goal to work.
I couldn't help thinking there was a cleaner way to allow you to fetch multiple pages at once, and then recursively keep fetching if the last page indicated there were additional pages to fetch. You may still need to add some additional logic to filter out any pages that you batch fetch that don't meet whatever criteria you're looking for, but this should get you most of the way:
async function getIds(startingPage, pages) {
const pagePromises = Array(pages).fill(null).map((_, index) => {
const page = startingPage + index;
// set the page however you do it with axios query params
config.page = page;
return axios(config);
});
// get the last page you attempted, and if it doesn't meet whatever
// criteria you have to finish the query, submit another batch query
const lastPage = await pagePromises[pagePromises.length - 1];
// the result from getIds is an array of ids, so we recursively get the rest of the pages here
// and have a single level array of ids (or an empty array if there were no more pages to fetch)
const additionalIds = !lastPage.done ? [] : await getIds(startingPage + pages, pages);
// now we wait for all page queries to resolve and extract the ids
const resolvedPages = await Promise.all(pagePromises);
const resolvedIds = [].concat(...resolvedPages).map(elem => elem.id);
// and finally merge the ids fetched in this methods invocation, with any fetched recursively
return [...resolvedIds, ...additionalIds];
}

Node js how to run axios.get every 2 seconds?

I am kinda of a newbie to node js, Here is what i am trying to do: i am looping through a json file full of links of our website via the map function (around 3000 links), inside the loop i am doing a axios get for each link and getting the response status code(will do other things in the future). But i want to run the axios get only like every 2 seconds or 5 seconds otherwise i am overwhelming the webserver. I am trying to input async await but it's still too fast and server is taking a hit (i am technically DDos-ing my own website). I put a SetTimeout around the axios but that doesn't seem like it worked, since in the console results are printing way too fast. so the question is, how do i make each axios.get request wait every 2 seconds before running in the map loop?.
var axios = require('axios');
const fs = require('fs');
var statusCheck = 0;
var main = [];
let rawdata = fs.readFileSync('C:/Users/jay/Documents/crawl/filtered2.json');
let jsonParsed = JSON.parse(rawdata);
jsonParsed.map(async(line) => {
var encodeLink = encodeURI(line.link);
const response = await axios.get(encodeLink).catch((err) => {
var Status_ErrorsCatchaxios = {
"status Code": err.response.status ? err.response.status : "No status code available",
"Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
"Original Link": encodeLink ? encodeLink : "No Original Link Available",
"errorCode": err
}
main.push(Status_ErrorsCatchaxios)
})
try {
console.log(response.status)
statusCheck = statusCheck + 1;
console.log("Link: ", statusCheck)
} catch (error) {
console.log(error)
}
})

The [].map function doesn't wait for your items to resolve, so your code is currently dispatching all the requests (as you said, around 3000) in parallel.
You can use for...of instead to only run one request at a time. For example:
async function makeRequests (lines) {
for (const line of lines) {
const encodedLink = encodeURI(line.link)
const response = await axios.get(encodedLink)
// ...your response handling code here...
}
}
makeRequests(jsonParsed)
If you want to wait for 2s between each request, you can add this line of code inside your for...of loop:
await new Promise(resolve => setTimeout(resolve, 2000))
Better solution
The solution above works, but I assume your webserver can probably take more than one request at a time, so maybe the ideal scenario would be to limit your code to make only up to N requests in parallel at a given time. This way you don't flood your server but you're able to get your results faster than just doing one request at a time.
The bluebird NPM module allows you to do that with their Promise.map function.
This function receives your list of items as the first argument, a function that executes something and returns a promise for each item as the second argument, and an object with a concurrency key describing how many items you want to allow to be handled in parallel as the third argument.
Here's how it could work:
const bluebird = require('bluebird')
async function makeRequests (lines) {
await bluebird.map(
lines,
async (line) => {
const encodedLink = encodeURI(line.link)
const response = await axios.get(encodedLink)
// ...your response handling code here...
},
{ concurrency: 3 }
)
}
makeRequests(jsonParsed)

Ditch the map, replace with a for ... of, await a promise that takes 2s to resolve, wrap everything inside an async IIFE for the await to be legal.
// dummy data
const fakeJson = new Array(5).fill();
const fakeRequest = () => console.log(`request at ${new Date().toUTCString()}`);
// iteration with 2s in between
(async () => {
for (let line of fakeJson) {
await new Promise(r => setTimeout(r, 2000));
fakeRequest();
}
})()
You can also use more classically use setInterval but HTTP requests are asynchronous, so might as well start with a structure that handles well async and loops.

You are hitting all at once because .map,.forEach,.reduce etc doesn't wait for the Promise to resolve. Use Simple For loop, it will wait for each promise to resolve or reject.
for(let i=0;i<jsonParsed.length;i++) {
var encodeLink = encodeURI(line.link);
const response = await axios.get(encodeLink).catch(...)
try {
....
} catch (error) {
...
}
})
Why it doesn't work?
If we imitate the forEach loop it will be something like,
function forEach(arr, cb){
for(let i=0;i<arr.length;i++){
cb(arr[i], i, cb);
}
}
So you see it doesn't await the cb.

The reason why timeout wont work in a loop is because it will fire all the requests/functions all at once after the timeout delay.
The idea is to put delay in each iteration and only after delay start the next iteration.
you can run a self invoking function which calls itself after the delay. So to run the function every 2 seconds, you can try this:
let jsonParsed = JSON.parse(rawdata);
let len = jsonParsed.length;
(function requestLoop (i) {
setTimeout(function () {
let line = jsonParsed[len-i]
var encodeLink = encodeURI(line.link);
const response = await axios.get(encodeLink).catch((err) => {
var Status_ErrorsCatchaxios = {
"status Code": err.response.status ? err.response.status : "No status code available",
"Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
"Original Link": encodeLink ? encodeLink : "No Original Link Available",
"errorCode": err
}
main.push(Status_ErrorsCatchaxios)
})
try {
console.log(response.status)
statusCheck = statusCheck + 1;
console.log("Link: ", statusCheck)
} catch (error) {
console.log(error)
}
let jsonParsed = JSON.parse(rawdata);
if (--i) requestLoop(i);
}, 2000)
})(len);

You can use
var i=0;
jsonParsed.map(async(line) => {
i++
setTimeout(async() => {
},i*2000)
}

you can use setTimeout function for running codes every 2 second!
setTimeout(async() => {
// await postRequest()
},2000)

Next block of code execute earlier before the fetch how can I fixed that

It should have given me not null output but I am getting null output. It is working fine if I console inside the block of code but outside it gives me null value
Here is the code:
app.post("/downloadDb",async (req,res)=>{
var docData = [];
var idList = [];
console.log("downloadBd");
const mahafuzCol = firestore.collection("Mahafuz")
await mahafuzCol.listDocuments()
.then( listDoc=>{
//List of id fetch
listDoc.forEach(data=>{
idList.push(data.id)
});
}).catch(e=>console.log(e));
//document is fetched
await idList.forEach(id=>{
mahafuzCol.doc(id).get().then(
doc=>{
docData.push(doc.data());
//Here I get desire output w=if I log with console
}
);
});
//Here I get null output
await console.log(docData);
});

Ok, looking at your piece of code, I would like to point out a few things.
You are using the latest and greatest ES7 async and await feature which is great. Why are you stuck with the old way of defining variables? Try to use let and const instead of var. Don't mix the ECMAScript versions like this. This is considered a bad practice.
Loops in Node.js are synchronous as of now (though asynchronous loops are in the pipeline of Node and we would see them soon). You cannot put asynchronous code inside a loop and expect them to work as expected. There is a whole concept of event loop in Node.js and how Node handles asynchronous tasks. So, if you have time, you should definitely go through the event loop concepts.
Here is the better way to write the code:
app.post('/downloadDb', async (req, res) => {
// always wrap asynchronous code in async/await in try/catch blocks
console.log('downloadBd');
const mahafuzCol = firestore.collection('Mahafuz');
try {
// assuming that listDocuments returns a promise
// await on it until it gets resolved
// all the listed documents will be assigned to docs
// once the promise is resolved
const docs = await mahafuzCol.listDocuments();
const idList = docs.map(data => data.id);
// again assuming that get() returns a promise
// pushing all the promises to an array so that
// we can use Promise.all to resolve all the promises
// at once
const promisesList = idList.map(id => mahafuzCol.doc(id).get());
// fetching document is here
// once all the promises are resolved, data contains
// the result of all the promises as an array
const data = await Promise.all(promisesList);
const docData = data.map(doc => doc.data());
console.log(docData);
// return some response
return res.status(200).send();
} catch (error) {
console.log('error: ', error);
// return some response
return res.status(500).send();
}
});
PS:
If you still somehow want to use asynchronous loops, have a look at this library
https://caolan.github.io/async/docs.html#each

Since forEach is async in nature, as soon as you write a promise in a forEach() loop. The next command gets executed, which in your case is:
// Here I get null output
await console.log(docData);
You'd need the typical for loop for this task:
Try the snippet below:
let idListLen = idList.length;
for (let i =0; i< idListLen; i++) {
mahafuzCol.doc(id).get().then(
doc=>{
docData.push(doc.data());
//Here I get desire output w=if I log with console
}
);
}
console.log(docData) //prints the list of data

Promisified API calls don't execute

I'm writing a service that makes an API call to get all bookings, then from those bookings gets a list all the coworkerIds who made those bookings. I then take the unique ids and use them to make another set of calls, to get an array of all the coworker records. Once this step is done I'll do the same for line item calls, however for some reason I can't get my list of coworker objects.
Here's my code:
apiService.getBookingsList(`?size=1000&from_Booking_FromTime=${weekAgo.toISOString()}`).then(bookings => {
const bookingCoworkerIds = bookings.map(booking => booking.CoworkerId);
const bookingCoworkerIdsUnique = bookingCoworkerIds.filter(recordParser.onlyUnique);
const getCoworker = function getCoworkerInfo(coworkerId) {
return apiService.getSingleCoworker(coworkerId);
}
const bookingCoworkerCalls = bookingCoworkerIdsUnique.map(coworkerId => getCoworker(coworkerId));
const bookingCoworkers = Promise.all(bookingCoworkerCalls);
bookingCoworkers.then(coworkers => {
console.log(coworkers.length);
console.log(coworkers[0]);
});
});
And here's the code for apiService.getSingleCoworker:
getSingleCoworker(coworkerId) {
// returns a single coworker object given their ID
return httpsRequest.createRequest(this.URL.coworkerList + `?Coworker_Id=${coworkerId}`, {}, this.requestHeaders, 'GET')
.then(result => JSON.parse(result).Records[0]);
}
I can also post my https code but I don't think that's the issue.
I suspect I'm doing something wrong with my promise pattern, as I'm still new to promises and async code in general. Neither of these console logs are reached, and instead the only output is:
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): SyntaxError: Unexpected end of JSON input
So what am I doing wrong here?

The idea of working with promises is chaining "then" calls (or commands).
In order to do that, a promise must return another promise or a value.
Like this:
apiService.getBookingsList(`?size=1000&from_Booking_FromTime=${weekAgo.toISOString()}`)
.then(bookings => {
const bookingCoworkerIds = bookings.map(booking => booking.CoworkerId);
const bookingCoworkerIdsUnique = bookingCoworkerIds.filter(recordParser.onlyUnique);
const bookingPromises = bookingCoworkerIdsUnique.map(coworkerId => apiService.getSingleCoworker(coworkerId));
return Promise.all(bookingPromises);
}).then(coworkers => {
console.log(coworkers.length);
console.log(coworkers[0]);
}).catch(err => {
// do something with error
});
inside the first "then" callback I return Promise.all which generates a promise.
The next "then" receives all the resolved values by the "all" promise.
Bear in mind that promises should always contain a "catch" call at the end in order to capture promise errors.
EDIT:
getSingleCoworker(coworkerId) {
const self = this;
return new Promise(function(resolve, reject) {
httpsRequest.createRequest(self.URL.coworkerList + `?Coworker_Id=${coworkerId}`, {}, this.requestHeaders, 'GET')
.then(result => {
const jsonVal = JSON.parse(result).Records[0];
resolve(jsonVal);
}).catch(reject);
});
}
Hope this helps

Node async loop - how to make this code run in sequential order?

I know there are several posts about this, but according to those I've found, this should work correctly.
I want to make an http request in a loop and I do not want the loop to iterate until the request callback has been fired. I'm using the async library like so:
const async = require("async");
const request = require("request");
let data = [
"Larry",
"Curly",
"Moe"
];
async.forEachOf(data, (result, idx, callback) => {
console.log("Loop iterated", idx);
let fullUri = "https://jsonplaceholder.typicode.com/posts";
request({
url: fullUri
},
(err, res, body) => {
console.log("Request callback fired...");
if (err || res.statusCode !== 200) return callback(err);
console.log(result);
callback();
});
});
What I see is:
Loop iterated 0
Loop iterated 1
Loop iterated 2
Request callback fired...
Curly
Request callback fired...
Larry
Request callback fired...
Moe
What I need to see is:
Loop iterated 0
Request callback fired...
Curly
Loop iterated 1
Request callback fired...
Larry
Loop iterated 2
Request callback fired...
Moe
Also, if there's a built-in way to do the same thing (async/await? Promise?) and the async library could be removed, that'd be even better.
I've seen some examples of recursion out there that are clever, but when I put this to use in a much more complex situation (e.g. multiple request calls per-loop, etc.) I feel like that approach is hard to follow, and isn't as readable.

You can ditch async altogether and go for async/await quite easily.
Promisify your request and use async/await
Just turn request into a Promise so you can await on it.
Better yet just use request-promise-native that already wraps request using native Promises.
Serial example
From then on it's a slam dunk with async/await:
const rp = require('request-promise-native')
const users = [1, 2, 3, 4]
const results = []
for (const idUser of users) {
const result = await rp('http://foo.com/users/' + idUser)
results.push(result)
}
Parallel example
Now, the problem with the above solution is that it's slow - the requests run serially. That's not ideal most of the time.
If you don't need the result of the previous request for the next request, just go ahead and do a Promise.all to fire parallel requests.
const users = [1, 2, 3, 4]
const pendingPromises = []
for (const idUser of users) {
// Here we won't `await` on *each and every* request.
// We'll just prepare it and push it into an Array
pendingPromises.push(rp('http://foo.com/users/' + idUser))
}
// Then we `await` on a a `Promise.all` of those requests
// which will fire all the prepared promises *simultaneously*,
// and resolve when all have been completed
const results = await Promise.all(pendingPromises)
Error handling
Error handling in async/await is provided by plain-old try..catch blocks, which I've omitted for brevity.

If you have many (thousands) of urls to process it's best to define a batch size and recursively call the process function to process one batch.
It's also best to limit amount of active connections, you can use this to throttle active connections or connections within a certain time (only 5 per second).
Last but not least; if you use Promise.all you want to make sure not all successes are lost when one promise rejects. You can catch rejected requests and return a Fail type object so it'll then resolve with this Fail type.
The code would look something like this:
const async = require("async");
//lib comes from: https://github.com/amsterdamharu/lib/blob/master/src/index.js
const lib = require("lib");
const request = require("request");
const Fail = function(reason){this.reason=reason;};
const isFail = o=>(o&&o.constructor)===Fail;
const requestAsPromise = fullUri =>
new Promise(
(resolve,reject)=>
request({
url: fullUri
},
(err, res, body) => {
console.log("Request callback fired...");
if (err || res.statusCode !== 200) reject(err);
console.log("Success:",fullUri);
resolve([res,body]);
})
)
const process =
handleBatchResult =>
batchSize =>
maxFunction =>
urls =>
Promise.all(
urls.slice(0,batchSize)
.map(
url=>
maxFunction(requestAsPromise)(url)
.catch(err=>new Fail([err,url]))//catch reject and resolve with fail object
)
)
.then(handleBatch)
.catch(panic=>console.error(panic))
.then(//recursively call itself with next batch
_=>
process(handleBatchResult)(batchSize)(maxFunction)(urls.slice(batchSize))
);
const handleBatch = results =>{//this will handle results of a batch
//maybe write successes to file but certainly write failed
// you can retry later
const successes = results.filter(result=>!isFail(result));
//failed are the requests that failed
const failed = results.filter(isFail);
//To get the failed urls you can do
const failedUrls = failed.map(([error,url])=>url);
};
const per_batch_1000_max_10_active =
process (handleBatch) (1000) (lib.throttle(10));
//start the process
per_batch_1000_max_10_active(largeArrayOfUrls)
.then(
result=>console.log("Process done")
,err=>console.error("This should not happen:".err)
);
In your handleBatchResult you can store failed requests to a file to try later const [error,uri] = failedResultItem; you should give up if a high amount of requests are failing.
After handleBatchResult there is a .catch, that is your panic mode, it should not fail there so I'd advice to pipe errors to a file (linux).