We Keep Coding

Best way to make a http request and then potentially another one

Title isn't so clear but to elaborate, I need to make a HTTP request to an API endpoint, and so far I'm using a function that looks something like this:
function getPostsFromAPI(argOne, argTwo) {
const apiUrl = `https://www.exampleapi.com/v1/userposts`
apiGet(`${apiUrl}?name=argOne&something=argTwo`).then(userPosts => {
// do stuff with userPosts
return userPostData
}).catch(handleError)
}
However, the API response can include the following:
{
//...
"has_more": true,
"next_offset": 10
}
In which case, I'd need to send the API call a second time, this time with the &offset=10 argument.
The promise would need to continue making API calls until has_more: true is no longer present. My initial thought would be to just re-run getPostsFromAPI() based on an if statement from inside itself, but I can't figure out how to make that work cleanly inside a promise. Ultimately, the promise should keep making requests until the API says that it's ran out of data to give (I'll implement my own limit).
What would be the best way to achieve this?

The algorithm to achieve this is much more obvious if you use async/await. You can just create an empty array, and gradually append to it in a loop until the server indicates there are no more results.
async function getPostsFromAPI(argOne, argTwo) {
const apiUrl = `https://www.exampleapi.com/v1/userposts`
let results = [];
let offset = 0;
while (true) {
let response = await apiGet(`${apiUrl}?name=argOne&something=argTwo&offset=${offset}`);
results = results.concat(response.records);
if (response.has_more) {
offset = response.next_offset;
} else {
return results;
}
}
}
If you can't use async/await and have to stick to promises, you can use recursion to have a method invoke itself each time a response indicates there are more records:
function getPostsFromAPI(argOne, argTwo) {
return new Promise((resolve, reject) => {
const apiUrl = `https://www.exampleapi.com/v1/userposts`;
let results = [];
let offset = 0;
const getNextPage = (offset = 0) => {
apiGet(`${apiUrl}?name=argOne&something=argTwo&offset=${offset}`).then((response) => {
results = results.concat(response.records);
if (response.has_more) {
getNextPage(response.next_offset);
} else {
resolve(results);
}
}).catch(reject);
}
getNextPage(0);
});
}
Note that as a matter of general good practice you should never construct a query string through concatenation or template strings. You should use URLSearchParams.toString() to ensure your query string is properly encoded. You can do so indirectly by creating a new URL:
const url = new URL(`https://www.exampleapi.com/v1/userposts`)
url.searchParams.append("argOne", argOne);
url.searchParams.append("argTwo", argTwo);
url.searchParams.append("offset", offset);
url.toString()

This is a great use case for an async generator.
Would look something like the following
async function* getPostsFromAPI(arg1, arg2) {
const apiUrl = `https://www.exampleapi.com/v1/userposts`
let response = { next_offset: 0 };
do {
response = await apiGet(`${apiUrl}?name=${arg1}&something=${arg2}&offset=${response.next_offset}`)
response.items.forEach((item) => {
yield item
})
} while (response.has_more)
}

Synchronize API calls in a for loops VueJS

I am trying to get synchronized flow inside a for loop which each loop having to call a API.
The flow is summarized as follows:
A Demo1() function containing forloop executes one by one, however in each loop it does a API call.,
A MainFunction() retrieves after the for loop has executed and does the final API call.
Supporting API calls for the same function is shown in code and seems quite self explanatory
The code structure is as follows:
</script>
...
async MainFunction() {
if (dataPresent) {
let clientFieldDetails = await this.Demo1()
.then(() => {
//This does the final API call based on the results fetched from for loop
this.processFinalAPICall();
})
.catch(err => {
console.log("Something went wrong... ", err);
});
} else {
console.log(
"No data.."
);
}
},
async Demo1() {
//Traversing around each fieldInfo
//this.dataPresent.items.forEach(item => {
//Replacing with normal for loop
for(item of this.dataPresent.items){
if (item.model) {
//Upload item model API Call
this.uploadItem(item.model)
.then(response => {
//PUT API Call
this.putItemModel().then(response => {
var result = response.data;
//Add itemModel fetched from API response
models.push(result);
console.log(result)
});
})
.catch(err => console.log("Axios err: ", err));
} else {
//Inside item price details
console.log("inside item price");
//API call for fetching price info
this.getitemPriceInfo(item.id);
}
});
},
getitemPriceInfo(itemid){
if(itemid){
//API call for fetching price info
this.getPriceinEuro(itemid);
itemsPrice.push(result)
}
else{
this.getPriceWorldWide().then(response => {
if(response.data === "item not created")
//Create new item API call
this.createNewItem(item.id)
else{
var result = response.data;
//Fetched API response for itemsPrice
itemsPrice.push(result);
console.log(result);
//Update item API call
this.updatePriceItem(item.id);
}
});
}
},
//EDIT: Adding API call
async getPriceinEuro(itemId) {
await this.$axios
.get("/auth/getEuroPrice", {
params: {
itemID: itemId
}
})
.then(response => {
console.log(" Resp :" + response.data);
let result = response.data;
//This gives me the price
itemsPrice.push(result)
});
},
processFinalAPICall(){
//get itemsPrice and models price pushed
var apiBodyModel = [];
this.models.forEach(model=>{
var uri = {
url: /modelInfo/+model
}
apiBodyModel.push(uri)
})
var apiBodyPrice = [];
this.itemsPrice.forEach(price=>{
var uri = {
url: /priceInfo/+price
}
apiBodyPrice.push(uri)
})
//Create a post body request from above data and POST
....
}
...
</script>
The code currently loops in for loop and doesnt wait for the API calls to finish. It executes processFinalCall() first and then the API calls. I am not sure about async/await, if I used it wrong please excuse. How do i get the the forLoop executed first and then the initiating the processFinalAPICall() from MainFunction?
Please let me know if I am doing it the right way. Thanks :)
I am using Node version 8.11 due to our project dependencies.
EDITED: Added API function Call for reference

I'm fairly sure that the problem you have lies in your Demo1 function, which, very broadly, looks like this:
async Demo1() {
[ARRAY OF ITEMS].forEach(item => {
this.uploadItem(item)
.then(response => {
// do some stuff
});
.catch(err => /* log errors */);
});
}
You don't await the upload here, so when you call Demo1() in MainFunction(), it'll go through the uploads without waiting for the previous one to finish first. I think the easiest way to get around this would be to use a for-of loop instead, since you pass a function to .forEach, and that just complicates things.
So instead of [ARRAY OF ITEMS].forEach(item => { ..., you can do this:
async Demo1() {
for(let item of [ARRAY OF ITEMS]) {
await this.uploadItem(item);
}
}
And your modified Demo1 function would look like this:
async Demo1() {
//Traversing around each fieldInfo
for (let item of this.dataPresent.items) {
if (item.model) {
//Upload item model API Call
await this.uploadItem(item.model)
//PUT API Call
let response = await this.putItemModel();
var result = response.data;
//Add itemModel fetched from API response
models.push(result);
console.log(result)
} else {
//Inside item price details
console.log("inside item price");
//API call for fetching price info
this.getitemPriceInfo(item.id);
}
};
}
Note that I haven't tested this, because I don't have the full code to plug it into, so I can't fully guarantee that it'll work. If it doesn't, let me know, so I can help fix it
Here's a bad mockup of this solution.

I guess its a simple change over from what #marsnebulasoup added in his mockup solution. From your comments I tried to replace upload method with axios call. You are missing return keyword after the function call , i.e,
upload(item) {
console.log("Iten value:"+ item);
return axios.get("https://jsonplaceholder.typicode.com/posts/" +item)
.then(response => {
console.log("Done processing API call (Times:"+ item)
console.log(response.data);
})
.catch(console.log);
A simple code changeover is designed based on previous answer by #marsnebulasoup in Replit:
EDIT: Added correct reference:
Code Reference: https://replit.com/join/comkahdm-gagangowda89
Cheers.

Loading data from Firebase asynchronously

I am developing a business manager web app in React with Firebase back-end. I am also coding a local API to simplify Firebase functions. I created this method, which loads data from a Firebase collection and returns an array of documents.
getDocuments(collection) {
var documents = [];
firebase.firestore().collection(collection).get().then(snapshot => {
snapshot.forEach(doc => {
documents.push(doc);
});
}).then(() => {
return documents;
})
}
However, when I call the method and assign it to a variable which I later print to the console, it says undefined.
var employees = getDocuments("employees");
console.log(employees);
What I want to do is to use .then() after calling the method to print the already loaded data to the console. Something like this:
var employees = getDocuments("employees").then(response => {
console.log(response);
})
Any help would be appreciated. Thank you.

Your getDocuments function seems to be unnecessarily complicated. This:
getDocuments(collection) {
return firebase.firestore().collection(collection).get().then(snapshot=>snapshot.docs)
}
yields exactly the same intended result (an Array of docs wrapped in a promise) as your function but performs faster since it skips looping through all the documents in the snapshot https://firebase.google.com/docs/reference/js/firebase.firestore.QuerySnapshot#docs
Afterwards just extract the value from the Promise returned by this function in your preferred way:
Option 1 (async/await)
let employees= await getDocuments('employees')
console.log(employees)
Option 2 (chaining)
let employees =[]
getDocuments('employees').then(response => {
employees=response
console.log(employees)
})
Explanation
When you are doing this:
var employees = getDocuments("employees").then(response => {
console.log(response);
})
you aren't receiving any value from getDocuments since you didn't return anything in the first place.
getDocuments(collection) {
var documents = [];
firebase.firestore().collection(collection).get().then(snapshot => {
snapshot.forEach(doc => {
documents.push(doc);
});
}).then(() => {
return documents; <-- this is returning the value of documents to the parent scope which is 'getDocuments', since the 'then' is related to the 'get' function
})
}

You should assign employees in the then like this
var employees = [];
getDocuments("employees").then(response => {
employees = response;
console.log(response);
})
Or if you are in an asynchronous function, you could go for something like this
var employees = await getDocuments("employees");
console.log(employees);
But the await keyword has to be done in an async function

Run HTTP requests in chunks

I want to run 1 thundered http requests in configurable chunks, and set configurable timeout between chunk requests. The request is based on the data provided with some.csv file.
It doesn't work because I am getting a TypeError, but when I remove () after f, it doesn't work either.
I would be very grateful for a little help. Probably the biggest problem is that I don't really understand how exactly promises work, but I tried multiple solutions and I wasn't able to achieve what I want.
The timeout feature will probably give me even more headache so I would appreciate any tips for this too.
Can you please help me to understand why it doesn't work?
Here is the snippet:
const rp = require('request-promise');
const fs = require('fs');
const { chunk } = require('lodash');
const BATCH_SIZE = 2;
const QUERY_PARAMS = ['clientId', 'time', 'changeTime', 'newValue'];
async function update(id, time, query) {
const options = {
method: 'POST',
uri: `https://requesturl/${id}?query=${query}`,
body: {
"prop": {
"time": time
}
},
headers: {
"Content-Type": "application/json"
},
json: true
}
return async () => { return await rp(options) };
}
async function batchRequestRunner(data) {
const promises = [];
for (row of data) {
row = row.split(',');
promises.push(update(row[0], row[1], QUERY_PARAMS.join(',')));
}
const batches = chunk(promises, BATCH_SIZE);
for (let batch of batches) {
try {
Promise.all(
batch.map(async f => { return await f();})
).then((resp) => console.log(resp));
} catch (e) {
console.log(e);
}
}
}
async function main() {
const input = fs.readFileSync('./input.test.csv').toString().split("\n");
const requestData = input.slice(1);
await batchRequestRunner(requestData);
}
main();
Clarification for the first comment:
I have a csv file which looks like below:
clientId,startTime
123,13:40:00
321,13:50:00
the file size is ~100k rows
the file contains information how to update time for a particular clientId in the database. I don't have an access to the database but I have access to an API which allows to update entries in the database.
I cannot make 100k calls at once, because: my network is limited (I work remotely because of coronavirus), it comsumpts a lot of memory, and API can also be limited and can crash if I will make all the requests at once.
What I want to achieve:
Load csv into memory, convert it to an Array
Handle api requests in chunks, for example take first two rows from the array, make API call based on the first two rows, wait 1000ms, take another two rows, and continue processing until the end of array (csv file)

Well, it seems like this is a somewhat classic case of where you want to process an array of values with some asynchronous operation and to avoid consuming too many resources or overwhelming the target server, you want to have no more than N requests in-flight at the same time. This is a common problem for which there are pre-built solutions for. My goto solution is a small piece of code called mapConcurrent(). It's analagous to array.map(), but it assumes a promise-returning asynchronous callback and you pass it the max number of items that should ever be in-flight at the same time. It then returns to you a promise that resolves to an array of results.
Here's mapConcurrent():
// takes an array of items and a function that returns a promise
// returns a promise that resolves to an array of results
function mapConcurrent(items, maxConcurrent, fn) {
let index = 0;
let inFlightCntr = 0;
let doneCntr = 0;
let results = new Array(items.length);
let stop = false;
return new Promise(function(resolve, reject) {
function runNext() {
let i = index;
++inFlightCntr;
fn(items[index], index++).then(function(val) {
++doneCntr;
--inFlightCntr;
results[i] = val;
run();
}, function(err) {
// set flag so we don't launch any more requests
stop = true;
reject(err);
});
}
function run() {
// launch as many as we're allowed to
while (!stop && inflightCntr < maxConcurrent && index < items.length) {
runNext();
}
// if all are done, then resolve parent promise with results
if (doneCntr === items.length) {
resolve(results);
}
}
run();
});
}
Your code can then be structured to use it like this:
function update(id, time, query) {
const options = {
method: 'POST',
uri: `https://requesturl/${id}?query=${query}`,
body: {
"prop": {
"time": time
}
},
headers: {
"Content-Type": "application/json"
},
json: true
}
return rp(options);
}
function processRow(row) {
let rowData = row.split(",");
return update(rowData[0], rowData[1], rowData[2]);
}
function main() {
const input = fs.readFileSync('./input.test.csv').toString().split("\n");
const requestData = input.slice(1);
// process this entire array with up to 5 requests "in-flight" at the same time
mapConcurrent(requestData, 5, processRow).then(results => {
console.log(results);
}).catch(err => {
console.log(err);
});
}
You can obviously adjust the number of concurrent requests to whatever number you want. I set it to 5 here in this example.

Async Promise returns undefined or zone aware promise

When calling a function that returns a promise, comes back as undefined unless async operators are removed, then returns ZoneAwarePromise, but contains no data.
I know the query returns data when the function executes, it however does not seem to pass that data to the actual return part of the function call.
I have looked at several Stack questions that have not answered this question including this question:
Async/Await with Request-Promise returns Undefined
This is using a REST endpoint to pull data, the console.logs do show the data is correct, however return comes back as undefined
this.allPeople.forEach(async person => {
const dodString = await this.getRelatedRecords(person); //undefined
}
This is the main function that returns a promise / data
async getRelatedRecords(person) {
// function truncated for clarity
// ...
//
console.warn('This async should fire first');
selPeopleTable.relationships.forEach(relationship => {
allRelationshipQueries.push(
arcgisService.getRelatedTableData(
selPeopleTable.url, [person[oidField.name]], relationship.id, relationship.name),
);
});
await Promise.all(allRelationshipQueries).then(allResults => {
console.log('Inside the Promise');
// The Specific node I am looking for
const data = allResults[1].results.relatedRecordGroups[0].relatedRecords[0].attributes.dod;
console.log(data); // Shows correctly as the data I am looking for
return data;
}).catch(function(data){
console.log('there might be data missing', data);
});
}
Removing the ASYNC operators cause the getRelatedRecords() to fire after the containing function and / or return a 'ZoneAwarePromise' which contains no data. I need getRelatedRecords() to fire first, then to run the rest of the code.
I can provide more snippets if need be.
Zone Aware Promise
When the Async operators are (I think) setup correctly

You need to return this as well:
await Promise.all(allRelationshipQueries).then(allResults => {
console.log('Inside the Promise');
// The Specific node I am looking for
const data = allResults[1].results.relatedRecordGroups[0].relatedRecords[0].attributes.dod;
console.log(data); // Shows correctly as the data I am looking for
return data;
})
return in the above block is returning but all of this is in the scope of the arrow function which is then(allResults => { so you also need to return this function like this:
return await Promise.all(allRelationshipQueries).then(allResults => {
Approach #2:
Second way would be to store that into variable like this:
let dataToReturn = await Promise.all(allRelationshipQueries).then(allResults => {
console.log('Inside the Promise');
// The Specific node I am looking for
const data = allResults[1].results.relatedRecordGroups[0].relatedRecords[0].attributes.dod;
console.log(data); // Shows correctly as the data I am looking for
return data;
}).catch(function(data){
console.log('there might be data missing', data);
});
return dataToReturn;