We Keep Coding

Using Bottleneck to rate-limit API requests in a library

I am writing an API wrapper in TypeScript. I would like the code to be asynchronous in order to maximally meet the rate limit of the API in question. The API wants requests to be submitted at a maximum rate of 1/second.
I intend to implement an API wrapper which is instantiated once, and allows the use of objects to reach the different endpoints. For instance, within the greater API there is a post and pool endpoint. I would like to access them like post_object.post.submit_request(argument1, ...) or post_object.pool.submit_request(argument1, ...).
I have created an object called state_info which is passed between the various objects, within which is contained a user-agent header, login information if provided, and a rate-limiter object from the Bottleneck library.
The issue I'm running into while testing is that my program doesn't seem to actually be limiting the rate of requests; no matter what I change the limit to in the arguments for Bottleneck, the requests all happen in about .600 seconds every time.
I am thinking this has something to do with passing around the rate-limiter object, or in accessing it from multiple places, but I'm unsure.
First, here is the code for the Model object, which represents access into the API.
import axios, { AxiosRequestConfig } from "axios";
import { StateInfo, Method } from "./interfaces";
export class Model {
public stateInfo: StateInfo;
constructor(stateInfo: StateInfo) {
// Preserve rate limiter, user agent, etc.
this.stateInfo = stateInfo;
}
//Updated to funcName = () => {} syntax to bind "this" to this class context.
private submit_request = (query_url: string, method: Method) => {
if (this.stateInfo.username && this.stateInfo.api_key) {
const axiosConfig: AxiosRequestConfig = {
method: method,
url: query_url,
headers: { "User-Agent": this.stateInfo.userAgent },
auth: {
username: this.stateInfo.username,
password: this.stateInfo.api_key,
},
};
return axios(axiosConfig);
} else {
const axiosConfig: AxiosRequestConfig = {
method: "get",
url: query_url,
headers: { "User-Agent": this.stateInfo.userAgent },
};
return axios(axiosConfig);
}
};
public submit_throttled_request = (url: string, method: Method) => {
return this.stateInfo.rateLimiter.schedule(
this.submit_request,
url,
method
);
};
}
Then, the code from which I call this class:
import { Model } from "./models/model";
import Bottleneck from "bottleneck";
const limiter: Bottleneck = new Bottleneck({ mintime: 1000, maxconcurrent: 1 });
const stateInfo = {
rateLimiter: limiter,
userAgent: "email#website.com | API Dev",
};
let modelObj: Model = new Model(stateInfo);
async function makeRequest() {
try {
let response = await modelObj.submit_throttled_request(
"https://www.website.com/api",
"get"
);
console.log(response.data.id + "|" + Date.now());
} catch (err) {
console.log(err);
}
}
let start = new Date();
for (let i = 0; i < 20; i++) {
makeRequest();
}
My expectation is that the operation would take, at a minimum, 10 seconds if only one request can be submitted per second. Yet I'm averaging half that, no matter what I include for mintime.

I've learned the answer to my own question after much head-scratching.
It turns out, in the "gotchas" section of the bottleneck API reference they note:
If you're passing an object's method as a job, you'll probably need to bind() the object:
with the following code:
// instead of this:
limiter.schedule(object.doSomething);
// do this:
limiter.schedule(object.doSomething.bind(object));
// or, wrap it in an arrow function instead:
limiter.schedule(() => object.doSomething());
This is the issue into which I was running. I was handing off axios(axiosContext) without binding the scope, so nothing was being sent off to the bottleneck ratelimiter. By wrapping is like so: this.state_info.rateLimiter.schedule(() => axios(axiosContext)); I have managed to correctly bind the context as needed.

Loading images in vue.js from an API

Consider this:
An API loads a manifest of image metadata. The images have an ID, and with another API call returns a base64 image from the DB. The model for the manifest is attachmentRecord and the ID is simply a field.
I would rather not preload these large strings into an array (that would work).
so I have this (which lazy loads on any manifest change):
<div v-for="(attachment, index) in attachmentRecord" :key="index">
<img :src="fetchImage(attachment.id)" />
</div>
fetchimage() is a wrapper for an axios function which returns back from a promise. (writing this from memory):
this.axios({
method: "get",
url: url,
}).then(res => res.data)
.catch(() => {
alert("Unable to load raw attachment from this task and ID");
});
}
Now, the network calls go thru fine, the ID passes in correctly, I can see the base 64data, but they don't seem to make it to wrapper function or the src attribute. It always comes up blank. I tried wrapping it in another promise,only to get a promise back to the src attribute. What would be a best practice for this situation in Vue?
Ok, so far I made these changes with Constantin's help:
I tried to strip it down without a helper function:
Vue template Code:
<div v-for="(attachment, index) in attachmentRecord" :key="index">
<img :src="getAttachmentFromTask(attachment.id)" />
base method:
async getAttachmentFromTask(attachmentID) {
if (!attachmentID) alert("Unknown Attachment!");
let sendBack = "";
let url = "/server/..."
await this.axios({
method: "get",
url: url
})
.then(res => {
sendBack = res.data;
})
.catch(() => {
alert("Unable to load raw attachment from this task and ID");
});
// >>>>>>>>>alerts base64 correctly; Vue loads [object Promise] in img
alert(sendBack);
return sendBack;
}

It turns out that Vue doesn't handle async / await as well as I thought. Therefore, you have to save the image data to each attachment in attachmentRecord. This getAttachmentFromTask method now handles this when accessed the first time and populates a data property for the corresponding attachment object. On successive calls, that property is returned if it is already populated. Note the usage of Vue.set() because the property is not available in the initial data, but we want it to be reactive. You can even set up a fallback image like a loader, see the shortly flickering SO logo without text before the larger logo appears:
new Vue({
el: '#app',
data: {
attachmentRecord: [{
id: 1
}]
},
methods: {
getAttachmentFromTask(attachmentIndex, attachmentID) {
let record = this.attachmentRecord[attachmentIndex];
if (!record.data) {
Vue.set(record, 'data', null);
axios.get('https://kunden.48design.de/stackoverflow/image-base64-api-mockup.json').then((result) => {
Vue.set(record, 'data', result.data);
});
}
return this.attachmentRecord[attachmentIndex].data;
}
}
});
img {
max-width: 100vw;
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.19.0/axios.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.6.10/vue.min.js"></script>
<div id="app">
<div v-for="(attachment, index) in attachmentRecord" :key="index">
<img :src="getAttachmentFromTask(index, attachment.id) || 'https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon.png'" />
</div>
</div>
old answer: (Unfortunately doesn't work that way with Vue currently)
Axios requests are asynchronous by default. So the function doesn't wait for then() to return the value. You could add the async keyword before your fetchImage function name and add the await keyword before this.axios. Then make the then callback assign the return value to a variable in the fetchImage function scope and have the function return it.
async fetchImage() {
let returnValue;
await this.axios({
method: "get",
url: url,
}).then(res => { returnValue = res.data; })
.catch(() => {
alert("Unable to load raw attachment from this task and ID");
});
return returnValue;
}

Apify web scraper ignoring URL Fragment

I have a list of URL that I want to scrape, so i put it into the startUrls like this
"startUrls": [
{
"url": "https://www.example.com/sample#000000",
"method": "GET"
},
{
"url": "https://www.example.com/sample#111111",
"method": "GET"
}
]
And this is the excerpt from my pageFunction code.
async function pageFunction(context) {
const { request } = context;
var name;
try {
name = document.querySelector('h1').textContent;
} catch (e) {
name = "null";
}
return {
link: request.url,
name
};
}
It's working fine with URLs that can be differentiated with either the domain, or the path. But if the only difference is in the fragment, only the first URL is processed as the second URL is considered a duplicate and therefore skipped.
i've tried adding this bit of code at the second line of the pageFunction
await context.enqueueRequest({
url: context.request.url,
keepUrlFragment: true,
});
But it leads up to another problem that it's producing duplicate results for each URL.
So what should I do to make this work correctly? Is there another way than calling enqueueRequest to set the keepUrlFragment to true ?

Unfortunately, you cannot set keepUrlFragment directly in startUrls now. So I propose to not use them at all. You can instead pass them as an array in customData. Then you can use page function like this with a dummy startUrl like http://example.com and label START
async function pageFunction(context) {
const { request, customData } = context;
if (request.userData.label === 'START') {
for (const url of customData) {
await context.enqueueRequest({
url,
keepUrlFragment: true,
});
}
} else {
// Your main scraping logic here
}
}

Nesting API calls and mapping to same returned object ES6 / ReactJs

I'm just starting out with React and ES6 so apologies if this is a bit of a simple one.
I'm currently playing round with the FoursquareAPI. I'm using ES6 fetch to return a series of objects (they're actually different venues in different parts of the world) which are then mapped and returned and stored in the application's state. This works fine and returns what I want:
// method to call api
getVenues: (searchTerm) => {
const fetchVenuesURL = `${urlExplore}${searchTerm}&limit=10&client_id=${clientId}&client_secret=${clientSecret}&v=20180602`;
return fetch(fetchVenuesURL).then( response => {
return response.json();
}).then( jsonResponse => {
if (jsonResponse.response.groups[0].items) {
return jsonResponse.response.groups[0].items.map(item => (
// populate venues
{
id: item.venue.id,
name: item.venue.name,
address : item.venue.location.address,
city : item.venue.location.city,
country : item.venue.location.country,
icon : item.venue.categories[0].icon
}
));
} else {
return [];
}
});
// method in App.js to setState
search(term){
Foursquare.getVenues(term).then(foursquareResponse => {
this.setState({venues: foursquareResponse});
});
}
The problem arises when I need to fetch photographs associated with each of the 'venues' returned by the original fetch. These come from a different endpoint. I'm not sure what the best approach is.
One way would be to have two separate api calling methods and then somehow populate an empty photos field of the first with the photos from the second back in App.js but that seems clunky.
My instinct is to somehow nest the Api calls but I'm uncertain about how to go about this. I'm hoping to do something along the lines of somehow applying a method to each iteration of the first mapped object, something along the lines of but not sure how to link them together so that the second goes into the photo property of the first:
{
id: item.venue.id,
name: item.venue.name,
address : item.venue.location.address,
city : item.venue.location.city,
country : item.venue.location.country,
icon : item.venue.categories[0].icon
photos : []
}
const fetchPhotosURL = `${urlPhotos}${venueId}/photos?limit=10&client_id=${clientId}&client_secret=${clientSecret}&v=20180602`;
return fetch(fetchPhotosURL).then( response => {
return response.json();
}).then( jsonResponse => {
if (jsonResponse.response.photos.items) {
console.log(jsonResponse.response.photos.items[0].venue)
return jsonResponse.response.photos.items.map(item => (
{
id : item.id,
created: item.createdAt,
prefix: item.prefix,
suffix: item.suffix,
width: item.width,
height: item.height,
venue: item.venue
}
));
} else {
return [];
}
})
Can anyone point me in the right direction with this. I'm guessing that it's one of those things that isn't that hard once you've done it once but I'm finding it difficult.
Thanks in advance.

nodejs recursively call same api and write to excel file sequentially

I need to call an API recursively using request promise after getting result from API need to write in an excel file , API sample response given below
{
"totalRecords": 9524,
"size": 20,
"currentPage": 1,
"totalPages": 477,
"result": [{
"name": "john doe",
"dob": "1999-11-11"
},
{
"name": "john1 doe1",
"dob": "1989-12-12"
}
]
}
Now I want to call this API n times, here n is equal to totalPages, after calling each API I want to write response result to the excel files.
First write page 1 response result to excel then append page 2 response result to excel file and so on..
I have written some sample code given below
function callAPI(pageNo) {
var options = {
url: "http://example.com/getData?pageNo="+pageNo,
method: 'GET',
headers: {
'Content-Type': 'application/json'
},
json: true
}
return request(options)
}
callAPI(1).then(function (res) {
// Write res.result to excel file
}).catch(function (err) {
// Handle error here
})
But facing problem calling recursively API and maintaining sequentially like write page 1 result first to excel file then page 2 result append to excel and so on..
Any code sample how to achieve in nodejs

You want to do something like this:
function getAllPages() {
function getNextPage(pageNo) {
return callAPI(pageNo).then(response => {
let needNextPage = true;
if (pageNo === 1) {
// write to file
} else {
// append to file
}
if (needNextPage) {
return getNextPage(pageNo+1);
} else {
return undefined;
}
});
}
return getNextPage(1);
}
Obviously change that 'needNextPage' to false to stop the recursion when you're done

So you want to do 477 requests in sequence? How long do you wanna wait for this to finish? Even in paralell, this would be still too long for me.
Best: write an API that can return you a batch of pages at once. Reducing the number of requests to the backend. Maybe something like http://example.com/getData?pages=1-100 and let it return an Array; maybe like
[
{
"totalRecords": 9524,
"currentPage": 1,
"totalPages": 477,
"result": [...]
},
{
"totalRecords": 9524,
"currentPage": 2,
"totalPages": 477,
"result": [...]
},
...
]
or more compact
{
"totalRecords": 9524,
"totalPages": 477,
"pages": [
{
"currentPage": 1,
"result": [...]
},
{
"currentPage": 2,
"result": [...]
},
...
]
}
Sidenote: writing the size of the results array into the json is unnecessary. This value can easily be determined from data.result.length
But back to your question
Imo. all you want to run in sequence is adding the pages to the sheet. The requests can be done in paralell. That already saves you a lot of overall runtime for the whole task.
callApi(1).then(firstPage => {
let {currentPage, totalPages} = firstPage;
//`previous` ensures that the Promises resolve in sequence,
//even if some later request finish sooner that earlier ones.
let previous = Promise.resolve(firstPage).then(writePageToExcel);
while(++currentPage <= totalPages){
//make the next request in paralell
let p = callApi(currentPage);
//execute `writePageToExcel` in sequence
//as soon as all previous ones have finished
previous = previous.then(() => p.then(writePageToExcel));
}
return previous;
})
.then(() => console.log("work done"));
or you wait for all pages to be loaded, before you write them to excel
callApi(1).then(firstPage => {
let {currentPage, totalPages} = firstPage;
let promises = [firstPage];
while(++currentPage < totalPages)
promises.push(callApi(currentPage));
//wait for all requests to finish
return Promise.all(promises);
})
//write all pages to excel
.then(writePagesToExcel)
.then(() => console.log("work done"));
or you could batch the requests
callApi(1).then(firstPage => {
const batchSize = 16;
let {currentPage, totalPages} = firstPage;
return Promise.resolve([ firstPage ])
.then(writePagesToExcel)
.then(function nextBatch(){
if(currentPage > totalPages) return;
//load a batch of pages in paralell
let batch = [];
for(let i=0; i<batchSize && ++currentPage <= totalPages; ++i){
batch[i] = callApi(currentPage);
}
//when the batch is done ...
return Promise.all(batch)
//... write it to the excel sheet ...
.then(writePagesToExcel)
//... and process the next batch
.then(nextBatch);
});
})
.then(() => console.log("work done"));
But don't forget to add the error handling. Since I'm not sure how you'd want to handle errors with the approaches I've posted, I didn't include the error-handling here.
Edit:
can u pls modify batch requests, getting some error, where you are assigning toalPages it's not right why the totalPages should equal to firstPage
let {currentPage, totalPages} = firstPage;
//is just a shorthand for
let currentPage = firstPage.currentPage, totalPages = firstPage.totalPages;
//what JS version are you targeting?
This first request, callApi(1).then(firstPage => ...) is primarily to determine currentIndex and totalLength, as you provide these properties in the returned JSON. Now that I know these two, I can initiate as many requests in paralell, as I'd want to. And I don't have to wait for any one of them to finish to determine at what index I am, and wether there are more pages to load.
and why you are writing return Promise.resolve([ firstPage ])
To save me some trouble and checking, as I don't know anything about how you'd implement writePagesToExcel.
I return Promise.resolve(...) so I can do .then(writePagesToExcel). This solves me two problems:
I don't have to care wether writePagesToExcel returns sync or a promise and I can always follow up with another .then(...)
I don't need to care wether writePagesToExcel may throw. In case of any Error, it all ends up in the Promise chain, and can be taken care of there.
So ultimately I safe myself a few checks, by simply wrapping firstPage back up in a Promise and continue with .then(...). Considering the amounts of data you're processing here, imo. this ain't too much of an overhead to get rid of some potential pitfalls.
why you are passing array like in resolve
To stay consistent in each example. In this example, I named the function that processes the data writePagesToExcel (plural) wich should indicate that it deals with multiple pages (an array of them); I thought that this would be clear in that context.
Since I still need this seperate call at the beginning to get firstPage, and I didn't want to complicate the logic in nextBatch just to concat this first page with the first batch, I treat [firstPage] as a seperate "batch", write it to excel and continue with nextBatch

function callAPI(pageNo) {
var options = {
url: "http://example.com/getData?pageNo="+pageNo,
method: 'GET',
headers: {
'Content-Type': 'application/json'
},
json: true
}
return request(options)
}
function writeToExcel(res){console.log(res)} //returns promise.
callAPI(1).then(function (res) {
if(res){
writeToExcel(res).then(() => {
var emptyPromise = new Promise(res => setTimeout(res, 0));
while(res && res.currentPage < res.totalPages){
emptyPromise = emptyPromise.then(() => {
return callAPI(res.currentPage).then(function (res){
if(res){
writeToExcel(res)
}
});
}
}
return emptyPromise;
});
}
}).catch(function (err) {
// Handle error here
})