I'm fetching data from an API that has a limit of 50 records per call. There are 10,000 records in total that I need to retrieve so looking at using pagination via the offset parameter to achieve this.
I'm not too familiar with the syntax I need to use but I would like to keep looping through the fetch until the number of records returned for a single call is < 50 to signal the final page.
I need my JSON responses from each call to be appended together, in addition to a final 'count' that gives me 10,000
let offset = 0
fetch(`https://api.opensea.io/api/v1/assets?collection=womenandweapons&format=json&offset=${offset}&limit=50&order_direction=desc`,
{
method: 'GET',
headers: {
'Accept': 'application/json'
}
})
.then(response => response.json())
.then(data => {
let len = Object.keys(data.assets).length
console.log(len)
console.log(data.assets)
})
offset += 50
I've had some luck doing something like this:
const main = async () => {
const url = new URL(AIRTABLE_URL);
let data = await fetcher(url, AIRTABLE_API_KEY);
storiesTotal = mapData(data.records);
while (data.offset) {
data = await fetcher(url, AIRTABLE_API_KEY, data.offset);
storiesTotal = [...storiesTotal, ...mapData(data.records)];
}
}
main();
Related
Context
I'm retrieving data from the ESPN API to fetch weekly NFL matchup data. So, I'm making 18 api calls each time I need to fetch this data to account for all 18 weeks in the NFL season. I'm then creating an array with the data I need from the responses to those calls and writing out 18 files that align with each week in the NFL season (week1.json, week2.json, etc.).
Problem
The problem is that when I call my endpoint, I am seeing 2 things intermittently, and not necessarily at the same time:
(1) Some of the json files(week1.json, week2.json, etc.) include only a portion of the expected array. So, instead of 16 objects in the array, I may see only 4, or only 6, etc. Why would I only see a portion of the response data written to the array that's ultimately written to the .json files?
(2) Not all files are written to each time the endpoint is called. So, I may see that only week1-week5's .json files are written. Why aren't all of them updated?
Problem Code
// iterate 18 times
for (let i = 0; i < 18; i++) {
let weekNumber;
weekNumber = i + 1;
const week = fs.readFileSync(`./pickem/week${weekNumber}.json`, 'utf8');
const weekJson = JSON.parse(week);
// empty weekJson.games array
weekJson.games = []
// get all items
axios.get(`https://sports.core.api.espn.com/v2/sports/football/leagues/nfl/seasons/2022/types/2/weeks/${weekNumber}/events?lang=en®ion=us`)
.then(response => {
const schedule = [];
// get all items from response
const items = response.data.items
// console.log(response.data.items)
items.forEach(item => {
// make get call to $ref
axios.get(item.$ref)
.then(response => {
// get name
const name = response.data.name
// get date
const date = response.data.date
// get event id
const eventid = response.data.id
// get team ids
let team1 = response.data.competitions[0].competitors[0].id
let team2 = response.data.competitions[0].competitors[1].id
// create new object
const newObject = {
name: name,
date: date,
eventid: eventid,
team1: team1,
team2: team2
}
// add games for week
weekJson.games.push(newObject);
fs.writeFileSync(`./pickem/week${weekNumber}.json`, JSON.stringify(weekJson));
})
.catch(error => {
console.log(error)
})
})
}).catch(error => {
console.log(error)
})
}
Updated Code
router.get('/getschedules', (req, res) => {
async function writeGames() {
// iterate 18 times
for (let i = 0; i < 18; i++) {
let weekNumber;
weekNumber = i + 1;
const week = fs.readFileSync(`./pickem/week${weekNumber}.json`, 'utf8');
const weekJson = JSON.parse(week);
// empty weekJson.games array
weekJson.games = []
// get all items
// Add await keyword to wait for a week to be processed before going to the next one
await axios.get(`https://sports.core.api.espn.com/v2/sports/football/leagues/nfl/seasons/2022/types/2/weeks/${weekNumber}/events?lang=en®ion=us`)
.then(async (response) => { // add async to be able to use await
const schedule = [];
// get all items from response
const items = response.data.items
console.log(response.data.items)
// Use standard loop to be able to benefit from async/await
for (let item of items) {
// make get call to $ref
// wait for an item to be processed before going to the next one
await axios.get(item.$ref)
.then(response => {
// get name
const name = response.data.name
// get date
const date = response.data.date
// get event id
const eventid = response.data.id
// get team ids
let team1 = response.data.competitions[0].competitors[0].id
let team2 = response.data.competitions[0].competitors[1].id
// create new object
const newObject = {
name: name,
date: date,
eventid: eventid,
team1: team1,
team2: team2
}
// add games for week
weekJson.games.push(newObject);
})
.catch(error => {
console.log(error)
})
}
// moved out of the for loop since you only need to write this once
fs.writeFileSync(`./pickem/week${weekNumber}.json`, JSON.stringify(weekJson));
}).catch(error => {
console.log(error)
})
}
}
writeGames();
})
Your issue might come from the fact that you are looping over an array of item that triggers parallel asynchronous calls and write weekJson before you get the entire data. (But theoretically your code should work if writeSyncFile is really synchronous, maybe there are locks on the file system that prevents node to write properly?)
You could try to make everything sequential and only write weekJson once instead of everytime you go over an item:
EDIT
I updated my original code proposition by keeping parallel calls and it worked for me (it's similar to OP's code but I only write the json file once per week).
Then I tried to run OP's code and it was working fine as well. So this makes me think that the problem isn't from the code itself but rather how it's called. As a pure node script, there doesn't seem to be any issue. But I just noticed that OP is using it server side as the result of an API call.
Having an API write so many JSON concurrently is probably not the best idea (especially if the api is called multiple times almost simultaneously). You could either
just return the games in the response
or precompute the results
or fetch and write them only once then cache the result to be reused
Then I wonder if due to the server context, there is not some kind of timeout since OP said that with my initial solution, only the first week was created.
const axios = require("axios");
const fs = require("fs");
async function writeGames() {
const writeWeekGamesPromises = [];
// iterate 18 times
for (let weekNumber = 1; weekNumber < 19; weekNumber++) {
// give week a default value in case the json file doesn't exist (for repro purpose)
let week = "{}";
try {
week = fs.readFileSync(`./pickem/week${weekNumber}.json`, "utf8");
} catch (e) {
console.log(`error reading week ${weekNumber} json file:`, e);
// file doesn't exist yet
}
const weekJson = JSON.parse(week);
// empty weekJson.games array
const games = [];
weekJson.games = games;
// get all items
// Add await keyword to wait for a week to be processed before going to the next one
writeWeekGamesPromises.push(axios
.get(
`https://sports.core.api.espn.com/v2/sports/football/leagues/nfl/seasons/2022/types/2/weeks/${weekNumber}/events?lang=en®ion=us`
)
.then(async (eventListResponse) => {
// add async to be able to use await
const schedule = [];
console.log(JSON.stringify(eventListResponse.data),'\n');
// get all items from response
const items = eventListResponse.data.items;
// console.log(eventListResponse.data.items); // this seems to be useless since we log the whole data just above
// parallelize calls and wait for all games from a week to be fetched before writing the file
await Promise.all(
items.map((item) => {
// we return the promise so that Promise.all will wait for all games to be pushed before going on writing the file
return axios
.get(item.$ref)
.then((response) => {
// get name, date and eventid
const {name, date, id: eventid} = response.data;
// get team ids
let team1 = response.data.competitions[0].competitors[0].id;
let team2 = response.data.competitions[0].competitors[1].id;
games.push({ name, date, eventid, team1, team2 });
})
.catch((error) => {
console.log(error);
});
})
);
// Now that all game data is ready, write in the file
fs.writeFileSync(
`./pickem/week${weekNumber}.json`,
JSON.stringify(weekJson)
);
})
.catch((error) => {
console.log(error);
}));
}
// Waiting for all games from all weeks to be processed
await Promise.all(writeWeekGamesPromises);
}
async function runAndLogTime() {
const start = Date.now();
await writeGames();
console.log(`took ${(Date.now() - start) / 1000}s to write all json files`);
}
runAndLogTime();
I'm trying to write a Discord bot that basically checks if each Discord User has a valid membership (that hasn't expired) inside a membership database using HTTP requests so I wrote something like the following
function checkmemberships() {
const memberships = fs.readFileSync('emailtest.txt').toString().toLowerCase().replace(/(?:\\[rn]|[\r\n]+)+/g, " ").split(" ");
const tenantId = 'example';
var today = new Date().toISOString().slice(0, 10);
for (i = 0; i < memberships.length; i += 3)
{
let contactId = memberships[i];
const membershipnumber = memberships[i + 1];
fetch(`https://rolodex.api.rhythmsoftware.com/contacts/${tenantId}/number/${contactId}`,
{
method: 'GET',
headers: {
'Content-Type': 'application/json',
'Authorization': global.apikey //this is generated somewhere else
},
}
)
.then((res) => res.json())
.then((res) => {
if (res.errorMessage == "RecordNotFound: Could not find a contact with the supplied number")
{
//Still To Do but not important
} else
{
if (res.id)
{
contactId = res.id; //Number can actually be different from what the user originally posts
fetch(`https://membership.api.rhythmsoftware.com/memberships/${tenantId}/contact/${contactId}`,
{
method: 'GET',
headers: {
'Content-Type': 'application/json',
'Authorization': global.apikey
},
}
)
.then((resp) => resp.json())
.then((resp) => {
console.log(resp);
if (resp.expiration_date)
{
let guild = client.guilds.cache.get('890266470641201172');
let member = guild.members.cache.get(membershipnumber); //At this point the membership isn't found because at this point it's undefined
if (resp.expiration_date <= today) {
member.roles.remove("890270511261696031");
member.roles.remove("890270660239175700");
}
}
})
}
}
})
}
}
This works when checking one membership but when I start introducing other memberships, I notice that the for loop is being completed before I have even got a response back for the first membership at which point then membershipnumber is no longer defined.
How can I change the code above so that the for loop waits for the HTTP response to be processed before it then does the next iteration?
I would use await fetch() to make sure the API response is completed before you can do anything with the data. This will prevent you from working with the data before the response is even complete.
So, in your case, you should change your code so that your await fetch is done first and outside of the loop, which is the opposite of how you have it now. Here is a really good article on how to use Await Fetch.
https://dmitripavlutin.com/javascript-fetch-async-await/#2-fetching-json
Await in the loop will make the checks sequentially. If they don't depend on one another, run the checks concurrently with Promise.all.
function checkmemberships() {
const memberships = fs.readFileSync('emailtest.txt').toString().toLowerCase().replace(/(?:\\[rn]|[\r\n]+)+/g, " ").split(" ");
const tenantId = 'example';
var today = new Date().toISOString().slice(0, 10);
let promises = [];
for (i = 0; i < memberships.length; i += 3) {
let contactId = memberships[i];
const membershipnumber = memberships[i + 1];
promises.push(checkMembership(tenentId, membershipnumber, today);
}
return Promise.all(promises);
}
function checkMembership(tenentId, membershipnumber, today) {
// .... from the op
return fetch(`https://rolodex.api.rhythmsoftware.com/contacts/${tenantId}/number/${contactId}`, // ...
// .then do json parse
// .then do expiration check
// return something, like a bool if the member is in good standing
}
I managed to get sensor data into a sqlite3 database. Every 2 minutes new values appear in the database. Now I want to display it with chart.js and therefore make use of sql.js
The code I use is:
const config_sqljs = {
locateFile: filename => (
'https://cdnjs.cloudflare.com/ajax/libs/sql.js/1.5.0/sql-wasm.wasm'
)
}
const sqlPromise = initSqlJs(config_sqljs)
const dataPromise = fetch((
'/sensors.db'
))
.then(res => res.arrayBuffer())
Promise.all([sqlPromise, dataPromise])
.then(([SQL, buf]) => {
const db = new SQL.Database(new Uint8Array(buf))
//const res = db.exec('select time_,temp,humi from sensors where topic=\"/x/sensors\"')
const res = db.exec('select time_,temp,humi from sensors where topic =\"/y/sensors\" and (date=\"29.11.2021\" or date=\"30.11.2021\" or date=\"01.12.2021\");')
const values = res[0].values;
let labelx = values.map((arr) => arr[0]);
let y = values.map((arr) => arr[1]);
let y2 = values.map((arr) => arr[2]);
})
It works well at first, but then I realized that the values in the arrays labelx, y, and y2 are not the newest one, they lack beheind, about half a day. The new values alwasy appear in the database. I think it has todo something with Promise.all() and .then but unfortunally I don't know much about js. The code is executet each time the site loads or reloads, even I I use Strg+F5 to force a complete, uncached reload it stays the same. Can you help me? Thanks!
=== EDIT ===
I adapted your suggestion:
const dataPromise = fetch((
'/db.db', {
method: "GET",
"cache-control": "no-store"}
))
.then(res => res.arrayBuffer())ยด
Sadly I get the error:
Uncaught (in promise) Error: file is not a database
the relevant line:
const res = db.exec('select x from db;')
Fetch might be caching the responses. Try:
fetch('/sensors.db', {
method: "GET",
"cache-control": "no-store"
})
I have a NIH API that can call only 50 pages at a time. The code below works for the first page of 50 items.
loadNIH() {
let offset = 0;
axios({
method: "GET",
url:
"https://api.federalreporter.nih.gov/v1/projects/search?query=orgName:Medical University of South Carolina$fy:2016$&offset=" + offset +"&limit=50"
})
.then(res => {
this.NIHData = res.data.items;
})
.catch(function(error) {
console.log(error);
});
},
I need to loop all of the pages until I get all of the data. Each time the offset needs to increase by the number of pages received (requested 50) 9 pages in this call. I need it to ADD data to the array of this.NIHData. I got one working so I need some help creating the loop.
Thanks in advance
You should repeatedly call the API until you get an empty resultset. This can be most easily achieved with the help of setTimeout()
loadNIH() {
let params =
{
offset: 0
}
this.NIHData = [];
this.fetchPages(params);
},
fetchPages(args)
{
axios.get("https://api.federalreporter.nih.gov/v1/projects/search?query=orgName:Medical University of South Carolina$fy:2016$&offset=" + args.offset +"&limit=50"
)
.then(res => {
this.NIHData.push(res.data.items);
args.offset += res.data.items.length;
if (res.data.items.length > 0) setTimeout(this.fetchPages, 2);
})
.catch(function(error) {
console.log(error);
});
}
I need combine multple API calls on a final object, this because API have limits to be consumed, any have an idea how is possible combine multiple calls in same final object, next is an example of my code, I need all data in this.lista but is not working:
created(){
this.$http.get('/api/transactions?senderId=8642612272713533685S&limit=1&offset=000')
.then( function(res){
console.log(res.body.count);
let limit = Math.ceil(res.body.count/1000);
console.log(limit);
let m = {};
let off = 0;
for (var i = 0; i <= limit; i++) {
this.$http.get('/api/transactions?senderId=8642612272713533685S&limit=1000', {params:{offset: off}})
.then( function(data){
this.lista = { ...this.lista, ...data.body.transactions }
} )
off = off + 1000;
}
}
);
}
any help will be appreciated
Using Promise.all is most likely what you are looking for. I will write just enough code for you to understand where to go.
// Populate your array array with URLs you want to get
let urls = ["url1", "url2", "...and so on"];
// Make into web request promises
let httpReqPromises = urls.map( url => this.$http.get(url) )
// Wait for all of them to resolve
Promise.all(httpReqPromises).then(allResponses => {
// Put them all together
this.lista = allResponses.reduce((a, b) => ({...a, ...b}, {})
})
The only work I leave up to you is how you populate the url variable.