We Keep Coding

How to recursively fetch data from paginated API then combine into one array

Below I tried to write a conditional that would fetch a paginated api and then map it to another api that is being fetched. One issue that is coming up is that it's not continue to loop after it pulls one paginated page or one next page. The second issue is that that the data being fetched from the pages aren't being combined into one array. What am I doing wrong or missing?
const fetchURL = `${baseURL}?owner=${accounts[0]}`;
fetch(fetchURL, {
method: 'GET',
redirect: 'follow',
})
.then(resp => resp.json())
.then(data => {
console.log(data);
const pageKey = data.pageKey
if (pageKey !== 0) {
fetch(`${baseURL}?owner=${accounts[0]}&pageKey=${pageKey}`, {
method: 'GET',
redirect: 'follow',
})
.then(resp => resp.json())
.then(data => {
console.log(data)
})
return data.ownedNfts.concat(data.ownedNfts)
} else {
return data
}
const responses = data.ownedNfts.map((ownedNfts) =>
fetch(`${baseURL1}stats?address=${ownedNfts.contract.address}`)
.then((res) => res.json()),
);

To manage pagination from api, you could try a recursive like this.
You should have a script with a request loop with increment params, and a threshold to break the loop. You have to manage the request delay from your api with a time sleep or something like this.
This example bellow work in a node env with axios, you can try it and adapt it with your environnement.
const { default: axios } = require('axios');
// Init a bigData array to push new data on each iteration
const bigData = [];
async function fetchAllPaginateData(
pageKey = 0 /** init by default page index 0 */,
) {
try {
const fetchURL = `https://api.instantwebtools.net/v1/passenger?page=${pageKey}&size=1`;
const response = await axios.get(fetchURL);
const { data } = response;
const { totalPages } = data; // Your api should give you a total page count, result or something to setup your iteration
bigData.push(data); // push on big data response data
// if current page isn't the last, call the fetch feature again, with page + 1
if (
pageKey < totalPages &&
pageKey < 10 // (this is a test dev condition to limit for 10 result) */
) {
pageKey++;
await new Promise((resolve) => setTimeout(resolve, 200)); // setup a sleep depend your api request/second requirement.
console.debug(pageKey, '/', totalPages);
return await fetchAllPaginateData(pageKey);
}
console.clear();
return console.info('Data complete.');
} catch (err) {
console.error(err);
}
}
fetchAllPaginateData().then(() => console.table(bigData));

I modified the previous answer slightly to be self contained in one method instead of modifying a payload in the larger scope.
const axios = require('axios');
// These outer brackets are "simulating" global variables
//// `API_URL` would probably be configured globally on your request module
//// `LIMIT` might be set based on the current environment
((API_URL, LIMIT = 10) => {
// This method could take any structure of parameters
// Placing them in an object may actually be better at this point
// Defaults to an empty `combineData` array
const fetchAllPaginateData = async (accountId, reqSize = 20, currentPage = 0, combineData = []) => {
try {
const fetchURL = `${API_URL}/passenger?owner=${accountId}&size=${reqSize}&page=${currentPage}`;
const response = await axios.get(fetchURL);
const { totalPages, totalPassengers, data } = response;
console.debug(`${totalPassengers} Passengers`, '/', `Needing ${totalPages} pages`, '/', `${reqSize} passengers per page`);
// Here's the secret sauce,
// Combine data on each recursion and return it at the end
combineData = combineData.concat(data);
if (currentPage < totalPages && currentPage < LIMIT) {
currentPage++;
console.debug(currentPage, 'of', totalPages, 'pages');
// Slow down requests so as not to DOS the API
await new Promise((resolve) => setTimeout(resolve, 200));
return await fetchAllPaginateData(reqSize, currentPage, combineData);
}
// Send big data back
return combineData;
} catch (err) {
console.error(err);
}
}
// Presumably this would be called from other methods or files
// providing the owners account number for the request query
fetchAllPaginateData(1007).then(data => console.table(data));
})('https://api.instantwebtools.net/v1', 2);

How to store API response data for analysis in javascript

I am attempting to build a bot that will periodically poll an API using axios for the price of multiple cryptocurrencies across multiple exchanges. I need to be able to then look at this stored data to analyse price differences of a given token between multiple exchanges to calculate if there is profit to be made if I were to purchase it on one and then sell on another.
So far I am able to get access to the price data and console.log it as the request is returned, however I need to add this data to an array so that it can be analysed at a later point. I understand that I will need to use promises to handle this but I find it slightly confusing.
The following code immediately outputs an empty array and then outputs all the individual prices. How can I gain access to those values at a later point in the code?
const axios = require('axios');
const { exchanges } = require('./resources/exchanges.json');
const { currencies } = require('./resources/currencies.json');
const buyCurrency = currencies.buy[0];
const poll = async () => {
const data = new Array();
await currencies.sell.forEach(async sellCurrency => {
await exchanges.forEach(async exchange => {
try {
const allOtherExchanges = exchanges.filter(x => x !== exchange).map(x => `,${x}`).join();
const response = await axios.get(`https://api.0x.org/swap/v1/quote?buyToken=${buyCurrency}&sellToken=${sellCurrency}&sellAmount=1000000000000000000&excludedSources=0x${allOtherExchanges}`)
if (response && response.data.price) {
console.log(exchange, sellCurrency, response.data.price)
data.push({
exchange,
price: response.data.price
});
}
} catch {}
});
});
console.log(data);
};
poll()

One of the solutions would be as follows:
const axios = require('axios');
const { exchanges } = require('./resources/exchanges.json');
const { currencies } = require('./resources/currencies.json');
const buyCurrency = currencies.buy[0];
const poll = async () => {
const data = new Array();
const promises = [];
currencies.sell.forEach(sellCurrency => {
exchanges.forEach(exchange => {
const allOtherExchanges = exchanges.filter(x => x !== exchange).map(x => `,${x}`).join();
promises.push(
axios.get(`https://api.0x.org/swap/v1/quote?buyToken=${buyCurrency}&sellToken=${sellCurrency}&sellAmount=1000000000000000000&excludedSources=0x${allOtherExchanges}`)
.then(response => {
if (response && response.data &&response.data.price) {
console.log(exchange, sellCurrency, response.data.price)
data.push({
exchange,
price: response.data.price
});
}
}).catch(err => console.error(err))
);
});
});
await Promise.all(promises);
console.log(data);
};
poll();

React script stop working after changing API call

I have a script which calls API from React and then triggers email notification function.
I was changing one part of it to call whole array of parameters instead of calling one parameter after another.
Here is part before change(working one). Console log shows correct response and I receive email notification as well.
const getApiData = () => {
const apiCall = (symbol) => {
return `https://min-api.cryptocompare.com/data/pricemulti?fsyms=${symbol}&tsyms=USD&api_key=API-KEY-HERE`
}
const MAX_CHARACKTERS = 300
let bucketArray = ['']
for (let i=0; i < assets.length - 1; i += 1) {
const symbol = `${bucketArray[bucketArray.length - 1]},${assets[i]}`
if (i === 0) {
bucketArray[0] = assets[i]
continue
}
if (symbol.length < MAX_CHARACKTERS) {
bucketArray[bucketArray.length - 1] = symbol
} else {
bucketArray[bucketArray.length] = assets[i]
}
}
const getData = () => {
Promise.all(
bucketArray.map(req => {
return axios(apiCall(req))
.then(({ data }) => data)
})
).then((data) => setDataApi(data))
}
getData()
};
Here is problematic one.
const getApiData = () => {
const getString = symbol =>
`https://min-api.cryptocompare.com/data/pricemulti?fsyms=${symbol}&tsyms=USD&api_key=API-KEY-HERE`;
function getAxious(id) {
const url = getString(id);
return axios.get(url);
}
const BUCKET_SIZE = 150;
const bucketArray = assets.reduce(
(arr, rec) => {
if (arr[arr.length - 1].length < BUCKET_SIZE) {
arr[arr.length - 1] = [...arr[arr.length - 1], rec];
return arr;
}
return [...arr, [rec]];
},
[[]]
);
bucketArray
.reduce((acc, rec) => {
return acc.then(results => {
return Promise.all(
rec.map(item =>
getAxious(item).then(({ data }) => {
return {
Symbol: item,
Open: data
};
})
)
).then(x => {
return [...x, ...results];
});
});
},
Promise.resolve([]))
.then(res => {
setDataApi(res);
});
};
Here in console I receive empty array - [] no errors showed, but email notification also stops from working.
I'm changing the code since I need to call whole array from API in one call. Before I was calling one symbol after another.
What I did wrong that console doesn't show the correct response?
EDIT1
Here is bucketArray value
const assets = ['ADA','KAVA','DOGE'];

I was not able to understand completely, but I think you want to collect all the results together and set it to the data using setDataApi.
Check the below code and let me know if it helps:
async function getApiData() {
const getString = (arr) =>
`https://min-api.cryptocompare.com/data/pricemulti?fsyms=${arr.join(
","
)}&tsyms=USD&api_key=API_KEY`;
function getAxious(arr) {
const url = getString(arr);
return axios.get(url);
}
const BUCKET_SIZE = 150;
const bucketArray = assets.reduce(
(arr, rec) => {
if (arr[arr.length - 1].length < BUCKET_SIZE) {
arr[arr.length - 1] = [...arr[arr.length - 1], rec];
return arr;
}
return [...arr, [rec]];
},
[[]]
);
const res = await getAxious(bucketArray);
console.log("res", res);
return res;
// after this you can set setDataApi(res);
}
// keep this useEffect sepearate
const [timer, setTimer] = useState(null);
useEffect(() => {
async function getApiDatahandler() {
const res = await getApiData();
console.log(res);
const timerId = setTimeout(() => {
getApiDatahandler();
}, 1000 * 60);
setTimer(timerId);
setDataApi(res)
// set the data setDataApi(res);
}
getApiDatahandler();
return () => {
window.clearTimeout(timer);
};
}, []);
// useEffect(() => {
// const timerId = setTimeout(() => {
// getApiData();
// }, 1000 * 60);
// }, [])

Checkout this codepen for a possible solution.
https://codepen.io/bcaure/pen/RwapqZW?editors=1011
In short, I don't know how to fix your code because it's quite a callback hell.
// Mock API and data
const bucketArray = [[{item: 'item1'}], [{item: 'item2'}], [{item: 'item3'}]];
const getAxious = item => {
return new Promise((resolve, reject) => resolve({data: 'API data'}));
}
// Return promise that combines API data + input item
const recToPromise = rec => rec.map(item => {
return new Promise((resolve, reject) => getAxious(item)
.then(data => resolve({item, data})));
});
// Flatten array
const recPromisesFlatten = bucketArray.flatMap(recToPromise);
Promise.all(recPromisesFlatten)
.then(res => {
const flattenRes = res.flatMap(({item, data}) => ({ Symbol: item, Open: data }));
console.log(JSON.Stringify(flattenRes))
});
What I'm suggesting to debug errors:
build your promise array first
then run Promise.all
then combine your data
Bonus: you can see flatMap instead of reduce for better readability.

Merging various backend requests in the express res.send()

I'm trying to make several asynchronous backend calls to generate a JSON response in my express API. Because of the nature of the API, I have 3 requests that are being made that are dependent on each other in some way.
Request 1: Returns an Array of values that are used to make request 2. Each value will be used as a mapping for the remaining requests. That is to say, it will be a unique identifier used to map the response from the requests in Request 3.
Request 2 (Parallel Batch): A request is made using each value from the Array returned in request 1. Each of these returns a value to be used in each of the Request 3s. That is to say, it's a 1-to-1
Request 3 (Parallel Batch): This request takes the response from Request 2, and makes a 1-to-1 follow up request to get more data on that specific mapping (the id from request 1)
I would like the final data I send to the consumer to look like this:
{
id1: details1,
id2: details2,
id3: details3,
...
}
Here is the code I have so far...
app.get("/artists/:artist/albums", (req, res) => {
console.log("#############")
const artistName = req.params.artist
let response = {};
let s3Promise = s3.listAlbums(artistName)
let albumDetailsPromises = []
s3Promise
.then((data) => {
data.map((album) => {
// Each album name here will actually be used as the unique identifier for
// the final response
// Build an Array of promises that will first fetch the albumId, then use
// that album id to fetch the details on the album
albumDetailsPromises.push(
discogs.getAlbumId(artistName, album).then( // Returns a promise
({ data }) => {
let masterId = data.results[0].id
let recordName = data.results[0].title
// Storing the album name to carry as a unique id alongside the promise
return [album, discogs.getAlbumDetails(masterId) // Returns a promise ]
}
)
)
})
})
.then(() => {
// When all the albumIds have been fetched, there will still exist a promise in the
// second index of each element in the albumDetailsPromises array
Promise.all(albumDetailsPromises)
.then((namedPromises) => {
namedPromises.map(
(album) => {
let albumName = album[0] // Unique Id
let albumDetailPromise = album[1]
// Resolving the albumDetailsPromise here, and storing the value on
// a response object that we intend to send as the express response
albumDetailPromise
.then(
({ data }) => {
response[albumName] = data
})
.catch(err => response[albumName] = err)
})
})
})
.catch((err) => console.log(err))
})
As of now, everything seems to be working as expected, I just can't seem to figure out how to "await" the response object being updated at the end of all these Promises. I've omitted res.send(response) from this example because it's not working, but that's of course my desired outcome.
Any advice is appreciated! New to javascript...

I would recommend rewriting this using async/await as it helps to reduce nesting. You can also extract the logic the get the album-details into a separate function, as this also increases the readability of the code. Something like this (this still needs error-handling, but it should give you a start):
app.get("/artists/:artist/albums", async (req, res) => {
const artistName = req.params.artist;
const albumNames = await s3.listAlbums(artistName);
const result = {};
const albumDetailPromises = albumNames.map(albumName => requestAlbumDetails(discogs, artistName, albumName));
const resolvedAlbumDetails = await Promise.all(albumDetailPromises);
// map desired response structure
for(const albumDetail of resolvedAlbumDetails) {
result[albumDetail.albumName] = albumDetail.albumDetails;
}
res.json(result);
});
async function requestAlbumDetails(service, artistName, albumName) {
const albumInfo = await service.getAlbumId(artistName, albumName);
const masterId = albumInfo.results[0].id;
const albumDetails = await service.getAlbumDetails(masterId);
return { albumName, albumDetails };
}
To answer your question how you could do it with your code:
You'd need to wait for all details to be fulfilled using another Promise.all call and then just send the response in the then-handler:
Promise.all(albumDetailsPromises)
.then((namedPromises) => {
const detailsPromises = namedPromises.map(
(album) => {
let albumName = album[0];
let albumDetailPromise = album[1];
return albumDetailPromise
.then(({ data }) => {
response[albumName] = data;
})
.catch(err => response[albumName] = err);
});
return Promise.all(detailsPromises)
.then(() => res.json(response));
})

Refactored using async/await...
app.get("/artists/:artist/albums", async (req, res) => {
const artistName = req.params.artist
let response = {};
let albums = await s3.listAlbums(artistName)
const promises = albums.map(async (album) => {
let result = await discogs.getAlbumId(artistName, album)
try {
let masterId = result.data.results[0].id
let tempRes = await discogs.getAlbumDetails(masterId)
return [album, tempRes.data]
} catch (error) {
return [album, { "msg": error.message }]
}
})
responses = await Promise.all(promises)
responses.map(data => { response[data[0]] = data[1] })
res.send(response)
})

How to get a reference to firebase image to be able to delete it?

so I have this function to retrieve all my photos
const getPhotos = async () => {
const storage = firebaseApp.storage()
let gsReference = storage.refFromURL('gs://my-app/')
gsReference = gsReference.child(`${id}/${country}`)
const { items } = await gsReference.listAll()
const urlPromises = items.map((pr) =>
pr
.getDownloadURL()
.then((url) => fetch(url))
.catch((error) => error),
)
let urls = await Promise.all(urlPromises)
urls = urls.filter((result) => !(result instanceof Error))
setPhotos(urls)
}
this works all good
now the problem is when I want to delete an image. ive got this function
const deletePhoto = (photoRef) => {
console.log('PHOTO:', photoRef)
var storage = firebaseApp.storage()
var storageRef = storage.ref(`${id}/${country}`)
}
the problem is I have no way of knowing the reference to the image I want to delete?
I am mapping over all the images and for each image I'm attaching the delete function and passing in the photo as an arg. however this just returns me this
body: ReadableStream {locked: false}
bodyUsed: false
headers: Headers {append: function, delete: function, get: function, has: function, set:
ok: true
redirected: false
status: 200
statusText: ""
type: "cors"
url: "https://firebasestorage.googleapis.com/v0/b/my-app.appspot.com/o/F2DFB6714-8082-4AF9-89…"
it looks like the url does reference the path but it cuts some of it off at the end. plus I don't really want to be splicing that down to find the image. there must be an easy way to set the reference??

You can combine Response and Reference object in another then callback after catch.
const getPhotos = async () => {
const storage = firebaseApp.storage()
let gsReference = storage.refFromURL('gs://my-app/')
gsReference = gsReference.child(`${id}/${country}`)
const { items: references } = await gsReference.listAll()
const result = references.map(async (reference) => {
const url = await reference.getDownloadURL();
let response = null;
try {
response = await fetch(url);
}
catch(error) {
response = error;
}
return {
response,
url,
reference
}
}
)
let referencesWithUrls = await Promise.all(result)
referencesWithUrls = referencesWithUrls.filter((result) => !(result.response instanceof Error))
setPhotos(referencesWithUrls.map(pr => pr.response))
}