I'm currently update useState array with sequential call to backend API with promise chaining using Axios.
I want to sequentially call 3 different API endpoints and gradually add elements so the total number of items in the array is 20.
My code currently looks something like this:
const [data, setData] = useState([])
function getData() {
const target = 'http://localhost:5000/api/backend/target/'
axios.get(target + 'one')
.then(res => {
setData([...data, ...res.data].slice(0, 20))
})
.then(axios.get(target + 'two'))
.then(res => {
if (data.length < 20) {
setData([...data, ...res.data].slice(0, 20))
}
})
.then(axios.get(target + 'three'))
.then(res => {
if (data.length < 20) {
setData([...data, ...res.data].slice(0, 20))
}
})
}
useEffect(() => {
getData()
}, [])
This works (not really) if the first endpoint returns more than 20 entries, but doesn't work if the first endpoint returns less than 20 entries, which means that the call to other two endpoints are simply not working.
You have two problems. First there's this:
.then(axios.get(target + 'two'))
This code immediately calls axios.get(target + 'two'), and passes its result into .then. So you're actually doing all the .gets right away, and since .then is expecting a function not a promise, it won't run anything later.
Instead, you need to pass a function into .then, which tells it what you want to eventually do, as in:
.then(() => axios.get(target + 'two'))
Secondly, there's this:
if (data.length < 20) {
setData([...data, ...res.data].slice(0, 20))
}
The data variable here is the empty array from the first render of the component. Any changes you've made to the state since then are not in here, so data.length will always be less than 20, and you'll always be copying an empty array.
Instead, you need to use the function version of setState to make sure you have the latest version of the state:
setData(prev => {
if (prev.length < 20) {
return [...prev, ...res.data].slice(0, 20);
} else {
return prev;
}
});
Related
In my code below I get an empty array on my console.log(response) but the console.log(filterdIds) inside the getIds function is showing my desired data. I think my resolve is not right.
Note that I run do..while once for testing. The API is paged. If the records are from yesterday it will keep going, if not then the do..while is stopped.
Can somebody point me to the right direction?
const axios = require("axios");
function getToken() {
// Get the token
}
function getIds(jwt) {
return new Promise((resolve) => {
let pageNumber = 1;
const filterdIds = [];
const config = {
//Config stuff
};
do {
axios(config)
.then((response) => {
response.forEach(element => {
//Some logic, if true then:
filterdIds.push(element.id);
console.log(filterdIds);
});
})
.catch(error => {
console.log(error);
});
} while (pageNumber != 1)
resolve(filterdIds);
});
}
getToken()
.then(token => {
return token;
})
.then(jwt => {
return getIds(jwt);
})
.then(response => {
console.log(response);
})
.catch(error => {
console.log(error);
});
I'm also not sure where to put the reject inside the getIds function because of the do..while.
The fundamental problem is that resolve(filterdIds); runs synchronously before the requests fire, so it's guaranteed to be empty.
Promise.all or Promise.allSettled can help if you know how many pages you want up front (or if you're using a chunk size to make multiple requests--more on that later). These methods run in parallel. Here's a runnable proof-of-concept example:
const pages = 10; // some page value you're using to run your loop
axios
.get("https://httpbin.org") // some initial request like getToken
.then(response => // response has the token, ignored for simplicity
Promise.all(
Array(pages).fill().map((_, i) => // make an array of request promisess
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${i + 1}`)
)
)
)
.then(responses => {
// perform your filter/reduce on the response data
const results = responses.flatMap(response =>
response.data
.filter(e => e.id % 2 === 0) // some silly filter
.map(({id, name}) => ({id, name}))
);
// use the results
console.log(results);
})
.catch(err => console.error(err))
;
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
The network tab shows the requests happening in parallel:
If the number of pages is unknown and you intend to fire requests one at a time until your API informs you of the end of the pages, a sequential loop is slow but can be used. Async/await is cleaner for this strategy:
(async () => {
// like getToken; should handle err
const tokenStub = await axios.get("https://httpbin.org");
const results = [];
// page += 10 to make the snippet run faster; you'd probably use page++
for (let page = 1;; page += 10) {
try {
const url = `https://jsonplaceholder.typicode.com/comments?postId=${page}`;
const response = await axios.get(url);
// check whatever condition your API sends to tell you no more pages
if (response.data.length === 0) {
break;
}
for (const comment of response.data) {
if (comment.id % 2 === 0) { // some silly filter
const {name, id} = comment;
results.push({name, id});
}
}
}
catch (err) { // hit the end of the pages or some other error
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
Here's the sequential request waterfall:
A task queue or chunked loop can be used if you want to increase parallelization. A chunked loop would combine the two techniques to request n records at a time and check each result in the chunk for the termination condition. Here's a simple example that strips out the filtering operation, which is sort of incidental to the asynchronous request issue and can be done synchronously after the responses arrive:
(async () => {
const results = [];
const chunk = 5;
for (let page = 1;; page += chunk) {
try {
const responses = await Promise.all(
Array(chunk).fill().map((_, i) =>
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${page + i}`)
)
);
for (const response of responses) {
for (const comment of response.data) {
const {name, id} = comment;
results.push({name, id});
}
}
// check end condition
if (responses.some(e => e.data.length === 0)) {
break;
}
}
catch (err) {
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
(above image is an except of the 100 requests, but the chunk size of 5 at once is visible)
Note that these snippets are proofs-of-concept and could stand to be less indiscriminate with catching errors, ensure all throws are caught, etc. When breaking it into sub-functions, make sure to .then and await all promises in the caller--don't try to turn it into synchronous code.
See also
How do I return the response from an asynchronous call? and Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference which explain why the array is empty.
What is the explicit promise construction antipattern and how do I avoid it?, which warns against adding a new Promise to help resolve code that already returns promises.
To take a step back and think about why you ran into this issue, we have to think about how synchronous and asynchronous javascript code works together. Your
synchronous getIds function is going to run to completion, stepping through each line until it gets to the end.
The axios function invocation is returning a Promise, which is an object that represents some future fulfillment or rejection value. That Promise isn't going to resolve until the next cycle of the event loop (at the earliest), and your code is telling it to do some stuff when that pending value is returned (which is the callback in the .then() method).
But your main getIds function isn't going to wait around... it invokes the axios function, gives the Promise that is returned something to do in the future, and keeps going, moving past the do/while loop and onto the resolve method which returns a value from the Promise you created at the beginning of the function... but the axios Promise hasn't resolved by that point and therefore filterIds hasn't been populated.
When you moved the resolve method for the promise you're creating into the callback that the axios resolved Promise will invoke, it started working because now your Promise waits for axios to resolve before resolving itself.
Hopefully that sheds some light on what you can do to get your multi-page goal to work.
I couldn't help thinking there was a cleaner way to allow you to fetch multiple pages at once, and then recursively keep fetching if the last page indicated there were additional pages to fetch. You may still need to add some additional logic to filter out any pages that you batch fetch that don't meet whatever criteria you're looking for, but this should get you most of the way:
async function getIds(startingPage, pages) {
const pagePromises = Array(pages).fill(null).map((_, index) => {
const page = startingPage + index;
// set the page however you do it with axios query params
config.page = page;
return axios(config);
});
// get the last page you attempted, and if it doesn't meet whatever
// criteria you have to finish the query, submit another batch query
const lastPage = await pagePromises[pagePromises.length - 1];
// the result from getIds is an array of ids, so we recursively get the rest of the pages here
// and have a single level array of ids (or an empty array if there were no more pages to fetch)
const additionalIds = !lastPage.done ? [] : await getIds(startingPage + pages, pages);
// now we wait for all page queries to resolve and extract the ids
const resolvedPages = await Promise.all(pagePromises);
const resolvedIds = [].concat(...resolvedPages).map(elem => elem.id);
// and finally merge the ids fetched in this methods invocation, with any fetched recursively
return [...resolvedIds, ...additionalIds];
}
I'm working on a small Javascript application that uses an API from pokeapi.co. Basically, it is supposed to fetch a few datas from the API and then display them on an HTML page. Here is the main part of the code :
const searchInput= document.querySelector(".recherche-poke input");
let allPokemon= [];
let tableauFin= [];
const listePoke= document.querySelector('.liste-poke');
function fetchPokemonBase(){
fetch("https://pokeapi.co/api/v2/pokemon?limit=75")
.then(reponse => reponse.json())
.then((allPoke) =>{
allPoke.results.forEach((pokemon) =>{
fetchPokemonComplet(pokemon);
})
})
}
fetchPokemonBase();
function fetchPokemonComplet(pokemon){
let objPokemonFull = {};
let url = pokemon.url;
let nameP = pokemon.name;
fetch(url)
.then(reponse => reponse.json())
.then((pokeData) => {
objPokemonFull.pic = pokeData.sprites.front_default;
objPokemonFull.type = pokeData.types[0].type.name;
objPokemonFull.id = pokeData.id;
fetch(`https://pokeapi.co/api/v2/pokemon-species/${nameP}`)
.then(reponse => reponse.json())
.then((pokeData) => {
objPokemonFull.name= pokeData.names[4].name;
allPokemon.push(objPokemonFull);
if(allPokemon.length === 75){
tableauFin= allPokemon.sort((a, b) => {
return a.id - b.id;
}).slice(0, 21);
createCard(tableauFin);
}
})
});
}
function createCard(arr){
for(let i= 0; i< arr.length; i++){
console.log(i + '\n');
const carte= document.createElement("li");
const txtCarte= document.createElement('h5');
txtCarte.innerText= arr[i].name;
const idCarte= document.createElement('p');
idCarte.innerText= `ID# ${arr[i].id}`;
const imgCarte= document.createElement('img');
imgCarte.src= arr[i].pic;
carte.appendChild(imgCarte);
carte.appendChild(txtCarte);
carte.appendChild(idCarte);
listePoke.appendChild(carte);
}
}
Here's what the code does : it gather a list of 75 pokemons from the pokeapi API. Then it fetches the data about each pokemon from the same site and stores each data into one element of the allPokemon array, when the length of this array reaches 75, we begin to create the HTML elements to display the data.
The ambigious part here is :
if(allPokemon.length === 75){
tableauFin= allPokemon.sort((a, b) => {
return a.id - b.id;
}).slice(0, 21);
createCard(tableauFin);
}
This code works but only when none of the requests fail. Otherwise, the length of the array allPokemon never reaches 75 and the rest of the code won't be executed. When I run the code, I run on XHR GET errors and the script stops before displaying the datas and it's (I think) what caused one of the promises to fail. I tried things like if(allPokemon.length === 74) (if I have one error, for example) and the code works just fine but that is surely not the solution.
Is there a way for me to "count" the errors I get from my requests so that I can do something like if(allPokemon.length === 75 - errorsCount) or maybe there is a smarter way to write my code?
Thanks in advance.
I think you can use promise all for this.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
function fetchPokemonBase(){
const promises = []
fetch("https://pokeapi.co/api/v2/pokemon?limit=75")
.then(reponse => reponse.json())
.then((allPoke) =>{
allPoke.results.forEach((pokemon) =>{
promises.push(fetchPokemonComplet(pokemon).catch(error => console.error(error)));
})
})
.then(() => {
Promise.all(promises)
.then(() => {
tableauFin= allPokemon.sort((a, b) => {
return a.id - b.id;
}).slice(0, 21);
createCard(tableauFin);
})
})
}
fetchPokemonBase();
let counter = 0
function fetchPokemonComplet(pokemon){
let objPokemonFull = {};
let url = pokemon.url;
let nameP = pokemon.name;
return fetch(url)
.then(reponse => reponse.json())
.then((pokeData) => {
counter++;
objPokemonFull.pic = pokeData.sprites.front_default;
objPokemonFull.type = pokeData.types[0].type.name;
objPokemonFull.id = pokeData.id;
return fetch(`https://pokeapi.co/api/v2/pokemon-species/${nameP}`)
.then(reponse => reponse.json())
.then((pokeData) => {
objPokemonFull.name= pokeData.names[4].name;
allPokemon.push(objPokemonFull);
})
});
}
So what you do is, instead of executing the fetch in the foreach, we push each fetch to an array of promises. Then we use the promise.all to execute them, and for each promise we catch the errors. So if one fails, the next promise will just continue.
With this code we push every fetch for the individual pokemons to an array:
promises.push(fetchPokemonComplet(pokemon).catch(error => console.error(error)));
Then we have an array of promises, which are the fetches to the server.
With the following code we execute these promises.
Promise.all(promises)
.then(() => {
tableauFin= allPokemon.sort((a, b) => {
return a.id - b.id;
}).slice(0, 21);
createCard(tableauFin);
})
the then on promise.all will be executed when all the promises are done.
And since we catch on every individual promise we don't have to worry about that.
And we don't care about the length and don't have to keep count.
Let me know if it works, or need any help.
From what I see, you will receive a list of between 0 and 75 pokemons from the first API call. Then, you fetch each one, however many it returned, up to 75 entries.
So, I think you want to just make sure your list isn't empty:
if(allPokemon.length > 0)
I've made a multidimensional array by doing an array.push on the inner and outer arrays. and I'm trying to get the array[0]'s length. I know I can't do that because .length in javascript only works if the array values are numerical. array[0] also returns undefined so am I not working with arrays as I think I am but some sort of objects instead? Sorry if the question is vague. I just don't know what I'm looking at.
what the array looks like in the console
getMapInfo = filterArray => {
var paramArray = [];
var locationArr = [];
var namesArr = [];
var ratingsArr = [];
var addressesArr = [];
var placeIDArr = [];
axios
.get(
`${"https://cors-anywhere.herokuapp.com/"}https://api.yelp.com/v3/businesses/search?`,
{
headers: {
Authorization: `Bearer ${process.env.REACT_APP_API_KEY}`
},
params: {
categories: "coffee, libraries",
latitude: this.state.currentLocation.lat,
longitude: this.state.currentLocation.lng,
limit: 20
}
}
)
// set state for locations, names
.then(res => {
for (var key in res.data.businesses) {
var addressesBase = res.data.businesses[key].location;
locationArr.push(res.data.businesses[key].coordinates);
placeIDArr.push(res.data.businesses[key].id);
namesArr.push(res.data.businesses[key].name);
ratingsArr.push(res.data.businesses[key].rating);
addressesArr.push(
"" +
addressesBase.address1 +
" " +
addressesBase.city +
", " +
addressesBase.state +
" " +
addressesBase.zip_code
);
}
if (filterArray === undefined) {
this.setState({
placeID: placeIDArr,
locations: locationArr,
names: namesArr,
ratings: ratingsArr,
addresses: addressesArr
});
}
paramArray.push(placeIDArr);
paramArray.push(locationArr);
paramArray.push(namesArr);
paramArray.push(ratingsArr);
paramArray.push(addressesArr);
})
.catch(err => {
console.log("Yelp API call error");
});
if (filterArray !== undefined) {
//console.log of paramArray[0] returns undefined
// console.log of paramArray is what is seen in the image
return new Promise((resolve, reject) => {
resolve(paramArray);
});
}
};
You are working with arrays correctly. The reason paramArray[0] returns undefined is because you are pushing to the array inside your .then function, which happens after the Axios call finishes - your console.log statements are outside of the .then function, causing them to run immediately instead of waiting for the Axios call (and therefore the array.push).
The simple solution is to move this part of your code inside the .then() callback:
if (filterArray !== undefined) {
//console.log of paramArray[0] returns undefined
// console.log of paramArray is what is seen in the image
return new Promise((resolve, reject) => {
resolve(paramArray);
});
}
Why is this necessary? Here's a simpler example to demonstrate:
let x = 0
// wait 1 second, then set x to 1
setTimeout(() => {
x = 1
}, 1000)
console.log(x)
What will be logged in the above example? You might initally think 1 would be logged, but actually it will log 0. This is because the code inside the callback is asynchronous (just like your Axios call), meaning it will wait in the background to run at a later time. Meanwhile, your code will continue executing, resulting in the console.log to be different than you expect.
What if we want to wait until x is set to 1? Then we would move our log into the callback.
let x = 0
// wait 1 second, then set x to 1
setTimeout(() => {
x = 1
console.log(x)
}, 1000)
Now, x will log 1. This is a simpler version of the exact problem you faced with Axios.
To read deeper on this topic, look for information on the JavaScript Call Stack. Here's a great article on it: https://www.freecodecamp.org/news/understanding-the-javascript-call-stack-861e41ae61d4/
I am new to rxjs and can't seem to find the correct operator for what I am trying to do.
In my example, I have an array I need to populate with results from an another observable, and once I have enough results in that array from making subscribe calls, I want to break and return the array.
// for reference, there is a class variable called keys
result = getResults(data, index).subscribe((result: any[]) => {
doSomethingWith(result);
});
getResults(data: any[], index: number) : Observable<any[]> {
obsFunctionThatGetsMeData(keys[index]).subscribe(result =>
data = data.concat(result);
if(data.length >= NEEDED_NUM_DATA) {
return Observable.of(data);
} else {
/* need to call obsFunctionThatGetsMeData(keys[index++]) and do the same logic as above. */
}
);
}
I know it is bad practice to put a subscribe in a subscribe, this is just the idea of what I'm looking for. I know takeWhile works on a condition, but I don't see how I can make extra calls if that condition fails. Does anyone know what operator is best for this kind of behavior?
Thanks!
obsFunctionThatGetsMeData returns Observable
Solved my own question
Using recursion & switchmap
getResults(data: any[], index: number): Observable<any> {
return obsFunctionThatGetsMeData(keys[index]).pipe(
switchMap(result => {
if (result) {
data= data.concat(result);
if (data.length < NEEDED_NUM_DATA && index < keys.length) {
return getResults(data, ++index);
} else {
return Observable.of(data);
}
}
}));
}
I'm not an expert in RxJS, but I think something like this should be possible:
Rx.Observable.from(obsFunctionThatGetsMeData)
.take(NEEDED_NUM_DATA)
.subscribe(doSomethingWith);
public take(count: number): Observable Emits only the first count
values emitted by the source Observable.
It's just an ideat to think about and doesn't pretend to be a working solution.
Looks like obsFunctionThatGetsMeData emits once (like an HTTP request), and you want to collect results from NEEDED_NUM_DATA number of these requests into array.
And also it should append to whatever you already have in data parameter?
Try this:
getResults( data: any[], index: number ): Observable<any[]>
{
return Observable.range( index, NEEDED_NUM_DATA )
.concatMap( i => obsFunctionThatGetsMeData( keys[i] ) )
.reduce(( acc: any[], value ) => acc.concat( value ), data );
}
Promises are your friend here. (because you are working with single values).
Although you think you are working with a stream of results (the observables), you are actually just working with a single value at a time. Each value is independently operated on in a loop that pushes the result into an array. So NEEDED_NUM_DATA independent variables consolidated into a a collection.
This code is much easier to rationalize about and to realize your goal:
var NEEDED_NUM_DATA = 4;
var collection = [];
async function getResults() {
let url = 'https://jsonplaceholder.typicode.com/posts/';
let i = 1;
while(collection.length <= NEEDED_NUM_DATA) {
let result = await getsMeData(url + i++);
collection.push(result);
}
}
async function getsMeData(url) {
const response = await fetch(url);
const json = await response.json();
return json;
}
getResults()
.then(()=>console.log(collection));
I have a component that fetches content from a service to process it. The thing is I can have multiple calls to this function, which results in duplicates on my array. I the following workaround:
getOptions(): Observable<PickQuality[]> {
console.log("MY LENGTH: ", this.options.length) // <=== Always returns 0 because the callback hasn't run yet
if(this.options.length == 0) {
this.championService.getChampions()
.subscribe(champions => {
champions.forEach(champion => this.options.push(new PickQuality(champion, 0)));
this.reevaluate();
this.optionsSubject.next(this.options);
});
return this.optionsSubject.asObservable();
}
else
return Observable.of(this.options);
}
and it didn't work, and then I tried the following trick inside the callback (where the this.options.length is correctly recognized):
if(this.options.length != 0) return; // <=== Looks bad!
which actually worked but seemed extremely inefficient to me, since the call to my service is still executed. How can I fix this?
I'd recommend to restructure your code a little:
if (this.options.length == 0) {
let source = this.championService.getChampions()
.share();
source.subscribe(champions => {
// ... whatever
this.options = whateverResult;
});
return source;
} else {
return Observable.of(this.options);
}
Now you can avoid using Subjects and return the source Observable which represents the HTTP request and is shared via the share() operator. This means there's only one HTTP request and its result is sent to this internal subscribe() call as well as to the subscriber outside this method.
Check for duplicates before pushing them.
this.championService.getChampions()
.subscribe(champions => {
champions.forEach(champion => {
if (champions.indexOf(champion) == -1)
this.options.push(new PickQuality(champion, 0));
});
this.reevaluate();
this.optionsSubject.next(this.options);
});