Swapi API pagination using (data.next) vanilla JS - javascript

I have this async function to get three separate requests from the swapi API to retrieve data. However, I'm only getting back the first page of data as it's paginated. I know I have to create a loop for data.next to make new requests but I'm unsure the best way to run it through my function.
(async function getData() {
//Utility Functions for fetch
const urls = ["https://swapi.co/api/planets/", "https://swapi.co/api/films/", "https://swapi.co/api/people/"];
const checkStatus = res => res.ok ? Promise.resolve(res) : Promise.reject(new Error(res.statusText));
const parseJSON = response => response.json();
//Get Data
await Promise.all(urls.map(url => fetch(url)
.then(checkStatus)
.then(parseJSON)
.catch(error => console.log("There was a problem!", error))))
.then(data => {
let planets = data[0].results,
films = data[1].results,
people = data[2].results;
buildData(films, planets, people);
});
})();

You are trying to access all the data.results keys in the loop, which misses the point of using Promise.all. Promise.all collects all the results from promises and stores it in a single array when all the promises are resolved.
So wait for the promises to resolve and use the array returned from Promise.all to build your data.
To get all the pages you need to have a recursive function. Which means that this function will keep calling itself until a condition is met. Sort of like a loop but with callbacks.
Every time you fetch a page check if the there is a next page by checking the next property in the response object. If there is call the getAllPages again until there are no more pages left. At the same time all the results are concatenated in a single array. That array is passed on to the next call which concatenates it again with the result. And at the end the collection variable, which contains all the concatenated arrays, is returned.
Let me know if you have any questions regarding the code.
(async function getData() {
//Utility Functions for fetch
const urls = ["https://swapi.co/api/planets/", "https://swapi.co/api/films/", "https://swapi.co/api/people/"];
const checkStatus = res => res.ok ? Promise.resolve(res) : Promise.reject(new Error(res.statusText));
const parseJSON = response => response.json();
// Get a single endpoint.
const getPage = url => fetch(url)
.then(checkStatus)
.then(parseJSON)
.catch(error => console.log("There was a problem!", error));
// Keep getting the pages until the next key is null.
const getAllPages = async (url, collection = []) => {
const { results, next } = await getPage(url);
collection = [...collection, ...results];
if (next !== null) {
return getAllPages(next, collection);
}
return collection;
}
// Select data out of all the pages gotten.
const [ planets, films, people ] = await Promise.all(urls.map(url => getAllPages(url)));
buildData(films, planets, people);
})();

Related

Learning Promises, Async/Await to control execution order

I have been studying promises, await and async functions. While I was just in the stage of learning promises, I realized that the following: When I would send out two requests, there was no guarantee that they would come in the order that they are written in the code. Of course, with routing and packets of a network. When I ran the code below, the requests would resolve in no specific order.
const getCountry = async country => {
await fetch(`https://restcountries.com/v2/name/${country}`)
.then(res => res.json())
.then(data => {
console.log(data[0]);
})
.catch(err => err.message);
};
getCountry('portugal');
getCountry('ecuador');
At this point, I hadn't learned about async and await. So, the following code works the exact way I want it. Each request, waits until the other one is done.
Is this the most simple way to do it? Are there any redundancies that I could remove? I don't need a ton of alternate examples; unless I am doing something wrong.
await fetch(`https://restcountries.com/v2/name/${country}`)
.then(res => res.json())
.then(data => {
console.log(data[0]);
})
.catch(err => err.message);
};
const getCountryData = async function () {
await getCountry('portugal');
await getCountry('ecuador');
};
getCountryData();
Thanks in advance,
Yes, that's the correct way to do so. Do realize though that you're blocking each request so they run one at a time, causing inefficiency. As I mentioned, the beauty of JavaScript is its asynchronism, so take advantage of it. You can run all the requests almost concurrently, causing your requests to speed up drastically. Take this example:
// get results...
const getCountry = async country => {
const res = await fetch(`https://restcountries.com/v2/name/${country}`);
const json = res.json();
return json;
};
const getCountryData = async countries => {
const proms = countries.map(getCountry); // create an array of promises
const res = await Promise.all(proms); // wait for all promises to complete
// get the first value from the returned array
return res.map(r => r[0]);
};
// demo:
getCountryData(['portugal', 'ecuador']).then(console.log);
// it orders by the countries you ordered
getCountryData(['ecuador', 'portugal']).then(console.log);
// get lots of countries with speed
getCountryData(['mexico', 'china', 'france', 'germany', 'ecaudor']).then(console.log);
Edit: I just realized that Promise.all auto-orders the promises for you, so no need to add an extra sort function. Here's the sort fn anyways for reference if you take a different appoach:
myArr.sort((a, b) =>
(countries.indexOf(a.name.toLowerCase()) > countries.indexOf(b.name.toLowerCase())) ? 1 :
(countries.indexOf(a.name.toLowerCase()) < countries.indexOf(b.name.toLowerCase()))) ? -1 :
0
);
I tried it the way #deceze recommended and it works fine: I removed all of the .then and replaced them with await. A lot cleaner this way. Now I can use normal try and catch blocks.
// GET COUNTRIES IN ORDER
const getCountry = async country => {
try {
const status = await fetch(`https://restcountries.com/v2/name/${country}`);
const data = await status.json();
renderCountry(data[0]); // Data is here. Now Render HTML
} catch (err) {
console.log(err.name, err.message);
}
};
const getCountryData = async function () {
await getCountry('portugal');
await getCountry('Ecuador');
};
btn.addEventListener('click', function () {
getCountryData();
});
Thank you all.

javascript: Update the DOM only when the result is ready

I have some api endpoint.
one returns all server details (https://dnscheck.io/api/serverDetails/)
others are server specific endpoint. (https://dnscheck.io/api/query/?id=2&type=A&hostname=test.com) for each server_Id(which I got from serverDetails endpoint), I have to call each api endpoint.
what I have done is.
I loop over the results array (which I got from serverDetails endpoint)
and for each iteration of loop, I call each endpoint for getting the ip.
loop:
for (const [index, item] of data.entries()) {
const res = await fetch(
`https://dnscheck.io/api/query/?id=${item.id}&type=${query.type}&hostname=${query.host}`
);
const result = await res.json();
renderResult(result, item, index);
}
render-function:
const renderResult = (result, data, index) => {
const ip = document.querySelector(`.ip-address${index + 1}`);
ip.innerHTML = result.answers[0].address;
};
In this way, results are displayed in the DOM in a sync way. (one after another)
But, what I want is, update the dom with the result, as soon as the result is ready.
what can I do?
Don't use await, as that blocks the for loop and orders the results. Use .then() instead.
for (const [index, item] of data.entries()) {
fetch(
`https://dnscheck.io/api/query/?id=${item.id}&type=${query.type}&hostname=${query.host}`
).then(res => res.json())
.then(result => renderResult(result, item, index));
}
You can do them in parallel by using map on the array and using fetch within. You can know when they've all finished by using Promise.all to observe the overall result:
await Promise.all(
data.entries().map(async (index, item) => {
const res = await fetch(
`https://dnscheck.io/api/query/?id=${item.id}&type=${query.type}&hostname=${query.host}`
);
// You need to check `res.ok` here
const result = await res.json();
renderResult(result, item, index);
)
);
Note that Promise.all will reject its promise immediately if any of the input promises rejects. If you want to know what succeeded and what failed, use allSettled instead:
const results = await Promise.allSettled(
data.entries().map(async (index, item) => {
const res = await fetch(
`https://dnscheck.io/api/query/?id=${item.id}&type=${query.type}&hostname=${query.host}`
);
// You need to check `res.ok` here
const result = await res.json();
renderResult(result, item, index);
)
);
// Use `results` here, it's an array of objects, each of which is either:
// {status: "fulfilled", value: <the fulfillment value>}
// or
// {status: "rejected", reason: <the rejection reason>}
About my "You need to check res.ok here" note: this is unfortunately a footgun in the fetch API. It only rejects its promise on network failure, not HTTP errors. So a 404 results in a fulfilled promise. I write about it here. Typically the best thing is to have wrapper functions you call, for instance:
function fetchJSON(...args) {
return fetch(...args)
.then(response => {
if (!response.ok) {
throw new Error(`HTTP error ${response.status}`); // Or an error subclass
}
return response.json();
});
}

Making multiple web api calls synchronously without nesting in Node.js with Axios

Is there any way I can make the below code run synchronously in a way where I can get all of the productLine ids and then loop through and delete all of them, then once all of this is complete, get all of the productIds and then loop through and delete all of them?
I really want to be able to delete each set of items in batch, but the next section can't run until the first section is complete or there will be referential integrity issues.
// Delete Product Lines
axios.get('https://myapi.com/ProductLine?select=id')
.then(function (response) {
const ids = response.data.value
ids.forEach(id => {
axios.delete('https://myapi.com/ProductLine/' + id)
})
})
.catch(function (error) {
})
// Delete Products (I want to ensure this runs after the above code)
axios.get('https://myapi.com/Product?select=id')
.then(function (response) {
const ids = response.data.value
ids.forEach(id => {
axios.delete('https://myapi.com/Product/' + id)
})
})
.catch(function (error) {
})
There's a lot of duplication in your code. To reduce code duplication, you can create a helper function that can be called with appropriate arguments and this helper function will contain code to delete product lines and products.
async function deleteHelper(getURL, deleteURL) {
const response = await axios.get(getURL);
const ids = response.data.value;
return Promise.all(ids.map(id => (
axios.delete(deleteURL + id)
)));
}
With this helper function, now your code will be simplified and will be without code duplication.
Now to achieve the desired result, you could use one of the following ways:
Instead of two separate promise chains, use only one promise chain that deletes product lines and then deletes products.
const prodLineGetURL = 'https://myapi.com/ProductLine?select=id';
const prodLineDeleteURL = 'https://myapi.com/ProductLine/';
deleteHelper(prodLineGetURL, prodLineDeleteURL)
.then(function() {
const prodGetURL = 'https://myapi.com/Product?select=id';
const prodDeleteURL = 'https://myapi.com/Product/';
deleteHelper(prodGetURL, prodDeleteURL);
})
.catch(function (error) {
// handle error
});
Use async-await syntax.
async function delete() {
try {
const urls = [
[ prodLineGetURL, prodLineDeleteURL ],
[ prodGetURL, prodDeleteURL ]
];
for (const [getURL, deleteURL] of urls) {
await deleteHelper(getURL, deleteURL);
}
} catch (error) {
// handle error
}
}
One other thing that you could improve in your code is to use Promise.all instead of forEach() method to make delete requests, above code uses Promise.all inside deleteHelper function.
Your code (and all other answers) are executing delete requests sequentially, which is huge waste of time. You should use Promise.all() and execute in parallel...
// Delete Product Lines
axios.get('https://myapi.com/ProductLine?select=id')
.then(function (response) {
const ids = response.data.value
// execute all delete requests in parallel
Promise.all(
ids.map(id => axios.delete('https://myapi.com/ProductLine/' + id))
).then(
// all delete request are finished
);
})
.catch(function (error) {
})
All HTTP request are asynchronous but you can make it sync-like. How? Using async-await
Suppose you have a function called retrieveProducts, you need to make that function async and then await for the response to keep processing.
Leaving it to:
const retrieveProducts = async () => {
// Delete Product Lines
const response = await axios.get('https://myapi.com/ProductLine?select=id')
const ids = response.data.value
ids.forEach(id => {
axios.delete('https://myapi.com/ProductLine/' + id)
})
// Delete Products (I want to ensure this runs after the above code)
const otherResponse = await axios.get('https://myapi.com/Product?select=id') // use proper var name
const otherIds = response.data.value //same here
otherIds.forEach(id => {
axios.delete('https://myapi.com/Product/' + id)
})
}
But just keep in mind that it's not synchronous, it keeps being async

Why can't I move "await" to other parts of my code?

Edit2: Solution at the bottom
I am using the chrome-console and I am trying to output fetched data, and I only get the desired output by writing "await" at exactly the right place, even though another solution can do it earlier and I don't know why/how it works.
solution() is the "official" solution from a web-course I am doing. Both functions return the same, currently. In myFunction I tried writing "await" in front of every used function and make every function "async", but I still can't replace the "await" inside log, even though the other solution can.
const urls = ['https://jsonplaceholder.typicode.com/users']
const myFunction = async function() {
// tried await before urls/fetch (+ make it async)
const arrfetched = urls.map( url => fetch(url) );
const [ users ] = arrfetched.map( async fetched => { //tried await in front of arrfetched
return (await fetched).json(); //tried await right after return
});
console.log('users', await users); // but can't get rid of this await
}
const solution = async function() {
const [ users ] = await Promise.all(urls.map(async function(url) {
const response = await fetch(url);
return response.json();
}));
console.log('users', users); // none here, so it can be done
}
solution();
myFunction();
I would think "await" works in a way that makes:
const a = await b;
console.log(a); // this doesn't work
the same as
const a = b;
console.log(await a); // this works
but it doesn't, and I don't understand why not. I feel like Promise.all does something unexpected, as simply writing "await" in the declaration can't do the same, only after the declaration.
Edit1: this does not work
const myFunction = async function() {
const arrfetched = await urls.map( async url => await fetch(url) );
const [ users ] = await arrfetched.map( async fetched => {
return await (await fetched).json();
});
console.log('users', users);
}
Edit2: Thanks for the help everyone, I tried putting ".toString()" on a lot of variables and switching where I put "await" in the code and where not.
As far as I understand it, if I don't use Promise.all then I need to await every time I want to use (as in the actualy data, not just use) a function or variable that has promises. It is insufficient to only have await where the data is being procensed and not further up.
In the Edit1 above users runs bevore any other await is complete, therefore no matter how many awaits i write in, none are being executed. Copying this code in the (in my case chrome-)console demostrates it nicely:
const urls = [
'https://jsonplaceholder.typicode.com/users',
]
const myFunction = async function() {
const arrfetched = urls.map( async url => fetch(url) );
const [ users ] = arrfetched.map( async fetched => {
console.log('fetched', fetched);
console.log('fetched wait', await fetched);
return (await fetched).json();
});
console.log('users', users);
console.log('users wait', await users);
}
myFunction();
// Output in the order below:
// fetched()
// users()
// fetched wait()
// users wait()
TL; DR: Promise.all is important there, but it's nothing magical. It just converts an array of Promises into a Promise that resolves with an array.
Let's break down myFunction:
const arrfetched = urls.map( url => fetch(url) );
This returns an array of Promises, all good so far.
const [ users] = arrfetched.map( async fetched => {
return (await fetched).json();
});
You're destructuring an array to get the first member, so it's the same as this:
const arr = arrfetched.map( async fetched => {
return (await fetched).json();
});
const users = arr[0];
Here we are transforming an array of promises into another array of promises. Notice that calling map with an async function will always result in an array of Promises.
You then move the first member of that array into users, so users now actually contains a single Promise. You then await it before printing it:
console.log('users', await users);
In contrast, the other snippet does something slightly different here:
const [ users ] = await Promise.all(urls.map(async function(url) {
const response = await fetch(url);
return response.json();
}));
Once again, let's separate the destructuring:
const arr = await Promise.all(urls.map(async function(url) {
const response = await fetch(url);
return response.json();
}));
const users = arr[0];
Promise.all transforms the array of Promises into a single Promise that results in an array. This means that, after await Promise.all, everything in arr has been awaited (you can sort of imagine await Promise.all like a loop that awaits everything in the array). This means that arr is just a normal array (not an array of Promises) and thus users is already awaited, or rather, it was never a Promise in the first place, and thus you don't need to await it.
Maybe the easiest way to explain this is to break down what each step achieves:
const urls = ['https://jsonplaceholder.typicode.com/users']
async function myFunction() {
// You can definitely use `map` to `fetch` the urls
// but remember that `fetch` is a method that returns a promise
// so you'll just be left with an array filled with promises that
// are waiting to be resolved.
const arrfetched = urls.map(url => fetch(url));
// `Promise.all` is the most convenient way to wait til everything's resolved
// and it _also_ returns a promise. We can use `await` to wait for that
// to complete.
const responses = await Promise.all(arrfetched);
// We now have an array of resolved promises, and we can, again, use `map`
// to iterate over them to return JSON. `json()` _also_ returns a promise
// so again you'll be left with an array of unresolved promises...
const userData = responses.map(fetched => fetched.json());
//...so we wait for those too, and destructure out the first array element
const [users] = await Promise.all(userData);
//... et voila!
console.log(users);
}
myFunction();
Await can only be used in an async function. Await is a reserved key. You can't wait for something if it isn't async. That's why it works in a console.log but not in the global scope.

Promise.all returning empty objects

I'm trying to get multiple data objects from The Movie Database at once using Promise.all. After I loop through all the results of the fetch call, and use .json() on each bit of data, I tried to log it to the console. However, rather than an array of objects with data, I'm getting an array of Promises. Nested in the promises, I can see my data, but I'm clearly missing a step in order to have an array of data objects, instead of just Promises.
What am I missing here?
//store movie API URLs into meaningful variables
const trending = `https://api.themoviedb.org/3/trending/all/day?api_key=${API_KEY}`;
const topRated = `https://api.themoviedb.org/3/movie/top_rated?api_key=${API_KEY}&language=en-US&page=1`;
const nowPlaying = `https://api.themoviedb.org/3/movie/now_playing?api_key=${API_KEY}&language=en-US&page=1`;
const upcoming = `https://api.themoviedb.org/3/movie/upcoming?api_key=${API_KEY}&language=en-US&page=1`;
//create an array of urls to fetch data from
const allMovieURLs = [trending, topRated, nowPlaying, upcoming];
const promiseURLs = allMovieURLs.map(url => fetch(url));
Promise.all(promiseURLs)
.then(responses => responses.map(url => url.json()))
.then(dataArr => console.log(dataArr));
};
Your .then(responses => responses.map(url => url.json())) resolves to an array of Promises, so you need to call Promise.all again if you want to wait for all to resolve:
Promise.all(promiseURLs)
.then(responses => Promise.all(responses.map(url => url.json())))
.then(dataArr => console.log(dataArr));
Or, you might consider using just one Promise.all, and having each URL fetch and the json, that way some items aren't idle in the middle of script execution:
const allMovieURLs = [trending, topRated, nowPlaying, upcoming];
const promiseURLs = allMovieURLs.map(url => fetch(url).then(res => res.json()));
Promise.all(promiseURLs)
.then(dataArr => console.log(dataArr));
try doing it this way
const promiseURLs = allMovieURLs.map(url => fetch(url).then(res => res.json()));
Promise.all(promiseURLs)
.then(responses => responses.forEach(response => { console.log(response)})

Categories