Been trying to fetch a link/image from an open API from the following guide: https://discordjs.guide/additional-info/rest-api.html#using-node-fetch
but it is not working. I keep getting an undefined response.
Already tried making async functions and so on but not getting any closer.
Also surrounded it by try-catch clausule to debug but not finding the answer.
module.exports = {
name: 'poes',
description: 'Laat een random poes foto zien',
async execute(message, args) {
const fetch = require('node-fetch');
const {body} = await fetch('https://aws.random.cat/meow').then(response => response.json());
message.channel.send(body.file);
},
};
And this is where it is used:
client.on('message', message => {
if (!message.content.startsWith(prefix) || message.author.bot) return;
const args = message.content.slice(prefix.length).split(/ +/);
const command = args.shift().toLowerCase();
if (!client.commands.has(command)) return;
try {
client.commands.get(command).execute(message, args);
} catch (error) {
console.error(error);
message.reply('there was an error trying to execute that command!');
}
}
);
Expected result following the Guide should be a random cat image.
The documentation you're using is incorrect in a couple of ways:
const {body} = await fetch('https://aws.random.cat/meow').then(response => response.json())
That line assumes fetch doesn't fail (with a 404, for instance). This is such a common mistake that I've written it up on my anemic little blog. fetch's promise only rejects on network errors, not HTTP errors. You have to check response.ok or response.status.
That the parsed result will have a body property.
It uses then in an async function, which makes very little sense.
But if I go to https://aws.random.cat/meow, I get this JSON:
{"file":"https:\/\/purr.objects-us-east-1.dream.io\/i\/img_20131111_094048.jpg"}
There's no body there, which is why you get undefined for it.
Here's an example fixing all three issues:
const response = await fetch('https://aws.random.cat/meow');
if (!response.ok) {
throw new Error("HTTP status " + response.status);
}
const body = await response.json();
// ^---^---- no { and }, we don't want to destructure
The response from the api is
{
"file": "https://purr.objects-us-east-1.dream.io/i/r958B.jpg"
}
And you are saying it is
{
"body" : {
"file" : ""
}
}
So you need to dump the brackets
const body = await fetch('https://aws.random.cat/meow')
.then(response => response.json());
or you need to look for file instead
const { file } = await fetch('https://aws.random.cat/meow')
.then(response => response.json());
console.log(file)
Related
I know this can be solved by writing all codes to async-await style, then can simply write let text = await res.text(); then try catch the JSON.parse(text) and then do decision.
But here I just want to know if there is any way we can achieve that in .then/.catch style.
Consider the below code:
async function test() {
try {
let n = await fetch("https://stackoverflow.com")
.then(res => {
return res.json()
})
.then(data => data.results.length)
.catch(e => {
console.error("Catch 2", e)
})
}
catch (e) {
console.error("Catch 3", e)
}
}
if we execute this function in the browser devtools(F12) with await test(), then there will be an error catch by the "Catch 2" clause. But in the error detail we can only see some logs like JSON parse error.
We cannot see the full text of the response body.
Is there any way that can get the text when the JSON parsing failed?
Your best bet is to look at the response in your devtools' network tab. That will show you the full response.
But if you want to do it in code, you can separate reading the response from parsing it by using the text method instead of the json method, then parsing the text yourself.
The parsing error may be down to the fact you aren't checking for HTTP success. As I noted on my old anemic blog here, fetch only rejects its promise on network errors, not HTTP errors (like 404, 500, etc.). To check for HTTP success, look at the ok or status properties.
Here's the minimal-changes version separating reading the response from parsing it, and checking for HTTP success before reading it at all:
async function test() {
try {
let n = await fetch("https://stackoverflow.com")
.then((res) => {
if (!res.ok) { // ***
throw new Error(`HTTP error ${res.status}`); // ***
} // ***
return res.text(); // ***
})
.then((text) => {
// *** you can look at `text` here in a debugger, or
// *** log it, save it, etc., before parsing below
// *** (which might throw an error)
try {
const data = JSON.parse(text); // ***
return data.results.length;
} catch (error) {
console.error("Parsing error", e);
console.error("Text we were parsing:", text);
}
})
.catch((e) => {
console.error("Catch 2", e);
});
// ...do something with `n`...
} catch (e) {
console.error("Catch 3", e);
}
}
But a couple of things there:
I wouldn't mix async/await with explicit promise callbacks like that.
With that and with your original code, errors will result in n receive the value undefined, because the catch handlers (and my new try/catch block in the then handler) don't return anything.
Instead:
async function test() {
try {
const res = await fetch("https://stackoverflow.com");
if (!res.ok) {
throw new Error(`HTTP error ${res.status}`);
}
const text = await res.text();
// *** you can look at `text` here in a debugger, or
// *** log it, save it, etc., before parsing below
// *** (which might throw an error)
try {
const data = JSON.parse(text);
const n = data.results.length;
// ...do something with `n`...
} catch (error) {
console.error("Parsing error", e);
console.error("Text we were parsing:", text);
}
} catch (e) {
console.error("Catch 3", e);
}
}
Or if you want to respond differently to the parsing error, wrap that bit in a try/catch, etc.
You shouldn't confuse the catch which catching errors in the fetch function itself - with the response errors
fetch("/developer.mozilla.org")
.then(res => {
if (!res.ok) {
console.log("there was an error also here") <= this code also runs
console.log("response is", res);
}
return res.json()
})
.then(data => data.results.length)
.catch(e => {
console.error("Catch 2", e);
})
In your case, you tried converting data -> JSON w/o success, it failed and dropped to the "catch" section.
but to inspect the response - you can dump it in the first section above where I added res.ok
I believe you could do something like this when using promise style Javascript:
const fetchDataPromise = () => {
fetch('https://stackoverflow.com').then((res) => {
res.json().then((jsonData) => {
console.log(jsonData)
}).catch((err) => {
console.error(err)
res.text().then((rawData) => {
console.log(rawData)
}).catch((err) => console.error(err))
})
})
}
Also more intuitive approach would be to use async/await (the trade-off is that you will have to do the API call again):
const fetchData = async () => {
try {
const res = await fetch('https://stackoverflow.com')
const jsonData = await res.json()
console.log(jsonData)
} catch (err) {
try {
console.error(err)
const res = await fetch('https://stackoverflow.com')
const rawData = await res.text()
console.log(rawData)
} catch (rawError) {
console.error(rawError)
}
}
}
I have a task that requires fetching api data, with the constraint of only one outstanding api request at a time. Must receive a response, or time out, before issuing the next one. Since fetch (or axios) returns a promise, I can’t figure out how to wait for each promise to fulfill before issuing the next fetch.
I'm handed a large array of api url's that must all be resolved in this one-at-a-time manner before continuing.
I’m using create-react-app’s bundled dev server, and Chrome browser.
Curiously, accomplishing this via a node script is easy, because ‘await fetch’ actually waits. Not so in my browser environment, where all the fetch requests blast out at once, returning promises along the way.
Here’s a simple loop that results in the desired behavior as a node script. My question is how to achieve this one-outstanding-request-at-a-time synchronous serialization in the browser environment?
const fetchOne = async (fetchUrl) => {
try {
const response = await fetch(fetchUrl, { // Or axios instead
"headers": {
'accept': 'application/json',
'X-API-Key': 'topSecret'
},
'method': 'GET'
})
const data = await response.json();
if (response.status == 200) {
return (data);
} else {
// error handling
}
} catch(error) {
// different error handling
}
}
const fetchAllData = async (fetchUrlArray) => {
let fetchResponseDataArray = new Array();
let fetchResponseDataObject = new Object(/*object details*/);
for (var j=0; j<fetchUrlArray.length; j++) { // or forEach or map instead
// Node actually synchronously waits between fetchOne calls,
// but react browser environment doesn't wait, instead blasts them all out at once.
// Question is how to achieve the one-outstanding-request-at-a-time synchronous
// serialization in the browser environment?
fetchResponseDataObject = await fetchOne(fetchUrlArray[j]);
fetchResponseDataArray.push(fetchResponseDataObject);
}
return(fetchResponseDataArray);
}
If there's a problem, it's with code you haven't shown (perhaps in one of your components, or maybe in your project configuration).
Here's an runnable example derived from the problem you described, which mocks fetch and an API, showing you how to iterate each network request synchronously (and handle potential errors along the way):
Note, handling potential errors at the boundaries where they might occur is a better practice than only having a top level try/catch: by doing so, you can make finer-grained decisions about what to do in response to each kind of problem. Here, each failed request is stored as [url, error] in a separate array so that you can programmatically make decisions if one or more requests failed. (Maybe you want to retry them in a subsequent step, or maybe you want to show something different in the UI, etc.). Note, there's also Promise.allSettled(), which might be useful to you now or in the future.
<div id="root"></div><script src="https://unpkg.com/react#17.0.2/umd/react.development.js"></script><script src="https://unpkg.com/react-dom#17.0.2/umd/react-dom.development.js"></script><script src="https://unpkg.com/#babel/standalone#7.16.4/babel.min.js"></script>
<script type="text/babel" data-type="module" data-presets="env,react">
const {useEffect, useState} = React;
const successChance = {
fetch: 0.95,
server: 0.95,
};
function mockApi (url, chance = successChance.server) {
// Simulate random internal server issue
const responseArgs = Math.random() < chance
? [JSON.stringify({time: performance.now()}), {status: 200}]
: ['Oops', {status: 500}];
return new Response(...responseArgs);
}
function mockFetch (requestInfo, _, chance = successChance.fetch) {
return new Promise((resolve, reject) => {
// Simulate random network issue
if (Math.random() > chance) {
reject(new Error('Network error'));
return;
}
const url = typeof requestInfo === 'string' ? requestInfo : requestInfo.url;
setTimeout(() => resolve(mockApi(url)), 100);
});
}
// Return an object containing the response if successful (else an Error instance)
async function fetchOne (url) {
try {
const response = await mockFetch(url);
if (!response.ok) throw new Error('Response not OK');
const data = await response.json();
return {data, error: undefined};
}
catch (ex) {
const error = ex instanceof Error ? ex : new Error(String(ex));
return {data: undefined, error};
}
}
async function fetchAll (urls) {
const data = [];
const errors = [];
for (const url of urls) {
const result = await fetchOne(url);
if (result.data) data.push([url, result.data]);
else if (result.error) {
// Handle this however you want
errors.push([url, result.error]);
}
}
return {data, errors};
}
function Example () {
const [data, setData] = useState([]);
const [loading, setLoading] = useState(false);
useEffect(() => {
const fetchData = async () => {
setLoading(true);
try {
const {data, errors} = await fetchAll([
'https://my.url/api/0',
'https://my.url/api/1',
'https://my.url/api/2',
'https://my.url/api/3',
'https://my.url/api/4',
'https://my.url/api/5',
'https://my.url/api/6',
'https://my.url/api/7',
'https://my.url/api/8',
'https://my.url/api/9',
]);
setData(data);
}
catch (ex) {
console.error(ex);
}
setLoading(false);
};
fetchData();
}, []);
return (
<div>
<div>Loading: {loading ? '...' : 'done'}</div>
<ul>
{
data.map(([url, {time}]) => (<li
key={url}
style={{fontFamily: 'monospace'}}
>{url} - {time}</li>))
}
</ul>
</div>
);
}
ReactDOM.render(<Example />, document.getElementById('root'));
</script>
I'm trying to write a test for the sad path of this function:
const awaitFirstStreamForPage = async page => {
try {
await page.waitForSelector('[data-stream="true"]', {
timeout: MAX_DELAY_UNTIL_FIRST_STREAM,
})
} catch (e) {
throw new Error(`no stream found for ${MAX_DELAY_UNTIL_FIRST_STREAM}ms`)
}
}
I managed to write a test that passes, but it takes 10 seconds to run because it actually waits for the test to finish.
describe('awaitFirstStreamForPage()', () => {
it('given a page and no active stream appearing: should throw', async () => {
jest.setTimeout(15000)
const browser = await puppeteer.launch({ headless: true })
const page = await getPage(browser)
let error
try {
await awaitFirstStreamForPage(page)
} catch (err) {
error = err
}
const actual = error.message
const expected = 'no stream found for 10000ms'
expect(actual).toEqual(expected)
await browser.close()
jest.setTimeout(5000)
})
})
There is probably a way to solve it using Jest's fake timers, but I couldn't get it to work. Here is my best attempt:
const flushPromises = () => new Promise(res => process.nextTick(res))
describe('awaitFirstStreamForPage()', () => {
it('given a page and no active stream appearing: should throw', async () => {
jest.useFakeTimers()
const browser = await puppeteer.launch({ headless: true })
const page = await getPage(browser)
let error
try {
awaitFirstStreamForPage(page)
jest.advanceTimersByTime(10000)
await flushPromises()
} catch (err) {
error = err
}
const actual = error.message
const expected = 'no stream found for 10000ms'
expect(actual).toEqual(expected)
await browser.close()
jest.useRealTimers()
})
})
which fails and throws with
(node:9697) UnhandledPromiseRejectionWarning: Error: no stream found for 10000ms
Even though I wrapped the failing function in a try/catch. How do you test a function like this using fake timers?
It's impossible to catch a rejection from awaitFirstStreamForPage(page) with try..catch if it's not awaited.
A rejection should be caught but after calling advanceTimersByTime and potentially after flushPromises.
It can be:
const promise = awaitFirstStreamForPage(page);
promise.catch(() => { /* suppress UnhandledPromiseRejectionWarning */ });
jest.advanceTimersByTime(10000)
await flushPromises();
await expect(promise).rejects.toThrow('no stream found for 10000ms');
The problem doesn’t seem to be the use of fake timers: the error you expected is the one being thrown. However, when testing functions that throw errors in Jest, you should wrap the error-throwing code in a function, like this:
expect(()=> {/* code that will throw error */}).toThrow()
More details here: https://jestjs.io/docs/en/expect#tothrowerror
Edit: For an async function, you should use rejects before toThrow; see this example: Can you write async tests that expect toThrow?
I am learning to use async / await and am having issues when trying to make an api request to set up my twilio device. Given the below code block, when I call device(), I get the following error message:
Uncaught
TwilioException {message: "Capability token is not valid or missing."}
message
:
"Capability token is not valid or missing."
__proto__
:
Object
I believe this is due to the fact that the json returned in the device function is still shown as pending. How do I resolve this and what am I doing wrong? Thanks.
Code block:
import {Device} from 'twilio-client';
const api = async () => {
try {
const response = await fetch('/api/twilio');
const json = await response.json();
if (response.status === 403) {
twilioConnectionFailure('Twilio has not been purchased.');
}
return json;
} catch (error) {
console.log(`Connection failed: ${error.message}`);
throw Error(error.message);
}
};
const device = () => {
const json = api();
Device.setup(json.token);
return Device;
};
export default device;
The api function is still asynchronous and returns a promise - you need to wait for it:
export default async function device() {
const json = await api();
// ^^^^^
Device.setup(json.token);
return Device;
}
tl;dr - if you have to filter the promises (say for errored ones) don't use async functions
I'm trying to fetch a list of urls with async and parse them, the problem is that if there's an error with one of the urls when I'm fetching - let's say for some reason the api endpoint doesn't exists - the program crushes on the parsing with the obvious error:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: ext is not iterable
I've tried checking if the res.json() is undefined, but obviously that's not it as it complains about the entire 'ext' array of promises not being iterable.
async function fetchAll() {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
Question 1:
How do I fix the above so it won't crash on an invalid address?
Question 2:
My next step is to write the extracted data to the database.
Assuming the data size of 2-5mgb of content, is my approach of using Promise.all() memory efficient? Or will it be more memory efficient and otherwise to write a for loop which handles each fetch then on the same iteration writes to the database and only then handles the next fetch?
You have several problems with your code on a fundamental basis. We should address those in order and the first is that you're not passing in any URLS!
async function fetchAll(urls) {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
First you have several try catch blocks on DEPENDANT DATA. They should all be in a single try catch block:
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url=>fetch(url)))
let ext = await Promise.all(data.map(res => {
// also fixed the ==! 'undefined'
if (res.json() !== undefined) { return res.json()}
}))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Next is the problem that res.json() returns a promise wrapped around an object if it exists
if (res.json() !== undefined) { return res.json()}
This is not how you should be using the .json() method. It will fail if there is no parsable json. You should be putting a .catch on it
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url => fetch(url).catch(err => err)))
let ext = await Promise.all(data.map(res => res.json ? res.json().catch(err => err) : res))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now when it cannot fetch a URL, or parse a JSON you'll get the error and it will cascade down without throwing. Now your try catch block will ONLY throw if there is a different error that happens.
Of course this means we're putting an error handler on each promise and cascading the error, but that's not exactly a bad thing as it allows ALL of the fetches to happen and for you to distinguish which fetches failed. Which is a lot better than just having a generic handler for all fetches and not knowing which one failed.
But now we have it in a form where we can see that there is some better optimizations that can be performed to the code
async function fetchAll(urls) {
try {
let ext = await Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.catch(error => ({ error, url }))
)
)
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now with a much smaller footprint, better error handling, and readable, maintainable code, we can decide what we eventually want to return. Now the function can live wherever, be reused, and all it takes is a single array of simple GET URLs.
Next step is to do something with them so we probably want to return the array, which will be wrapped in a promise, and realistically we want the error to bubble since we've handled each fetch error, so we should also remove the try catch. At that point making it async no longer helps, and actively harms. Eventually we get a small function that groups all URL resolutions, or errors with their respective URL that we can easily filter over, map over, and chain!
function fetchAll(urls) {
return Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.then(data => ({ data, url }))
.catch(error => ({ error, url }))
)
)
}
Now we get back an array of similar objects, each with the url it fetched, and either data or an error field! This makes chaining and inspecting SUPER easy.
You are getting a TypeError: ext is not iterable - because ext is still undefined when you caught an error and did not assign an array to it. Trying to loop over it will then throw an exception that you do not catch.
I guess you're looking for
async function fetchAll() {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
const ext = await Promise.all(data.map(res => res.json()));
for (let item of ext) {
console.log(item);
}
} catch (err) {
console.log(err);
}
}
Instead of fetch(url) on line 5, make your own function, customFetch, which calls fetch but maybe returns null, or an error object, instead of throwing.
something like
async customFetch(url) {
try {
let result = await fetch(url);
if (result.json) return await result.json();
}
catch(e) {return e}
}
if (res.json()==! 'undefined')
Makes no sense whatsoever and is an asynchronous function. Remove that condition and just return res.json():
try {
ext = await Promise.all(data.map(res => res.json()))
} catch (err) {
console.log(err)
}
Whether or not your approach is "best" or "memory efficient" is up for debate. Ask another question for that.
You can have fetch and json not fail by catching the error and return a special Fail object that you will filter out later:
function Fail(reason){this.reason=reason;};
const isFail = o => (o&&o.constructor)===Fail;
const isNotFail = o => !isFail(o);
const fetchAll = () =>
Promise.all(
urls.map(
url=>
fetch(url)
.then(response=>response.json())
.catch(error=>new Fail([url,error]))
)
);
//how to use:
fetchAll()
.then(
results=>{
const successes = results.filter(isNotFail);
const fails = results.filter(isFail);
fails.forEach(
e=>console.log(`failed url:${e.reason[0]}, error:`,e.reason[1])
)
}
)
As for question 2:
Depending on how many urls you got you may want to throttle your requests and if the urls come from a large file (gigabytes) you can use stream combined with the throttle.
async function fetchAll(url) {
return Promise.all(
url.map(
async (n) => fetch(n).then(r => r.json())
)
);
}
fetchAll([...])
.then(d => console.log(d))
.catch(e => console.error(e));
Will this work for you?
If you don't depend on every resource being a success I would have gone back to basics skipping async/await
I would process each fetch individual so I could catch the error for just the one that fails
function fetchAll() {
const result = []
const que = urls.map(url =>
fetch(url)
.then(res => res.json())
.then(item => {
result.push(item)
})
.catch(err => {
// could't fetch resource or the
// response was not a json response
})
)
return Promise.all(que).then(() => result)
}
Something good #TKoL said:
Promise.all errors whenever one of the internal promises errors, so whatever advice anyone gives you here, it will boil down to -- Make sure that you wrap the promises in an error handler before passing them to Promise.all
Regarding question 1, please refer to this:
Handling errors in Promise.all
Promise.all is all or nothing. It resolves once all promises in the array resolve, or reject as soon as one of them rejects. In other words, it either resolves with an array of all resolved values, or rejects with a single error.