Issue
I need to download query results from an endpoint by streaming results to a CSV file. This is in an effort to support enormous ResultSets being sent through the browser at one time.
Is there a way to accomplish this using Axios in the context of a React App?
I have seen fetch() and know that it has the following characteristics:
returns ReadableStream
Is NOT supported by IE11
Does NOT allow for intercepting requests
The status of a response relates to the request itself, not the HTTP status
This means that the only way to receive an error would be to have something go wrong with the stream ending prematurely
This definitely won't work for me since I have custom error-handling related to user permissions
Besides the ReadableStream response type, the rest of the characteristics listed are not permissible. I will need to support IE11 and allow for intercepting requests / reading the HTTP status to determine how to handle the traffic.
Example with fetch:
// The promise returned by `fetch` rejects if the fetch was unable to make HTTP-request
// e.g. network problems, or there’s no such site.
// Abnormal HTTP-statuses, such as 404 or 500 do not cause an error.
const results = await fetch(`${URL}/data`, {
method: 'post', // HTTP POST to send query to server
headers: {
Accept: 'application/json, text/plain, */*', // indicates which files we are able to understand
'Content-Type': 'application/json', // indicates what the server actually sent
},
body: JSON.stringify(query), // server is expecting JSON
credentials: 'include', // sends the JSESSIONID cookie with the address
}).then(res => res.json()) // turn the ReadableStream response back into JSON
.then((res) => {
if (res.ok) {
// boolean, true if the HTTP status code is 200-299.
console.log('response.ok!');
} else if (res.status === 401) {
throw Error(`You are not authenticated. Please login.`);
} else if (res.status === 403) {
throw Error(`You are not authorized to access this data.`);
} else {
throw Error(`Request rejected with status ${res.status}`);
}
})
.catch((error) => {
// catches error case and if fetch itself rejects
error.response = {
status: 0,
statusText:
'Cannot connect. Please make sure you are connected to internet.',
};
throw error;
});
console.log(results);
Example with axios (not streaming)
Axios instance
import ...
const Api = axios.create({
baseURL: `${URL}`,
withCredentials: true,
});
// attach interceptors to requests and responses
// these are defined elsewhere and imported
Api.interceptors.request.use((request) => requestHandler(request));
Api.interceptors.response.use((response) => successHandler(response), (error) => errorHandler(error));
export default Api;
Axios request
const query = {"selections":{"TABLE_A":["COLUMN1"]},"filters":[{"predicates":[]}],"joins":[],"sorts":[],"limit":100,"offset":0}
const response = await Api.post('/data', query);
// further transformations to response to get formatted csv results required
Questions about Axios
Is it possible to have a ReadableStream in Axios same as fetch?
Is streaming in Axios only possible when assuming that it will be supported by Node in a server-side only setting?
Sites like this appear to say that using responseType: 'stream' isn't something that can be done in the browser, only with Node.js using fs
Is it possible to use fetch or something else in conjunction with Axios?
Streaming a response from the browser is not currently supported :
https://github.com/axios/axios/issues/479
Since we're dealing with XMLHttpRequests in the browser, Axios is limited to the specification set by whatwg. :
https://xhr.spec.whatwg.org/#interface-xmlhttprequest
https://github.com/whatwg/xhr
Specifically, these are the only supported types :
enum XMLHttpRequestResponseType {
"",
"arraybuffer",
"blob",
"document",
"json",
"text"
};
stream is accepted when setting a responseType in axios, but this is misleading. The adapter is going to be xhr.js implicitly since we are using the browser which relies on XMLHttpRequests. HttpRequests are made on the server-side and will allow axios to use the http.js adapter. THEN you can use stream as a ResponseType with Node.js.
Using the fetch API seems to be the only solution with a ReadableStream as a response body type.
If you just need to download a file, using blob in the responseType options is definitely okay.
axios.post(url, param,
{ header: {...}, responseType: 'blob' }
)
.then(res => {
const link = document.createElement('a');
link.href = URL.createObjectURL(res);
link.click();
})
Related
I have an axios request interceptor that adds bearer authorization via an access token. I also have a response interceptor that catches 'token expired' responses, gets a new token via a refresh mechanism, and retries the original request. This seems like it should work, except when the original request is retried, it seems to have lost all its headers. This confuses my API backend as it expects a Content-Type (which was there in the original request).
Note: I'm familiar with the large number of questions about why axios doesn't respect a custom Content-Type header. That's not my problem -- I'm not setting one -- axios does a fine job determining that on its own. I'm just confused at why the Content-Type header that axios sets itself is getting removed when resending the original request.
Relevant code:
const api = axios.create({
baseURL: "https://www.mycoolapi.com",
timeout: 5000,
withCredentials: true,
});
api.interceptors.request.use((config) => {
// Add the auth header (not messing with any other headers)
config.headers!.Authorization = `Bearer ${accessToken}`;
return config;
}, (error) => Promise.reject(error));
api.interceptors.response.use((response) => response,
async (error) => {
if (error.response.status == 401 && refreshToken && !error.config._isRetry) {
// Token expired? Refresh, then retry the original request
error.config._isRetry = true;
await refreshLogIn();
return api(error.config);
}
throw error;
});
async function refreshLogIn() {
// Use the refresh token to get a new token pair
const response = await api.post( "/token",
new URLSearchParams({
grant_type: "refresh_token",
refresh_token: refreshToken,
})
);
accessToken = response.data["access_token"];
refreshToken = response.data["refresh_token"];
}
If I put a breakpoint on return api(error.config) in the response interceptor, and inspect the original request (which is in error.config), I get the following headers (reminder, I added only the Authorization header, the rest are axios defaults or calculated from the requeset):
If I then step through to the request interceptor where the original response is being retried, I get these headers instead:
Content-Type has been removed, along with what was in Symbol(defaults) (whatever that is).
What am I missing here?
As #Phil noted in the comments, this is a bug in axios >= 1.0.0. I suppose one could downgrade to 0.27.2, but I just rewrote using good old fetch. My use case was simple enough anyway, and one less dependency is never a bad thing.
I am new to javascript. I was trying to make an api call.
My code
const options = {
method: 'GET',
headers: {
Authorization: 'Basic dW5kZWZpbmVkOnVuZGVmaW5lZA==',
'content-type': 'application/json',
}
};
fetch(
'https://www.eraktkosh.in/BLDAHIMS/bloodbank/nearbyBB.cnt?hmode=GETNEARBYSTOCKDETAILS&stateCode=21&districtCode=378&bloodGroup=all&bloodComponent=11&lang=0',
options
)
.then((response) => response.json())
.then((response) => console.log(response))
.catch((err) => console.error(err));
but I encountered with an error saying
Error: Failed to fetch
This api call works perfectly with Hoppscotch
If I try to hit the url right on my url bar, it also works fine.
Any help is strongly appreciated. Thank you from Manoranjan
As other People already mentioned, you can't pass a Body when doing a GET HTTP call, instead you can pass Query Params
Notice this part on the URL
hmode=GETNEARBYSTOCKDETAILS&stateCode=21&districtCode=378&bloodGroup=all&bloodComponent=11&lang=0
Still looking into the code it seems the server have a cors policy, look at this sandbox
See this codesandbox -> https://codesandbox.io/s/peaceful-mcclintock-exuzol?file=/src/index.js
Summary:
GET accept body/payload but it could cause errors, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/GET
Using the Web API (new headers, new request) for doing the HTTP call
It is better to just avoid sending payloads in GET requests.
Please don't use body with a get request. The GET request is purely meant to collect back data from server, which allows you to sent Queries, not data on the request. Just remove body:'false' or use body:false. The best way is to remove the body from your request so unexpected input is not sent via this GET request.
I understand that there are many similar questions, but I am posting this because I feel it is slightly different.
I am trying to send a GET request to the Slack API using an HTTP request.
Specifically, the code looks like the following.
import useSWR from "swr";
const useSlackSearch = (query: string) => {
const token = process.env.NEXT_PUBLIC_SLACK_API_USER_TOKEN;
const myHeaders = new Headers();
myHeaders.append("Authorization", "Bearer " + token);
const slackURL = `https://slack.com/api/search.messages?query=${query}`;
const fetcher = async (url: string) => {
const response = await fetch(url, {
headers: myHeaders,
}).then((res) => res.json());
return response;
};
const { data, error } = useSWR(slackURL, fetcher, {
revalidateOnFocus: true,
revalidateOnReconnect: true,
});
if (error) {
return console.log(`Failed to load: ${error}`);
} else if (!data) {
return console.log("Loading...");
} else {
console.log(data);
return data;
}
};
export default useSlackSearch;
The environments I'm using are as follows.
Device: MacBook Air
OS: macOS
Browser: Chrome
From: localhost:3000
To: Slack API html page (https://slack.com/api/search.messages)
After reading the MDN articles like below, I understood that
There is such a thing as a simple HTTP request as defined by MDN
If the request you want to send does not correspond to this simple request, the browser will send a preflight request
In the response to that preflight request, there is a header called Access-Control-Allow-Headers.
Only headers set to the value of this Access-Control-Allow-Headers header can be used as headers in the main request after preflighting.
In this case, I tried to use the Authorization header, but it was trapped by the above restriction.
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests
https://developer.mozilla.org/en-US/docs/Glossary/Preflight_request
That's all I understand.
However, on the official Slack API page for the method in question, it says to specify the token in the Authorization header, so I'm having trouble.
I also don't understand how to specify the Access-Control-Request-Headers in the preflight header, as described in another questioner's thread. The reason is that the only thing that communicates to the Slack API is the browser in this case, and the only relevant source is JavaScript (React / Next.js to be exact)!
After that, I found preflight response from Slack API as follows;
access-control-allow-headers: slack-route, x-slack-version-ts, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags
As I thought, I understand that Authorization is not allowed because it is not included as a value. So the question is how to solve it.
Furthermore, I found out later that the preflight request from the browser properly declared that it wanted to use Authorization as an actual request header. However, the preflight response did not contain the value.
Following CBroe's advice, I was able to contact the Slack help center directly, so I asked this problem. What I found out as a result is that HTTP requests from browsers are not supported as of the end of February 2022. Of course, they have received quite a lot of requests regarding this, so they hope to address it at some point.
This time, the browser sent Access-Control-Request-Headers:Authorization in the preflight request. But the Slack API server side did not allow the Authorization header in the request from the browser. Therefore, Authorization was not set in the Access-Control-Allow-Headers in the preflight response from the Slack API side.
As a result, the response from the Slack API side returned Invalid Auth, even though Authorization was added as a header when making an actual request from the browser.
Through this error, I gained a deeper understanding of HTTP requests such as CORS and preflighting, but since it is not explicitly written on the official Slack website, I left it here.
What is Preflight: https://developer.mozilla.org/en-US/docs/Glossary/Preflight_request
What is Access-Control-Allow-Header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers
What is CORS simple request: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests
I could not get the Authorization header to work either. However, Slack provided this example for adding token authentication to the Post body following the deprecation of the query parameters method.
This worked for me to make Web API calls to Slack from the browser (for testing) so that Slack would read the token for authentication. Note, according to Slack's best practices for security, user and bot tokens should be stored with care and not used in client-side Javascript:
try {
const res = await fetch("https://slack.com/api/conversations.list", {
method: "POST",
body: `token=${TOKEN}`, // body data type must match "Content-Type" header
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
}).catch((error) => {
console.log(error);
});
if (!res.ok) {
throw new Error(`Server error ${res.status}`);
} else {
const data = await res.json();
console.log(data);
}
} catch (error) {
console.log(error);
}
using token in request body instead of Authorization header worked for me.
axios({
method: 'post',
url: 'https://slack.com/api/chat.postMessage',
data: `text=Hi&channel=D048GGYTJUK&token=${process.env.TOKEN}`
})
I'm in the process of attempting to verify a JWT access_token against OneLogin's api as described here. My code is as follows:
const client_id = MY_CLIENT_ID
const client_secret = MY_CLIENT_SECRET
const token = MY_ONE_LOGIN_JWT_ACCESS_TOKEN
axios
.post(
"https://my-endpoint-dev.onelogin.com/oidc/2/token/introspection",
{ token, client_id, client_secret, token_type_hint: "access_token" },
{
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
}
)
.then((response) => {
console.log("response");
console.log(response);
})
.catch((err) => {
console.log("err");
console.log(err);
});
The endpoint appears to work fine, in fact when the JWT has become expired it gives me an error stating as such and I need to update the token I'm passing along. However, whenever I make a standard request as shown above with valid credentials and tokens I get the following error response:
{error: "invalid_request", error_description: "no client authentication mechanism provided"}
There's no documentation on the provided page that describes what is wrong with the request when that error is received. From the documentation, so far as I can tell, my request is formatted correctly.
I have verified that the Token Endpoint in OneLogin is set to POST, so my assumption that the client_secret should be in the body is documented as correct (though I did try it as Basic just to verify):
I've attempted searching for a solution, but the only thing close I've found advises that the Content-Type header may not be supplied. I've made sure to add that to the list of headers and have verified it shows up in the request, but still the error persists.
Any thoughts to what I may be missing here?
EDIT:
Attempted to do a cURL request and received a 200 response back with the same information. Leading me to believe it's something with the axios call that I have incorrect.
I get this message when I don't provide either the client_id or the client_secret. Hopefully you can validate that you are actually sending both in your request. Maybe you can try the request via postman to double check.
I ran into the same issue and finally figured out you have to turn the data into a query string: https://axios-http.com/docs/urlencoded
For example:
import qs from 'qs';
const data = { 'bar': 123 };
const options = {
method: 'POST',
headers: { 'content-type': 'application/x-www-form-urlencoded' },
data: qs.stringify(data),
url,
};
axios(options);
I are trying to fetch data from our API. The API has enabled CORS support and returns the below response to the OPTIONS request:
Access-Control-Request-Headers:content-type
Access-Control-Allow-Origin:*
The API doesn't allow 'Content-type' anything other than 'application/json'.
Using this limitation, I am trying to use the fetch method of React-Native to get the data.
Method 1 (no-cors):
{
method: 'POST',
mode: "no-cors",
headers: {
'content-type': 'application/json'
}
With this method, the browser automatically sends the content-type as 'text/plain'. I assume this is because CORS allow just one of the three headers by default. However, since the server doesn't support this content-type, it returns an error back for unsupported content type.
Method 2 (with cors or with nothing):
{
method: 'POST',
mode: "cors", // or without this line
redirect: 'follow',
headers: {
'content-type': 'application/json'
}
}
...
.then(response => console.log(response))
In this scenario, using Chrome's F12 network tool, I can see the server returning data : the first request to the server is a fetch for OPTIONS. To this, the server replies back with an empty object along with the above headers set. The next call is the actual POST API call, to which the server responds back with a proper JSON response containing some data. However, the response which is getting on the console via my code is {}. I assume this is because the react's fetch API is returning back the response of the OPTIONS call instead of the actual POST call.
Is there any way to ignore the response of the OPTIONS request and get the then method to process the response of the subsequent request?
The immediate problem you’re hitting is, your code as currently written expects the response to be JSON but the response is actually a Promise that you need to handle to get the JSON.
So you need to instead do something like this:
fetch("https://example.com")
.then(response => response.json())
.then(jsondata => console.log(jsondata))