Downloading large files with axios - javascript

I am currently downloading large files to my client code using the axios lib. Files are around 600MB. But during the download process the page crashes, like it runs out of memory or similar.
I need to hold the file in the memmory because the content is encrypted and I need to decrypt it before passing it to the user.
I use the REST GET Http request like this:
axios.get(url, {
headers: {
"Authorization": authHeader().Authorization,
"Accept" : "application/octet-stream, application/json, text/plain, */*"
}, responseType: 'arraybuffer'
})
.then(function(response) {
console.log(response);
Are there any common workaround around the problem. So far I wasn't able to find any.

Open the url in the new tab on the client side using
window.open(url)
Let the browser handle the document automatically,
If you want to decrypt the data, please try to decrypt on server side since you'll be giving out decryption key on the client side, which can have security issues.

Do you actually need to do it with axios? There is the Fetch API, which can serve the purpose. Here's how I do it for files in the same size range as yours (media files and ZIPs of up to 1 GB):
fetch(url, {
mode: 'no-cors', // to allow any accessible resource
method: 'GET',
})
.then((response) => {
console.debug('LOAD_FROM_URL::response', response);
//NOTE: response URL is possibly redirected
See https://github.com/suterma/replayer-pwa/blob/main/src/store/actions.ts for more context.
It's working flawless so far for me.

Can you please give a try by setting maxContentLength and maxBodyLength to Infinity in the axios call.
axios.get(url, {
headers: {
"Authorization": authHeader().Authorization,
"Accept" : "application/octet-stream, application/json, text/plain, */*"
},
responseType: 'arraybuffer',
maxContentLength: Infinity,
maxBodyLength: Infinity
})
.then(function(response) {
console.log(response);
}
You can also have a look into this axios issue forum for the same.

HTTP Protocol: Range Parameter
Use the 'Range' header for getting big files part by part.
Example Curl HTTP Request : https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
curl http://i.imgur.com/z4d4kWk.jpg -i -H "Range: bytes=0-1023"
Example JS
axios.get(url, {
headers: {
"Authorization": authHeader().Authorization,
"Accept" : "application/octet-stream, application/json, text/plain, */*",
//--------------------------|
"Range" : "bytes=0-1023" // <------ ADD HERE
//--------------------------|
},
responseType: 'arraybuffer'
}).then(function(response) {
console.log(response);
}

You need to revisit the design of your frontend. Either its ReactJs or Angular or plain JS, just Offload such large downloads as well as other long running scritps to a concept of JavaScript Web Workers. This is latest way to offloading javascript tasks to the background so the page responsiveness remains intact.
Otherwise JavaScript is single threaded so there is no way to bypass the issue. WebWorkers are only thing to make exceptions here
It is as of today now supported on all latest browsers.
More on this: link

Related

not able to fetch remote json link from local html file in chrome

I want to fetch a json from link [https://www.nseindia.com/api/equity-stockIndices?index=NIFTY%2050]
When I open in browser it dumps the json on screen; good as expected but when I try to fetch this link from javascript it gives me cors error.
Please help me by giving a working solution. I am not a pro programmer.
Preferably I don't want a server in middle. Server solution is considered as last option if not possible without it. Currently I am using nodejs server in middle.
Error I get when I run code
I tried with this code in javascript
fetch('https://www.nseindia.com/api/equity-stockIndices?index=NIFTY%2050', {
method: 'GET',
headers: {
'Accept': 'application/json',
},
})
.then(response => response.json())
.then(response => console.log(JSON.stringify(response)))
This is due to cors header missing. CORS stands for cross orging resource sharing. This header tells the server/backend to permit any origins (like www.google.com) except its own from which a browser should permit loading resources.
Here is the fix to your code
fetch('https://www.nseindia.com/api/equity-stockIndices?index=NIFTY%2050', {
method: 'GET',
headers: {
'Accept': 'application/json',
'Access-Control-Allow-Origin':'*',
},
})
.then(response => response.json())
.then(response => console.log(JSON.stringify(response)))

Postman request works but not axios CORS

In a bit of a pickle at the moment , I could do a postman request like that and I get my data response back :
URL : https://hiddenurlforexample.com
Authorization : Bearer XXXXXXXX-XXXX-XXXX-XXXX
When I do it on Axios on my website though I get a 401 CORS error. Any idea what the difference is ? This is how my axios request looks like :
axios
.request({
url: 'test/url',
method: 'get',
baseURL: 'https://hiddenurlforexample.com',
headers: {
"Access-Control-Allow-Origin" : "*",
"Content-type": "Application/json",
"Authorization": "Bearer XXXXXXXX-XXXX-XXXX-XXXX"
}
})
.then(response => {
console.log(response.data)
})
.catch(function (error) {
console.log(error)
})
I am a Frontend Developer, I have been told that there was nothing to do in the backend .
What Chris G said, and next to that Postman ignores the CORS validation because it is a dev tool.
Your backend server should return the correct CORS headers. While developing you could go with a wildcard for the CORS headers but it's highly recommended to add the specific domain you're calling the backend from (i.e. the domain of your website).
Also note that the CORS headers are returned via an OPTIONS call, again, your backend should support that. What backend are you running?

Adding headers to fetch, solving cors, running cryptojs client side

I'm working on a project where I need first to get a authentication token from a server.
For this I need to send a GET request to the authorisation server with two key-value pairs included in the header: 1/ the client id (const) and 2/ a HMAC SHA1 calculated value based on client ID timestamp and so on.
This is working fine with Postman. (I calculate the sha1 on an online calculator)
Problem 1: (cryptojs client side)
As a node app I included the cryptojs library and the calculation works. But even with RequireJS I can not get cryptojs to run in the browser.
Error: Module name "crypto-js" has not been loaded yet for context: _. Use require([])
Problem 2: (cors)
Apparently chrome refuses the connection as the server does not accept all incoming connections.
Adding mode: 'no-cors' to the fetch request does not solve the problem.
Problem 3: (headers)
I need to add two key - value pairs to the get request headers. In postman this is no problem but I'm not sure this works with append or just adding them to my headers: { }
I constantly get a server error as if no headers where added.
I have already tried REquireJS for the cryptojs problem.
I have added the headers to a myHeaders object
const myHeaders = new Headers();
myHeaders.append('ClientID', CLIENTID);
myHeaders.append('Clientsecret', hashedToken);
and also just added the values to:
headers: {
...
'ClientID': CLIENTID,
'Clientsecret': hashedToken,
}
Both don't seem to help.
My code:
function getToken(){
hashedToken = getHashedSecret(); //this won't work client side as cryptojs can not be loaded
const CLIENTID = "CLIENTID";
const AUTHURL = "https://authorization.server.com/api/CLIENTID/authorization/";
var TIMESTAMP = getTimeStamp();
const myHeaders = new Headers();
// myHeaders.append('Content-Type', 'application/json');
myHeaders.append('ClientID', CLIENTID);
myHeaders.append('Clientsecret', hashedToken);
console.log(myHeaders);
let response = fetch(AUTHURL+TIMESTAMP, {
method: 'GET',
headers: {
myHeaders,
'Accept': 'application/json',
'Content-Type': 'application/json',
'Origin': '',
'Host': 'authorization.server.com',
include: 'ClientID', CLIENTID
},
mode: 'no-cors',
})
.then(response => response.json())
.then(data => {
console.log(data);
document.getElementById('output').innerHTML = data;
})
.catch(error => console.error(error))
console.log('data');
return data;
}
I should get a token from the server
It sounds like https://authorization.server.com doesn't allow access from your page's origin. Remember that in browsers, the Same Origin Policy prevents scripts from one origin from requesting information from other origins by default. (postman, not being a browser, is not subject to this restriction). This is so that scripts on Site A (a malicious actor) can't steal your personal information from Site B (perhaps your online banking) by making requests (from your browser, thus with your authentication information) to Site B.
For this to work, server code at https://authorization.server.com will need to respond to requests from the browser using the Cross-Origin Resource Sharing to allow access from your origin. You cannot do it from your client-side code (for obvious reasons).
Alternately, you can run a server on your origin, make the requests to that server, and it can make the requests to https://authorization.server.com and pass back the responses to you. Your server, not being a browser, is not subject to the SOP.

Request JS Uri requesting incorrectly

I am trying to POST to a server using Request JS. I am having issues with parts of the path.
return await request.get({
method: 'POST',
uri: `${domain}/info/test/`,
body: bodyAsString,
headers: {
Authorization: `bearer ${token}`,
},
}).pipe(res);
This api has two ways to hit it: /info/test as a POST and /info/test/:GUID as a GET. For some reason, when I hit the API with Request it complains that 'test' is not a valid GUID like I am one level too low on the path. I can hit it in Postman just fine.
you should use request.post instead of request.get

Uncaught (in promise) TypeError: Failed to fetch and Cors error

having a problem with getting data back from database. I am trying my best to explain the problem.
1.If I leave "mode":"no-cors" inside the code below, then I can get data back from server with Postman, but not with from my own server. Thinking it has to be my client side error
When I remove "mode":"no-cors" then I am getting 2 errors:
-Fetch API cannot load http://localhost:3000/. Request header field access-control-allow-origin is not allowed by Access-Control-Allow-Headers in preflight response.
-Uncaught (in promise) TypeError: Failed to fetch
Quick Browsing suggested to put in the "mode":"no-cors" which fixed this error, but it does not feel right thing to do.
So I thought maybe somebody has a suggestion how to approach this problem.
Really hope I was clear enough, but pretty sure I am not giving clear explanation here :S
function send(){
var myVar = {"id" : 1};
console.log("tuleb siia", document.getElementById('saada').value);
fetch("http://localhost:3000", {
method: "POST",
headers: {
"Access-Control-Allow-Origin": "*",
"Content-Type": "text/plain"
},//"mode" : "no-cors",
body: JSON.stringify(myVar)
//body: {"id" : document.getElementById('saada').value}
}).then(function(muutuja){
document.getElementById('väljund').innerHTML = JSON.stringify(muutuja);
});
}
Adding mode:'no-cors' to the request header guarantees that no response will be available in the response
Adding a "non standard" header, line 'access-control-allow-origin' will trigger a OPTIONS preflight request, which your server must handle correctly in order for the POST request to even be sent
You're also doing fetch wrong ... fetch returns a "promise" for a Response object which has promise creators for json, text, etc. depending on the content type...
In short, if your server side handles CORS correctly (which from your comment suggests it does) the following should work
function send(){
var myVar = {"id" : 1};
console.log("tuleb siia", document.getElementById('saada').value);
fetch("http://localhost:3000", {
method: "POST",
headers: {
"Content-Type": "text/plain"
},
body: JSON.stringify(myVar)
}).then(function(response) {
return response.json();
}).then(function(muutuja){
document.getElementById('väljund').innerHTML = JSON.stringify(muutuja);
});
}
however, since your code isn't really interested in JSON (it stringifies the object after all) - it's simpler to do
function send(){
var myVar = {"id" : 1};
console.log("tuleb siia", document.getElementById('saada').value);
fetch("http://localhost:3000", {
method: "POST",
headers: {
"Content-Type": "text/plain"
},
body: JSON.stringify(myVar)
}).then(function(response) {
return response.text();
}).then(function(muutuja){
document.getElementById('väljund').innerHTML = muutuja;
});
}
In my case, the problem was the protocol. I was trying to call a script url with http instead of https.
try this
await fetch(url, {
mode: 'no-cors'
})
See mozilla.org's write-up on how CORS works.
You'll need your server to send back the proper response headers, something like:
Access-Control-Allow-Origin: http://foo.example
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
Bear in mind you can use "*" for Access-Control-Allow-Origin that will only work if you're trying to pass Authentication data. In that case, you need to explicitly list the origin domains you want to allow. To allow multiple domains, see this post
you can use solutions without adding "Access-Control-Allow-Origin": "*", if your server is already using Proxy gateway this issue will not happen because the front and backend will be route in the same IP and port in client side but for development, you need one of this three solution if you don't need extra code
1- simulate the real environment by using a proxy server and configure the front and backend in the same port
2- if you using Chrome you can use the extension called Allow-Control-Allow-Origin: * it will help you to avoid this problem
3- you can use the code but some browsers versions may not support that so try to use one of the previous solutions
the best solution is using a proxy like ngnix its easy to configure and it will simulate the real situation of the production deployment
Sometimes, please check your port number. If localhost port number is mismatch, you will get the same error as well.
I was getting this error and realized my server.js wasn't running.

Categories