I'm working against an HTTP server that sometimes returns a body that is longer than the response's Content-length header. For example, curl shows me this:
curl -v -k -H "Cookie: VerySecretCookie" https://fqdn/path
[...]
< HTTP/1.1 200 200
< Date: Thu, 01 Jul 2021 10:45:32 GMT
[...]
< X-Frame-Options: SAMEORIGIN
<
* Excess found in a non pipelined read: excess = 83, size = 2021, maxdownload = 2021, bytecount = 0
I'm trying to issue a similar request via axios and read the entire body, even beyond the Content-length. However, it seems to stop after Content-length bytes have been read.
This is the config I'm using:
const config: AxiosRequestConfig = {
method: 'GET',
responseType: 'stream',
maxRedirects: 0,
decompress: false,
timeout: 60000,
};
And I then retrieve the response via:
private async streamToString(stream: Stream): Promise<string> {
return new Promise((res, rej) => {
let data = '';
stream.on('end', () => {
res(data);
});
stream.on('error', (err) => {
rej(err);
});
stream.on('data', (chunk) => {
data += chunk;
});
});
}
const data = await this.streamToString(response.data);
From a quick glance at axios' code, I couldn't find a place where it inspects the Content-length header, so perhaps this is done by Node's http? Is there a way to tell it to ignore Content-length and just read the entire stream until it closes (assuming HTTP keepalive is not used)?
Related
For my little Javascript app I wrote serverside API function with CGI.
I made it very simple, and full example script looks like that:
#!/usr/bin/env perl
use strict; use warnings; use 5.014;
use CGI;
use JSON;
use Data::Dumper;
my $q = new CGI;
my %p = $q->Vars;
_api_response();
sub _api_response {
my ( $error ) = #_;
my $res;
my $status = 200;
my $type = 'application/json';
my $charset = 'utf-8';
if ( $error ) {
$status = 500;
$res->{data} = {
status => 500,
};
$res->{error} = {
error => 'failure',
message => $error,
detail => Dumper \%p,
};
} else {
$res->{data} = {
status => 200,
};
}
print $q->header(
-status => $status,
-type => $type,
-charset => $charset,
);
my $body = encode_json( $res );
print $body;
}
When I call it from JS script with fetch, it gets no response body. If I checked from Developers Tools/Network, it has also no response body. If I enter the same URL into browser, it shows JSON body. If I use curl as
curl -v 'https://example.com/my_api?api=1;test=2;id=32'
response seems have also correct body:
< HTTP/2 200
< date: Mon, 13 Sep 2021 14:04:42 GMT
< server: Apache/2.4.25 (Debian)
< set-cookie: example=80b7b276.5cbe0f250c6c7; path=/; expires=Thu, 08-Sep-22 14:04:42 GMT
< cache-control: max-age=0, no-store
< content-type: application/json; charset=utf-8
<
* Connection #0 to host example.com left intact
{"data":{"status":200}}
Why fetch does not see it as a body?
For sake of completeness, I include JS part also:
async function saveData(url = '', data = {}) {
const response = await fetch(url, {
method: 'GET',
mode: 'no-cors',
cache: 'no-cache',
credentials: 'omit',
headers: {
'Content-Type': 'application/json'
},
redirect: 'follow',
referrerPolicy: 'no-referrer',
});
console.log(response); // body is null
return response.json();
}
Using the function as:
saveData('https://example.com/my_api?api=1;test=2;id=32', { answer: 42 })
.then(data => {
console.log(data);
})
.catch( error => {
console.error( error );
});
On console I see error:
SyntaxError: Unexpected end of input
One possible reason for this error is empty JSON string.
I was able to reproduce your problem, and then I was able to fix it.
It was a CORS issue. You need to enable CORS on both the front and the back end.
On the front end you need to set the content security policy with a meta tag in your page's <head>:
<meta http-equiv="Content-Security-Policy" content="default-src *; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' 'unsafe-eval' http://localhost">
(Don't forget to change localhost to whatever your real domain is.)
On the back you need to add the CORs header:
print $q->header(
-status => $status,
-type => $type,
-charset => $charset,
-access_control_allow_origin => '*', # <-- add this line
);
As a side note, none of the settings you're passing into fetch are necessary. And since you're awaiting the response and then returning another promise anyway, there is really no reason for that to even be a an async function.
Until you're ready to do something with the unused data argument, the following code will suffice:
function saveData(url = '', data = {}) {
return fetch(url).then(response=>response.json());
}
You have to await for response.json() too.
Try return await response.json();, instead of return response.json();
My react app is supposed to attain a JSON object to display information onto dashboard. There is an object when I check postman using the URL which is what I want to use however when trying to access the fields from the object I am getting a type error of undefined javascript. Even when I console log the object it is an empty array.
The below is the fetch request which has worked for all other aspects of the application but not for this part of it.
API call in Postman using the same URL gives the following response:
[
{
"average_hb": "85.27",
"average_os": "92.84",
"max_hb": "86.35",
"max_os": "96.54"
}
]
get/ call response on chrome developer tools:
Request URL: http://127.0.0.1:8000/dashboard/utilisation/T/detail/get/
Request Method: GET
Status Code: 200 OK
Remote Address: 127.0.0.1:8000
Referrer Policy: strict-origin-when-cross-origin
Access-Control-Allow-Origin: http://localhost:3000
Allow: GET, HEAD, OPTIONS
Content-Length: 79
Content-Type: application/json
Date: Tue, 07 Sep 2021 21:07:37 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.7.7
Vary: Accept, Origin, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
The outcome I would like from this is to be able to display the values of the four variables (average_hb etc.) onto application. These fields come from Django back end which as mentioned, does return an object when looking in postman.
function UtilisationDetail(props){
var average_hb = 0;
var average_os = 0;
var max_hb = 0;
var max_os = 0;
const [utilisations, setUtilisation] = useState([]);
useEffect(()=>{
fetch(`http://127.0.0.1:8000/dashboard/utilisation/${props.warehouseState}/detail/get/`, {
method:'GET',
headers: {
'Content-Type': 'application/json'
}
})
.then( resp => resp.json())
.then(resp => setUtilisation(resp))
// .catch( error => console.log(error) )
}, [props.warehouseState])
function setDetails(){
try{
console.log(utilisations)
average_hb = utilisations[0].average_hb;
average_os = utilisations[0].average_os;
max_hb = utilisations[0].max_hb;
max_os = utilisations[0].max_os;
}
catch(e){
}
}
return(
<div>
{setDetails}
<h4>Space</h4>
<p>Average HB {average_hb}</p>
<p>Average OS {average_os}</p>
<p>Max HB {max_hb}</p>
<p>Max OS {max_os}</p>
</div>
)
}
export default UtilisationDetail;
You need to use another useEffect for setDetails() that is dependent on utilisations, and have a guard against the utilisations.length
useEffect(() => {
if(utilisations.length) setDetails()
}, [utilisations])
And then handle your return statement differently to render the output.
I'm a complete Javascript beginner. I'm trying to poll a JSON for a status page with this code :
setInterval(refresh, 5000);
var beat_state = 0;
function refresh()
{
fetch("http://localhost:8080/", {headers: {"Accept": "application/json"}, mode: 'no-cors'}).then(
function(response) {
if (response.ok) {
document.getElementById("hearth").textContent = "❤️";
response.json().then(update_display);
} else {
document.getElementById("hearth").textContent = "💔️";
}
}
);
}
function update_display(json_health)
{
/* ... */
}
The server return this response :
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 186
Date: Sun, 06 Sep 2020 20:25:59 GMT
Server: lighttpd/1.4.55
{"socket_listening": true,"quick_shutdown": false,"workers_online": 20,"workers_sleeping": 19,"workers_backlog": 0,"connections": [{"id": 4,"active_requests": 1,"socket_insane": false}]}
I'm doing test with a standalone on-disk HTML page.
Given that the Firefox 80.0.1 console show that the request is successful with no error. Why is reponse.ok false ?
when you have mode:'no-cors'
you will get a response with type: "opaque"
which returns status:0
and since ok is only true on status in the range 200-299 you get ok:false
sources:
OK status
what type opaque does
opaque being response for no-cors
I would try to change the fetch syntax, it seems that you've done some nested ".then"s, that may cause the fetch not to work
In order to have the process run and catch errors, maybe something like this will work
setInterval(refresh, 5000);
var beat_state = 0;
function refresh()
{
fetch("http://localhost:8080/", {headers: {"Accept": "application/json"}, mode: 'no-cors'})
.then(function(response) { if (response.ok) {
document.getElementById("hearth").textContent = "❤️";
return response.json();
} else {
document.getElementById("hearth").textContent = "💔️";
}
})
.then(function(json) {
update_display();
}
.catch(error => {
console.log(error.message);
})
I wold like to know how to make a GET request in node.js sending a body.
const options = {
hostname: 'localhost',
port: 3000,
path: '/abc',
method: 'GET'
}
http.get(options, (res) => {
res.on('data', (chunk) => {
console.log(String(chunk))
})
})
As it says in the documentation:
Since most requests are GET requests without bodies, Node.js provides this convenience method. The only difference between this method and http.request() is that it sets the method to GET and calls req.end() automatically.
So the answer is to use http.request directly. http.request has an example using POST, but it's the same for GET (start the request with http.request, use write to send the body data, use end when done sending data), other than the fact that (as noted above) GET usually doesn't have any body. In fact, RFC 7231 notes that:
A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request.
Using the standard http:
`const http = require('http');
https.get('http://localhost:3000/abc', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
console.log(JSON.parse(data).explanation);
});
}).on("error", (err) => {
console.log("Error: " + err.message);
});`
Hope this helps
Using Body in GET request is not recommended at all cause it is not the suggest behavior by HTTP 1.1 but you can use the following method:
const data = JSON.stringify({
"userId": 1,
"id": 1,
"title": "delectus aut autem",
"completed": false
});
const https = require('https')
const options = {
hostname: 'jsonplaceholder.typicode.com',
port: 443,
path: '/posts',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': data.length
}
}
const req = https.request(options, (res) => {
console.log(`statusCode: ${res.statusCode}`)
res.on('data', (d) => {
process.stdout.write(d)
})
})
req.on('error', (error) => {
console.error(error)
})
req.write(data)
req.end()
I made a go server that can do the basics. Now I want to do a request to my server from my node.js frontend (Axios) get a cookie back (for my login system) here is the code for putting the cookie in my response:
var hashKey = []byte("testkey") //for testing purpopes
var blockKey = []byte(securecookie.GenerateRandomKey(32))
var s = securecookie.New(hashKey, blockKey)
if encoded, err := s.Encode("cookie-name", value); err == nil {
cookie := &http.Cookie{
Name: "cookie-name",
Value: encoded,
Path: "/",
Secure: true,
HttpOnly: true,
}
http.SetCookie(*w, cookie) // w = *http.ResponseWriter
...
when I use my REST tool to see what I get I can see that the 'set-cookie' header is present. The same is If I inspect in Microsoft Edge I can see the set-cookie header. But if I inspect in Google Chrome then I can't see the header. Also if I look in the cookies tab in both Chrome and edge the cookie is not set.
this is my function that is ran for the request:
async post( url, data, ct ) {
try {
const res = await axios.post(url, data, {
headers: {
'Content-Type': (ct || "text/plain")
},
withCredentials: true
});
if (res.status === 200) {
return res.data;
}
} catch (e) {
console.error(e);
return false;
}
}
my Response Headers:
server: nginx/1.14.0 (Ubuntu)
date: Thu, 17 Jan 2019 14:29:07 GMT
content-type: text/plain charset=utf-8
content-length: 4
connection: keep-alive
setcookie:cookiename=MTU0NzczNTM0N3xGOTJYUUw5TFNXZHI2dU9jT3hCeTZUTE5TaTBFNU1XN1F 5WGMzb3c1dGZRUENEU2xPZHFwTXJQLW8zND18_VCYxNVRbIAUrs9_8EcGpTUEiqVyYL_2M5Olbjhnkeg =; Path=/
access-control-allow-origin:https://beta.bvwitteveen.nl
access-control-allow-methods:GET, POST, OPTIONS
access-control-allow-credentials:true
access-control-allow-headers:DNT,User-Agent,X-Requested-With,If-
ModifiedSince,Cache-Control,Content-Type,Range,Set-Cookie
access-control-expose-headers:Content-Length,Content-Range
Why is my cookie behaving so weird? What am I doing wrong here?