Get/examine headers of some website in console - javascript

So, I want to send a GET request from my FFQuantum console and examine the response (the header) that I receive as response. I just want to check the fields, nothing more.
Now, when I run this script on this website (https://stackoverflow.com/), with this code:
var req = new XMLHttpRequest();
var web_adress = 'https://stackoverflow.com/';
req.open('GET', web_adress, false);
req.send(null);
var headers = req.getAllResponseHeaders().toLowerCase();
alert(headers);
I get the header just right, but when I'm for example, on Google, then I get the error:
For results, I expect to see filled popup but I get the empty one. That is for when I'm on Google and trying to fetch the Stack's header.
What am I getting?
Cross-Origin Request Blocked: The Same Origin Policy disallows reading
the remote resource at https://stackoverflow.com/. (Reason: CORS
header ‘Access-Control-Allow-Origin’ missing).
Is there a way to get the headers from StackOverflow when I'm on Googles pages?

No, you can't. Is is because of security. Browser don't allow you to send requests across domains! To allow this action stackoverflow must return something like this:
Access-Control-Allow-Origin: https://google.com
But there isn't this header in response. If you want to do something like this - try to use special browsers which do not use The Same Origin Policy or some other soft.
Phantomjs for example or nodejs (if you want to do it with JS) or curl request etc.
But probaby you can do something like this:
google.com => yourdomain.com => stackoverflow.com
Here you use your host like a proxy (remember that you need to set Access-Control-Allow-Origin: *)
The problem is in browser security =)

Related

Why does my JavaScript code receive a "No 'Access-Control-Allow-Origin' header is present on the requested resource" error while the header is given

I'm trying to get informations from a SystemLinkServlet.
So I tried to execute this JavaScript code from a Nintex Forms (Sharepoint) :
var http = new XMLHttpRequest();
var url = 'www.exampleservlet.com';
var params = "anyxml"
http.open('POST', url, true)
http.setRequestHeader('Content-type', 'application/xml');
http.setRequestHeader('Access-Control-Allow-Origin', '*');
http.onreadystatechange = function() {
if(http.readyState == 4 && http.status == 200) {
alert(http.responseText);
}
};
http.send(params);
But I still got this error in my console :
Access to XMLHttpRequest at 'www.exampleservlet.com' from origin 'www.exampleorigin.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
It seems that the header is ignored or maybe I can't set multiple request headers?
It works on Postman.
Update
So It worked with an extension but apparently, I can't set headers with JavaScript code in my Nintex Forms.
I'm trying to find to pass those headers without using an extension.
If you are using PHP, try adding the following code at the beginning of the php file:
If you are using localhost, try this:
header("Access-Control-Allow-Origin: *");
If you are using external domains such as server, try this:
header("Access-Control-Allow-Origin: http://www.webiste.com");
Also you I suggest you to use this extension:
https://chrome.google.com/webstore/detail/cors-unblock/lfhmikememgdcahcdlaciloancbhjino?hl=en
Postman or other similar tools provide you development environments. In this way, you can ignore and pass CORS rule while sending request and getting response by changing tool settings. But if you sending request via browser(chrome, firefox etc.), browsers always add some preflight controls.
For example, browser send options message to get server side rule before your http request. So that invalid or wrong requests are blocked by browser before processing your http request.
In your case, server side must include your domain information. You can not change this communication rule from client side by adding just "Access-Control-Allow-Origin: *" or "Access-Control-Allow-Origin: http://www.webiste.com" statements.

Geoserver access REST in javascript - Why 403 Forbidden? [duplicate]

I am building a web API. I found whenever I use Chrome to POST, GET to my API, there is always an OPTIONS request sent before the real request, which is quite annoying. Currently, I get the server to ignore any OPTIONS requests. Now my question is what's good to send an OPTIONS request to double the server's load? Is there any way to completely stop the browser from sending OPTIONS requests?
edit 2018-09-13: added some precisions about this pre-flight request and how to avoid it at the end of this reponse.
OPTIONS requests are what we call pre-flight requests in Cross-origin resource sharing (CORS).
They are necessary when you're making requests across different origins in specific situations.
This pre-flight request is made by some browsers as a safety measure to ensure that the request being done is trusted by the server.
Meaning the server understands that the method, origin and headers being sent on the request are safe to act upon.
Your server should not ignore but handle these requests whenever you're attempting to do cross origin requests.
A good resource can be found here http://enable-cors.org/
A way to handle these to get comfortable is to ensure that for any path with OPTIONS method the server sends a response with this header
Access-Control-Allow-Origin: *
This will tell the browser that the server is willing to answer requests from any origin.
For more information on how to add CORS support to your server see the following flowchart
http://www.html5rocks.com/static/images/cors_server_flowchart.png
edit 2018-09-13
CORS OPTIONS request is triggered only in somes cases, as explained in MDN docs:
Some requests don’t trigger a CORS preflight. Those are called “simple requests” in this article, though the Fetch spec (which defines CORS) doesn’t use that term. A request that doesn’t trigger a CORS preflight—a so-called “simple request”—is one that meets all the following conditions:
The only allowed methods are:
GET
HEAD
POST
Apart from the headers set automatically by the user agent (for example, Connection, User-Agent, or any of the other headers with names defined in the Fetch spec as a “forbidden header name”), the only headers which are allowed to be manually set are those which the Fetch spec defines as being a “CORS-safelisted request-header”, which are:
Accept
Accept-Language
Content-Language
Content-Type (but note the additional requirements below)
DPR
Downlink
Save-Data
Viewport-Width
Width
The only allowed values for the Content-Type header are:
application/x-www-form-urlencoded
multipart/form-data
text/plain
No event listeners are registered on any XMLHttpRequestUpload object used in the request; these are accessed using the XMLHttpRequest.upload property.
No ReadableStream object is used in the request.
Have gone through this issue, below is my conclusion to this issue and my solution.
According to the CORS strategy (highly recommend you read about it) You can't just force the browser to stop sending OPTIONS request if it thinks it needs to.
There are two ways you can work around it:
Make sure your request is a "simple request"
Set Access-Control-Max-Age for the OPTIONS request
Simple request
A simple cross-site request is one that meets all the following conditions:
The only allowed methods are:
GET
HEAD
POST
Apart from the headers set automatically by the user agent (e.g. Connection, User-Agent, etc.), the only headers which are allowed to be manually set are:
Accept
Accept-Language
Content-Language
Content-Type
The only allowed values for the Content-Type header are:
application/x-www-form-urlencoded
multipart/form-data
text/plain
A simple request will not cause a pre-flight OPTIONS request.
Set a cache for the OPTIONS check
You can set a Access-Control-Max-Age for the OPTIONS request, so that it will not check the permission again until it is expired.
Access-Control-Max-Age gives the value in seconds for how long the response to the preflight request can be cached for without sending another preflight request.
Limitation Noted
For Chrome, the maximum seconds for Access-Control-Max-Age is 600 which is 10 minutes, according to chrome source code
Access-Control-Max-Age only works for one resource every time, for example, GET requests with same URL path but different queries will be treated as different resources. So the request to the second resource will still trigger a preflight request.
Please refer this answer on the actual need for pre-flighted OPTIONS request: CORS - What is the motivation behind introducing preflight requests?
To disable the OPTIONS request, below conditions must be satisfied for ajax request:
Request does not set custom HTTP headers like 'application/xml' or 'application/json' etc
The request method has to be one of GET, HEAD or POST. If POST, content type should be one of application/x-www-form-urlencoded, multipart/form-data, or text/plain
Reference:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
When you have the debug console open and the Disable Cache option turned on, preflight requests will always be sent (i.e. before each and every request). if you don't disable the cache, a pre-flight request will be sent only once (per server)
Yes it's possible to avoid options request. Options request is a preflight request when you send (post) any data to another domain. It's a browser security issue. But we can use another technology: iframe transport layer. I strongly recommend you forget about any CORS configuration and use readymade solution and it will work anywhere.
Take a look here:
https://github.com/jpillora/xdomain
And working example:
http://jpillora.com/xdomain/
For a developer who understands the reason it exists but needs to access an API that doesn't handle OPTIONS calls without auth, I need a temporary answer so I can develop locally until the API owner adds proper SPA CORS support or I get a proxy API up and running.
I found you can disable CORS in Safari and Chrome on a Mac.
Disable same origin policy in Chrome
Chrome: Quit Chrome, open an terminal and paste this command: open /Applications/Google\ Chrome.app --args --disable-web-security --user-data-dir
Safari: Disabling same-origin policy in Safari
If you want to disable the same-origin policy on Safari (I have 9.1.1), then you only need to enable the developer menu, and select "Disable Cross-Origin Restrictions" from the develop menu.
As mentioned in previous posts already, OPTIONS requests are there for a reason. If you have an issue with large response times from your server (e.g. overseas connection) you can also have your browser cache the preflight requests.
Have your server reply with the Access-Control-Max-Age header and for requests that go to the same endpoint the preflight request will have been cached and not occur anymore.
I have solved this problem like.
if($_SERVER['REQUEST_METHOD'] == 'OPTIONS' && ENV == 'devel') {
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Headers: X-Requested-With');
header("HTTP/1.1 200 OK");
die();
}
It is only for development. With this I am waiting 9ms and 500ms and not 8s and 500ms. I can do that because production JS app will be on the same machine as production so there will be no OPTIONS but development is my local.
You can't but you could avoid CORS using JSONP.
you can also use a API Manager (like Open Sources Gravitee.io) to prevent CORS issues between frontend app and backend services by manipulating headers in preflight.
Header used in response to a preflight request to indicate which HTTP headers can be used when making the actual request :
content-type
access-control-allow-header
authorization
x-requested-with
and specify the "allow-origin" = localhost:4200 for example
After spending a whole day and a half trying to work through a similar problem I found it had to do with IIS.
My Web API project was set up as follows:
// WebApiConfig.cs
public static void Register(HttpConfiguration config)
{
var cors = new EnableCorsAttribute("*", "*", "*");
config.EnableCors(cors);
//...
}
I did not have CORS specific config options in the web.config > system.webServer node like I have seen in so many posts
No CORS specific code in the global.asax or in the controller as a decorator
The problem was the app pool settings.
The managed pipeline mode was set to classic (changed it to integrated) and the Identity was set to Network Service (changed it to ApplicationPoolIdentity)
Changing those settings (and refreshing the app pool) fixed it for me.
OPTIONS request is a feature of web browsers, so it's not easy to disable it. But I found a way to redirect it away with proxy. It's useful in case that the service endpoint just cannot handle CORS/OPTIONS yet, maybe still under development, or mal-configured.
Steps:
Setup a reverse proxy for such requests with tools of choice (nginx, YARP, ...)
Create an endpoint just to handle the OPTIONS request. It might be easier to create a normal empty endpoint, and make sure it handles CORS well.
Configure two sets of rules for the proxy. One is to route all OPTIONS requests to the dummy endpoint above. Another to route all other requests to actual endpoint in question.
Update the web site to use proxy instead.
Basically this approach is to cheat browser that OPTIONS request works. Considering CORS is not to enhance security, but to relax the same-origin policy, I hope this trick could work for a while. :)
One solution I have used in the past - lets say your site is on mydomain.com, and you need to make an ajax request to foreigndomain.com
Configure an IIS rewrite from your domain to the foreign domain - e.g.
<rewrite>
<rules>
<rule name="ForeignRewrite" stopProcessing="true">
<match url="^api/v1/(.*)$" />
<action type="Rewrite" url="https://foreigndomain.com/{R:1}" />
</rule>
</rules>
</rewrite>
on your mydomain.com site - you can then make a same origin request, and there's no need for any options request :)
It can be solved in case of use of a proxy that intercept the request and write the appropriate headers.
In the particular case of Varnish these would be the rules:
if (req.http.host == "CUSTOM_URL" ) {
set resp.http.Access-Control-Allow-Origin = "*";
if (req.method == "OPTIONS") {
set resp.http.Access-Control-Max-Age = "1728000";
set resp.http.Access-Control-Allow-Methods = "GET, POST, PUT, DELETE, PATCH, OPTIONS";
set resp.http.Access-Control-Allow-Headers = "Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since";
set resp.http.Content-Length = "0";
set resp.http.Content-Type = "text/plain charset=UTF-8";
set resp.status = 204;
}
}
What worked for me was to import "github.com/gorilla/handlers" and then use it this way:
router := mux.NewRouter()
router.HandleFunc("/config", getConfig).Methods("GET")
router.HandleFunc("/config/emcServer", createEmcServers).Methods("POST")
headersOk := handlers.AllowedHeaders([]string{"X-Requested-With", "Content-Type"})
originsOk := handlers.AllowedOrigins([]string{"*"})
methodsOk := handlers.AllowedMethods([]string{"GET", "HEAD", "POST", "PUT", "OPTIONS"})
log.Fatal(http.ListenAndServe(":" + webServicePort, handlers.CORS(originsOk, headersOk, methodsOk)(router)))
As soon as I executed an Ajax POST request and attaching JSON data to it, Chrome would always add the Content-Type header which was not in my previous AllowedHeaders config.

Why sending a POST by AJAX is interpreted by the HTTP Server as OPTIONS and sending by CURL is effectively a PUT?

I'm testing a HTTP Server that I have developed myself with C++ and Boost libraries. More specifically, I'm testing an endpoint where a JSON is received by PUT.
To test the RESTFul webservice I use Curl with the following command:
curl -H "Content-Type: application/json" -H "Content-Length: 34" -H "Connection: close" -X PUT --data "#response_json" http://localhost:8080/answer
where response_json is a file with the json to be sent. This works fine, the server receives the request as a PUT and do what is supposed to do.
However, when I test the webservice from AJAX with this:
function sendPut2() {
var http = new XMLHttpRequest();
var url = 'http://localhost:8080/answer';
var data = JSON.stringify({"question": "a", "answer": "b"});
http.open("PUT", url, true);
http.setRequestHeader("Content-type", "application/json");
http.setRequestHeader("Content-Length", data.length);
http.setRequestHeader("Connection", "close");
http.onreadystatechange = function() {
if(http.readyState == 4 && http.status == 200) {
alert(http.responseText);
}
}
http.send(data);
}
the server receives it as OPTIONS and does not work. Moreover, in Firebug console I can see: "NetworkError: 404 Not Found - http://localhost:8080/answer".
I have tried with Firefox and Chrome. What is wrong in my javascript code?
This is the Firebug with the request from Javascript:
The browser has a same origin policy for security reasons. When you request a Ajax PUT in the browser from a different origin than the current web page was loaded from, then the request is subject to that same origin policy. The destination site can choose to support CORS (cross origin resource sharing) which is a specific scheme that the browser implements that lets it ask the target site if a specific cross origin request is OK or not.
Using the OPTIONS request before the PUT request is one such part of the CORS scheme. If the browser detects certain conditions on the original cross origin request, then it will first issue an OPTIONS request and, if it gets the right response from that, then it will issue the target request (a PUT in your case). Things that can trigger the browser to use the OPTIONS request are things like custom headers, certain types of authorization required, certain content types, certain types of requests, etc...
CURL, on the other hand, enforces no such same origin security (that is someting a browser invented for its own web page security model) so it just sends the PUT request right through without requiring the correct answer from the OPTIONS request first.
FYI, if the Javascript in the browser that is making the Ajax request is requesting from the same origin as the loaded web page that contains the Javascript, then it should not trigger the OPTIONS request because it would be a same origin request rather than a cross origin request. If you have a local server, make sure that the web page is being loaded from the local server (same hostname and port number) too, not from the file system and not one using an IP address and the other using localhost or something like that. As far as the browser is concerned, the hostname has to physically be the same, not just the same IP address.
Here's info from MDN on what requests are "preflighted" with the OPTIONS request:
Preflighted requests
Unlike simple requests (discussed above), "preflighted" requests first
send an HTTP request by the OPTIONS method to the resource on the
other domain, in order to determine whether the actual request is safe
to send. Cross-site requests are preflighted like this since they may
have implications to user data. In particular, a request is
preflighted if:
It uses methods other than GET, HEAD or POST. Also, if POST is used
to send request data with a Content-Type other than
application/x-www-form-urlencoded, multipart/form-data, or text/plain,
e.g. if the POST request sends an XML payload to the server using
application/xml or text/xml, then the request is preflighted. It sets
custom headers in the request (e.g. the request uses a header such as
X-PINGOTHER)
FYI, here's a pretty good explanation of the various aspects of CORS. Because your request is a PUT, it will be in the "not-so-simple request" part of that article.

What is wrong with my HTTP over AJAX (javascript)

My code:
var answer_array = [];
var req = new XMLHttpRequest();
req.onload = function() {
answer_array = answer_array.concat(JSON.parse(this.responseText).results);
console.log(answer_array);
}
req.open("GET", "https://api.comettracker.com/v1/gpsdata?fromdate=2015-10-13");
req.setRequestHeader("authorization", "Basic Base64 encoded credentials");
req.setRequestHeader("cache-control", "no-cache");
req.setRequestHeader("postman-token", "b94725ff-408b-c82e-a985-6c38feb380af");
req.send();
This is what is in my console:
scripts2.js:22 OPTIONS https://api.comettracker.com/v1/gpsdata?fromdate=2015-10-13 (anonymous function) # scripts2.js:22
2015-10-21 12:41:09.059 index.html:1 XMLHttpRequest cannot load https://api.comettracker.com/v1/gpsdata?fromdate=2015-10-13. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access. The response had HTTP status code 405.
When I go to the network tab on Chrome I see this:
gpsdata?fromdate=2015-10-13 OPTIONS 405 xhr scripts2.js:22 0 B 452 ms
This error message:
No 'Access-Control-Allow-Origin' header is present on the requested resource
means that you are running into a cross origin permission issue which means that you are trying to access a site that does not permit access from the domain that your page is on. If your page is on your local drive being accessed with a file:// URL, then the first thing you can do is to put it on an actual web server and try it there since file:// URLs have some additional restrictions on them.
If that doesn't work either, then the issue is that the api.comettracker.com site is not allowing access from your particular site.
When I put your code into a jsFiddle and try it there and look at the network trace, what I see there is that the OPTIONS method which is used to pre-flight a cross origin request is being rejected by api.comettracker.com which tells the browser the cross origin request as currently formatted is not permitted.
I get a different error if your custom headers are removed from the request so I think that there's something incorrect about your custom headers. Since I don't know that particular API, don't have your access credentials or know how to use them, I don't know what exactly to suggest for the headers, but I think that's the place to start.

Why does No 'Access-Control-Allow-Origin' is not enforced when I call third party services, but it is when I call my own server?

I'm building a RESTful service on a JAX-RS server and some clients that will be attached to it.
The hour came to start testing the endpoints on the clients and I tried first on JavaScript since until now, it has been very easy for me to make requests to third party resources with this code:
function httpGet(theUrl){
var xmlHttp = new XMLHttpRequest();
xmlHttp.open( "GET", theUrl, false );
xmlHttp.send( null );
return xmlHttp.responseText;
}
I know I shouldn't do synchronous requests but that's off topic.
On Firefox, the error I get is:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://someurl.com/someresource/. (Reason: CORS header 'Access-Control-Allow-Origin' missing).
The requests don't work both on my local server and on the deployment server.
I've found that most solutions to this problem have to do something with setting a header Access-Control-Allow-Origin: *. I've tried this and it hasn't worked for me.
At first I thought it was a problem with my server configuration, but now I think it's the browser that is not letting me execute the request because of the Same Origin Policy. Is this correct? If it's correct, why does the exact same code as above, with no Access-Control-Allow-Headers: *, works for third party services (Google, Facebook, etc.)?
Is there a whitelist of sites that are always allowed to break the Same Origin Policy?
If the answer to the last question is no, then they must have some specific configutation on their server side code to allow Cross Origin communications to happen. What could this configuration be?
At first I thought it was a problem with my server configuration
It is.
but now I think it's the browser that is not letting me execute the request because of the Same Origin Policy.
That's true. The browser is disallowing the request per the SOP because your server isn't configured to allow cross-origin requests.
It's much more than just passing back a single header. Full details in the spec, but basically it comes down to:
Responding to OPTIONS requests, not just GET, POST, and so on.
Responding with all of the necessary headers.
Responding with the correct values for those headers.
The headers you'll have to send back are at least:
Access-Control-Allow-Origin
Access-Control-Allow-Methods
Access-Control-Allow-Headers
and you may also need Access-Control-Allow-Credentials. The values you need to supply for these are usually derived from the headers with similar (but slightly different) names that accompany the request.
You must supply the headers both in response to the OPTIONS call (if there is one), and also the subsequent GET or POST etc. call.
If it's correct, why does the exact same code as above, with no Access-Control-Allow-Headers: *, works for third party services (Google, Facebook, etc.)?
The browser fills in the request headers automatically; what's different is the server responding to the request.
Is there a whitelist of sites that are always allowed to break the Same Origin Policy?
No, of course not.
If the answer to the last question is no, then they must have some specific configutation on their server side code to allow Cross Origin communications to happen. What could this configuration be?
The above.
Here's some pseudocode for the server side granting access via CORS (it's written in JavaScript, since I know you're familiar with JavaScript, but it is pseudocode, and you need to do this on your server):
var origin, method, headers;
origin = getRequestHeader("Origin");
if (origin /* and you want to grant access to it */) {
addResponseHeader("Access-Control-Allow-Origin", origin);
method = getRequestHeader("Access-Control-Request-Method");
if (method) {
// Note the request header is singular, but the response header is plural
addResponseHeader("Access-Control-Allow-Methods", method);
}
headers = getRequestHeader("Access-Control-Request-Headers");
if (headers) {
addResponseHeader("Access-Control-Allow-Headers", headers);
}
if (/* You want to allow the origin to provide credentials and cookies*/) {
addResponseHeader("Access-Control-Allow-Credentials", "true");
}
}
At first I thought it was a problem with my server configuration, but now I think it's the browser that is not letting me execute the request because of the Same Origin Policy. Is this correct?
By default, the browser will enforce the Same Origin Policy and block your JavaScript from accessing the data.
The configuration of the server you are making the request to can set CORS headers (including Access-Control-Allow-Origin and Access-Control-Allow-Headers) to tell the browser not to enforce the Same Origin Policy for that request.
Is there a whitelist of sites that are always allowed to break the Same Origin Policy?
No
If the answer to the last question is no, then they must have some specific configutation on their server side code to allow Cross Origin communications to happen. What could this configuration be?
The configuration sets the response headers that are described in the error message you quoted.

Categories