We Keep Coding

How to fetch Wikipedia site?Access to fetch at * from origin * has been blocked by CORS policy [duplicate]

This question related to another question, which was asked year ago. Author asked how to make cros-origin request using JavaScript and Wikipedia API and one comment was:
en.wikipedia.org doesn't seem to allow CORS
and he was advised to use JSONP instead.
I know I can use JSONP, but I prefer CORS if I can use it.
I tried on jsfiddle
var url = "https://en.wikipedia.org/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json";
$.ajax({
url: url,
data: 'query',
dataType: 'json',
type: 'POST',
headers: { 'Api-User-Agent': 'Example/1.0' },
origin: 'https://jsfiddle.net/',
success: function (data) {
console.log(data);
//do something with data
}});
and get the following error:
XMLHttpRequest cannot load
https://en.wikipedia.org/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json.
Response to preflight request doesn't pass access control check: No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'https://fiddle.jshell.net' is therefore not allowed
access.
Request Header:
authority:en.wikipedia.org
method:OPTIONS
path:/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json
scheme:https
accept:/
accept-encoding:gzip, deflate, sdch
accept-language:en-US,en;q=0.8,fr-CA;q=0.6,fr;q=0.4,fr-FR;q=0.2,ru;q=0.2,uk;q=0.2
access-control-request-headers:accept, api-user-agent, content-type
access-control-request-method:POST
origin:https://fiddle.jshell.net
referer:https://fiddle.jshell.net/_display/
user-agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
Response Header:
accept-ranges:bytes
age:0
backend-timing:D=33198 t=1462749020308717
cache-control:no-cache
content-encoding:gzip
content-length:20
content-type:text/html
date:Sun, 08 May 2016 23:10:20 GMT
p3p:CP="This is not a P3P policy! See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
server:mw1114.eqiad.wmnet
set-cookie:CP=H2; Path=/; secure
set-cookie:GeoIP=US:MA:Waltham:42.37:-71.24:v4; Path=/; secure; Domain=.wikipedia.org
set-cookie:WMF-Last-Access=08-May-2016;Path=/;HttpOnly;secure;
Expires=Thu, 09 Jun 2016 12:00:00 GMT
status:200
strict-transport-security:max-age=31536000; includeSubDomains; preload
vary:Accept-Encoding
via:1.1 varnish, 1.1 varnish
x-analytics:https=1;nocookies=1
x-cache:cp1066 pass+chfp(0), cp1055 frontend pass+chfp(0)
x-client-ip:146.115.167.51
x-content-type-options:nosniff
x-powered-by:HHVM/3.12.1
x-varnish:2807049448, 2537048470
So, I need confirmation that CORS doesn't work for Wikipedia API and I need use JSONP.

To make JavaScript Fetch/XHR requests to Wikipedia, add origin=* to the URL query params.
So the base of the URL in the question should be like this:
https://en.wikipedia.org/w/api.php?origin=*&action=query…
See the CORS-related docs for the Wikipedia backend:
For anonymous requests, origin query string parameter can be set to * which will allow requests from anywhere.
2016-05-09 original answer
See “Enable cross-domain API requests in API's JSON responses”, an open bug for Wikimedia sites that indicates that they currently only support CORS requests from different Wikimedia sites themselves to other Wikimedia sites—but they do not support CORS requests from external sites.
See in particular https://phabricator.wikimedia.org/T62835#2191138 (from Apr 8, 2016) which is a summary that indicates they are considering to make a change to allow CORS request from external sites, but they have not yet enabled it.
2016-07-12 update
It seems they will be deploying CORS support today:
unauthenticated cross-domain API requests are now possible. This
should be deployed to WMF wikis with 1.128.0-wmf.10, see
https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap for the schedule
https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap indicates the 1.128.0-wmf.10 deployment dates are 12 July 2016 to 14 July 2016.
2016-08-05 update
As torvin notes in a comment below:
to trigger the new behaviour, you need to specify origin=* in your url params. This is currently buried in the T62835 discussion and is not stated in the documentation yet.

Fetch is not sending custom headers [duplicate]

This question already has answers here:
Response to preflight request doesn't pass access control check Laravel and Ajax call
(1 answer)
XMLHttpRequest cannot load XXX No 'Access-Control-Allow-Origin' header
(11 answers)
Closed 5 years ago.
I'm trying to send a custom header within a fetch call, but it seems that the headers aren't being sent for some reason. I found some questions that seemed to indicate that 'cors' mode needs to be set as a fetch option, but I tried that and it hasn't made a difference.
In the console I'm getting this error:
Fetch API cannot load http://localhost:8000/GroupRoutePermission. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8082' is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
However if I remove the x-api-key header from my fetch request, I don't get any CORS console error and get a JSON response just fine -- my JSON with error that says api key is not set (as expected).
I've also hit my endpoint with Postman with x-api-key set, and it works fine. Strangely enough I've ripped the below code out of a previous project of mine, and in that project the custom header gets sent just fine (even without cors mode), so I'm at a loss as to what else to try.
let apiKey = ""
if (typeof localStorage.apiKey != 'undefined')
apiKey = localStorage.apiKey
else
window.location = "/login"
console.log(apiKey)
fetch(url,{
credentials: 'include',
mode: 'cors',
headers: new Headers({
'Content-Type': 'text/plain',
'x-api-key': localStorage.apiKey
})
})
Chrome Network tab Request Headers:
Accept:*/*
Accept-Encoding:gzip, deflate, sdch, br
Accept-Language:en-US,en;q=0.8,fr-CA;q=0.6,fr;q=0.4,en-CA;q=0.2
Access-Control-Request-Headers:x-api-key
Access-Control-Request-Method:GET
Cache-Control:max-age=0
Connection:keep-alive
Host:localhost:8000
Origin:http://localhost:8082
Referer:http://localhost:8082/lists/ResearchTrial
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.90 Safari/537.36
Response Headers with X-Api-Key sent:
HTTP/1.1 200 OK
Host: localhost:8000
Connection: close
X-Powered-By: PHP/5.5.38-4+deb.sury.org~xenial+1
Allow: GET,HEAD
Cache-Control: no-cache
Content-Type: text/html; charset=UTF-8
Date: Tue, 12 Sep 2017 19:30:58 GMT
Response Headers if I remove X-Api-Key in request:
HTTP/1.1 200 OK
Host: localhost:8000
Connection: close
X-Powered-By: PHP/5.5.38-4+deb.sury.org~xenial+1
Access-Control-Allow-Origin: http://localhost:8082
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Content-Length, Accept- Encoding, X-Api-Key
Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 12 Sep 2017 19:28:29 GMT
Please help!

I've ripped the below code out of a previous project of mine, and in that project the custom header gets sent just fine (even without cors mode), so I'm at a loss as to what else to try.
Was that project also making cross-domain requests?
My guess is that the API will send cors headers when auth fails, but will not send the headers when auth succeeds. This doesn't affect Postman, which doesn't have to worry about cors.
You should be able to confirm this in Postman by sending an authenticated request and checking the response headers.

Understanding Page Response

I am trying to understand how the whole CORS policy works. To explain my confusion, let me give you an example:
$.get("https://www.google.com", function(response) { alert(response) });
The above request will return with the following error:
XMLHttpRequest cannot load https://www.google.com/. Redirect from 'https://www.google.com/' to 'https://www.google.ca/?gfe_rd=cr&ei=TlqUWeGEH5HRXqW6utgI' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://fiddle.jshell.net' is therefore not allowed access.
Now in order for that to work, google would have to white-list https://fiddle.jshell.net.
Now, if I were to try the same thing on a restful API page, that will work. My question is really simple, Why?
Trying to analyze this, I tried hitting an API and analyzing its response:
https://apigee.com/console/bing?req=%7B%22resource%22%3A%22web_search%22%2C%22params%22%3A%7B%22query%22%3A%7B%22query%22%3A%22sushi%22%2C%22sources%22%3A%22web%22%7D%2C%22template%22%3A%7B%22format%22%3A%22json%22%7D%2C%22headers%22%3A%7B%7D%2C%22body%22%3A%7B%22attachmentFormat%22%3A%22mime%22%2C%22attachmentContentDisposition%22%3A%22form-data%22%7D%7D%2C%22verb%22%3A%22get%22%7D
Response:
HTTP/1.1 200
Date:
Wed, 16 Aug 2017 14:31:32 GMT
Content-Length: 266
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Server: Apigee Router
X-Content-Type-Options: nosniff
I came to conclusion that it must be the headers. Specifically I belive that it is this header: Content-Type: application/json; But I don't know for sure, I am trying to understand this and hoping somebody here can explain to me.

So I did 2 tests: running your code $.get("https://www.google.com", function(response) { alert(response) }); snippet from the console and requesting https://www.google.com from https://apigee.com/console/others
I think what happens in the 1st case is the fact that the request is done from the client, next request headers are sent:
:authority:www.google.com
:method:GET
:path:/?_=1502896196820
:scheme:https
accept:*/*
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8
origin:https://stackoverflow.com
referer:https://stackoverflow.com/questions/45717044/understanding-page-response
user-agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3187.0 Safari/537.36
x-chrome-uma-enabled:1
x-client-data:CJG2yQEIo7bJAQiMmMoBCKudygEIs53KAQjRncoBCKiiygE=
Since Google does not reply with 'Access-Control-Allow-Origin: *' - client, and in the request I have origin:https://stackoverflow.com, Chrome in my case throws CORS error.
In the 2nd test, using https://apigee.com/console/others and requesting https://www.google.com , apigee.com seems to overwrite headers and sends:
GET / HTTP/1.1
Host:
www.google.com
X-Target-URI:
https://www.google.com
Connection:
Keep-Alive
Also, from DEV console, I can see it does server to server call so no client involved in throwing CORS, thus I am getting the responses with Google page.
UPDATE:
Regarding JSON API requests, here is some interesting info from Google CloudPlatform about CORS
Note: CORS configuration applies only to XML API requests. For JSON
API requests, Cloud Storage returns the Access-Control-Allow-Origin
header with the origin of the request.
Thus, if the request is performed from the client, a client should not throw CORS errors since it gets Access-Control-Allow-Origin with the same origin it sent.
However, different APIs and clients might process requests differently. Thus, sometimes Firefox throws CORS while Chrome does not.

$http post call OPTIONS not post?

Hello so i'm trying to call my backend and getting some strange issue with calling post method.
Remote Address:192.168.58.183:80
Request URL:http://192.168.58.183/ESService/ESService.svc/CreateNewAccount
Request Method:OPTIONS
Status Code:405 Method Not Allowed
Request Headersview source
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8,pl;q=0.6
Access-Control-Request-Headers:accept, content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:192.168.58.183
Origin:http://localhost:8100
Referer:http://localhost:8100/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93
Safari/537.36
Response Headersview source
Access-Control-Allow-Headers:Content-Type, Accept
Access-Control-Allow-Methods:POST,GET,OPTIONS
Access-Control-Allow-Origin:*
Access-Control-Max-Age:1728000
Allow:POST
Content-Length:1565
Content-Type:text/html; charset=UTF-8
Date:Mon, 02 Feb 2015 09:11:17 GMT
Server:Microsoft-IIS/7.5
X-Powered-By:ASP.NET
And my code looks like here i think this call is okay can someone review it?
$scope.RegisterUser = function(){
var us = {
UserName:$scope.userName,
Password:$scope.password,
UserRoleID:null,
Company:$scope.company,
Terms:$scope.terms,
ID:null,
BuyerID:app.buyerId
};
$http({method:'POST', url:app.wcf + '/CreateNewAccount', data:{us:us}})
.then(
function(resp){
app.Logger(resp.data);
},
function(err){
app.Logger(err);
})};
So maybe i'm doing something wrong or i need to pass optional config to http?

Normally, browsers will not allow your site's JavaScript to read the data from a cross-origin request. This is because your site might be instructing the browser to get information from the user's online banking, company intranet, or some other private site. This is called The Same Origin Policy.
A standard called CORS allows a site to give permission to another site to read data from it.
Since POST requests can have side effects, an additional layer of security is added. Before the browser will make the POST request, it will make a pre-flight OPTIONS request to ask for permission to make the POST request.
Your server is not configured to handle that OPTIONS request (and probably isn't configured to return the CORS headers for the POST request either).
You need to set up CORS support if you want to allow your JavaScript to make requests to it from a different origin.

This is a pre flight request and is used to enable CORS there is no need to be concerned this is normal.

jQuery getJSON doesnt send cookies

i am including JS on domain1 form domain2
<script type="text/javascript" src="http://www.domain2.com/script.js"></script>
that script doesn onload and on button click a JSONP request to domain2
$.getJSON( 'http://www.domain2.com/process?callback=?',
function(data){
if ( data ) processData( data );
}
);
and then displaying the data on domain1.
So here is my problem:
The getJSON request doesnt send cookies to the domain2.
The weirdest thing is that it does send the cookies half a day and the other half not. :-)
This is how the request looks like when it doesnt work:
Request details
GET /ajax/embed-user-library?detail=98&callback=jsonp1312398534998 HTTP/1.1
User-Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.9.168 Version/11.50
Host: www.floowie.com
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en,sk-SK;q=0.9,sk;q=0.8
Accept-Encoding: gzip, deflate
Referer: http://www.sokker.cz/en/test2
Connection: Keep-Alive
Response details
HTTP/1.1 200 OK
Date: Wed, 03 Aug 2011 19:06:51 GMT
Server: Apache/2.2.16 (Debian)
X-Powered-By: PHP/5.3.5-0.dotdeb.1
Set-Cookie: SESSID=64292b70dc28d7c6c9f13f70070353d8; path=/; domain=.floowie.com
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Content-Length: 34
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/json
And this when it works(nothing changed in the scripts):
Request details
GET /ajax/embed-user-library?detail=99&test=1&callback=jsonp1312398534999 HTTP/1.1
User-Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.9.168 Version/11.50
Host: test1.floowie.com
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en,sk-SK;q=0.9,sk;q=0.8
Accept-Encoding: gzip, deflate
Referer: http://www.sokker.cz/en/test2
Cookie: __utma=254918925.1489796832.1301725317.1312260335.1312298033.44; __utmz=254918925.1312298033.44.11.utmcsr=sokker.cz|utmccn=(referral)|utmcmd=referral|utmcct=/en/test2; lang=en; FLWSESSID=ddd1bc696f83f5a70b5f0f3ae30b4691; __utma=121955676.1030804516.1282595153.1312390656.1312397285.194; __utmb=121955676.8.10.1312397285; __utmc=121955676; __utmz=121955676.1312397285.194.21.utmcsr=floowie.crmserver.cz|utmccn=(referral)|utmcmd=referral|utmcct=/index.php
Connection: Keep-Alive
Response details
HTTP/1.1 200 OK
Date: Wed, 03 Aug 2011 19:07:45 GMT
Server: Apache/2.2.16 (Debian)
X-Powered-By: PHP/5.3.5-0.dotdeb.1
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Content-Length: 20
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/json
Did someone see such a behaviour?
Is it solvable?
Thank you

If you want to use AJAX petitions over different domains/subdomains you have to implement Cross Origin Requests.
References:
http://hacks.mozilla.org/2009/07/cross-site-xmlhttprequest-with-cors/
https://developer.mozilla.org/en/http_access_control
Examples:
http://arunranga.com/examples/access-control/
Your server needs to send this headers:
Access-Control-Allow-Origin: test1.floowie.com
Access-Control-Allow-Credentials: true // allow cookie/session credentials
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
You can return the Access-Control-Allow-Origin globally or set specifically dependent of your input Origin ($_SERVER['HTTP_ORIGIN']) request header. Also apply for Access-Control-Allow-Methods.
You must implement the OPTIONS petition. Before the first AJAX call, modern browsers call that URL with an OPTIONS method to retrieve the above headers.
Ok this is the first part, the second is with jQuery. Read very carefully this page: http://api.jquery.com/jQuery.ajax/
You will need to add some options to every AJAX call, you can do it globally:
$(document).ajaxSend(function (event, xhr, settings) {
settings.xhrFields = {
withCredentials: true
};
});
Or specific:
$.ajax({
url: a_cross_domain_url,
xhrFields: {
withCredentials: true
}
});
This issue made ​​me lose many hours... hope it helps.
Note that you won't need to set your cookie domain as ".floowie.com" if you want.

You must properly implement CORS requests with credentials to send and receive cookies via Ajax. See developer.mozilla.org, specifically under the section titled "Requests with credentials."
First off, here is a simple CORS Ajax request with credentials, using jQuery 1.5.1+:
$.ajax({
url: "http://www.domain2.com/process",
xhrFields: {
withCredentials: true
}
}).done(function (data) { console.log(data); });
Note the withCredentials flag in the xhrFields. This flag tells the browser to send cookies with the request for the external domain, not the origin domain. In your case, cookies for www.domain2.com will be sent, and you will have access to them server-side.
On the server-side, you need to add certain headers to the response:
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: www.domain1.com
Important: requests with credentials cannot set the Access-Control-Allow-Origin header to global (Access-Control-Allow-Origin: *). It must specify domains (Access-Control-Allow-Origin: www.domain1.com).
It's obviously better if you specify a domain for the Access-Control-Allow-Origin header. But if you don't know or care where the CORS request is coming from, you could use the Origin header from the request and simply set the Access-Control-Allow-Origin header of your response to that. In C#, this is how we did this:
this.Response.AddHeader("Access-Control-Allow-Origin", this.Request.Headers["Origin"]);
After doing all of this, cookies that you set server-side will be sent back with the response, and the browser will be able to properly handle them and insert them into the browser's cookie store for www.domain2.com. And any subsequent CORS requests you send will send these cookies in the request as well.
If you are sending a request other than with the GET, POST, or HEAD methods, you will need to implement Preflighted requests (see under section titled "Preflighted requests"):
Unlike simple requests (discussed above), "preflighted" requests first send an HTTP request by the OPTIONS method to the resource on the other domain, in order to determine whether the actual request is safe to send. Cross-site requests are preflighted like this since they may have implications to user data. In particular, a request is preflighted if:
It uses methods other than GET, HEAD or POST. Also, if POST is used to send request data with a Content-Type other than application/x-www-form-urlencoded, multipart/form-data, or text/plain, e.g. if the POST request sends an XML payload to the server using application/xml or text/xml, then the request is preflighted.
It sets custom headers in the request (e.g. the request uses a header such as X-PINGOTHER)
Side-note about IE8 and IE9:
The Ajax call above will fail in IE8 and 9. I included the JS file from MoonScript/jQuery-ajaxTransport-XDomainRequest on my page, and this automagically allowed CORS requests to work in those old IE versions. But sadly, the XDomainRequest object that MS created for IE8 and 9 does not allow cookies to be sent or received. (see this MSDN blog post for more information)

You have different hosts. In the first example the host is "Host: www.floowie.com". In the second it is "Host: test1.floowie.com".
I'm guessing that the cookies are originally set by 'test1.floowie.com' and you haven't specified that they should be available to '.floowie.com' (i.e. the whole domain and all subdomains).
Can you post the code that sets the cookies in the first place?
If you get this fixed, it should at least show consistent behaviour. However, IE will probably still not pass cookies across subdomains. That's what I'm wrestling with at the moment, which is how I can across your question.