We Keep Coding

How to fetch Wikipedia site?Access to fetch at * from origin * has been blocked by CORS policy [duplicate]

This question related to another question, which was asked year ago. Author asked how to make cros-origin request using JavaScript and Wikipedia API and one comment was:
en.wikipedia.org doesn't seem to allow CORS
and he was advised to use JSONP instead.
I know I can use JSONP, but I prefer CORS if I can use it.
I tried on jsfiddle
var url = "https://en.wikipedia.org/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json";
$.ajax({
url: url,
data: 'query',
dataType: 'json',
type: 'POST',
headers: { 'Api-User-Agent': 'Example/1.0' },
origin: 'https://jsfiddle.net/',
success: function (data) {
console.log(data);
//do something with data
}});
and get the following error:
XMLHttpRequest cannot load
https://en.wikipedia.org/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json.
Response to preflight request doesn't pass access control check: No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'https://fiddle.jshell.net' is therefore not allowed
access.
Request Header:
authority:en.wikipedia.org
method:OPTIONS
path:/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json
scheme:https
accept:/
accept-encoding:gzip, deflate, sdch
accept-language:en-US,en;q=0.8,fr-CA;q=0.6,fr;q=0.4,fr-FR;q=0.2,ru;q=0.2,uk;q=0.2
access-control-request-headers:accept, api-user-agent, content-type
access-control-request-method:POST
origin:https://fiddle.jshell.net
referer:https://fiddle.jshell.net/_display/
user-agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
Response Header:
accept-ranges:bytes
age:0
backend-timing:D=33198 t=1462749020308717
cache-control:no-cache
content-encoding:gzip
content-length:20
content-type:text/html
date:Sun, 08 May 2016 23:10:20 GMT
p3p:CP="This is not a P3P policy! See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
server:mw1114.eqiad.wmnet
set-cookie:CP=H2; Path=/; secure
set-cookie:GeoIP=US:MA:Waltham:42.37:-71.24:v4; Path=/; secure; Domain=.wikipedia.org
set-cookie:WMF-Last-Access=08-May-2016;Path=/;HttpOnly;secure;
Expires=Thu, 09 Jun 2016 12:00:00 GMT
status:200
strict-transport-security:max-age=31536000; includeSubDomains; preload
vary:Accept-Encoding
via:1.1 varnish, 1.1 varnish
x-analytics:https=1;nocookies=1
x-cache:cp1066 pass+chfp(0), cp1055 frontend pass+chfp(0)
x-client-ip:146.115.167.51
x-content-type-options:nosniff
x-powered-by:HHVM/3.12.1
x-varnish:2807049448, 2537048470
So, I need confirmation that CORS doesn't work for Wikipedia API and I need use JSONP.

To make JavaScript Fetch/XHR requests to Wikipedia, add origin=* to the URL query params.
So the base of the URL in the question should be like this:
https://en.wikipedia.org/w/api.php?origin=*&action=query…
See the CORS-related docs for the Wikipedia backend:
For anonymous requests, origin query string parameter can be set to * which will allow requests from anywhere.
2016-05-09 original answer
See “Enable cross-domain API requests in API's JSON responses”, an open bug for Wikimedia sites that indicates that they currently only support CORS requests from different Wikimedia sites themselves to other Wikimedia sites—but they do not support CORS requests from external sites.
See in particular https://phabricator.wikimedia.org/T62835#2191138 (from Apr 8, 2016) which is a summary that indicates they are considering to make a change to allow CORS request from external sites, but they have not yet enabled it.
2016-07-12 update
It seems they will be deploying CORS support today:
unauthenticated cross-domain API requests are now possible. This
should be deployed to WMF wikis with 1.128.0-wmf.10, see
https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap for the schedule
https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap indicates the 1.128.0-wmf.10 deployment dates are 12 July 2016 to 14 July 2016.
2016-08-05 update
As torvin notes in a comment below:
to trigger the new behaviour, you need to specify origin=* in your url params. This is currently buried in the T62835 discussion and is not stated in the documentation yet.

Understanding Page Response

I am trying to understand how the whole CORS policy works. To explain my confusion, let me give you an example:
$.get("https://www.google.com", function(response) { alert(response) });
The above request will return with the following error:
XMLHttpRequest cannot load https://www.google.com/. Redirect from 'https://www.google.com/' to 'https://www.google.ca/?gfe_rd=cr&ei=TlqUWeGEH5HRXqW6utgI' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://fiddle.jshell.net' is therefore not allowed access.
Now in order for that to work, google would have to white-list https://fiddle.jshell.net.
Now, if I were to try the same thing on a restful API page, that will work. My question is really simple, Why?
Trying to analyze this, I tried hitting an API and analyzing its response:
https://apigee.com/console/bing?req=%7B%22resource%22%3A%22web_search%22%2C%22params%22%3A%7B%22query%22%3A%7B%22query%22%3A%22sushi%22%2C%22sources%22%3A%22web%22%7D%2C%22template%22%3A%7B%22format%22%3A%22json%22%7D%2C%22headers%22%3A%7B%7D%2C%22body%22%3A%7B%22attachmentFormat%22%3A%22mime%22%2C%22attachmentContentDisposition%22%3A%22form-data%22%7D%7D%2C%22verb%22%3A%22get%22%7D
Response:
HTTP/1.1 200
Date:
Wed, 16 Aug 2017 14:31:32 GMT
Content-Length: 266
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Server: Apigee Router
X-Content-Type-Options: nosniff
I came to conclusion that it must be the headers. Specifically I belive that it is this header: Content-Type: application/json; But I don't know for sure, I am trying to understand this and hoping somebody here can explain to me.

So I did 2 tests: running your code $.get("https://www.google.com", function(response) { alert(response) }); snippet from the console and requesting https://www.google.com from https://apigee.com/console/others
I think what happens in the 1st case is the fact that the request is done from the client, next request headers are sent:
:authority:www.google.com
:method:GET
:path:/?_=1502896196820
:scheme:https
accept:*/*
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8
origin:https://stackoverflow.com
referer:https://stackoverflow.com/questions/45717044/understanding-page-response
user-agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3187.0 Safari/537.36
x-chrome-uma-enabled:1
x-client-data:CJG2yQEIo7bJAQiMmMoBCKudygEIs53KAQjRncoBCKiiygE=
Since Google does not reply with 'Access-Control-Allow-Origin: *' - client, and in the request I have origin:https://stackoverflow.com, Chrome in my case throws CORS error.
In the 2nd test, using https://apigee.com/console/others and requesting https://www.google.com , apigee.com seems to overwrite headers and sends:
GET / HTTP/1.1
Host:
www.google.com
X-Target-URI:
https://www.google.com
Connection:
Keep-Alive
Also, from DEV console, I can see it does server to server call so no client involved in throwing CORS, thus I am getting the responses with Google page.
UPDATE:
Regarding JSON API requests, here is some interesting info from Google CloudPlatform about CORS
Note: CORS configuration applies only to XML API requests. For JSON
API requests, Cloud Storage returns the Access-Control-Allow-Origin
header with the origin of the request.
Thus, if the request is performed from the client, a client should not throw CORS errors since it gets Access-Control-Allow-Origin with the same origin it sent.
However, different APIs and clients might process requests differently. Thus, sometimes Firefox throws CORS while Chrome does not.

Access Control Allow Origin header Error Only in Chrome,Safari but not Opera

I have enable Access-Control-Allow-Origin on my apache webserver. I am able to load all requests on the first load. However, subsequent loading/refreshing of the pages would leave half of the requests failed, leaving an error of,
XMLHttpRequest cannot load http://***. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:3000' is therefore not allowed access.
In Chrome Developer Tool, under the Network tab, enabling "Disable Cache" removes this error.
I am using AngularJS $http to call my request, I have changed the cache option to true/false but this abnormality persist. For your information, I am also using an angular-cache module, https://github.com/jmdobry/angular-cache, I have tried enabling/disabling this too, but to no avail.
Here is a copy of the request header:
GET ***Valid_Url_With_Parameters** HTTP/1.1
Host: ********
Connection: keep-alive
Accept: application/json, text/plain, */*
Origin: http://localhost:3000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Referer: http://localhost:3000/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I have a 2 different types response header with different HTTP status codes for the failed requests.
HTTP/1.1 200 OK
Content-Length: 564121
Content-Type: application/json
Server: Apache/2.2.15 (CentOS)
Vary: Accept
Last-Modified: Fri, 16 Sep 2016 12:41:41 GMT
Connection: keep-alive
Date: Fri, 16 Sep 2016 12:43:44 GMT
HTTP/1.1 206 Partial Content
Content-Type: application/json
Server: Apache/2.2.15 (CentOS)
Vary: Accept
Last-Modified: Fri, 16 Sep 2016 13:39:05 GMT
Content-Range: bytes 29200-29200/785592
Content-Length: 1
Connection: keep-alive
Date: Fri, 16 Sep 2016 14:20:48 GMT
It seems something to do with caching. The first time the page loads, all of the request have the correct access control header. Subsequent refresh/reload removes this header. How can I continue using caching and keep the access control header?
Edit:
Just to add on, I tried using Safari, the same abnormality appeared. It load on the first try but subsequent loading will throw a no access control allow origin error. However, this time, the response header in Safari shows blank.
Edit 2:
Abnormality appears in Firefox as well but not in Opera Beta version 40.0.2308.52. I am beginning to wonder if this is the cause of the browser. Access-Control-Allow-Origin header is present too.
Response Header in Opera
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Date: Sat, 17 Sep 2016 06:56:53 GMT
Server: Apache/2.2.15 (CentOS)
Vary: Accept
Transfer-Encoding: chunked
Edit 3:
I would like to add that the requests that I'm pulling is a large amount of data. I am pulling the last 7 days history of data, there are about 20 JSON files, each ranging from 500KB to 1000KB. I noticed that when I change the query parameter to pull in a small history of data, this error does not happen.
Any help is much appreciated.

One work around I've found is to dynamically generate a javascript file on the server as resource included into your page. This doesn't require AJAX the which gets blocked by CORS and can be cached.

We had this same problem in Chrome when returning a specific value in the 'Access-Control-Allow-Origin' header. We were able to solve it by also including Vary: Origin in the response headers.
It appears Chrome considers response headers, including 'Access-Control-Allow-Origin', as part of the cached response. So if it requests the same file from a different origin, that header from the cache will not match. But if you include Vary: Origin it lets Chrome know the response (just this one header in our case) will be different based on the Origin of the request.
This page on MDN explains it:
If the server sends a response with an Access-Control-Allow-Origin
value that is an explicit origin (rather than the "*" wildcard), then
the response should also include a Vary response header with the value
Origin — to indicate to browsers that server responses can differ
based on the value of the Origin request header.

I would recommend you to add chrome extension in your browser
Allow-Control-Allow-Origin: *

Javascript GET Request with 301 Redirect

After not being able to figure out what was going wrong, I thought that I'd try my luck on here to see if anyone knows what's going on.
I have an angularJS app with a GoLang/Gorilla mux server backend.
The web app is on http://localhost:8888/ and the server, http://localhost:8080/
Basically I have this simple Javascript GET request:
$.get('http://localhost:8080/api/v1'+'/locations/', {borough: "Queens"})
Using inspect element, in the response headers I can see the following:
Content-Length:68
Content-Type:text/html; charset=utf-8
Date:Sun, 17 Apr 2016 20:12:00 GMT
Location:/api/v1/locations?borough=queens
And the following in the console:
XMLHttpRequest cannot load http://localhost:8080/api/v1/locations/? borough=queens. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8888' is therefore not allowed access.
I try the exact same request using Postman and see a 401 with:
Access-Control-Allow-Origin →*
Content-Length →0
Content-Type →text/plain; charset=utf-8
Date →Sun, 17 Apr 2016 19:46:53 GMT
Which is what I expect. For some reason, it appears that my request is not even making it to the server, which does not offer a 301 request. The fact that Postman and the app give completely different responses is perplexing.
I had some issues with CORS but thought I resolved that. The fact that Postman gives me a different result with the Access-Control-Allow-Origin header makes me think this is something else.
I welcome any help.
Edit:
I made the server reply back with the standard Cors parameters. This is what I get now from Postman:
Access-Control-Allow-Headers →Origin, X-Requested-With, Content-Type, Accept, Authorization
Access-Control-Allow-Methods →GET, POST, PUT, DELETE
Access-Control-Allow-Origin →*
Content-Length →0
Content-Type →text/plain; charset=utf-8
Date →Sun, 17 Apr 2016 20:27:31 GMT
Still get the 301 for the JS call...
Edit2: (tried setting Postman call to exact same as network request)
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Host:localhost:8080
Origin:http://localhost:8888
Referer:http://localhost:8888/root/mainapp/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Still get the same 301 from inspect element and expected 401 with Postman. How can it be that in a Restful HTTP context, we have two identical HTTP calls and yet have completely different behavior out of it?!

So basically when I removed the final slash to make it:
from
$.get('http://localhost:8080/api/v1'+'/locations/', {borough: "Queens"})
to
$.get('http://localhost:8080/api/v1'+'/locations', {borough: "Queens"})
This worked!!! I have no bloody idea as to why. If I would love to hear from anyone that may know what happened here and why this works. I'll reiterate- because of the last slash, the request didn't even make it to the server.

how do I allow cross origin resource sharing? [duplicate]

This question already has answers here:
CORS & example.com
(3 answers)
Closed 7 years ago.
I don't know why my ajax CORS doesn't work..
ajax
$(document).ready(function(){
var xhr = new XMLHttpRequest();
$.ajax({
url: "SERVER_URL_AND_PARAMETERS",
type:"POST",
beforeSend:function(xhr){
xhr.setRequestHeader("Access-Control-Allow-Origin", "*");
xhr.setRequestHeader("Access-Control-Allow-Methods", "GET, POST");
},
dataType:"json",
crossDomain: true,
success:function(data, textStatus, xhr){
alert(data);
},
error:function(xhr,status,error){
alert("code:"+xhr.textStatus+"\n"+"message:"+error.responseText+"\n"+"error:"+error.log);
}
});
});
response headers
Allow:GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS
Cache-Control:no-cache="set-cookie, set-cookie2"
Connection:Keep-Alive
Content-Language:ko-KR
Content-Length:0
Content-Type:text/plain
Date:Mon, 02 Nov 2015 07:19:54 GMT
Expires:Thu, 01 Dec 1994 16:00:00 GMT
Keep-Alive:timeout=10, max=100
Set-Cookie:SOME_COOKIES; Expires=Tue, 01-Nov-16 07:19:53 GMT; Path=/
X-UA-Compatible:IE=EmulateIE8, requiresActiveX=true
request headers
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Access-Control-Request-Headers:accept, access-control-allow-headers, access-control-allow-methods, access-control-allow-origin
Access-Control-Request-Method:POST
Connection:keep-alive
Host:SERVER_URL
Origin:http://CLIENT_URL
Referer:http://CLIENT_URL/AND/JSP_FILE_PATH.jsp?lineCd=CODE1&prdtCode=CODE2
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
chrome error detail
MLHttpRequest cannot load SERVER_URL_AND_PARAMETER Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'CLIENT_URL' is therefore not allowed access.
I don't know what is the problem on my code. I am working on CLIENT_URL side web application.

CORS headers such as "Access-Control-Allow-Origin" MUST be set by the server, not by the client. It is the server that grants CORS access to clients, not the other way around. You can't give yourself CORS access from the browser.
From the MDN section on CORS, here's a descriptive quote:
The Cross-Origin Resource Sharing standard works by adding new HTTP
headers that allow servers to describe the set of origins that are
permitted to read that information using a web browser. Additionally,
for HTTP request methods that can cause side-effects on user data (in
particular, for HTTP methods other than GET, or for POST usage with
certain MIME types), the specification mandates that browsers
"preflight" the request, soliciting supported methods from the server
with an HTTP OPTIONS request method, and then, upon "approval" from
the server, sending the actual request with the actual HTTP request
method. Servers can also notify clients whether "credentials"
(including Cookies and HTTP Authentication data) should be sent with
requests.
In particular, note the part that says "allow servers to describe the set of origins that are permitted to read that information using a web browser".