I want to extract the source code of a webpage which is hosted by other website, but the problem is that O get an empty response, I tried to pull the source of multiple websites but the problem is from my code:
$(document).ready(function(){
$.get('http://www.xxxx.com', function(xdata) {
alert("content: "+xdata);
});
});
Is there any mistake?
Note: when I try to get the source of a local page, it works, but I don't know why it doesn't for an external one
Thanks
This isn't allowed, according to the Same Origin Policy.
The only way to approach this is to use some server-side pull of the data, which you would then process using your AJAX requests, this is known as a Cross-Domain Proxy.
you cant use AJAX across domains
You cannot use contain from an other domain because of the same origin policy
please look into JsonP.
Access-Control-Allow-Origin: * header need to set on external site to do cross domain access.
Because of the SOP (same origin policy), You can't use URL's from other domains. Try accessing a page from local server and don't use http.
If you're not interested in building your own proxy, there's a very easy-to-use public proxy (hosted on AppEngine) for this, with a JavaScript library. CurlJS: http://curljs.azoffdesign.com/
Your example could be done like this (after including the library):
curl("http://www.xxxx.com", function (status, xdata) {
alert("content:" + xdata);
});
Hope that helps!
I have a server using a virtual domain and created an apache proxy.
Super fast, works, no quirks.
Copy this, fix paths (mod_proxy, domains, etc...), add to your .conf file, restart server
LoadModule proxy_module /usr/local/zend/apache2/modules/mod_proxy.so
LoadModule proxy_http_module /usr/local/zend/apache2/modules/mod_proxy_http.so
ProxyRequests Off
ProxyPreserveHost On
ProxyPass /datadomain http://datadomain.com/webservices
ProxyPassReverse /datadomain http://datadomain.com/webservices
Now http://datadomain.com/webservices/data.php = http://yourdomain.com/datadomain/data.php
Enjoy!!!
Related
I have a page with some D3 javascript on. This page sits within a HTTPS website, but the certificate is self-signed.
When I load the page, my D3 visualisations do not show, and I get the error:
Mixed Content: The page at 'https://integration.jsite.com/data/vis' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://integration.jsite.com/data/rdata.csv'. This request has been blocked; the content must be served over HTTPS.
I did some research and all I found what the JavaScript will make the call with the same protocol that the page was loaded. So if page was loaded via https then the rdata.csv should also have been requested via https, instead it is requested as http.
Is this because the certificate is self-signed on the server? What I can do to fix this, other than installing a real SSL certificate?
What I can do to fix this (other than installing a real SSL certificate).
You can't.
On an https webpage you can only make AJAX request to https webpage (With a certificate trusted by the browser, if you use a self-signed one, it will not work for your visitors)
Steps to Allow Insecure Content in Chrome
To allow insecure content on individual sites within Chrome, click on the lock icon in the URL bar, then click 'Site settings'.
There you will see a list of various permissions the page has. Choose 'Allow' next to 'Insecure content'.
Now your HTTPS site can access HTTP endpoint
I had the same issue for my angular project, then I make it work in Chrome by changing the setting. Go to Chrome setting -->site setting -->Insecure content --> click add button of allow, then add your domain name
[*.]XXXX.biz
Now problem will be solved.
You will be able to solve the error by adding this code to your html file:
<meta http-equiv="Content-Security-Policy" content="upgrade-insecure-requests" />
If any solutions don't work, try this solution.
I solved the problem adding a slash at the end of the requesting url
This way: '/data/180/'
instead of: '/data/180'
As for me, I had same warning.
I fixed it at URL request.
I had excessive '/'.
Before:
const url = ${URL}search/movie/?api_key=${API_KEY}&query=${movie};
After:
const url = ${URL}search/movie?api_key=${API_KEY}&query=${movie};
I had the same problem but from IIS in visual studio, I went to project properties -> Web -> and project url change http to https
One solution here server side end point which you access via https, which then makes the call to whichever http url, and then and returns the result. In other words, making your own little HTTPS proxy to access the http resource
update core_config_data
set value='X-Forwarded-Proto'
where path='web/secure/offloader_header'
this is easy,
if you use .htaccess , check http: for https: ,
if you use codeigniter, check config : url_base -> you url http change for https.....
I solved my problem.
I have a page with some D3 javascript on. This page sits within a HTTPS website, but the certificate is self-signed.
When I load the page, my D3 visualisations do not show, and I get the error:
Mixed Content: The page at 'https://integration.jsite.com/data/vis' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://integration.jsite.com/data/rdata.csv'. This request has been blocked; the content must be served over HTTPS.
I did some research and all I found what the JavaScript will make the call with the same protocol that the page was loaded. So if page was loaded via https then the rdata.csv should also have been requested via https, instead it is requested as http.
Is this because the certificate is self-signed on the server? What I can do to fix this, other than installing a real SSL certificate?
What I can do to fix this (other than installing a real SSL certificate).
You can't.
On an https webpage you can only make AJAX request to https webpage (With a certificate trusted by the browser, if you use a self-signed one, it will not work for your visitors)
Steps to Allow Insecure Content in Chrome
To allow insecure content on individual sites within Chrome, click on the lock icon in the URL bar, then click 'Site settings'.
There you will see a list of various permissions the page has. Choose 'Allow' next to 'Insecure content'.
Now your HTTPS site can access HTTP endpoint
I had the same issue for my angular project, then I make it work in Chrome by changing the setting. Go to Chrome setting -->site setting -->Insecure content --> click add button of allow, then add your domain name
[*.]XXXX.biz
Now problem will be solved.
You will be able to solve the error by adding this code to your html file:
<meta http-equiv="Content-Security-Policy" content="upgrade-insecure-requests" />
If any solutions don't work, try this solution.
I solved the problem adding a slash at the end of the requesting url
This way: '/data/180/'
instead of: '/data/180'
As for me, I had same warning.
I fixed it at URL request.
I had excessive '/'.
Before:
const url = ${URL}search/movie/?api_key=${API_KEY}&query=${movie};
After:
const url = ${URL}search/movie?api_key=${API_KEY}&query=${movie};
I had the same problem but from IIS in visual studio, I went to project properties -> Web -> and project url change http to https
One solution here server side end point which you access via https, which then makes the call to whichever http url, and then and returns the result. In other words, making your own little HTTPS proxy to access the http resource
update core_config_data
set value='X-Forwarded-Proto'
where path='web/secure/offloader_header'
this is easy,
if you use .htaccess , check http: for https: ,
if you use codeigniter, check config : url_base -> you url http change for https.....
I solved my problem.
I'm developing a page that pulls images from Flickr and Panoramio via jQuery's AJAX support.
The Flickr side is working fine, but when I try to $.get(url, callback) from Panoramio, I see an error in Chrome's console:
XMLHttpRequest cannot load http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=processImages&minx=-30&miny=0&maxx=0&maxy=150. Origin null is not allowed by Access-Control-Allow-Origin.
If I query that URL from a browser directly it works fine. What is going on, and can I get around this? Am I composing my query incorrectly, or is this something that Panoramio does to hinder what I'm trying to do?
Google didn't turn up any useful matches on the error message.
EDIT
Here's some sample code that shows the problem:
$().ready(function () {
var url = 'http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=processImages&minx=-30&miny=0&maxx=0&maxy=150';
$.get(url, function (jsonp) {
var processImages = function (data) {
alert('ok');
};
eval(jsonp);
});
});
You can run the example online.
EDIT 2
Thanks to Darin for his help with this. THE ABOVE CODE IS WRONG. Use this instead:
$().ready(function () {
var url = 'http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&minx=-30&miny=0&maxx=0&maxy=150&callback=?';
$.get(url, function (data) {
// can use 'data' in here...
});
});
For the record, as far as I can tell, you had two problems:
You weren't passing a "jsonp" type specifier to your $.get, so it was using an ordinary XMLHttpRequest. However, your browser supported CORS (Cross-Origin Resource Sharing) to allow cross-domain XMLHttpRequest if the server OKed it. That's where the Access-Control-Allow-Origin header came in.
I believe you mentioned you were running it from a file:// URL. There are two ways for CORS headers to signal that a cross-domain XHR is OK. One is to send Access-Control-Allow-Origin: * (which, if you were reaching Flickr via $.get, they must have been doing) while the other was to echo back the contents of the Origin header. However, file:// URLs produce a null Origin which can't be authorized via echo-back.
The first was solved in a roundabout way by Darin's suggestion to use $.getJSON. It does a little magic to change the request type from its default of "json" to "jsonp" if it sees the substring callback=? in the URL.
That solved the second by no longer trying to perform a CORS request from a file:// URL.
To clarify for other people, here are the simple troubleshooting instructions:
If you're trying to use JSONP, make sure one of the following is the case:
You're using $.get and set dataType to jsonp.
You're using $.getJSON and included callback=? in the URL.
If you're trying to do a cross-domain XMLHttpRequest via CORS...
Make sure you're testing via http://. Scripts running via file:// have limited support for CORS.
Make sure the browser actually supports CORS. (Opera and Internet Explorer are late to the party)
You need to maybe add a HEADER in your called script, here is what I had to do in PHP:
header('Access-Control-Allow-Origin: *');
More details in Cross domain AJAX ou services WEB (in French).
For a simple HTML project:
Python 2
cd project
python -m SimpleHTTPServer 8000
Python 3
cd project
python -m http.server 8000
Then browse your file.
Works for me on Google Chrome v5.0.375.127 (I get the alert):
$.get('http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=?&minx=-30&miny=0&maxx=0&maxy=150',
function(json) {
alert(json.photos[1].photoUrl);
});
Also I would recommend you using the $.getJSON() method instead as the previous doesn't work on IE8 (at least on my machine):
$.getJSON('http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=?&minx=-30&miny=0&maxx=0&maxy=150',
function(json) {
alert(json.photos[1].photoUrl);
});
You may try it online from here.
UPDATE:
Now that you have shown your code I can see the problem with it. You are having both an anonymous function and inline function but both will be called processImages. That's how jQuery's JSONP support works. Notice how I am defining the callback=? so that you can use an anonymous function. You may read more about it in the documentation.
Another remark is that you shouldn't call eval. The parameter passed to your anonymous function will already be parsed into JSON by jQuery.
As long as the requested server supports the JSON data format, use the JSONP (JSON Padding) interface. It allows you to make external domain requests without proxy servers or fancy header stuff.
If you are doing local testing or calling the file from something like file:// then you need to disable browser security.
On MAC:
open -a Google\ Chrome --args --disable-web-security
It's the same origin policy, you have to use a JSON-P interface or a proxy running on the same host.
We managed it via the http.conf file (edited and then restarted the HTTP service):
<Directory "/home/the directory_where_your_serverside_pages_is">
Header set Access-Control-Allow-Origin "*"
AllowOverride all
Order allow,deny
Allow from all
</Directory>
In the Header set Access-Control-Allow-Origin "*", you can put a precise URL.
In my case, same code worked fine on Firefox, but not on Google Chrome. Google Chrome's JavaScript console said:
XMLHttpRequest cannot load http://www.xyz.com/getZipInfo.php?zip=11234.
Origin http://xyz.com is not allowed by Access-Control-Allow-Origin.
Refused to get unsafe header "X-JSON"
I had to drop the www part of the Ajax URL for it to match correctly with the origin URL and it worked fine then.
As final note the Mozilla documentation explicitly says that
The above example would fail if the header was wildcarded as:
Access-Control-Allow-Origin: *. Since the Access-Control-Allow-Origin explicitly mentions http://foo.example,
the credential-cognizant content is returned to the invoking web
content.
As consequence is a not simply a bad practice to use '*'. Simply does not work :)
Not all servers support jsonp. It requires the server to set the callback function in it's results. I use this to get json responses from sites that return pure json but don't support jsonp:
function AjaxFeed(){
return $.ajax({
url: 'http://somesite.com/somejsonfile.php',
data: {something: true},
dataType: 'jsonp',
/* Very important */
contentType: 'application/json',
});
}
function GetData() {
AjaxFeed()
/* Everything worked okay. Hooray */
.done(function(data){
return data;
})
/* Okay jQuery is stupid manually fix things */
.fail(function(jqXHR) {
/* Build HTML and update */
var data = jQuery.parseJSON(jqXHR.responseText);
return data;
});
}
I use Apache server, so I've used mod_proxy module. Enable modules:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
Then add:
ProxyPass /your-proxy-url/ http://service-url:serviceport/
Finally, pass proxy-url to your script.
For PHP - this Work for me on Chrome, safari and firefox
https://w3c.github.io/webappsec-cors-for-developers/#avoid-returning-access-control-allow-origin-null
header('Access-Control-Allow-Origin: null');
using axios call php live services with file://
I also got the same error in Chrome (I didn't test other browers). It was due to the fact that I was navigating on domain.com instead of www.domain.com. A bit strange, but I could solve the problem by adding the following lines to .htaccess. It redirects domain.com to www.domain.com and the problem was solved. I am a lazy web visitor so I almost never type the www but apparently in some cases it is required.
RewriteEngine on
RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]
Make sure you are using the latest version of JQuery. We were facing this error for JQuery 1.10.2 and the error got resolved after using JQuery 1.11.1
Folks,
I ran into a similar issue. But using Fiddler, I was able to get at the issue. The problem is that the client URL that is configured in the CORS implementation on the Web API side must not have a trailing forward-slash. After submitting your request via Google Chrome and inspect the TextView tab of the Headers section of Fiddler, the error message states something like this:
*"The specified policy origin your_client_url:/' is invalid. It cannot end with a forward slash."
This is real quirky because it worked without any issues on Internet Explorer, but gave me a headache when testing using Google Chrome.
I removed the forward-slash in the CORS code and recompiled the Web API, and now the API is accessible via Chrome and Internet Explorer without any issues. Please give this a shot.
Thanks,
Andy
There is a small problem in the solution posted by CodeGroover above , where if you change a file, you'll have to restart the server to actually use the updated file (at least, in my case).
So searching a bit, I found this one To use:
sudo npm -g install simple-http-server # to install
nserver # to use
And then it will serve at http://localhost:8000.
So, this Perl script:
http://hacheck.tel.fer.hr/xml.pl
will return a XML result based on the POST form-data that it receives.
I have a web-page on one of my domains (none of which are hacheck.tel.fer.hr) and I would like to use that Perl script via Ajax.
Now, the Same origin policy disallows me to send Ajax requests from my domain like so:
$.post('http://hacheck.tel.fer.hr/xml.pl', {'textarea': '...'}, function(data) {
// process data
});
The above code throws this error:
XMLHttpRequest cannot load
http://hacheck.tel.fer.hr/xml.pl.
Origin http://ecmazing.com is not
allowed by
Access-Control-Allow-Origin.
I would like to know what my options are (I would like to be able to use that Perl script). I know that placing my web-page onto the hacheck.tel.fer.hr domain would obviously solve my issue (and that may in fact be doable, but I'll have to contact the admin for that).
But are there any alternatives?
I've heard about CORS. Could it be used to solve my issue? If I understand correctly, with CORS you have to specify on the server that another domain is permitted, and than web-pages from that other domain can receive responses from your server (or something like that) :)?
I've heard about CORS. Could it be used to solve my issue?
Yes, but only in browsers that support it. The controller of hacheck.tel.fer.hr would have to set it up.
But are there any alternatives?
Proxy the request through your own server
I'm developing a page that pulls images from Flickr and Panoramio via jQuery's AJAX support.
The Flickr side is working fine, but when I try to $.get(url, callback) from Panoramio, I see an error in Chrome's console:
XMLHttpRequest cannot load http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=processImages&minx=-30&miny=0&maxx=0&maxy=150. Origin null is not allowed by Access-Control-Allow-Origin.
If I query that URL from a browser directly it works fine. What is going on, and can I get around this? Am I composing my query incorrectly, or is this something that Panoramio does to hinder what I'm trying to do?
Google didn't turn up any useful matches on the error message.
EDIT
Here's some sample code that shows the problem:
$().ready(function () {
var url = 'http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=processImages&minx=-30&miny=0&maxx=0&maxy=150';
$.get(url, function (jsonp) {
var processImages = function (data) {
alert('ok');
};
eval(jsonp);
});
});
You can run the example online.
EDIT 2
Thanks to Darin for his help with this. THE ABOVE CODE IS WRONG. Use this instead:
$().ready(function () {
var url = 'http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&minx=-30&miny=0&maxx=0&maxy=150&callback=?';
$.get(url, function (data) {
// can use 'data' in here...
});
});
For the record, as far as I can tell, you had two problems:
You weren't passing a "jsonp" type specifier to your $.get, so it was using an ordinary XMLHttpRequest. However, your browser supported CORS (Cross-Origin Resource Sharing) to allow cross-domain XMLHttpRequest if the server OKed it. That's where the Access-Control-Allow-Origin header came in.
I believe you mentioned you were running it from a file:// URL. There are two ways for CORS headers to signal that a cross-domain XHR is OK. One is to send Access-Control-Allow-Origin: * (which, if you were reaching Flickr via $.get, they must have been doing) while the other was to echo back the contents of the Origin header. However, file:// URLs produce a null Origin which can't be authorized via echo-back.
The first was solved in a roundabout way by Darin's suggestion to use $.getJSON. It does a little magic to change the request type from its default of "json" to "jsonp" if it sees the substring callback=? in the URL.
That solved the second by no longer trying to perform a CORS request from a file:// URL.
To clarify for other people, here are the simple troubleshooting instructions:
If you're trying to use JSONP, make sure one of the following is the case:
You're using $.get and set dataType to jsonp.
You're using $.getJSON and included callback=? in the URL.
If you're trying to do a cross-domain XMLHttpRequest via CORS...
Make sure you're testing via http://. Scripts running via file:// have limited support for CORS.
Make sure the browser actually supports CORS. (Opera and Internet Explorer are late to the party)
You need to maybe add a HEADER in your called script, here is what I had to do in PHP:
header('Access-Control-Allow-Origin: *');
More details in Cross domain AJAX ou services WEB (in French).
For a simple HTML project:
Python 2
cd project
python -m SimpleHTTPServer 8000
Python 3
cd project
python -m http.server 8000
Then browse your file.
Works for me on Google Chrome v5.0.375.127 (I get the alert):
$.get('http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=?&minx=-30&miny=0&maxx=0&maxy=150',
function(json) {
alert(json.photos[1].photoUrl);
});
Also I would recommend you using the $.getJSON() method instead as the previous doesn't work on IE8 (at least on my machine):
$.getJSON('http://www.panoramio.com/wapi/data/get_photos?v=1&key=dummykey&tag=test&offset=0&length=20&callback=?&minx=-30&miny=0&maxx=0&maxy=150',
function(json) {
alert(json.photos[1].photoUrl);
});
You may try it online from here.
UPDATE:
Now that you have shown your code I can see the problem with it. You are having both an anonymous function and inline function but both will be called processImages. That's how jQuery's JSONP support works. Notice how I am defining the callback=? so that you can use an anonymous function. You may read more about it in the documentation.
Another remark is that you shouldn't call eval. The parameter passed to your anonymous function will already be parsed into JSON by jQuery.
As long as the requested server supports the JSON data format, use the JSONP (JSON Padding) interface. It allows you to make external domain requests without proxy servers or fancy header stuff.
If you are doing local testing or calling the file from something like file:// then you need to disable browser security.
On MAC:
open -a Google\ Chrome --args --disable-web-security
It's the same origin policy, you have to use a JSON-P interface or a proxy running on the same host.
We managed it via the http.conf file (edited and then restarted the HTTP service):
<Directory "/home/the directory_where_your_serverside_pages_is">
Header set Access-Control-Allow-Origin "*"
AllowOverride all
Order allow,deny
Allow from all
</Directory>
In the Header set Access-Control-Allow-Origin "*", you can put a precise URL.
In my case, same code worked fine on Firefox, but not on Google Chrome. Google Chrome's JavaScript console said:
XMLHttpRequest cannot load http://www.xyz.com/getZipInfo.php?zip=11234.
Origin http://xyz.com is not allowed by Access-Control-Allow-Origin.
Refused to get unsafe header "X-JSON"
I had to drop the www part of the Ajax URL for it to match correctly with the origin URL and it worked fine then.
As final note the Mozilla documentation explicitly says that
The above example would fail if the header was wildcarded as:
Access-Control-Allow-Origin: *. Since the Access-Control-Allow-Origin explicitly mentions http://foo.example,
the credential-cognizant content is returned to the invoking web
content.
As consequence is a not simply a bad practice to use '*'. Simply does not work :)
Not all servers support jsonp. It requires the server to set the callback function in it's results. I use this to get json responses from sites that return pure json but don't support jsonp:
function AjaxFeed(){
return $.ajax({
url: 'http://somesite.com/somejsonfile.php',
data: {something: true},
dataType: 'jsonp',
/* Very important */
contentType: 'application/json',
});
}
function GetData() {
AjaxFeed()
/* Everything worked okay. Hooray */
.done(function(data){
return data;
})
/* Okay jQuery is stupid manually fix things */
.fail(function(jqXHR) {
/* Build HTML and update */
var data = jQuery.parseJSON(jqXHR.responseText);
return data;
});
}
I use Apache server, so I've used mod_proxy module. Enable modules:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
Then add:
ProxyPass /your-proxy-url/ http://service-url:serviceport/
Finally, pass proxy-url to your script.
For PHP - this Work for me on Chrome, safari and firefox
https://w3c.github.io/webappsec-cors-for-developers/#avoid-returning-access-control-allow-origin-null
header('Access-Control-Allow-Origin: null');
using axios call php live services with file://
I also got the same error in Chrome (I didn't test other browers). It was due to the fact that I was navigating on domain.com instead of www.domain.com. A bit strange, but I could solve the problem by adding the following lines to .htaccess. It redirects domain.com to www.domain.com and the problem was solved. I am a lazy web visitor so I almost never type the www but apparently in some cases it is required.
RewriteEngine on
RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]
Make sure you are using the latest version of JQuery. We were facing this error for JQuery 1.10.2 and the error got resolved after using JQuery 1.11.1
Folks,
I ran into a similar issue. But using Fiddler, I was able to get at the issue. The problem is that the client URL that is configured in the CORS implementation on the Web API side must not have a trailing forward-slash. After submitting your request via Google Chrome and inspect the TextView tab of the Headers section of Fiddler, the error message states something like this:
*"The specified policy origin your_client_url:/' is invalid. It cannot end with a forward slash."
This is real quirky because it worked without any issues on Internet Explorer, but gave me a headache when testing using Google Chrome.
I removed the forward-slash in the CORS code and recompiled the Web API, and now the API is accessible via Chrome and Internet Explorer without any issues. Please give this a shot.
Thanks,
Andy
There is a small problem in the solution posted by CodeGroover above , where if you change a file, you'll have to restart the server to actually use the updated file (at least, in my case).
So searching a bit, I found this one To use:
sudo npm -g install simple-http-server # to install
nserver # to use
And then it will serve at http://localhost:8000.