Getting the source code of web pages in the same network - javascript

The tool that I'm developing takes as parameter a url and retrieves the source code from that url to analyze. The tool will be hosted in our domain, and it should have the ability to analyze web pages of sites that are in the same internal network domain. I make this Jquery.ajax call to retrieve the source code :
$.ajax({
url: siteToanalyze,
type : 'GET',
dataType : 'html',
success: function (result) {
content = result;
analyzeCode(content, siteToanalyze);
},
error: function (jqxhr, status, errorThrown) {
alert("Failure, Unable to recieve content");
}
});
I can get the code using YQL for external web sites as well. I can get source code for web pages from the same origin. But what about web pages in the same internal network, that are not accessible to YQL.
For instance, if this js code is served from 172.20.1.160:5000 and we give a site in 172.20.5.68:5000 in the same network. The web console shows a HTTP 200 status request for the url, but results in the error function being run. Why does that happen? Or is there some other way to get source code of web pages other than the way I mentioned?

The issue was indeed because of the Same origin policy. The content I wanted to access is from a different origin than that of the above code. I solved the issues using CORS

Related

Scrape info from external XML url with jQuery

I need to scrape/parse some info from an external XML file (xml hosted on other domain) and place that info on my site.
I tried this, and didn't succeed:
jQuery(document).ready(function()
{
jQuery.ajax({
type: 'GET',
url: 'http://42netmedia.com/smart/signal_onAir10.xml',
crossDomain: true,
dataType: 'jsonp',
success: parseXml
});
});
function parseXml(xml)
{
jQuery(xml).find('nowOnAirTitle').each(function()
{
jQuery(".my-site-element").append(jQuery(this).find('nowOnAirTitle').text());
});
}
I hope I managed to explain properly.
PS. I am using jQuery because my site is hosted on WordPress
The short answer is that you cannot do what you're trying to do from Javascript running in a browser.
The same origin policy will block AJAX requests for URLs on different domains. There are a couple of exceptions to this policy:
If the target server supports CORS, it can indicate that cross-origin requests are OK. The URL you are trying to reach is on a server that does not support CORS.
If the target server supports JSONP, your script can retrieve data from the target server by using a <script> tag in your page with a src attribute pointing to the target URL and including a callback parameter to tell the server to wrap the data in a javascript function that returns the data. The URL you are trying to reach is on a server that does not support JSONP.
Yes, jQuery does allow you to make a JSONP request across to the other domain, but the response is not Javascript but XML. Your browser is trying to execute the XML as if it was Javascript and fails with a syntax error.
Any solution to this will require server-side support on the server which hosts your code. You could set up a proxy URL on your web server, but you'd need to do that very carefully because you don't want your server to be used as an open proxy. You could also use a cron job to grab the URL using curl and save it to a static file on your server, but that's probably a bit rude.

slideshare oembed api and Access-Control-Allow-Origin

I'd like to embed a slideshare iframe based on its URL. http://www.slideshare.net/developers/oembed says I should be able to get a JSON object about the resource and within that is the embed code to use. E.g. if I open
Reference Link
in the browser, I get JSON. If I open the same via jquery
$.getJSON('http://www.slideshare.net/api/oembed/2?url=http://www.slideshare.net/lyndadotcom/code-drivesworld12&format=json', function (obj) { console.log(obj) });
then I get
XMLHttpRequest cannot load http://www.slideshare.net/api/oembed/2?url=http://www.slideshare.net/lyndadotcom/code-drivesworld12&format=json&callback=myFunction. No 'Access-Control-Allow-Origin' header is present on the requested resource.
Ideally I'd like to also control slideshare from an external link - which this page - http://www.slideshare.net/developers - says is possible if you follow the SlideShare Player API - which is a 404 page.
Googling for slideshare developer just links me to many shared slideshows, not actual help that exists or works ... grrr
When you try to load JSON from another domain, it fails because there is a domain boundary you can not cross.To avoid this, you have to use JSON with Padding. JSONP or "JSON with padding" is a communication technique used in JavaScript programs running in web browsers to request data from a server in a different domain, something prohibited by typical web browsers because of the same-origin policy. Try this following code mate.Hope this might help you. :)
$.ajax({
type: "GET",
url: 'http://www.slideshare.net/api/oembed/2?url=http://www.slideshare.net/lyndadotcom/code-drivesworld12&format=jsonp',
dataType: 'jsonp',
success: function (data) {
$('.dynamic').html(data.html);
}
});
Fiddle here

No 'access-control-allow-origin' header is present in AJAX request

I am using the following URL to get data for a weather app.
http://wsf.cdyne.com/WeatherWS/Weather.asmx/GetCityWeatherByZIP
I've used this API before, but this time I'm trying to load the data in using AJAX so the rest of my page doesnt reload when this data is fetched.
Here is my javascript
<script>
$(function () {
var zip = 16001;
$.ajax({
type: 'GET',
crossDomain:'true',
url: 'http://wsf.cdyne.com/WeatherWS/Weather.asmx/GetCityWeatherByZIP' +zip,
success: function (data) {
console.log("Here is the data", data);
},
error: function () {
alert("Error loading data");
}
});
});
</script>
I keep recieving the following error in the console.
XMLHttpRequest cannot load http://wsf.cdyne.com/WeatherWS/Weather.asmx/GetCityWeatherByZIP16001. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:50733' is therefore not allowed access. The response had HTTP status code 500. weatherAjax.html:1
This makes it seem like their is an issue with accessing the data. But i've used this before with a simple form that the user input their zip code into. And it returned the data. The only difference now is I want to load the data using AJAX so the entire page doesn't reload. What am I doing wrong?
Previously you were sending the user's browser to someone else's website. They were leaving your site and all was fine.
Now you are trying to write JavaScript to instruct your visitor's browser to fetch data from someone else's website (which would use any auth/authz credentials they had) and give that data to your code.
Since the other website could be, for example, an online bank, this is forbidden unless the site you are requesting the data from gives explicit permission for you to access it.
For further reading, see the Same Origin Policy and HTTP access control (CORS).

consuming PHP service returning valid json with $.ajax

I've been stuck on consuming a web service created in PHP, not sure what I'm doing wrong.. Ive created a fiddle example here : http://jsfiddle.net/e97AV/
I've tried various combinations of things but keep on getting 404 not found feedback, when I specify jsonp i get no error message, but in the web console i can see a 404 error.. in the browser when I visit the url it is returning valid json
My question is how would I know when to use jsonp or json? Also these service have been provided to me from an external source other than agreeing on json being returned how would I know if the problem is on my side or theirs?
heres the ajax code
baseUrl = "http://exclusivegetaways.co.za/api.php";
$.ajax({
type: "GET",
url: baseUrl,
data: {something : "something"},
//contentType: "application/json; charset=utf-8",
dataType: "json",
success: function (result) {
alert("works");
alert(result);
},
error: function (a,b,cc) {
alert(a+b+cc);
}
});
I've since been able to pull json data from the ajax error object?? like so:
baseUrl = "http://exclusivegetaways.co.za/api.php?something=something";
$.ajax({
type: "GET",
url: baseUrl,
dataType: "json",
success: function (res) {
alert("worked");
//alert(res);
},
error: function(jqxhr) {
try {
f = JSON.parse(jqxhr.responseText);
...valid json returned here
} catch(err) {alert(err);}
}
});
This is because of a security restriction that prevents Ajax from querying remote locations.
As a workaround to enable access to a remote location via Ajax, you could build a custom URL in your webApp (in PHP for instance) which queries the distant API and returns JSON.
Then, in your JavaScript, you call this URL (from your application) via Ajax.
First: Always look at your JavaScript error console.
XMLHttpRequest cannot load http://exclusivegetaways.co.za/api.php?location=provinces.
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://fiddle.jshell.net' is therefore not allowed access.
See also Ways to circumvent the same-origin policy
I've tried various combinations of things but keep on getting 404 not found feedback, when I specify jsonp i get no error message, but in the web console i can see a 404 error. in the browser when I visit the url it is returning valid json
This suggests that:
They don't support JSONP
They look at the HTTP headers and 404 your request to block access from Ajax (this isn't a good way to do that, the error code is misleading)
My question is how would I know when to use jsonp or json?
Usually by reading the documentation for the server you are trying to use
Also these service have been provided to me from an external source other than agreeing on json being returned how would I know if the problem is on my side or theirs?
Usually by working with whatever support is provided by the API provider (i.e. start with their documentation, then fall back to whatever means they provide for communicating with a human).
Due to Same Origin Policy your ajax request is allowed only if:
domain name, application layer protocol, and (in most browsers) port
number of the HTML document running the script are the same
In your case the application layer protocol is different, that's why your script fails.
Possible solutions are:
JSONP, which has to be provided by the server
CORS, which is a more 'elegant' and clean solution, but is not yet fully supported by IE (IE7 doesn't support it, IE8 has some limitations)
Answer taken from this link

Error "Origin null is not allowed by Access-Control-Allow-Origin" when using JQuery to update OData

I've created a database locally, and used Microsoft's WCF Data Services to create an OData service. I've managed to figure out how to read the data, but when attempting to update, Google Chrome gives this error:
"Origin null is not allowed by Access-Control-Allow-Origin."
This only happens when I open my HTML page directly from my C drive (without a web server). If I go via my web server, then it works. Any ideas as to how I can get this to work without using a web server?
Here's my code:
var results=BOData.StephenBO1;
results[0].txtLastStage = $("#txtLastStage").val();
results[0].txtTeamCode = $("#txtTeamCode").val();
results[0].txtClientName = $("#txtClientName").val();
var url = "http://localhost/odata/StephenService.svc/CL_Darwin1('0900000000000000000000000000276')";
var json = JSON.stringify(results[0]);
$.ajax({
url: url,
data: json,
type: "PUT",
contentType: "application/json; charset=utf-8",
dataType: "json",
success: function (result) {
alert("Saved StephenBO1");
},
error: function (result) {
alert("Update Failure - Status Code=" +
result.status + ", Status=" + result.statusText);
}
});
Any wise and intelligent comments would be appreciated... and let me know if you need any additional info.
Thanks,
Stephen
In my opinion this SO Question (and answer) can resolve your problem.
From http://datajs.codeplex.com/documentation:
Browsers have a policy (commonly referred to as the same origin policy. that blocks requests across domain boundaries. Because of this restriction update operations cannot be performed if the web page is served by a domain and the target OData endpoint is in a different one. Users have the ability to disable this policy in the browser, however it is typically turned on by default. datajs is designed with such an assumption. The following options are available to support this scenario:
Have the web server provide a relaying mechanism to redirect requests to the appropriate OData endpoint.
Use an XDomainRequest object. This option is not available in all browsers.
Use cross origin XMLHttpRequest object. This option is also not available in all browsers.
Prompt for consent on first use. This option is not available in all browsers and generally provides a poor user experience.

Categories