How can we get the source code of a webpage from a webpage in php and/or javascript?
In Javascript without using unnecessary frameworks (in the example api.codetabs.com is a proxy to bypass Cross-Origin Resource Sharing):
fetch('https://api.codetabs.com/v1/proxy?quest=google.com').then((response) => response.text()).then((text) => console.log(text));
Thanks to:
#PLB
#Shadow Wizard
Getting the source code of an iframe
http://www.frihost.com/forums/vt-32602.html
#Matt Coughlin.
First, you must know that you will never be able to get the source code of a page that is not on the same domain as your page in javascript. (See http://en.wikipedia.org/wiki/Same_origin_policy).
In PHP, this is how you do it :
file_get_contents($theUrl);
In javascript, there is three ways :
Firstly, by XMLHttpRequest : http://jsfiddle.net/635YY/1/
var url="../635YY",xmlhttp;//Remember, same domain
if("XMLHttpRequest" in window)xmlhttp=new XMLHttpRequest();
if("ActiveXObject" in window)xmlhttp=new ActiveXObject("Msxml2.XMLHTTP");
xmlhttp.open('GET',url,true);
xmlhttp.onreadystatechange=function()
{
if(xmlhttp.readyState==4)alert(xmlhttp.responseText);
};
xmlhttp.send(null);
Secondly, by iFrames : http://jsfiddle.net/XYjuX/1/
var url="../XYjuX";//Remember, same domain
var iframe=document.createElement("iframe");
iframe.onload=function()
{
alert(iframe.contentWindow.document.body.innerHTML);
}
iframe.src=url;
iframe.style.display="none";
document.body.appendChild(iframe);
Thirdly, by jQuery : http://jsfiddle.net/edggD/2/
$.get('../edggD',function(data)//Remember, same domain
{
alert(data);
});
Following Google's guide on fetch() and using the D.Snap answer, you would have something like this:
fetch('https://api.codetabs.com/v1/proxy?quest=URL_you_want_to_fetch')
.then(
function(response) {
if (response.status !== 200) {
console.log('Looks like there was a problem. Status Code: ' +
response.status);
return;
}
// Examine the text in the response
response.text().then(function(data) {
// data contains all the plain html of the url you previously set,
// you can use it as you want, it is typeof string
console.log(data)
});
}
)
.catch(function(err) {
console.log('Fetch Error :-S', err);
});
This way you are using a CORS Proxy, in this example it is Codetabs CORS Proxy.
A CORS Proxy allows you to fetch resources that are not in your same domain, thus avoiding the Same-Origin policies blocking your requests.
You can take a look at other CORS Proxys:
https://nordicapis.com/10-free-to-use-cors-proxies/
Ajax example using jQuery:
// Display the source code of a web page in a pre tag (escaping the HTML).
// Only works if the page is on the same domain.
$.get('page.html', function(data) {
$('pre').text(data);
});
If you just want access to the source code, the data parameter in the above code contains the raw HTML source code.
Related
when i debug this code in to chrome console then its not show any output or alert! please help me to complete this code! i need to get read my read.txt file text in to console.log....
the code was i try one is shows below.
function loadText() {
fetch('C:\Windows\Temp\read.txt')
.then(function(response){
return response.text();
})
.then(function(data){
console.log(data);
alert(data)
})
.catch(function(error){
console.log(error);
alert(data)
})
}
Try below code and indicate your directory like below
async function fetchText() {
let response = await fetch('../demo.txt');
console.log(response.status); // 200
console.log(response.statusText); // OK
if (response.status === 200) {
let data = await response.text();
console.log(data);
// handle data
}
}
fetchText();
This seems like a duplicate of this issue - AJAX request to local file system not working in Chrome?
The problems are the same, however you are using the fetch API (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) not an XMLHttp request (https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest)
When i run loadText i get the following error:-
Fetch API cannot load . URL scheme must be "http" or "https" for CORS request.
You cannot make requests to the filesystem in Chrome. However you can disable Chrome security using flags (--allow-file-access-from-files)
see - Allow Google Chrome to use XMLHttpRequest to load a URL from a local file however this is not advised.
You will also need to update your path in the fetch function by prefixing with file:/// this tells it to look in the file system, and changed the protocol from http or https.
Question
I need to parse an RSS feed and display the parsed details in an HTML page.
Solution I Found
How to parse an RSS feed using JavaScript? is a very similar question and I followed it.
Using above question, I build the following code.
<script>
$(document).ready(function() {
//feed to parse
var feed = "https://feeds.feedburner.com/raymondcamdensblog?format=xml";
$.ajax(feed, {
accepts:{
xml:"application/rss+xml"
},
dataType:"xml",
success:function(data) {
//Credit: http://stackoverflow.com/questions/10943544/how-to-parse-an-rss-feed-using-javascript
$(data).find("item").each(function () { // or "item" or whatever suits your feed
var el = $(this);
document.write("------------------------");
document.write("title : " + el.find("title").text());
document.write("link : " + el.find("link").text());
document.write("description: " + el.find("description").text());
});
}
});
});
</script>
The Error
Failed to load
https://feeds.feedburner.com/raymondcamdensblog?format=xml: No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://localhost' is therefore not allowed access.
What I need
How can I change my code to read RSS feeds using JavaScript without getting above error?
You could use something like https://rss2json.com.
It parses the feed to json for javascript:
var feedURL = "https://feeds.feedburner.com/raymondcamdensblog?format=xml";
$.ajax({
type: 'GET',
url: "https://api.rss2json.com/v1/api.json?rss_url=" + feedURL,
dataType: 'jsonp',
success: function(result) {
console.log(result);
}
});
You're getting that error because of the same-origin policy. See below and/or read the full article at MDN:
For security reasons, browsers restrict cross-origin HTTP requests
initiated from within scripts. For example, XMLHttpRequest and the
Fetch API follow the same-origin policy. This means that a web
application using those APIs can only request HTTP resources from the
same origin the application was loaded from, unless the response
from the other origin includes the right CORS headers.
So your script is making a cross-origin HTTP request (which uses XMLHttpRequest through jQuery.ajax()) to https://feeds.feedburner.com/raymondcamdensblog?format=xml, but the CORS header of Access-Control-Allow-Origin is not being set by FeedBurner, therefore you get the "Failed to load ..." error. (But even if the header was set, if it didn't include your origin (localhost or some-domain.com), you'd still get the same error.)
So how can you change your code to read the RSS feeds using JavaScript without getting that error?
Use a third-party web service, just like what #Saeed suggested.
Create a server-side script (e.g. using PHP) that fetches the feed content and make AJAX requests to that script instead of directly requesting it from FeedBurner, or the actual source URL. See below for a simple example.
If I really had to, I'd probably ask FeedBurner to set the appropriate CORS headers...
Sample of a very simple PHP script for fetching the feed content:
<?php
// Set the feed URL.
$feed_url = 'https://feeds.feedburner.com/raymondcamdensblog?format=xml';
// Fetch the content.
// See http://php.net/manual/en/function.file-get-contents.php for more
// information about the file_get_contents() function.
$content = file_get_contents( $feed_url );
// Set the Content-Type header.
header( 'Content-Type: application/rss+xml' );
// Display the content and exit.
echo $content;
exit;
?>
So for example, you could save that to fetch-feed.php, and then in your JavaScript/jQuery script code, change the value of the feed variable like so:
var feed = "http://localhost/path/to/fetch-feed.php";
That way (i.e. using your own server-side script), you could at least be sure that the browser would always grant your XMLHttpRequest (or AJAX) request. (i.e. you wouldn't get the "No 'Access-Control-Allow-Origin' header" error)
You can also use jquery-rss or Vanilla RSS, which comes with nice templating and is super easy to use:
// Example for jquery.rss
$("#your-div").rss("https://feeds.feedburner.com/raymondcamdensblog?format=xml", {
limit: 3,
layoutTemplate: '<ul class="inline">{entries}</ul>',
entryTemplate: '<li>[{author}#{date}] {title}<br/>{shortBodyPlain}</li>'
})
// Example for Vanilla RSS
const RSS = require('vanilla-rss');
const rss = new RSS(
document.querySelector("#your-div"),
"https://feeds.feedburner.com/raymondcamdensblog?format=xml",
{
// options go here
}
);
rss.render().then(() => {
console.log('Everything is loaded and rendered');
});
See http://jsfiddle.net/sdepold/ozq2dn9e/1/ for a working example.
It's a CORS related error. You are getting that error because the URL from where you are requesting data does not have CORS enabled. CORS stands for 'Cross-Origin Resource Sharing'. If CORS is enabled on a server, your browser will let you make requests to that server. Otherwise, it will not.
https://feeds.feedburner.com/raymondcamdensblog?format=xml does not have CORS enabled, that's why your browser will not allow you to make ajax requests to that server. You can get around it by making the requests on your server and provide the data to the browser from your own server or a server that has CORS enabled.
i want to make a script that makes every video's comment section look like the ones that still have the old kind.
for example, videos on this channel:https://www.youtube.com/user/TheMysteryofGF/videos
in Firebug, in the Net tab, i noticed the comment JSON file's URL it is requested from is different.
i tried to run a code on the youtube watch page which would request the file the same way, but it doesnt work, and in firebug it says it was forbidden.
the URL is the same, they are both POST, and i cant figure out what is different. i can even resend the original request in firebug and it works... so anyway, here is a code i tried on a video with "1vptNpkysBQ" video url.
var getJSON = function(url, successHandler, errorHandler) {
var xhr = typeof XMLHttpRequest != 'undefined'
? new XMLHttpRequest()
: new ActiveXObject('Microsoft.XMLHTTP');
xhr.open('post', url, true);
xhr.onreadystatechange = function() {
var status;
var data;
// https://xhr.spec.whatwg.org/#dom-xmlhttprequest-readystate
if (xhr.readyState == 4) { // `DONE`
status = xhr.status;
if (status == 200) {
data = JSON.parse(xhr.responseText);
successHandler && successHandler(data);
} else {
errorHandler && errorHandler(status);
}
}
};
xhr.send();
};
getJSON('https://www.youtube.com/watch_fragments_ajax?v=1vptNpkysBQ&tr=time&frags=comments&spf=load', function(data) {
alert('Your public IP address is: ' + data);
}, function(status) {
alert('Something went wrong.');
});
You are using Ajax to get data. Ajax has 1 restriction: You can only get data from your own server. When you try to get data from another server/domain, you get a "Access-Control-Allow-Origin" error.
Any time you put http:// (or https://) in the url, you get this error.
You'll have to do it the Youtube way.
That's why they made the javascript API. Here is (the principal of) how it works. You can link javascript files from other servers, with the < script > tag
So if you could find a javascript file that starts with
var my_videos = ['foo', 'bar', 'hello', 'world'];
then you can use var my_videos anywhere in your script. This can be used both for functions and for data. So the server puts this (dynamically generated) script somewhere, on a specific url. You, the client website can use it.
If you want to really understand it, you should try building your own API; you'll learn a lot.
Secondary thing: Use GET.
POST means the client adds data to the server (example: post a comment, upload a file, ...). GET means you send some kind of ID to the server, then the server returns its own data to the client.
So what you are doing here, is pure GET.
I want to retrieve a HTML page as document inside a Firefox/Greasemonkey userscript.
Edit: This is not a cross-domain request.
Here's my example code:
var r = new XMLHttpRequest();
r.open("GET", document.location.href, true);
r.responseType = "document";
r.send(null);
This looks just like the example in https://developer.mozilla.org/en/HTML_in_XMLHttpRequest ,
but r.send(null) causes a TypeError. Causes, not throws! Wrapping the line in a try...catch won't change anything, it seems like a callback or an event handler raises the exception:
TypeError: document.location is null
The traceback refers to a Firefox-internal event.js file, but not to my script.
Removing the line setting the responseType gets rid of the exception, adding callbacks does not.
However, the response is valid and responseXML provides a DOM tree.
I'm using FF 13.0.1.
Am I missing something or is this a bug?
Solution: This had something to do with an extension created by Mozilla's Addon Builder, not Firefox.
The script is running on google.com and you are trying to fetch google.de, right? That's a cross-domain request. (Also, the question code is not a valid synch or asynch use of XMLHttpRequest.)
To do cross-domain (or not) AJAX in a Greasemonkey script (Or Chrome), use GM_xmlhttpRequest().
Note that GM_xmlhttpRequest() does not currently let you specify responseType, but you don't need to do that in this case anyway. If you want a nice parsed document, use DOMParser.
Putting it all together:
GM_xmlhttpRequest ( {
method: 'GET',
//url: 'https://www.google.de/',
url: location.href, // self get, checking for updates
onload: function (respDetails) {
processResponse (respDetails);
}
} );
function processResponse (respDetails) {
// DO ALL RESPONSE PROCESSING HERE...
var parser = new DOMParser ();
var doc = parser.parseFromString (respDetails.responseText, "text/html");
//--- Example showing that the doc is fully parsed/functional...
console.log (doc.querySelectorAll ("p") );
}
PS: Since this is not cross-domain after all, the original code, corrected would be:
var r = new XMLHttpRequest();
r.onload = function () {
// DO ALL RESPONSE PROCESSING HERE...
console.log (this.response.querySelectorAll ("div") );
}
r.open ("GET", location.href, true);
r.responseType = "document";
r.send (null);
for an asynchronous request.
Unfortunately, you cannot do Ajax from one domain to another:
http://en.wikipedia.org/wiki/Same_origin_policy
You can read into CORS:
http://en.wikipedia.org/wiki/Cross-origin_resource_sharing
or JSONP as possible solutions:
http://en.wikipedia.org/wiki/JSONP
However, browsers are designed in such a way so that people can't just randomly create Ajax requests across domains due to this being a security issue.
If you absolutely need to grab content off a different domain, I'd look into creating your own server API using cURL, serving your own content on the same domain, and then using Ajax there. Otherwise, you'll have to see if Google will grant CORS access or has some sort of built in JSONP request.
So I have a bit of a problem. When I ask MooTools to send a request it comes back as failed every time. I can't seem to diagnose the problem either because if I try to get the returned header info the console just gives me "Refused to get unsafe header 'Status'" Message. The only thing I can think of is that the server isn't letting me access outside resources but maybe I just coded it wrong.
Here's the request code:
var finfo = current.textFontData();
var url = 'http://antiradiant.com/clients/TMW/rbwizard/mailer.php?s='+current.size+'&b='+current.box+'&l='+current.lidWood+'&c='+current.cartID+'&f='+finfo.font+'&l1='+finfo.line1+'&l2='+finfo.line2;
console.log(url);
var req = new Request({
url: url,
onSuccess: function() {
console.log('success');
//atc2.send();
},
onFailure: function() {
console.log('failure');
console.log(this.getHeader('Status'));
//atc2.send();
},
onException: function(headerName, value) {
console.log('exception');
console.log(headerName+': '+value);
}
});
req.send();
This code is derived from the resource rb_wizard.js (lines 81-103) on http://tylermorriswoodworking.myshopify.com/pages/recipe-box-wizard?b=maple&l=cherry&s=3x5&c=42042892
Mootools has a class called Request.JSONP that will help with your cross domain problem. Its sub class of the Request class, so your methods should work the same. I believe you need to call .post() or .get() at the end instead of send, but thats about all that should chnge. I'm not sure what version you're running on but here is the link tot he docs Mootools Request.JSONP
The error message "Refused to get unsafe header 'Status'" is spat out by WebKit based browsers (Safari, Chrome, etc) when you violate the cross-domain security model.
Therefore, it seems likely that the code you pasted is located on a domain other than antiradiant.com, and therefore is not allowed (by the browser) to request sites on antiradiant.com.
What I ended up doing was just using an iframe. All I really had to do was send data to another site and not receive any so it worked out.