Is there any way (in Javascript) to download a remote website (i.e. like with Curl), read it into a string variable and further process it?
You can only download a file from the same domain, as per the Same Origin Policy. You can download content from the same domain though, using the XMLHTTPRequest object:
var xhReq = createXMLHttpRequest();
xhReq.open("GET", "page.html", true);
xhReq.onreadystatechange = onResponse;
xhReq.send(null);
...
function onResponse() {
if (xhReq.readyState != 4) { return; }
var serverResponse = xhReq.responseText;
...
}
There are ways to circumvent the policy, some of them listed in the same Wikipedia page. But it's a hack at best and illegal at worst.
Sure- The url must be from the same domain, unless the url has a cross domain policy or you create a server side proxy script.
The following code is an example of an ajax call to any domain through a proxy PHP script:
var xmlhttp = new XMLHttpRequest();
xmlhttp.open("POST","http://localhost/proxy.php?url=http://google.com", true);
xmlhttp.onreadystatechange = function() {
if (request.readyState == 4 && request.status == 200) {
// ensure we have a response...
if (xmlhttp.responseText) {
var html = xmlhttp.responseText;
// do your processing here...
}
}
};
xmlhttp.send();
You then would make your proxy.php script connect to the given url via Curl (or whatever url library your sever side language has) and then simply echo the content from your domain...
<?php
// proxy.php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$_GET["url"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);
echo $result;
?>
hope that all makes sense.
You can use the Yahoo Query Language to query any page on the web.
For example, if you want the full source of the Google homepage, you could use:
select * from html where url="http://google.com" and xpath='/html' limit 1
You'd have to use their JSON callback and reserialize the returned object, but you'd be able to get a full view of the page.
Mostly you won't be allowed. Javascript will prevent you doing this for security reasons. However, you can request json data from other domains using jQuery. Here is an example from the jquery docs that gets some cat pictures from flickr...
$.getJSON("http://api.flickr.com/services/feeds/photos_public.gne?tags=cat&tagmode=any&format=json&jsoncallback=?",
function(data){
$.each(data.items, function(i,item){
$("<img/>").attr("src", item.media.m).appendTo("#images");
if ( i == 4 ) return false;
});
});
You can find this code in the jQuery Docs. As you can see, this makes a request, gets the data back and updates some image tags in the DOM with the cat pictures...
Related
I'm working on a real-time timetable for a scool. Now when i'm testing it i'm getting an Error only in internet explorer. I use ie v.11 and charset UTF-8
When i send the data of the current student to a php-script there are some characters like ä,ö,ü which get parse to a ?. But only in InternetExplorer.
if i sent a URL like this: http://adopraesenz.ipwin.ch/data/students.php?q=getWirth Nalia;3:Kürzel;0;0;4;0;0;0;
and want to receive them in the php like this
$q = $_REQUEST["q"];
echo $q;
i get this output: getWirth Nalia;3:K?rzel;0;0;4;0;0;0;
I'm sending data like this:
function getData(str) {
var requestURL = "http://adopraesenz.ipwin.ch/data/students.php?q=" +str;
var request = new XMLHttpRequest();
request.onreadystatechange = function() {
if(this.readyState === 4 && this.status === 200){
loadJson(request);
}
};
request.open("GET", requestURL, true);
request.send();
}
for more code or information please ask.
In PHP you should always use urlencode and urldecode if you want to send data over a GET request. First encode your parameters, and after receiving decode them again. Most webservers will do the decoding for you, so you only have to call the encode code most of the time.
Also check that your form is using the same charset in html and http-header (utf-8 prefered)
Examples from official php documentation
encode
<?php
echo '<a href="mycgi?foo=', urlencode($userinput), '">';
decode (webserver will do this for you most of the time)
<?php
$query = "my=apples&are=green+and+red";
foreach (explode('&', $query) as $chunk) {
$param = explode("=", $chunk);
if ($param) {
printf("Value for parameter \"%s\" is \"%s\"<br/>\n", urldecode($param[0]), urldecode($param[1]));
}
}
An easier solution would be to use a post request instead of get, this is the prefered way for sending json data.
so I have a self written php web service for an app.
Now I want to do the same with the web. But I struggle with the login.
I need to call 2 functions of the web service. One to get a login token another one to verify that and request a access token.
So I call them with javascript:
<?php
if(isset($_POST['email']) && isset($_POST['password']))
{
?>
<script>
var xhttp = new XMLHttpRequest();
xhttp.open("POST", "http://webservice.com/requestLoginToken", false);
xhttp.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
xhttp.send(<?php echo "'email=" . urlencode($_POST['email']) . "'"?>);
if(xhttp.status == 401)
document.getElementById("message").innerHTML = xhttp.responseText;
else
{
var loginToken = xhttp.responseText;
var pwHash = loginToken + <?php echo "'" . sha1($_POST['password']) . "'" ?>;
pwHash = sha1(pwHash);
xhttp = new XMLHttpRequest();
xhttp.open("POST", "http://webservice.com/authenticate", false);
xhttp.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
xhttp.send("loginToken=" + loginToken + "&publicHash=" + pwHash);
if(!xhttp.status == 200)
document.getElementById("message").innerHTML = xhttp.responseText;
else
{
var accesstoken = xhttp.responseText;
}
}
</script>
<?php
}
?>
So now I have the access token. But how can I set this now as a session variable of $_SESSION?
Or what is the best way so save this access token during the stay?
I need this token whenever I call the API.
Not a direct answer to your question, but you should definitely consider using a proper framework or library, even for something that small. Code Igniter as far I remember is easy to get started with.
For instance, that line: urlencode($_POST['email']) is a big security issue, allowing XSS attacks.
Now, back to your question. What you need is to set a session cookie, then the browser will take care of it for you. Setting a cookie from JS is considered a bad practice (as it prevents using HttpOnly cookies, a recommended security flag), the easiest would be that the server answering to the login AJAX call set a cookie on the response.
That cookie will be issued to any subsequent calls to the same server. Again, if you use a proper framework, all the authentication stuff will be handled for you.
i want to make a script that makes every video's comment section look like the ones that still have the old kind.
for example, videos on this channel:https://www.youtube.com/user/TheMysteryofGF/videos
in Firebug, in the Net tab, i noticed the comment JSON file's URL it is requested from is different.
i tried to run a code on the youtube watch page which would request the file the same way, but it doesnt work, and in firebug it says it was forbidden.
the URL is the same, they are both POST, and i cant figure out what is different. i can even resend the original request in firebug and it works... so anyway, here is a code i tried on a video with "1vptNpkysBQ" video url.
var getJSON = function(url, successHandler, errorHandler) {
var xhr = typeof XMLHttpRequest != 'undefined'
? new XMLHttpRequest()
: new ActiveXObject('Microsoft.XMLHTTP');
xhr.open('post', url, true);
xhr.onreadystatechange = function() {
var status;
var data;
// https://xhr.spec.whatwg.org/#dom-xmlhttprequest-readystate
if (xhr.readyState == 4) { // `DONE`
status = xhr.status;
if (status == 200) {
data = JSON.parse(xhr.responseText);
successHandler && successHandler(data);
} else {
errorHandler && errorHandler(status);
}
}
};
xhr.send();
};
getJSON('https://www.youtube.com/watch_fragments_ajax?v=1vptNpkysBQ&tr=time&frags=comments&spf=load', function(data) {
alert('Your public IP address is: ' + data);
}, function(status) {
alert('Something went wrong.');
});
You are using Ajax to get data. Ajax has 1 restriction: You can only get data from your own server. When you try to get data from another server/domain, you get a "Access-Control-Allow-Origin" error.
Any time you put http:// (or https://) in the url, you get this error.
You'll have to do it the Youtube way.
That's why they made the javascript API. Here is (the principal of) how it works. You can link javascript files from other servers, with the < script > tag
So if you could find a javascript file that starts with
var my_videos = ['foo', 'bar', 'hello', 'world'];
then you can use var my_videos anywhere in your script. This can be used both for functions and for data. So the server puts this (dynamically generated) script somewhere, on a specific url. You, the client website can use it.
If you want to really understand it, you should try building your own API; you'll learn a lot.
Secondary thing: Use GET.
POST means the client adds data to the server (example: post a comment, upload a file, ...). GET means you send some kind of ID to the server, then the server returns its own data to the client.
So what you are doing here, is pure GET.
I'm a real noob in js. I've seen a lot of questions about this, but none of them seems got it clear to me. They all falls into json, that is not available.
I'm trying to access a very poorly documented API(it is in portuguese...). It does not support Json nor Jsonp. It is only xml, via SOAP, HTTP GET e HTTP POST. And of course I'm getting:
XMLHttpRequest cannot load
http://www.camara.gov.br/SitCamaraWS/Deputados.asmx/ObterDeputados.
Origin http://kubrusly.com is not allowed by
Access-Control-Allow-Origin.
my code is:
$.ajax({
type: "GET",
url: "http://www.camara.gov.br/SitCamaraWS/Deputados.asmx/ObterDeputados",
dataType: "xml",
success: parseXml
});
When working locally this goes fine, but once uploaded to my server, it won't retrieve nothing...
Curious enough if i try dataType: "jsonp" :P i get an error complaining about unexpected char '<'. And the xml doc is visible in Safari's console. So the data, in xml format, is arriving in browser, it is there, but i can't access it...
So if json is not an option, how can i go to retrieve this data? It is so frustrating.
I pasted the testing code here:
http://jsfiddle.net/HwP2S/1/
#Kevin B
Thanks for your answer. I did a little research on CORS. I got a little confused with that. Is this supposed to happend on the client side? Any way I got an .htaccess from html5 boilerplate, uncommented the
[EDIT] - solved.
Ok, so I get it going, thanks for the help of you guys. I'll let my solution registered here...
I found that CORS stuff is done in the server side, not my case.
I did not try YQL, you know one more variable, syntax... I let this for last, but did not need it.
I managed to set a proxy and go through it, i spend a lot of time before setting the user agent, and that did it. Before that I aways got 403 error...
this page from yahoo, helped my a lot, some good code examples, both js and PHP.
here the code that is working for me:
js:
// Based on script from yahoo's example # http://developer.yahoo.com/javascript/samples/ajax/sample_proxy_ajax.html
// tweaked by vk
// end point without domain...
var endPtpath = '/SitCamaraWS/Deputados.asmx/ObterDeputados?';
// PHP proxy
var proxiedUrl = 'http://kubrusly.com/yxd/php_proxy_simple.php?yws_path=' + encodeURIComponent(endPtpath);
// mind browsers...
var xmlhttp = null;
if (window.XMLHttpRequest) {
xmlhttp = new XMLHttpRequest();
//overrideMimeType if supported
if ( typeof xmlhttp.overrideMimeType != 'undefined') { xmlhttp.overrideMimeType('text/xml'); }
} else if (window.ActiveXObject) {
xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
} else {
alert('Perhaps your browser does not support xmlhttprequests?');
}
// Create an HTTP GET request
xmlhttp.open('GET', proxiedUrl, true);
// Set the callback function
xmlhttp.onreadystatechange = function() {
if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
// Output the results
console.log(xmlhttp.responseText);
} else {
// waiting for the call to complete
console.log("waiting...");
}
};
// Make the actual request
xmlhttp.send(null);
PHP
<?php
// PHP Proxy --- only 'GET'
// Based on script from yahoo's example # http://developer.yahoo.com/javascript/samples/proxy/php_proxy_simple.txt
// tweaked by vk
// hostname - just base domain
define ('HOSTNAME', 'http://www.camara.gov.br');
// Get the REST call path from the AJAX application - js;
$path = $_GET['yws_path'];
// assign yo url
$url = HOSTNAME.$path;
//Open the Curl session
$session = curl_init($url);
// Don't return HTTP headers. Do return the contents of the call
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_HTTPGET, true);
//set user agent, that did it!!! copied from browsers...
curl_setopt($session, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36');
// The web service returns XML. Set the Content-Type appropriately
header("Content-Type: text/xml");
//// verbose mode for debuging good tool!
// $fp_err = fopen('verbose_file.txt', 'ab+');
// fwrite($fp_err, date('Y-m-d H:i:s')."\n\n"); //add timestamp to theverbose log
// curl_setopt($session, CURLOPT_VERBOSE, 1);
// curl_setopt($session, CURLOPT_FAILONERROR, true);
// curl_setopt($session, CURLOPT_STDERR, $fp_err);
// Make the call
$xml = curl_exec($session);
// done, shutDown.
curl_close($session);
?>
I'm sending a POST from a chrome extension content script to a server I control. I setup the permissions in the manifest ok. Here is my XHR code. (I want to avoid jQuery for this). Its sending an empty responseText
var xhr = new XMLHttpRequest();
xhr.open("POST",'http://mysite.com/make',true);
xhr.onreadystatechange=function() {
if (xhr.readyState == 4) {
var res = JSON.parse(xhr.responseText);
console.log(res);
}
}
xhr.send({'textbox':data[0].user,'from':'extension'});
data[0].user is an object I got directly from the Twitter API
in my CI controller I have
$user = $this->input->get_post('textbox', TRUE);
$from = $this->input->get_post('from', TRUE);
$fullURL = 'http://www.google.com'; //example of a URL from code.
$json = $this->output->set_content_type('application/json');
$json->set_output(json_encode(array('URL' => $fullURL)));
The response text is empty
a jquery call on the other hand works fine
$.post("http://mysite.com/make", { 'textbox': data[0].user, 'from':'jquery' },
function(data) {
console.log(data);
});
Reason is simple, JQuery post method can accept JSON and then convert it to string and send to the server.
What you are trying to do is to directly send JSON here :
xhr.send({'textbox':data[0].user,'from':'extension'}) // Incorrect way
send method should either accept NULL or a string which is generally made up of QueryString Parameters like.
xhr.send("textbox="+ data[0].user + "&from=extension"); // Correct way
This will ensure that your data goes to the appropriate URL with textbox and from as post request parameters.
and queryString will be generated like textbox=username1234&from=extension in the packet's body unlike one goes in Get with the headers along side the URL.
jQuery's post method makes it simpler for you to format data you send using JSON and then internally it converts that to a queryString to send parameters.
You can't directly send Javascript object like that with an XHR object!
Also checkout this example:
http://beradrian.wordpress.com/2007/07/19/passing-post-parameters-with-ajax/