So i'm trying to create a web spider. I've run into a website, that has some javascript, and I want to trick the browser into thinking that an event has been fired and that it must call the corresponding javascript code to handle the event. How would I be able to do this in Perl? using the WWW::Mechanize or WWW::Scripter::Plugin::Javascript?
Also, it would be very appreciated I someone could put up an example of how to use WWW::Scripter::Plugin::Javascript.
Thanks in advance. Also if someone has a better way to word the question please go ahead and edit it
In a normal browser setup, the JavaScript is in the browser, not on the server. It's the client that executes the JavaScript.
That means you need manually figure out what the JavaScript code does and code that in Perl, or you need to load a JavaScript engine.
Here are three JavaScript engines:
WWW::Mechanize::Firefox
Win32::IE::Mechanize
WWW::Scripter::Plugin::JavaScript
Using WWW::Mechanize and Live Http Headers, I did a Live Http Replay.
From the replay, I copied the headers (ie Connection: keep-alive to $agent->add_header( "Connection" => "keep-alive"); ) and then copied the Post content to my $content = '..
Then $agent->post( $url_of_the_site, Content => $content);
This worked to click a link like 2 on a aspx site.
I used this code as a guide http://pastie.org/1728196/wrap
Related
I have built a simple webpage for a touchscreen kiosk (Win7, XAMPP).
The interface is built up of 9 tiles (windows metro style). HTML, PHP and CSS only. Each of the tiles are simple links
What I would like to do is track how many times each of the tiles have been clicked.
Examples of my pages are;
www.example.com/help.html
www.example.com/contact.html
www.example.com/map/floor1.html
The kiosk will be running on localhost and I feel that Google Analytics, Piwik or AWStats are too resource intensive for such a small task. Obviously as the kiosk is running on localhost the IPs, location, browser etc... aren't important.
Are there any other ways I could track the clicks to a log file or similar?
Any advice appreciated.
You can use onclick functions on the links, and use javascript to write a log file. This might help you creating a log file through javascript
I would say this data can be found inside Apaches access logs if you only want to know how many times a page has been accessed. This can be easliy done by using a tool such as Apache Log Viewer.
If you actually want to log link clicks, you probably have to use javascript action handlers. Because I consider writing from JavaScript ugly, I would probably send an ajax request to my PHP server every time.
Edit:
Another way would be, to convert all you html files into php and log from there (I can also add an example how).
Example:
<html>
<?php
$count= include 'count.php';
$count['count-'. __FILE__]= $count['count-'.__FILE__] + 1;
file_put_contents('count.php', '<?php return ' . var_export($count, true) .'; ?>');
?>
</html>
We have an HTML/JS animation which is basically a fancy counter. The actual counter part is a div the value for which gets loaded from a PHP script that spits out a random number. It looks like this:
<?php
$val = file_get_contents("num.txt");
$val += rand(1, 5);
file_put_contents("num.txt", $val);
print $val;
We are hosting this PHP file on an external domain, because this animation will be packaged as part of an iPad publication. The idea is that when our animation gets loaded it will call to our PHP file and get it's value from there.
I was wondering what the most straightforward way of loading this data would be.
The div for the text part of the counter that we are changing is #Stage_Text. I've tried the following approach in the html of the animation, which unfortunately did not work:
$(document).ready(function () {
$.ajax({
url : "http://ourdomain.com/stat.php",
dataType: "text",
success : function (result) {
$("#Stage_Text").html(result);
}
});
});
Any assistance would be greatly appreciated.
I've just tested your code, and it seems to work fine.
Is you ajax request completing okay? You can check this in Firebug or your developer tools.
That's my only thought without other information.
------UDPATE-----
Sorry, I didn't see that your Ajax request was to a different domain. Yeah, this won't work because of the same-origin policy - in essence, you can only make ajax calls to the domain that served the javascript file.
Lot's of ideas to get around it here.
The easiest way is probably to use a server-side proxy.
Thanks for the responses! In the end we solved this by simply hosting the entire animation on the server and calling it within the app. This way the code I posted originally worked beautifully, and solved a few other challenges that we had - for example what happens if the user viewing the app doesn't have an internet connection? Doing it as I originally intended meant that the value would simply be blank (you could have a placeholder but it's still not ideal). With our workaround the animation simply does not get loaded and we have another, more static element underneath.
I'd say that in this particular case this is the absolutely most straightforward and easiest way to achieve what we wanted.
What would I need to do to get this example running on my machine?
http://www.w3schools.com/ajax/tryit.asp?filename=tryajax_httprequest_js (page no longer available)
I'm looking to access the XML file hosted on w3schools (and not move it to my machine), but run the HTML and Javascript code on my machine. I tried changing the third to last line from:
<button onclick="loadXMLDoc('note.xml')">Get XML</button>
to:
<button onclick="loadXMLDoc('http://www.w3schools.com/ajax/note.xml')">Get XML</button>
thinking this would make it work, but it didn't seem to help. Any suggestions?
Just put the full URL into your browser window which will let your browser get it, then copy/paste and save locally. Javascript won't fetch stuff from outside the domain it's served from (without a fair bit of extra work), due to the Same Origin policy ( a security feature).
You can't go cross domain using AJAX. You should move the XML file to the same server that you have the site files stored on and call it that way.
https://developer.mozilla.org/en/Same_origin_policy_for_JavaScript
You need to use the following code in the function that does the AJAX:
try {
netscape.security.PrivilegeManager.enablePrivilege("UniversalPreferencesRead");
} catch (e) {
alert("error");
}
This only works for Firefox! There are other options which can be passed to enablePrivilege that may be useful.
I have a web service which the customers use by inserting an external JavaScript (hosted on my servers). Recently, due to server outage - the external JavaScript became unavailable and my customers' websites came to a crawl as browser didn't load rest of the website until it loaded the JS (it goes into header of the websites).
I am trying to work out methods so that customers' website don't slow down even if my server goes down and for that I wanted to simulate a condition where the my server isn't responding. Note that if I specify a wrong URL, browser won't load the JS but in case URL is right and server isn't responding, browser will stall loading rest of the page. I want to simulate the last case. Any ideas how can I go about it?
PS: On server side, I am using the LAMP stack.
Create a script that sleeps for a configurably long time
Something like
<?php
$how_long = $_GET['seconds'];
sleep($how_long);
echo "alert('Finished sleeping!');";
?>
Then you just access this script instead, for example by putting this in your HTML code
<script src="http://example.com/hang_for.php?seconds=3600" />. That would sleep for an hour. There will be another timeouts that'll trigger first configured in php.ini, but that's exactly what you want to test, no?
If the "P" in your LAMP is PHP, you could use the sleep function (documented here). Then, have your test page load your PHP script as the source of your Javascript to see what happens.
Did you try looping back the server into itself (or any other HTTP server w/o the webservice on)?
unplugging is pretty drastic, the off-button should do.
Unplug the server. Having no power makes a server unresponsive...
Hey everyone, I'm working on a widget for Apple's Dashboard and I've run into a problem while trying to get data from my server using jquery's ajax function. Here's my javascript code:
$.getJSON("http://example.com/getData.php?act=data",function(json) {
$("#devMessage").html(json.message)
if(json.version != version) {
$("#latestVersion").css("color","red")
}
$("#latestVersion").html(json.version)
})
And the server responds with this json:
{"message":"Hello World","version":"1.0"}
For some reason though, when I run this the fields on the widget don't change. From debugging, I've learned that the widget doesn't even make the request to the server, so it makes me think that Apple has some kind of external URL block in place. I know this can't be true though, because many widgets phone home to check for updates.
Does anyone have any ideas as to what could be wrong?
EDIT: Also, this code works perfectly fine in Safari.
As requested by Luca, here's the PHP and Javascript code that's running right now:
PHP:
echo $_GET["callback"].'({"message":"Hello World","version":"1.0"});';
Javascript:
function showBack(event)
{
var front = document.getElementById("front");
var back = document.getElementById("back");
if (window.widget) {
widget.prepareForTransition("ToBack");
}
front.style.display = "none";
back.style.display = "block";
stopTime();
if (window.widget) {
setTimeout('widget.performTransition();', 0);
}
$.getJSON('http://nakedsteve.com/data/the-button.php?callback=?',function(json) {
$("#devMessage").html(json.message)
if(json.version != version) {
$("#latestVersion").css("color","red")
}
$("#latestVersion").html(json.version)
})
}
In Dashcode click Widget Attributes then Allow Network Access make sure that option is checked. I've built something that simply refused to work, and this was the solution.
Cross-domain Ajax requests ( Using the XMLHttpRequest / ActiveX object ) are not allowed in the current standard, as per the W3C spec:
This specification does not include
the following features which are being
considered for a future version of
this specification:
Cross-site XMLHttpRequest;
However there's 1 technique of doing ajax requests cross-domain, JSONP, by including a script tag on the page, and with a little server configuration.
jQuery supports this, but instead of responding on your server with this
{"message":"Hello World","version":"1.0"}
you'll want to respond with this:
myCallback({"message":"Hello World","version":"1.0"});
myCallback must be the value in the "callback" parameter you passed in the $.getJSON() function. So if I was using PHP, this would work:
echo $_GET["callback"].'({"message":"Hello World","version":"1.0"});';
Apple has some kind of external URL block in place.
In your Info.plist you need to have the key AllowNetworkAccess set to true.
<key>allowNetworkAccess</key>
<true/>
Your code works in Safari because it is not constrained in the dashboard sever and it is not standards complient in that it DOES allow cross site AJAX. FF IS standards complient in that it DOES NOT allow cross site ajax.
If you are creating a dashboard widget, why don't you use the XMLHttpRequest Setup function in the code library of DashCode. Apple built these in so you don't need to install 3rd party JS libraries. I'm not sure about JSON support but perhaps starting here will lead you in a better direction.
So another solution is to create your own server side web service where you can control the CORS of, the users web browser can't access another site, but if you wrap that other site in your own web service (on the same domain) then it does not cause an issue.
Interesting that it works in Safari. As far as I know to do x-domain ajax requests you need to use the jsonp dataType.
http://docs.jquery.com/Ajax/jQuery.getJSON
http://bob.pythonmac.org/archives/2005/12/05/remote-json-jsonp/
Basically you need to add callback=? to your query string and jquery will automatically replace it with the correct method eg:
$.getJSON("http://example.com/getData.php?act=data&callback=?",function(){ ... });
EDIT: put the callback=? bit at the end of the query string just to be safe.