We Keep Coding

Unable to get local storage variable in Wicked Pdf

I have stored some data "xyz" in java script local storage. Now I'm trying to access that in wicked pdf but unable to do so. It's working when i display as html but does not print that in the pdf.
That's what i have saved:
localStorage.setItem("my_id", "xyz");
In file.pdf.erb:
<script type="text/javascript">
setTimeout((function () {
//console.log(localStorage.getItem("dashboard_bar_chart_actual_situation_control"));
document.getElementById("p1").innerHTML = "HELLo "+localStorage.getItem("my_id");
window.status = "FLAG_FOR_PDF";
}), 1000);
</script>

Are you setting the localStorage value on the same page that you are rendering to PDF, or are you setting it on a prior page, and then expecting it to be retrieved on the PDF page?
The latter won't work in the default PDF rendering mode, because your HTML is saved to a file on-disk, and opened in a virtual browser, without any context (localStorage, cookies) of what happened on other pages (unless you supply them as options to wkhtmltopdf, at least for cookies).
You could explicitly set it during render, so that it becomes available, something like this:
<script>
// Render it to the page, so it is executed when the JS runs:
localStorage.setItem("my_id", "<%= "xyz" %>")
// Use it:
document.getElementById("p1").innerHTML = "HELLo "+localStorage.getItem("my_id");
</script>
Alternatively, you can use the WickedPDF middleware, which attempts an in-place replacement of the HTML you see with a PDF version, without saving it to a temporary HTML file on-disk, but that also may not work well, depending on how the JS code gets executed.

PHP Get Rendered Javascript Page

I'm developing application using AngularJS. Everything seems to be nice until I meet something that leads me to headache: SEO.
From many references, I found out that AJAX content crawled and indexed by Google bot or Bing bot 'is not that easy' since the crawlers don't render Javascript.
Currently I need a solution using PHP. I use PHP Slim Framework so my main file is index.php which contains function to echo the content of my index.html. My question is:
Is it possible to make a snapshot of rendered Javascript in HTML?
My strategy is:
If the request query string contains _escaped_fragment_, the application will generate a snapshot and give that snapshot as response instead of the exact file.
Any help would be appreciated. Thanks.

After plenty of times searching and researching, I finally managed to solve my problem by mixing PHP with PhantomJS (version 2.0). I use exec() function in PHP to run phantomJS and create Javascript file to take get the content of the targeted URL. Here are the snippets:
index.php
// Let's assume that you have a bin folder under your root folder directory which contains phantomjs.exe and content.js
$script = __DIR__ ."/bin/content.js";
$target = "http://www.kincir.com"; // target URL
$cmd = __DIR__."/bin/phantomjs.exe $script $target";
exec($cmd, $output);
return implode("", $output);
content.js
var webPage = require('webpage');
var system = require('system');
var page = webPage.create();
var url = system.args[1]; // This will get the second argument from $cmd, in this example, it will be the value of $target on index.php which is "http://www.kincir.com"
page.open(url, function (status) {
page.onLoadFinished = function () { // Make sure to return the content of the page once the page is finish loaded
var content = page.content;
console.log(content);
phantom.exit();
};
});

I recently published a project that gives PHP access to a browser. Get it here: https://github.com/merlinthemagic/MTS. It also relies on PhantomJS.
After downloading and setup you would simply use the following code:
$myUrl = "http://www.example.com";
$windowObj = \MTS\Factories::getDevices()->getLocalHost()->getBrowser('phantomjs')->getNewWindow($myUrl);
//now you can either retrive the DOM and parse it, like this:
$domData = $windowObj->getDom();
//this project also lets you manipulate the live page. Click, fill forms, submit etc.

how to load html page variable into python variable using the web address

I found a guide here:
https://docs.python.org/2/library/htmlparser.html
but the function HTMLParser.feed(data) uses data as the html itself.
There is a way to do similar feed but only with the web address ?
something like this ...
HTMLParser.feed("www.a.com") ?
Generally, i want to take a variable from different web pages and load it into python variable with python script and compere between them.
Thanks.

import urllib2
f = urllib2.urlopen(url)
page_data = f.read()
# do stuff with html
HTMLParser.feed(page_data)
f.close()
This will return the raw html from the page. You can then parse it and find whatever you want. Not sure if there is a faster solution.

Maybe
python-requests?
import requests
r = requests.get("https://github.com/")
r.content
Later if you want to parse the content you can use lxml

Inject local .js file into a webpage?

I'd like to inject a couple of local .js files into a webpage. I just mean client side, as in within my browser, I don't need anybody else accessing the page to be able to see it. I just need to take a .js file, and then make it so it's as if that file had been included in the page's html via a <script> tag all along.
It's okay if it takes a second after the page has loaded for the stuff in the local files to be available.
It's okay if I have to be at the computer to do this "by hand" with a console or something.
I've been trying to do this for two days, I've tried Greasemonkey, I've tried manually loading files using a JavaScript console. It amazes me that there isn't (apparently) an established way to do this, it seems like such a simple thing to want to do. I guess simple isn't the same thing as common, though.
If it helps, the reason why I want to do this is to run a chatbot on a JS-based chat client. Some of the bot's code is mixed into the pre-existing chat code -- for that, I have Fiddler intercepting requests to .../chat.js and replacing it with a local file. But I have two .js files which are "independant" of anything on the page itself. There aren't any .js files requested by the page that I can substitute them for, so I can't use Fiddler.

Since your already using a fiddler script, you can do something like this in the OnBeforeResponse(oSession: Session) function
if ( oSession.oResponse.headers.ExistsAndContains("Content-Type", "html") &&
oSession.hostname.Contains("MY.TargetSite.com") ) {
oSession.oResponse.headers.Add("DEBUG1_WE_EDITED_THIS", "HERE");
// Remove any compression or chunking
oSession.utilDecodeResponse();
var oBody = System.Text.Encoding.UTF8.GetString(oSession.responseBodyBytes);
// Find the end of the HEAD script, so you can inject script block there.
var oRegEx = oRegEx = /(<\/head>)/gi
// replace the head-close tag with new-script + head-close
oBody = oBody.replace(oRegEx, "<script type='text/javascript'>console.log('We injected it');</script></head>");
// Set the response body to the changed body string
oSession.utilSetResponseBody(oBody);
}
Working example for www.html5rocks.com :
if ( oSession.oResponse.headers.ExistsAndContains("Content-Type", "html") &&
oSession.hostname.Contains("html5rocks") ) { //goto html5rocks.com
oSession.oResponse.headers.Add("DEBUG1_WE_EDITED_THIS", "HERE");
oSession.utilDecodeResponse();
var oBody = System.Text.Encoding.UTF8.GetString(oSession.responseBodyBytes);
var oRegEx = oRegEx = /(<\/head>)/gi
oBody = oBody.replace(oRegEx, "<script type='text/javascript'>alert('We injected it')</script></head>");
oSession.utilSetResponseBody(oBody);
}
Note, you have to turn streaming off in fiddler : http://www.fiddler2.com/fiddler/help/streaming.asp and I assume you would need to decode HTTPS : http://www.fiddler2.com/fiddler/help/httpsdecryption.asp
I have been using fiddler script less and less, in favor of fiddler .Net Extensions - http://fiddler2.com/fiddler/dev/IFiddlerExtension.asp

If you are using Chrome then check out dotjs.
It will do exactly what you want!

How about just using jquery's jQuery.getScript() method?
http://api.jquery.com/jQuery.getScript/

save the normal html pages to the file system, add the js files manually by hand, and then use fiddler to intercept those calls so you get your version of the html file

Access and change the attributes of <script> tag

How can I access and change the attributes of tag, specially the src?
something like this maybe:
document.scripts[i].src
of course it does not work!
I intend to alter it this way:
document.scripts[i].src += '?ver3'
in a loop for all the script on the page.

You could use document.getElementsByTagName('script') to get all the script elements in the page.
However, any script elements that you find will already have loaded their content, so it will be too late to change the URL that they use to load content.
If you want to alter the URLs, you should use a solution on the server side, so that the URLs are changed when they arrive to the browser.

With jQuery:
jQuery("script").each(function(){
var old_src = jQuery(this).attr('src');
jQuery(this).attr('src', old_src+'?ver3');
})
With good-old JS:
var scripts = document.getElementsByTagName('script');
for (var i=0; i<scripts.length; i++){
var old_src = scripts[i].getAttribute('src');
scripts[i].setAttribute('src', old_src +'?ver3');
}

Install httpd (Apache) and PHP.
Put into your hosts file (or hosts.txt) this line:
127.0.0.1 example.com
Write an php script named index.php and put it into server directory (/var/www/html or some like this; depends on configuration).
When you load in web browser http://example.com you will be redirected to 127.0.0.1 (your local server) and in PHP this variable $_SERVER["HTTP_HOST"] will be set to example.com.
Get this site (e.g. using cURL), change what you want (using string functions, regex, DOMDocument, SimpleXML, etc.) and send modified content to browser.
That's all. :)