We Keep Coding

getting OnLoad HTML/DOM for an HTML page in PHP

I am trying to get the HTML (ie what you see initially when the page completes loading) for some web-page URI. Stripping out all error checking and assuming static HTML, it's a single line of code:
function GetDisplayedHTML($uri) {
return file_get_contents($uri);
}
This works fine for static HTML, and is easy to extend by simple parsing, if the page has static file dependencies/references. So tags like <script src="XXX">, <a href="XXX">, <img src="XXX">, and CSS, can also be detected and the dependencies returned in an array, if they matter.
But what about web pages where the HTML is dynamically created using events/AJAX? For example suppose the HTML for the web page is just a brief AJAX-based or OnLoad script that builds the visible web page? Then parsing alone won't work.
I guess what I need is a way from within PHP, to open and render the http response (ie the HTML we get at first) via some javascript engine or browser, and once it 'stabilises', capture the HTML (or static DOM?) that's now present, which will be what the user's actually seeing.
Since such a webpage could continually change itself, I'd have to define "stable" (OnLoad or after X seconds?). I also don't need to capture any timer or async event states (ie "things set in motion that might cause web page updates at some future time"). I only need enough of the DOM to represent the static appearance the user could see, at that time.
What would I need to do, to achieve this programmatically in PHP?

To render page with JS you need to use some browser. PhantomJS was created for tasks like this. Here is simple script to run with Phantom:
var webPage = require('webpage');
var page = webPage.create();
var system = require('system');
var args = system.args;
if (args.length === 1) {
console.log('First argument must be page URL!');
} else {
page.open(args[1], function (status) {
window.setTimeout(function () { //Wait for scripts to run
var content = page.content;
console.log(content);
phantom.exit();
}, 500);
});
}
It returns resulting HTML to console output.
You can run it from console like this:
./phantomjs.exe render.js http://yandex.ru
Or you can use PHP to run it:
<?php
$path = dirname(__FILE__);
$html = shell_exec($path . DIRECTORY_SEPARATOR . 'phantomjs.exe render.js http://phantomjs.org/');
echo htmlspecialchars($html);
My PHP code assumes that PhantomJS executable is in the same directory as PHP script.

PHP Get Rendered Javascript Page

I'm developing application using AngularJS. Everything seems to be nice until I meet something that leads me to headache: SEO.
From many references, I found out that AJAX content crawled and indexed by Google bot or Bing bot 'is not that easy' since the crawlers don't render Javascript.
Currently I need a solution using PHP. I use PHP Slim Framework so my main file is index.php which contains function to echo the content of my index.html. My question is:
Is it possible to make a snapshot of rendered Javascript in HTML?
My strategy is:
If the request query string contains _escaped_fragment_, the application will generate a snapshot and give that snapshot as response instead of the exact file.
Any help would be appreciated. Thanks.

After plenty of times searching and researching, I finally managed to solve my problem by mixing PHP with PhantomJS (version 2.0). I use exec() function in PHP to run phantomJS and create Javascript file to take get the content of the targeted URL. Here are the snippets:
index.php
// Let's assume that you have a bin folder under your root folder directory which contains phantomjs.exe and content.js
$script = __DIR__ ."/bin/content.js";
$target = "http://www.kincir.com"; // target URL
$cmd = __DIR__."/bin/phantomjs.exe $script $target";
exec($cmd, $output);
return implode("", $output);
content.js
var webPage = require('webpage');
var system = require('system');
var page = webPage.create();
var url = system.args[1]; // This will get the second argument from $cmd, in this example, it will be the value of $target on index.php which is "http://www.kincir.com"
page.open(url, function (status) {
page.onLoadFinished = function () { // Make sure to return the content of the page once the page is finish loaded
var content = page.content;
console.log(content);
phantom.exit();
};
});

I recently published a project that gives PHP access to a browser. Get it here: https://github.com/merlinthemagic/MTS. It also relies on PhantomJS.
After downloading and setup you would simply use the following code:
$myUrl = "http://www.example.com";
$windowObj = \MTS\Factories::getDevices()->getLocalHost()->getBrowser('phantomjs')->getNewWindow($myUrl);
//now you can either retrive the DOM and parse it, like this:
$domData = $windowObj->getDom();
//this project also lets you manipulate the live page. Click, fill forms, submit etc.

Responding to button click events in Google Docs custom HTML dialog

I am an Admin of a Google Spreadsheet. Several times a week me and other admins have to add new users to the spreadsheet so I thought it would be easier if we could use a custom HTML dialog to do so.
I am displaying the dialog like so:
function AddRow()
{
var html = HtmlService.createHtmlOutputFromFile('AddRow')
.setSandboxMode(HtmlService.SandboxMode.IFRAME);
SpreadsheetApp.getUi().showModalDialog(html, 'Add Row')
}
File AddRow.html is a simple HTML form with text boxes, select elements, etc.
Now is there a way I can access the spreadsheet to add a new row with the values entered by the user?
In the HTML file AddRow.html I tried the following
<script>
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet = spreadsheet.getSheetByName('2015');
sheet.insertRows(2, 1);
</script>
However this does not insert a row. If I move that code of the HTML file into a *.gs file it does work so it must be a limitation on HTML files?

The script code within served HTML files is javascript, not Google Apps Script - the main difference for this question is that server-side Google Apps Script has access to the full set of APIs provided for Google Services, while the client-side javascript does not.
One option is to use the google.script.run Client-side API to invoke the function that you already have working on the server side (in the editor). For more information, see the guide to communicating with server functions in HTML service.
function insertRows(rowIndex, numRows) {
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet = spreadsheet.getSheetByName('2015');
sheet.insertRows(rowIndex, numRows);
return true; // Need to have a return value for google.script.run to work
// To return a failure to the client side, throw an error.
}
Client side script:
...
// Call insertRows on server side.
google.script.run
.withSuccessHandler( hooray ) // See function hooray below
.insertRows(2, 1);
...
/**
* This function will receive a call-back if the google.script.run
* call returns success.
*/
function hooray( result ) {
alert( "Received result <" + result + ">" );
}

create callback API

New Restful API's like Google, OpenStreetview use a simple call back mechanism.
Basically you call the API, adding a parameter &callback=my function.
When executing a call to this API, as a result my function is called passing a JSON dataset.
I am trying to create the same mechanisme for a API I am building for my personal use.
As far as I understood my API needs to return a javascript, that calls the function that is passed in a script.
For a test I created this:
function apiCall(URL,values, keyPair,cBackPair) {
// URL specifics URL to call
// keyPair: <keyname>=<key>; leave black if unneeded
// cBacPair: <callBackParametername>=<functionname>
// called is: URL?values&keypair&cBackPair
var request = (keyPair)?'&'+keyPair:'';
request = URL + '?'+ encodeURI(values) + request + '&' + cBackPair;
var script = document.createElement("script");
script.setAttribute("type", "text/javascript");
script.setAttribute("src", request);
document.body.appendChild(script);
}
function callAPI() {
apiCall('http://xllent.nl/map/ajax/answer.php','q=one','','s=doit');
}
function doit(result) {
alert(result);
}
To test I call callAPI onload.
The script answer.php is very basic:
<?$s = $_GET['s'];
?>
<script type="text/javascript">
doit('jeroen');
</script>
Later the script would use $s to call the right script, and of course supply user data.
For now I am just trying to get the script doit('jeroen'); to be run. But nothing happens.
Typing javascript:doit('jeroen'); in the browser window gives the result I would expect.
Any suggestions?

Don't surround your javascript with <script> tags. You are not generating a HTML file with a javascript body.. You should think of this as if you're generating a javascript file on fly.
Javascript files also don't start and end with <script>

Chrome extension regarding injected script + localstorage

I am puzzling my way through my first 'putting it all together' Chrome extension, I'll describe what I am trying to do and then how I have been going about it with some script excerpts:
I have an options.html page and an options.js script that lets the user set a url in a textfield -- this gets stored using localStorage.
function load_options() {
var repl_adurl = localStorage["repl_adurl"];
default_img.src = repl_adurl;
tf_default_ad.value = repl_adurl;
}
function save_options() {
var tf_ad = document.getElementById("tf_default_ad");
localStorage["repl_adurl"] = tf_ad.value;
}
document.addEventListener('DOMContentLoaded', function () {
document.querySelector('button').addEventListener('click', save_options);
});
document.addEventListener('DOMContentLoaded', load_options );
My contentscript injects a script 'myscript' into the page ( so it can have access to the img elements from the page's html )
var s = document.createElement('script');
s.src = chrome.extension.getURL("myscript.js");
console.log( s.src );
(document.head||document.documentElement).appendChild(s);
s.parentNode.removeChild(s);
myscript.js is supposed to somehow grab the local storage data and that determines how the image elements are manipulated.
I don't have any trouble grabbing the images from the html source, but I cannot seem to access the localStorage data. I realize it must have to do with the two scripts having different environments but I am unsure of how to overcome this issue -- as far as I know I need to have myscript.js injected from contentscript.js because contentscript.js doesn't have access to the html source.
Hopefully somebody here can suggest something I am missing.
Thank you, I appreciate any help you can offer!
-Andy

First of all: You do not need an injected script to access the page's DOM (<img> elements). The DOM is already available to the content script.
Content scripts cannot directly access the localStorage of the extension's process, you need to implement a communication channel between the background page and the content script in order to achieve this. Fortunately, Chrome offers a simple message passing API for this purpose.
I suggest to use the chrome.storage API instead of localStorage. The advantage of chrome.storage is that it's available to content scripts, which allows you to read/set values without a background page. Currently, your code looks quite manageable, so switching from the synchronous localStorage to the asynchronous chrome.storage API is doable.
Regardless of your choice, the content script's code has to read/write the preferences asynchronously:
// Example of preference name, used in the following two content script examples
var key = 'adurl';
// Example using message passing:
chrome.extension.sendMessage({type:'getPref',key:key}, function(result) {
// Do something with result
});
// Example using chrome.storage:
chrome.storage.local.get(key, function(items) {
var result = items[key];
// Do something with result
});
As you can see, there's hardly any difference between the two. However, to get the first to work, you also have to add more logic to the background page:
// Background page
chrome.extension.onMessage.addListener(function(message, sender, sendResponse) {
if (message.type === 'getPref') {
var result = localStorage.getItem(message.key);
sendResponse(result);
}
});
On the other hand, if you want to switch to chrome.storage, the logic in your options page has to be slightly rewritten, because the current code (using localStorage) is synchronous, while chrome.storage is asynchronous:
// Options page
function load_options() {
chrome.storage.local.get('repl_adurl', function(items) {
var repl_adurl = items.repl_adurl;
default_img.src = repl_adurl;
tf_default_ad.value = repl_adurl;
});
}
function save_options() {
var tf_ad = document.getElementById('tf_default_ad');
chrome.storage.local.set({
repl_adurl: tf_ad.value
});
}
Documentation
chrome.storage (method get, method set)
Message passing (note: this page uses chrome.runtime instead chrome.extension. For backwards-compatibility with Chrome 25-, use chrome.extension (example using both))
A simple and practical explanation of synchronous vs asynchronous ft. Chrome extensions