We Keep Coding

What is the best way to parse html in google apps script

var page = UrlFetchApp.fetch(contestURL);
var doc = XmlService.parse(page);
The above code gives a parse error when used, however if I replace the XmlService class with the deprecated Xml class, with the lenient flag set, it parses the html properly.
var page = UrlFetchApp.fetch(contestURL);
var doc = Xml.parse(page, true);
The problem is mostly caused because of no CDATA in the javascript part of the html and the parser complains with the following error.
The entity name must immediately follow the '&' in the entity reference.
Even if I remove all the <script>(.*?)</script> using regex, it still complains because the <br> tags aren't closed.
Is there a clean way of parsing html into a DOM tree.

I ran into this exact same problem. I was able to circumvent it by first using the deprecated Xml.parse, since it still works, then selecting the body XmlElement, then passing in its Xml String into the new XmlService.parse method:
var page = UrlFetchApp.fetch(contestURL);
var doc = Xml.parse(page, true);
var bodyHtml = doc.html.body.toXmlString();
doc = XmlService.parse(bodyHtml);
var root = doc.getRootElement();
Note: This solution may not work if the old Xml.parse is completely removed from Google Scripts.

In 2021, the best way to parse HTML on the .gs side that I know of is...
Click + next to Library
Enter 1ReeQ6WO8kKNxoaA_O0XEQ589cIrRvEBA9qcWpNqdOP17i47u6N9M5Xh0
Click "Look up"
Click Add
Sample usage:
const contentText = UrlFetchApp.fetch('https://www.somesite.com/').getContentText();
const $ = Cheerio.load(contentText);
$('.some-class').first().text();
That's it -- this is probably the closest we'll get to doing jQuery-like DOM selection in GAS. The .first() is important or else you may extract more content than you expected (think of it as using querySelector() instead of querySelectorAll()).
Credit where credit is due: https://github.com/tani/cheeriogs

As of May 2020, you can now use the Cheerio library for Google Apps Script to do this.
Returns the content of Wikipedia's Main Page
const content = getContent_('https://en.wikipedia.org');
const $ = Cheerio.load(content);
Logger.log($('#mp-right').text());
Returns the content of the first paragraph <p> of Wikipedia's Main Page
const content = getContent_('https://en.wikipedia.org');
const $ = Cheerio.load(content);
Logger.log($('p').first().text());
To add to your project:
Select Resources - Libraries... in the Google Apps Script editor. Enter the project key 1ReeQ6WO8kKNxoaA_O0XEQ589cIrRvEBA9qcWpNqdOP17i47u6N9M5Xh0 in the Add a library field, and click "Add". Select the highest version number, and click "Save".

I found that the best way to parse html in google apps is to avoid using XmlService.parse or Xml.parse. XmlService.parse doesn't work well with bad html code from certain websites.
Here a basic example on how you can parse any website easily without using XmlService.parse or Xml.parse. In this example, i am retrieving a list of president from "wikipedia.org/wiki/President_of_the_United_States"
whit a regular javascript document.getElementsByTagName(), and pasting the values into my google spreadsheet.
1- Create a new Google Sheet;
2- Click the menu Tools > Script editor... to open a new tab with the code editor window and copy the following code into your Code.gs:
function onOpen() {
var ui = SpreadsheetApp.getUi();
ui.createMenu("Parse Menu")
.addItem("Parse", "parserMenuItem")
.addToUi();
}
function parserMenuItem() {
var sideBar = HtmlService.createHtmlOutputFromFile("test");
SpreadsheetApp.getUi().showSidebar(sideBar);
}
function getUrlData(url) {
var doc = UrlFetchApp.fetch(url).getContentText()
return doc
}
function writeToSpreadSheet(data) {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheets()[0];
var row=1
for (var i = 0; i < data.length; i++) {
var x = data[i];
var range = sheet.getRange(row, 1)
range.setValue(x);
var row = row+1
}
}
3- Add an HTML file to your Apps Script project. Open the Script Editor and choose File > New > Html File, and name it 'test'.Then copy the following code into your test.html
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<input id= "mButon" type="button" value="Click here to get list"
onclick="parse()">
<div hidden id="mOutput"></div>
</body>
<script>
window.onload = onOpen;
function onOpen() {
var url = "https://en.wikipedia.org/wiki/President_of_the_United_States"
google.script.run.withSuccessHandler(writeHtmlOutput).getUrlData(url)
document.getElementById("mButon").style.visibility = "visible";
}
function writeHtmlOutput(x) {
document.getElementById('mOutput').innerHTML = x;
}
function parse() {
var list = document.getElementsByTagName("area");
var data = [];
for (var i = 0; i < list.length; i++) {
var x = list[i];
data.push(x.getAttribute("title"))
}
google.script.run.writeToSpreadSheet(data);
}
</script>
</html>
4- Save your gs and html files and Go back to your spreadsheet. Reload your Spreadsheet. Click on "Parse Menu" - "Parse". Then click on "Click here to get list" in the sidebar.

Xml.parse() has an option to turn on lenient parsing, which helps when parsing HTML. Note that the Xml service is deprecated however, and the newer XmlService doesn't have this functionality.

For simple tasks such as grabbing one value from a webpage, you could use a regular expression. Regex is notoriously bad for parsing HTML as there's all sorts of weird cases it can get tripped up, but if you're confident about the HTML you're accessing this can sometimes be the simplest way.
Here's an example that fetches the contents of the page's <title> tag:
var page = UrlFetchApp.fetch(contestURL);
var regExp = new RegExp("<title>(.*)</title>", "gi");
var result = regExp.exec(page.getContentText());
// [1] is the match group when using parenthesis in the pattern
var value = result ? result[1] : 'No title found';

I know it is not exactly what OP asked, but I found this question when I was looking for some html parsing options - so it might be useful for others as well.
There is an easy to use the library for TEXT parsing. It's useful if you want to get only one piece of information from the html(xml) code.
EDIT 2021: The script library id is:
1Mc8BthYthXx6CoIz90-JiSzSafVnT6U3t0z_W3hLTAX5ek4w0G_EIrNw
It works like in the picture above
function getData() {
var url = "https://chrome.google.com/webstore/detail/signaturesatori-central-s/fejomcfhljndadjlojamaklegghjnjfn?hl=en";
var fromText = '<span class="e-f-ih" title="';
var toText = '">';
var content = UrlFetchApp.fetch(url).getContentText();
var scraped = Parser
.data(content)
.from(fromText)
.to(toText)
.build();
Logger.log(scraped);
return scraped;
}

If you are using
Cheerio library for Google Apps Script
Source code
Library page (⭐ star it!)
Installation by library ID:
1ReeQ6WO8kKNxoaA_O0XEQ589cIrRvEBA9qcWpNqdOP17i47u6N9M5Xh0
A function to get current emojis from unicode.org:
function getEmojis() {
var t = new Date();
var url = 'https://unicode.org/emoji/charts/full-emoji-list.html';
var fetch = UrlFetchApp.fetch(url);
var contentText = fetch.getContentText();
//console.log(new Date() - t);
// Cherio
var $ = Cheerio.load(contentText);
var data = [];
$("table > tbody > tr").each((index, element) => {
var row = [];
$(element).find("td").each((index, child) => {
row.push($(child).text());
});
if (row.length > 0) {
data.push(row);
}
});
//console.log(data);
//console.log(new Date() - t);
// Result
return data;
}
↑ Sample code shows how to parse table and put it into [[array]]
May be used as a custom function:
Bonus
Parsing the site may be a time-consuming operation + you may reach the limit.
Here's a test file with a full version of the script:
https://docs.google.com/spreadsheets/d/1iO7YjYWyfseQu_YCfRbGDPg7NskOgMu_iO1iGjr7KxY/edit#gid=93365395
↑ it uses CasheService to reduce the number of calls.

Natively there's no way unless you do what you already tried which wont work if the html doesnt conform with the xml format.

There are two options
a) One is to use JavaScript's string functions. First locate your tag using string.indexOf() and then extract the data you want using string.substring().
b) The other option is to make use of the Xml Service.

It's not possible to create an HTML DOM server-side in Apps Script. Using regular expressions is likely your best option, at least for simple parsing.

Chrome.Local.Storage update for html5 localstorage Chrome Packaged apps

I'm trying to update my chrome apps to have some new manifest features but I have to rewrite some of code to do so. Here are examples of local storage get item that I believe I need to use either chrome.local.storage or sync.
var name = 'chrome-writer-files';
document.forms.editor.doc1.value = localStorage.getItem(name);
var name = 'chrome-writer-files2';
document.forms.editor.doc2.value = localStorage.getItem(name);
var name = 'chrome-writer-files3';
document.forms.editor.doc3.value = localStorage.getItem(name);
Please let me know how I can rewrite this so I don't get the error.

You need to use chrome.storage.local.get() instead.
Learn more: https://developer.chrome.com/extensions/storage.html#method-StorageArea-get
Full example: (Chrome - chrome.storage.local.get and set)
chrome.storage.local.set({'someItem': 'some value'});
chrome.storage.local.get('someItem', function (result) {
alert(result);
});

Pushing data to javascript array but the array is not growing

I'm trying to create a navigation system for an internal website.
To do so I'm creating an array by means of javascript that tracks the url of the page and with each new page I'm pushing that new url into the array.
Problem is, each new page seems to be overwriting the last page.
This is what is in my javascript file ... notice I only create a new array if the array doesn't already exist (it will be deleted when the person leaves the website).
var myURL = document.URL;
if (typeof myHistory == "undefined" || !(myHistory instanceof Array)) {
var myHistory = [];
}
myHistory.push(myURL);
var last_element = myHistory[myHistory.length - 1];
var number_rows = myHistory.length;
This what I'm using to see the values in the html ...
<script type="text/javascript">
<!--
document.write(last_element);
document.write(number_rows);
// -->
</script>
It's displaying the URL (last_element) as desired but number_rows remains at 1 when I browse between pages rather than go up to 2, 3, 4, etc which is what I hope to achieve.
Can anyone give me any pointers?

Every time you refresh a page, JavaScript is refreshed anew. If you need to have data persistence, you'll need to use cookies, localStorage, or server-side data storage.
All of those options will require that you serialize to and deserialize from strings.
Here's a quick example of how you could do this using localStorage:
//closure to prevent global pollution
//it's good to be green
(function () {
"use strict";
var history,
url;
//set url here
//I'm making the assumption that no url will have a ',' char in it
history = localStorage['history'].split(',');
history.push(url);
localStorage['history'] = history.join(',');
console.log(url, history.length);
}());

When you browse between pages the javascript environment is re-created on each page. So you are always starting with an empty array.

Can you pass URL parameters to a Javascript file?

<script src="myscript.js?someParameter=123"></script>
From within myscript.js, is there any way to obtain that someParameter was set to 123? Or is the only way to use server side scripts that generate the javascript file with the parameters in it?

Well, you get URL parameters from window.location.href. As the name says, it refers to the current window. What the <script> tag does it to embed the linked file into the current document, thus into the same window. If you parsed window.location.href from the linked JavaScript file, you'd only get the URL from the embedding document.
There are two ways to pass parameters to another JavaScript file:
As #Dave Newton suggested, just declare a variable, then embed the JS file like you did (without the parameters of course, because they have no effect).
Create an iframe, pass the parameters you want to the URL of the iframe, then embed the JavaScript file inside the iframe. An iframe will create a new window instance.

Jquery Address does this, so i've been checking their code out and this is the improved solution I just created myself:
$.each($('script'), function(id, val){ //loop trough all script-elements
var tmp_src = String($(this).attr('src'));//store the src-attr
var qs_index = tmp_src.indexOf('?');//check if src has a querystring and get the index
//Check if the script is the script we are looking for and if it has QS-params
if(tmp_src.indexOf('myscript.js') >= 0 && qs_index >= 0)
{
//this is myscript.js and has a querystring
//we want an array of param-pairs: var1 = value1, var2 = value2, ...
var params_raw = tmp_src.substr(qs_index + 1).split('&');
//create empty options array
var options = [];
//loop troug raw params
$.each(params_raw, function(id, param_pair){
//split names from values
var pp_raw = param_pair.split('=');
//store in options array
options[pp_raw[0]] = pp_raw[1];
});
//check the results out in the console!
console.log(options);
}
});
I hope this does what you need?

The answer is a definite "YES". I've been doing this on various projects for over a decade. The solution is actually easy, it's just non-intuitive (you have to generate an error). To be clear, the following code lets you do something like this:
<script src="https://example.com/script.js?id=1&bar=this works!" />
All you need to do is initiate a silent error, which takes less than 1/1000 of a second even on the worst outdated mobile browsers. You shouldn't do it a ton, but you only need to do it once. This error is processed, so it won't show up as an error in telemetry or 3rd party error trackers either.
// Generic function used to see if a param exists in a URL string.
// Provided here in case you don't know how to do it.
// This is not needed for the solution.
function getParameter (name, url) {
if (!url) url = scriptName()
name = name.replace(/[\[\]]/g, '\\$&')
var regex = new RegExp('[?&]' + name + '(=([^&#]*)|&|#|$)')
var results = regex.exec(url)
if (!results) return null
if (!results[2]) return ''
return decodeURIComponent(results[2].replace(/\+/g, ' '))
}
// Gets the name of this script (whatever this file runs in)
// You can use this name to get parameters just like you would for the window URL :)
function getScriptName () {
var error = new Error(),
source,
lastStackFrameRegex = new RegExp(/.+\/(.*?):\d+(:\d+)*$/),
currentStackFrameRegex = new RegExp(/getScriptName \(.+\/(.*):\d+:\d+\)/)
if ((source = lastStackFrameRegex.exec(error.stack.trim())) && source[1] !== '')
return source[1]
else if ((source = currentStackFrameRegex.exec(error.stack.trim())))
return source[1]
else if (error.fileName !== undefined)
return error.fileName
}

Change query argument of jQuery suggest plugin

This question is kind-of crappy because I try to get around some limitations:
Current JS sends an ajax query with the following code
jQuery('#searchbox').suggest('/?live=1');
What the server get is the following query string:
?live=1&q=searchstring
Problem:
The server expects the query string to be preceded with 's=' not 'q='
I have to use the existing scripts so what I'm trying to so is find a way to change 'q=' to 's=' in javascript, without altering the exisiting suggest plugin or the php search script.
Thanks.

The only way you will do that is by modifying the plugin, if you want to avoid changing the script forever and need this only for one page, override the function temporarily.
$.suggest.suggest = function() {
var q = $.trim($input.val());
if (q.length >= options.minchars) {
cached = checkCache(q);
if (cached) {
displayItems(cached['items']);
} else {
//This is the line we r changing
$.get(options.source, {s: q}, function(txt) {
$results.hide();
var items = parseTxt(txt, q);
displayItems(items);
addToCache(q, items, txt.length);
});
}
} else {
$results.hide();
}
}