We Keep Coding

Add page break before text in Apps Script

I'm trying to insert a page break before certain text.
I tried the solution in this post: Replace a text keyword with a "Page Break" element in Apps Script
Which adds the page break after the text, played around with the code and couldn't get it to add it before. As a workaround I was trying to append a paragraph text after I append the page Break, but couldn't get it work.

I believe your goal as follows.
You want to insert the pagebreak before a text word in Google Document using Google Apps Script.
In this case, I would like to propose the following sample script using Google Docs API. At Google Docs API, the page can be inserted to the middle of text using index. So I thought that this direction might be a bit simple and the process cost might be able to be also reduced. The flow of this script is as follows.
Retrieve all contents from Google Document using the method of "documents.get" in Docs API.
Create the request body for using the method of "documents.batchUpdate" in Docs API.
Request the request body to the method of "documents.batchUpdate" in Docs API.
Sample script:
Please copy and paste the following script to the script editor of Google Document, and please set searchPattern. And, please enable Google Docs API at Advanced Google services.
function myFunction() {
const searchText = "{{page break}}"; // Please set text. This script inserts the pagebreak before this text.
// 1. Retrieve all contents from Google Document using the method of "documents.get" in Docs API.
const docId = DocumentApp.getActiveDocument().getId();
const res = Docs.Documents.get(docId);
// 2. Create the request body for using the method of "documents.batchUpdate" in Docs API.
let offset = 0;
const requests = res.body.content.reduce((ar, e) => {
if (e.paragraph) {
e.paragraph.elements.forEach(f => {
if (f.textRun) {
const re = new RegExp(searchText, "g");
let p = null;
while (p = re.exec(f.textRun.content)) {
ar.push({insertPageBreak: {location: {index: p.index + offset}}});
}
}
})
}
offset = e.endIndex;
return ar;
}, []).reverse();
// 3. Request the request body to the method of "documents.batchUpdate" in Docs API.
Docs.Documents.batchUpdate({requests: requests}, docId);
}
In this sample script, const searchPattern = "{{page break}}" is used as the text for inserting the pagebreak. Please modify this for your actual situation.
Result:
When above script is run, the following result is obtained.
From:
To:
References:
Method: documents.get
Method: documents.batchUpdate
InsertPageBreakRequest
Added:
I confirmed that google-docs-api is also included in your tags of your question. So I proposed a sample script for using Google Docs API. But from your following replying, it seems that you wanted to use Google Docs API without enabling Google Docs API at Advanced Google services. I couldn't notice about this from your question and tags.
is there a way of doing this without having to enable the Google Docs API in the Apps Script environment? I get a Reference Error for Docs.Documents.get(docId) if I don't.
About your replying, I add one more sample script. In this sample script, Google Docs API is used with UrlFetchApp. So Google Docs API of Advanced Google services is not used. But, in this case, Google Docs API is required to be enabled at API console. So I propose 2 patterns for this.
Please link GCP to GAS project and enable Google Docs API at API console.
Please enable Google Docs API at Advanced Google services once and save the GAS project. Here, please wait for minutes. And then, please disable Google Docs API at Advanced Google services. In the current stage, it seems that even when Google Docs API is disabled at Advanced Google services, Google Docs API is not disabled at API console. But I'm not sure whether this is the permanent situation. But, now, I thought that this might be able to be used for this your situation.
Sample script:
Before you use this script, please enable Google Docs API at API console by doing one of them as I proposed above and run the script.
function myFunction() {
const searchText = "{{page break}}"; // Please set text. This script inserts the pagebreak before this text.
// 1. Retrieve all contents from Google Document using the method of "documents.get" in Docs API.
const accessToken = ScriptApp.getOAuthToken();
const docId = DocumentApp.getActiveDocument().getId();
const url1 = "https://docs.googleapis.com/v1/documents/" + docId;
const response1 = UrlFetchApp.fetch(url1, {headers: {authorization: "Bearer " + accessToken}});
const res = JSON.parse(response1.getContentText());
// 2. Create the request body for using the method of "documents.batchUpdate" in Docs API.
let offset = 0;
const requests = res.body.content.reduce((ar, e) => {
if (e.paragraph) {
e.paragraph.elements.forEach(f => {
if (f.textRun) {
const re = new RegExp(searchText, "g");
let p = null;
while (p = re.exec(f.textRun.content)) {
ar.push({insertPageBreak: {location: {index: p.index + offset}}});
}
}
})
}
offset = e.endIndex;
return ar;
}, []).reverse();
// 3. Request the request body to the method of "documents.batchUpdate" in Docs API.
const url2 = `https://docs.googleapis.com/v1/documents/${docId}:batchUpdate`;
UrlFetchApp.fetch(url2, {method: "post", payload: JSON.stringify({requests: requests}), contentType: "application/json", headers: {authorization: "Bearer " + accessToken}});
// DocumentApp.getActiveDocument(); // This is used for automatically adding a scope of https://www.googleapis.com/auth/documents by the script editor.
}
Note:
When an error related to Google Docs API occurs, please enable Google Docs API at API console again.

Web Scraping interactive map (javascript) with R and PhantomJS

I am trying to scrape data from an interactive map (looking to get crime data for a county). I am using R (rvest) and trying to use phantomjs too. I'm new to web scraping so I am not really understanding how all the elements work together (trying to get there).
The problem I believe I am having is that after I run the phantomjs and upload the html using R's rvest package, I end up with more scripts and no clear data in the html. My code is below.
writeLines("var url = 'http://www.google.com';
var page = new WebPage();
var fs = require('fs');
page.open(url, function (status) {
just_wait();
});
function just_wait() {
setTimeout(function() {
fs.write('cool.html', page.content, 'w');
phantom.exit();
}, 2500);
}
", con = "scrape.js")
A function that takes in the url that I want to scrape
s_scrape <- function(url = "https://gis.adacounty.id.gov/apps/crimemapper/",
js_path = "scrape.js",
phantompath = "/Users/alihoop/Documents/phantomjs/bin/phantomjs"){
# this section will replace the url in scrape.js to whatever you want
lines <- readLines(js_path)
lines[1] <- paste0("var url ='", url ,"';")
writeLines(lines, js_path)
command = paste(phantompath, js_path, sep = " ")
system(command)
}
Execute the js_scrape() function and get a html file saved as "cool.html"
js_scrape()
Where I am not understanding what to do next is the below R code:
map_data <- read_html('cool.html') %>%
html_nodes('script')
The output I get in the HTML via phantomjs is just scripts again. Looking for help on how to proceed when faced (in my mind) is javascript nested in javascript scripts(?)
Thank you!

This site uses javascript to make queries to the server. One solution is to reproduce the rest request and read the returning JSON file directly. This avoids the need to use Phantomjs.
From the developer tools options from your browser and looking through the xhr files, you will find a file(s) named "query" with a link similar to: "https://gisapi.adacounty.id.gov/arcgis/rest/services/CrimeMapper/CrimeMapperWAB/FeatureServer/11/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=1000"
Read this JSON response directly and convert to a list with the use of the jsonlite package:
library(jsonlite)
output<-jsonlite::fromJSON("https://gisapi.adacounty.id.gov/arcgis/rest/services/CrimeMapper/CrimeMapperWAB/FeatureServer/11/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=1000")
output$features
Find the first number in the link, (11 in this case) "FeatureServer/11/query?f=json". This number will determine which crime to query the server with. I found, it can take a value from 0 to 11. Enter 0 for arson, 4 for drugs, 11 for vandalism, etc.

Getting GeoJSON from Google Spreadsheets

I would like to have an option to save the geoJSON file from Google Spreadsheets.
The issue has been raised here:
https://gis.stackexchange.com/questions/140995/publishing-google-sheet-to-web-as-geojson-file
although the plugin available here:
https://github.com/mapbox/geo-googledocs/
doesn't work because the geocode receiving the error message:
https://github.com/mapbox/geo-googledocs/issues/42
UiApp has been deprecated. Please use HtmlService instead.
I tired to combine the existing code, provided by this plugin with the code proposed as a way to fixing this issue:
https://gist.github.com/hidrodixtion/c3a6b6ba7af624d1800625efb7a40fbd
The code name is Geo.js as per the plugin provided above
I put the function doGet above the Geocoders as per below:
// Global variables
function doGet() {
return HtmlService.createHtmlOutputFromFile('Index');
}
var ss = SpreadsheetApp.getActiveSpreadsheet(),
sheet = ss.getActiveSheet(),
activeRange = ss.getActiveRange(),
settings = {};
var geocoders = {
yahoo: {
... another part of the code available here:
https://raw.githubusercontent.com/mapbox/geo-googledocs/master/MapBox.js
unfortunately it doesn't work.
It seems like I don't know where I must superseed the depreciated UiApp function with the HTML service.
The problem is, that I don't know what part of the code in the Geo plugin should be replaced. It might be really helpful as the plugin can do the geocoding either.
My another part of combination was as follows:
I changed the code:
// Create a new UI
var app = UiApp.createApplication()
.setTitle('Export GeoJSON')
.setStyleAttribute('width', '460')
.setStyleAttribute('padding', '20');
into:
var app = HtmlService.createHtmlOutput().setWidth(800).setHeight(600);
.setTitle('Export GeoJSON') //line 105
.setStyleAttribute('width', '460')
.setStyleAttribute('padding', '20');
but once I save now I get this:
Syntax error. (line 105, file "Geo")
I picked up this code from here:
https://gist.github.com/hidrodixtion/c3a6b6ba7af624d1800625efb7a40fbd
line 141
Does anybody knows how to fix this issue?

How to load external js library in Jmeter?

I have the following code in a jsr223 sampler:
var key = "key";
var dateStamp = "20160329T134359Z";
var regionName = "us-east-1";
var serviceName = "execute-api";
var kDate= Crypto.HMAC(Crypto.SHA256, dateStamp, "AWS4" + key, { asBytes: true})
var kRegion= Crypto.HMAC(Crypto.SHA256, regionName, kDate, { asBytes: true });
var kService=Crypto.HMAC(Crypto.SHA256, serviceName, kRegion, { asBytes: true });
var kSigning= Crypto.HMAC(Crypto.SHA256, "aws4_request", kService, { asBytes: true });
vars.put("AWSKey", kSigning);
Now when I run it i get this error:
Response code: 500
Response message: javax.script.ScriptException: sun.org.mozilla.javascript.internal.EcmaError: ReferenceError: "Crypto" is not defined. (#6) in at line number 6
Obviously I dont have the crypto libs. However I am at loss on how to load them. I downloaded all the relavant js and put them in the /lib folder and still nothing.
I downloaded this file: https://github.com/Boussetta/CryptoJS-v3.1.2
Which handles the functions in the code above but for the life of me I have not idea how to import it.
TIA

If you want to go for JavaScript - there are 2 options:
Use Rhino load() method like:
load("crypto.js")
Use HmacUtils class from Apache Commons Codec from JavaScript
var rawhmac = org.apache.commons.codec.digest.HmacUtils.hmacSha1(key,data)
var encoded = org.apache.commons.codec.binary.Base64.encodeBase64String(rawhmac)
However I would recommend going for option 3 - switch to "groovy" language instead of JavaScript, that way you will be able to:
Re-use Amazon authentication samples in your test
Get maximum performance and confidence as groovy scripts can be compiled while other languages are interpreted so groovy implementation will take less resources and will work faster. See Beanshell vs JSR223 vs Java JMeter Scripting: The Performance-Off You've Been Waiting For! article for more details.

How to get google search output in the google application script environment?

If I use next function to get google output:
function myFunction() {
var post_url, result;
post_url = "http://www.google.com/search?q=stack+overflow";
result = UrlFetchApp.fetch(post_url);
Logger.log(result);
}
doesn't work.
P.S.
Sorry, I have to eхplore some dependences.
I take an example
function scrapeGoogle() {
var response = UrlFetchApp.fetch("http://www.google.com/search?q=labnol");
var myRegexp = /<h3 class=\"r\">([\s\S]*?)<\/h3>/gi;
var elems = response.getContentText().match(myRegexp);
for(var i in elems) {
var title = elems[i].replace(/(^\s+)|(\s+$)/g, "")
.replace(/<\/?[^>]+>/gi, "");
Logger.log(title);
}
}
and it works, than I begin to do some modifications and noticed that when I have some error in code it gives me an error
Request failed for http://www.google.com/search?q=labnol returned code
503.
So I did some researches without error's and it solution works. But when I began to form it to the function in lib it begans to throw me an error of 503 each time!
I'm very amazing of such behavior...
Here is short video only for fact. https://youtu.be/Lem9eiIVY0I
P.P.S.
Oh! I've broke some violations, so the google engine send me to stop list
so I run this:
function scrapeGoogle() {
var options =
{
'muteHttpExceptions': true
}
var response = UrlFetchApp.fetch("http://www.google.com/search?q=labnol", options);
Logger.log(response);
}
and get
About this pageOur systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen?
As I see I have to use some special google services to get the search output and not to be prohibited?

You can use simple regex to extract Google search results.
var regex = /<h3 class=\"r\">([\s\S]*?)<\/h3>/gi;
var items = response.getContentText().match(regex);
Alternatively, you can use the ImportXML function in sheets.
=IMPORTXML(GOOGLE_URL, "//h3[#class='r']")
See: Scrape Google Search with Sheets