We Keep Coding

Web Scraping interactive map (javascript) with R and PhantomJS

I am trying to scrape data from an interactive map (looking to get crime data for a county). I am using R (rvest) and trying to use phantomjs too. I'm new to web scraping so I am not really understanding how all the elements work together (trying to get there).
The problem I believe I am having is that after I run the phantomjs and upload the html using R's rvest package, I end up with more scripts and no clear data in the html. My code is below.
writeLines("var url = 'http://www.google.com';
var page = new WebPage();
var fs = require('fs');
page.open(url, function (status) {
just_wait();
});
function just_wait() {
setTimeout(function() {
fs.write('cool.html', page.content, 'w');
phantom.exit();
}, 2500);
}
", con = "scrape.js")
A function that takes in the url that I want to scrape
s_scrape <- function(url = "https://gis.adacounty.id.gov/apps/crimemapper/",
js_path = "scrape.js",
phantompath = "/Users/alihoop/Documents/phantomjs/bin/phantomjs"){
# this section will replace the url in scrape.js to whatever you want
lines <- readLines(js_path)
lines[1] <- paste0("var url ='", url ,"';")
writeLines(lines, js_path)
command = paste(phantompath, js_path, sep = " ")
system(command)
}
Execute the js_scrape() function and get a html file saved as "cool.html"
js_scrape()
Where I am not understanding what to do next is the below R code:
map_data <- read_html('cool.html') %>%
html_nodes('script')
The output I get in the HTML via phantomjs is just scripts again. Looking for help on how to proceed when faced (in my mind) is javascript nested in javascript scripts(?)
Thank you!

This site uses javascript to make queries to the server. One solution is to reproduce the rest request and read the returning JSON file directly. This avoids the need to use Phantomjs.
From the developer tools options from your browser and looking through the xhr files, you will find a file(s) named "query" with a link similar to: "https://gisapi.adacounty.id.gov/arcgis/rest/services/CrimeMapper/CrimeMapperWAB/FeatureServer/11/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=1000"
Read this JSON response directly and convert to a list with the use of the jsonlite package:
library(jsonlite)
output<-jsonlite::fromJSON("https://gisapi.adacounty.id.gov/arcgis/rest/services/CrimeMapper/CrimeMapperWAB/FeatureServer/11/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=1000")
output$features
Find the first number in the link, (11 in this case) "FeatureServer/11/query?f=json". This number will determine which crime to query the server with. I found, it can take a value from 0 to 11. Enter 0 for arson, 4 for drugs, 11 for vandalism, etc.

HTTP Request using Mozilla Rhino

I would like to write a JavaScript code processed with Mozilla Rhino that can do a simple HTTP GET request, which fetches a text string from a URL.
The problem is that, I couldn't find any support in Rhino to do any kind of HTTP requests. Besides, I don't have access to the Rhino instance itself, it's running via TopBraid Composer IDE for ontology modelling. I believe any idea about a simple library that I can import within my JavaScript file maybe a good solution.
Any help?
Thanks.

Okay, so it wasn't that difficult to figure it out. This one works via TopBraid Composer and without importing any JAVA libraries. Here's the answer in case anyone needs it later on.
var resourceURL = new java.net.URL(
'http://someurl');
var urlConnection = resourceURL.openConnection();
var inputStream = new java.io.InputStreamReader(urlConnection
.getInputStream());
var bufferedReader = new java.io.BufferedReader(inputStream);
var inputLine = bufferedReader.readLine();
bufferedReader.close();
var jsString = String(inputLine);
return jsString;

Using RHINO js engine to make http requests

I'm trying to use the Mozilla/Rhino js engine to test some SOAP requests in the command line. However, none of the normal objects for making requests (XMLHttpRequest, HttpRequest) seem to be available. Why is this? Can I import libraries?

I was able to get it to work using just Rhino with the following code.
var post = new org.apache.commons.httpclient.methods.PostMethod("https://someurl/and/path/");
var client = new org.apache.commons.httpclient.HttpClient();
// ---- Authentication ---- //
var creds = new org.apache.commons.httpclient.UsernamePasswordCredentials("username", "password");
client.getParams().setAuthenticationPreemptive(true);
client.getState().setCredentials(org.apache.commons.httpclient.auth.AuthScope.ANY, creds);
// -------------------------- //
post.setRequestHeader("Content-type", "application/xml");
post.setRequestEntity(new org.apache.commons.httpclient.methods.StringRequestEntity(buildXML(), "text/plain", "ASCII" ));
var status = client.executeMethod(post);
var br = new java.io.BufferedReader(new java.io.InputStreamReader(post.getResponseBodyAsStream()));
var response = "";
var line = br.readLine();
while(line != null){
response = response + line;
line = br.readLine();
}
post.releaseConnection();

You might possibly find a library to import, you could also write your own in Java and make them available to your rhino instance, depending on how your are using it. Keep in mind Rhino is just a Javascript language engine. It doesn't have a DOM, and is not inherently 'web aware' so to speak.
However, since it sounds like you are doing this for testing/experimentation purposes, and you will probably be more productive not having to reinvent the wheel to do so, I will strongly, strongly suggest that you just download Node.js and look into the request module (for making HTTP requests) or any of the various SOAP modules.
You can do a ton more with Node.js, but you can also use it as a very simple runner for Javascript files as well. Regardless you should move away from Rhino though. It is really old and not really supported anymore, especially now that with JDK8 even the javax.script support will switch to the Nashorn engine.
UPDATE: If you really want to give it a go (and if you are prepared to monkey around with Java), you might look at this SO question and its answers. But unless you are something of a masochist, I think you'll be happier taking a different path.

I was actually able to do this using Orchestrator 5.1 with the 'Scriptable task' object to interface with the Zabbix API:
var urlObject = new URL(url);
var jsonString = JSON.stringify({ jsonrpc: '2.0', method: 'user.login', params: { user: 'username', password: 'password' }, id: 1 });
urlObject.contentType = "application/json";
result = urlObject.postContent(jsonString);
System.log(result);
var authenticationToken = JSON.parse(result).result;

Does anyone knows how to import new classes when using Javascript instead of Groovy as default scripting language in SoapUI?

I have a important piece of code that modifies my requests written Javascript, but i want to add an external jar and use it from the scripts, any suggestions? It's defined a sort of import function or something like that?

In order to use your own jars in Setup Script in SOAPUI with javascript language you must do the next steps:
With SOAPUI closed, copy your jars to SOAPUI_HOME/bin/ext.
Start SOPAUI
On the project properties select Javascript for Script Language property.
Then in the Setup Script you have to reference the full package for your java classes like (I put a sample where I decode a base64 string):
java.lang.System.out.println("-----------");
java.lang.System.out.println("SAMPLE INIT");
java.lang.System.out.println("-----------\n");
var obj = new org.apache.commons.codec.binary.Base64();
var stringb64 = new java.lang.String("dGVzdA==")
var dataDecoded = obj.decodeBase64(stringb64.getBytes());
java.lang.System.out.println("RESULT " + new java.lang.String(dataDecoded) + "\n");
java.lang.System.out.println("-----------");
java.lang.System.out.println("SAMPLE END");
java.lang.System.out.println("-----------");
If you run soapui from .bat you can see the system out on the cmd:

How can I list websites on IIS7, from script, without using IIS6 compat pack (WMI veneer)

On IIS6, I can use WMI to list available websites, like this:
var iis = GetObject("winmgmts://localhost/root/MicrosoftIISv2");
var query = "SELECT * FROM IIsWebServerSetting"
// get the list of virtual servers
var results = iis.ExecQuery(query);
for(var e = new Enumerator(results); !e.atEnd(); e.moveNext()) {
var site = e.item();
// site.Name // W3SVC/1, W3SVC/12378398, etc
// site.Name.substr(6) // 1, 12378398, etc
// site.ServerComment) // "Default Web Site", "Site2", etc
// site.ServerBindings(0).Port // 80, 8080, etc
}
I know I can run this script on IIS7, if I have previously installed the IIS6 Compatibility Pack.
Is it possible to get the list of WebSites without requiring the compatibility pack as a pre-requisite?
I know I can run AppCmd to do this from the command line:
\Windows\system32\inetsrv\appcmd list sites
But... can I run that from a custom action in an MSI?
And... if not, how can I do the equivalent thing (list websites on IIS7) from javascript?
EDIT
Here's how I tried running the command from within Javascript.
function GetWebSites_IIS7()
{
var ParseOneLine = function(oneLine) {
...a bunch of regex parsing here....
};
LogMessage("GetWebSites_IIS7() ENTER");
var shell = new ActiveXObject("WScript.Shell");
var windir = shell.Environment("system")("windir");
// aka Session.Property("%WINDIR%")
var appcmd = windir + "\\system32\\inetsrv\\appcmd.exe list sites";
var oExec = shell.Exec(appcmd);
var sites = [];
while (!oExec.StdOut.AtEndOfStream) {
var oneLine = oExec.StdOut.ReadLine();
var line = ParseOneLine(oneLine);
LogMessage(" site: " + line.name);
sites.push(line);
}
return sites;
}
This works, but it briefly pops a visible console window, which then disappears. Doesn't look very polished. I think I can avoid the console window by using shell.Run() instead of shell.Exec(). But shell.Run() doesn't give access to the stdout, so I would have to redirect the output to a temporary file, then read the output. I haven't tried that yet. That may introduce some security issues; I'll have to see.
Related:
Where and how should my CustomAction create and read a temporary file?

Yes, you can run appcmd from the custom action the same way you do any custom action which runs exe. First off, you should author a DirectorySearch/FileSearch elements to find the full path to the executable. Next, add a custom action with ExeCommand attribute. You're probably trying to get feedback from a user, so leave it immediate. Also, think about using QuietExec in order not to show console window to your users.
By the way, if my guess is correct, you're trying to do something like this. Hope this helps.