We Keep Coding

Web scraping in R by first navigating through a JavaScript module

I looked up various questions and answers but unfortunately none of the problems I found dealt with a case that is similar to mine. In a typical question, the JavaScript table builds up directly when the website is loaded. In my case, however, I first have to navigate through the JavaScript module and select several criteria before I get the sought-after result.
This is my case: I have to scrape the exchange rates for various currencies from this website www.globocambio.co. To do that, I have (1) to navigate to “I WANT COLOMBIAN PESO”, (2) select the currency (e.g., “Chilean Peso”), (3) and the collection destination (e.g., “El Dorado International Airport”). Only then the respective exchange rate is being loaded. See this screenshot for illustration. I marked the three selection steps red. Green is the data point that I want to scrape for different currencies.
I am not very familiar with JavaScript but I tried to understand what is going on. Here is what I found out:
Using Chrome DevTools, I investigated the Network activity when loading an exchange rate. There is an XHR called “GetPrice” that requests the price using this URL: https://reservations.globocambio.co/DesktopModules/GlobalExchange/API/Widget/GetPrice and using the following Form Data
ISOAOrigen=CLP&cantidadOrigen=9000&ISOADestino=COP&cantidadDestino=0&centerId=27&operationType=OperationTypesBuying
I understand that the Form Data contains the information that I initially selected manually:
operationType=OperationTypesBuying: this is the “I WANT COLOMBIAN PESO” option
ISOAOrigen=CLP: this is the “Chilean Peso”
centerId=27: this is the “El Dorado International Airport”
The server responds to my request with the following information:
{“MonedaOrigen":{"ISOA":"CLP","Nombre":null,"Margen":0.1630000000,"Tramo":0.0,"Fixing":2.9000000000},"CantidadOrigen":9000.00,"MonedaDestino":{"ISOA":"COP","Nombre":null,"Margen":0.0,"Tramo":0.0,"Fixing":0.0},"CantidadDestino":21845.70,"TipoCambio":2.42730000000000000000,"MargenOrigen":0.0,"TramoOrigen":0.0,"FixingOrigen":0.0,"MargenDestino":0.0,"TramoDestino":0.0,"FixingDestino":0.0,"IdCentro":"27","Comision":null,"ComisionTramoSuperior":null,"ComisionAplicada":{"CodigoMoneda":null,"CodigoTipoMoneda":0,"ComisionFija":0.0,"ComisionVariable":0.0,"TramoInicio":0.0,"TramoFin":null,"Orden”:0}}
From this response, "TipoCambio":2.42730000000000000000 is then being written on the website using this line of HTML code: <span id="spTipoCambioCompra">2.427300</span>
This means that "TipoCambio" is the value that I am looking for.
So, I have to communicate somehow via R with the server using the Form Data as input variables. Can anyone tell me how to do this?
I mean, understand that I have to combine the URL https://reservations.globocambio.co/DesktopModules/GlobalExchange/API/Widget/GetPrice with the Form Data “ISOAOrigen=CLP&cantidadOrigen=9000&ISOADestino=COP&cantidadDestino=0&centerId=27&operationType=OperationTypesBuying” somehow but I do not know how it works..
Any help will be appreciated!
Update:
I still have no idea how to solve the above issue, yet. However, I try to approach it with small steps.
Using RSelenium, I am currently trying to find out how to click on the option “I WANT COLOMBIAN PESO”. My idea was to use the following code:
library(RSelenium)
remDr <- RSelenium::remoteDriver(remoteServerAddr = "localhost",
port = 4445L,
browserName = "chrome")
remDr$open()
remDr$navigate("https://www.globocambio.co/en/home")
webElem <- remDr$findElement("id", "tabCompra") #What is wrong here?
webElem$clickElement() # Click on "I WANT COLOMBIAN PESO"
But I get an error message after executing webElem <- remDr$findElement("id", "tabCompra"):
Selenium message:no such element: Unable to locate element: {"method":"css selector","selector":"#tabCompra"}
(Session info: chrome=81.0.4044.113)
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html
...
Error: Summary: NoSuchElement
Detail: An element could not be located on the page using the given search parameters.
class: org.openqa.selenium.NoSuchElementException
Further Details: run errorDetails method
What am I doing wrong here?

I solved my problem using selenium in Python:
from selenium import webdriver
driver = webdriver.Firefox(executable_path = '/your_path/geckodriver')
driver.get("https://www.globocambio.co/en/")
driver.switch_to.frame("iframeWidget");
elem = driver.find_element_by_id('tabCompra')
elem.click()
elem = driver.find_element_by_id('inputddlMonedaOrigenCompra')
elem.click()
elem.send_keys(Keys.CLEAR)
elem.send_keys("Chilean Peso")
elem.send_keys(Keys.ENTER)
elem.send_keys(Keys.ARROW_DOWN)
elem.send_keys(Keys.RETURN)
elem = driver.find_element_by_id('info-change-compra')
print(elem.text)

Trying to figure out realexpayments - js (rxp-js)

I'm trying to figure out the usage of the Realexpayments javascript sdk, found here: https://github.com/realexpayments/rxp-js (and using the hosted pay page solution).
The usage makes a lot of sense, I have a rxp-hpp.js file that I can use like this:
RealexHpp.init(payButtonId, merchantUrl, jsonFromServerSdk);
The part that I don't understand is
payButtonId - The ID of the button used to launch the lightbox.
merchantUrl - The URL to which the JSON response from Realex will be posted.
jsonFromServerSdk - The JSON output from the Realex HPP Server SDK.
the jsonFromServerSdk - how do I get\build that? I could see if it was json I'd need to build up on the client side, but jsonFromServerSdk would imply that I need to get that somehow (beforehand), and it isn't explained too well on the readme file.
I think I'm just missing something vital here. The idea is to have a payment handled purely on the realex site (and not my website) as to be PCI compliant (i.e. no credit card #'s on my site at all).
Any assistance\explanation would be appreciated, thanks.
I should be able to get away wtih avoiding a server-side call just for testing, but I'm trying this:
var myJson= JSON.stringify({
TIMESTAMP: '20170504081338',
'MERCHANT_ID': '<mymerchantid>',
ACCOUNT: 'internet',
ORDER_ID: 'testorderid',
AMOUNT: '11.95',
CURRENCY: 'CAD',
SHA1HASH: '<myhash>',
CUST_NUM: 'testcustnum',
HPP_LANG: 'US',
HPP_VERSION: '2',
CARD_PAYMENT_BUTTON: 'Pay.'
});
RealexHpp.init(payButtonId, myMerchantUrl, myJson);
And it seems to be trying to work: but I get a red error that says: Invalid MERCHANT_ID or ACCOUNT. Please contact the merchant.
Hard to tell if maybe my json is malformed, or what.

TypeError: Cannot read property "1" from undefined

I have some very simple code that is triggered by a google form (see http://goo.gl/lFHi3j) that asks for 2 email addresses. The code works fine when the email address is hardcoded (see commented email below) but when I try and obtain the email using e.values (when the form is submitted) is get the following error
Execution failed: TypeError: Cannot read property "1" from undefined.
(line 3, file "Code").
I've searched this forum but can't find a solution to this problem. Can anyone please help.
function onFormSubmit (e) {
// var email_address_ = "pian1966#gmail.com";
var email_address_ = e.values[1];
// Send the email
var subject_ = "Andy's Report";
var body_ = "Here is Andy's report...";
MailApp.sendEmail(email_address_, subject_, body_, {htmlBody: body_});
}

There is no e.values[1].
That is the answer to the question as to why you are getting that error.
Do a console.log(e) and familiarize yourself with what e consists of by looking at the log results in a browser debugger.
From there you can determine for yourself if e.values even exists as an array. Then go from there.

Thanks for the responses which all helped me find a solution to the problems I was having. Here's what I've discovered as I've worked on a solution:
The Google App Script must be created (as a blank script) from
responses spreadsheet and NOT out of the form itself!
In Google Forms, the developer has the option of unlinking their
responses spreadsheet and sending responses to new results
spreadsheet. If this option is selected then a new script needs to
be created from the new results spreadsheet and the code must be
copied/moved into the new script. Of course this means that a new
trigger must be created and the script needs to be authorized to
run.
Once a user submits a form, they can be given the option to
edit/correct their submission. Unless EVERY field is edited, the
script thinks the unedited fields are blank and writes "undefined"
into the resulting Google Doc. I've therefore disabled the "Edit"
option in my form to get around this.
If the user skips over irrelevant fields (leaves them blank) then
the e.values numbering order is thrown out. I have therefore made
every field compulsory with the comment that the user enter 'n/a' if a
field doesn't apply.
Having worked through these issues my form now works perfectly (albeit a little clumsily with all questions being compulsory). I would be very grateful if anyone has any suggestions on how to get around the Editing and and Blank Field problems.
(Thanks to TJ Houston for his original script)

json standards for structure of responses

I'm trying to decide on a standard for json responses for my applications going forwards.
So I'm thinking something along the lines of
Fake Json
{
Message:"hello, this is a reponse",
action: 0 or 1, or "Good" or "bad",
statusCode: 234,
}
I'm wondering if there is a good standard for this already, and also how I can keep my standards in sync between my front and back end?
Is there a clever way to do this?

I'm using a structure similar to this:
{ status: ("ok", "error" or "redirect"), message: "some message goes here.", fields: "field selectors goes here in case some highlighting is needed", url: "url to open or redirect to if needed" }
The status property is the key to tell what to do next;
The message can be used to show a popup message or an inline message, etc. Can also be a list of messages separated by a '|' character to split;
The field property is used to select (with jQuery or other JS library) the fields or other objects that need an action (think of this for field validation);
The url property is used to change the page, open a popup or post something back, depending on your needs.

Jira Gadget: Configuration isn't saved for reconfiguration

I am writing a gadget for Jira with some configuration options. One of these configuration options is a "project or filter picker".
My problem lies in the part, when I want to reconfigure the gadget's preferences. I have read the code of the timesince-gadget as an example and I think the relevant part is the following:
if (/^jql-/.test(gadget.getPref("projectOrFilterId"))){
projectAndFilterPicker =
{
userpref: "projectOrFilterId",
type: "hidden",
value: gadgets.util.unescapeString(this.getPref("projectOrFilterId"))
};
} else {
projectAndFilterPicker = AJS.gadget.fields.projectOrFilterPicker(gadget, "projectOrFilterId", args.options);
}
Basicly I've copied the code from the timesince-gadget. Unfortunately even if already configured, the javascript always enters the else part.
A problem is, that I ve no experience with jql and don't totally understand the if clause.
But usually (e.g. when calling the rest api and processing the config infos)
gadget.getPref("projectOrFilterId")
returns a string containing the id of the picked project or filter.
Question is now: How can I make my gadget remember the last configuration like it's done with some many other Jira gadgets?
I really hope anyone can help me with that.

It turnes out, the answer is even simplier then I thought.
First: In the descriptor you can totally forget the if part from above. Just
var projectAndFilterPicker = AJS.gadget.fields.projectOrFilterPicker(gadget, "projectOrFilterId", args.options);
is needed.
Second: Retrieve the project's or filter's name in your rest resource, which shouldn't be a problem, since you already want to use the processed id. Then return this name back to the view part of your javascript and type in something like
this.projectOrFilterName = args.myrestclasskey.projectOrFilterName;
And tada: reconfiguration will display the old configured name!

I had this problem once when I forgot to specify the option in the Gadget XML file. I solved it by adding this to the XML:
<UserPref name="projectOrFilterId" datatype="hidden"/>