We Keep Coding

Take selected text, send it over to Scryfall API, then take the link and put it in the selected text

I've been able to sort out the middle bit (the API seems to be called to just fine) along with the submenu displaying. Originally I thought that just the end part wasn't working but I'm now thinking that the selection part isn't either.
What am I doing wrong with the getSelection() and what do I need to do to insert a link into said selection? (to clarify, not to replace the text with a link, but to insert a link into the text)
//Open trigger to get menu
function onOpen(e) {
DocumentApp.getUi().createAddonMenu()
.addItem('Scry', 'serumVisions')
.addToUi();
}
//Installation trigger
function onInstall(e) {
onOpen(e);
}
//I'm not sure if I need to do this but in case; declare var elements first
var elements
// Get selected text (not working)
function getSelectedText() {
const selection = DocumentApp.getActiveDocument().getSelection();
if (selection) {
var elements = selection.getRangeElements();
Logger.log(elements);
} else {
var elements = "Lack of selection"
Logger.log("Lack of selection");
}
}
//Test run
// insert here
// Search Function
function searchFunction(nameTag) {
// API call + inserted Value
let URL = "https://api.scryfall.com/cards/named?exact=" + nameTag;
// Grabbing response
let response = UrlFetchApp.fetch(URL, {muteHttpExceptions: true});
let json = response.getContentText();
// Translation
let data = JSON.parse(json);
// Jackpot
let link = data.scryfall_uri;
// Output
Logger.log(link);
}
// Test run
searchFunction("Lightning Bolt");
//Let's hope this works how I think it works
function serumVisions() {
const hostText = getSelectedText();
const linkage = searchFunction(hostText);
// Unsure what class I'm supposed to use, this doesn't
const insertLink = DocumentApp.getActiveDocument().getSelection().newRichTextValue()
.setLinkUrl(linkage);
Logger.log(linkage);
}
For the first part, I tried the getSelection() and getCursor() examples from the Google documentation but they don't seem to work, they all just keep returning null.
For the inserting link bit, I read all those classes from the Spreadsheet section of the documentation, at the time I was unaware but now knowing, I haven't been able to find a version of the same task for Google Docs. Maybe it works but I'm writing it wrong as well, idk.

Modification points:
In your script, the functions of getSelectedText() and searchFunction(nameTag) return no values. I think that this might be the reason for your current issue of they all just keep returning null..
elements of var elements = selection.getRangeElements(); is not text data.
DocumentApp.getActiveDocument().getSelection() has no method of newRichTextValue().
In the case of searchFunction("Lightning Bolt");, when the script is run, this function is always run. Please be careful about this.
When these points are reflected in your script, how about the following modification?
Modified script:
Please remove searchFunction("Lightning Bolt");. And, in this case, var elements is not used. Please be careful about this.
From your script, I guessed that in your situation, you might have wanted to run serumVisions(). And also, I thought that you might have wanted to run the individual function. So, I modified your script as follows.
function getSelectedText() {
const selection = DocumentApp.getActiveDocument().getSelection();
var text = "";
if (selection) {
text = selection.getRangeElements()[0].getElement().asText().getText().trim();
Logger.log(text);
} else {
text = "Lack of selection"
Logger.log("Lack of selection");
}
return text;
}
function searchFunction(nameTag) {
let URL = "https://api.scryfall.com/cards/named?exact=" + encodeURIComponent(nameTag);
let response = UrlFetchApp.fetch(URL, { muteHttpExceptions: true });
let json = response.getContentText();
let data = JSON.parse(json);
let link = data.scryfall_uri;
Logger.log(link);
return link;
}
// Please run this function.
function serumVisions() {
const hostText = getSelectedText();
const linkage = searchFunction(hostText);
if (linkage) {
Logger.log(linkage);
DocumentApp.getActiveDocument().getSelection().getRangeElements()[0].getElement().asText().editAsText().setLinkUrl(linkage);
}
}
When you select the text of "Lightning Bolt" in the Google Document and run the function serumVisions(), the text of Lightning Bolt is retrieved, and the URL like https://scryfall.com/card/2x2/117/lightning-bolt?utm_source=api is retrieved. And, this link is set to the selected text of "Lightning Bolt".
Reference:
getSelection()

How to iterate on the links to google forms in the cells of a spreadsheet column to get their questions?

I have a Google spreadsheet with links to questionnaires. I wanted to know how to get the questions from each of the questionnaires.
I guess I have to do: at best, use the script editor and iterate on the lines, and at worst, do webscraping.
const puppeteer = require('puppeteer');
function appendString() {
var range = SpreadsheetApp.getActiveSheet().getActiveRange();
var numRows = range.getNumRows();
var numCols = 0;
for (var i = 1; i <= numRows; i++) {
for (var j = 1; j <= numCols; j++) {
var currentValue = range.getCell(i,j).getValue();
await page.goto(currentValue);
const pollFrame = page.frames().find() # From there I have some difficulties
}
}
}
But I get the following error:
SyntaxError: await is only valid in async function (ligne 10, fichier "Code.gs")
Not to mention the async problem or the buttonthat I still have to click, the selection looks like this:
<div class="freebirdFormviewerViewItemsItemItemTitle exportItemTitle freebirdCustomFont" id="i1" role="heading" aria-level="3" aria-describedby="i.desc.310938276">How often did you fly before the Covid-19 epidemic? </div>
But the IDs don't follow a logical numerical order, so I don't know how to extract them automatically.
Then I don't know how to do it. I wonder if it's simpler because they're products from the same supplier.
Here is the equivalent in csv format:
https://docs.google.com/forms/d/e/1FAIpQLSfzocEm6IEDKVzVGOlg8ijysWZyAvQur0NheJb_I_xozgKusA/viewform?usp=sf_link
https://docs.google.com/forms/d/e/1FAIpQLScrm0ZTrvlONf5MX37N93H_FajNzfbNy9ZtitX-Vq9PPuLPHA/viewform?usp=sf_link
https://docs.google.com/forms/d/e/1FAIpQLSeolFSh3OyS_XpX1lRIJP-8CH8WG0X0hL98SM9d85LqC22Bow/viewform?usp=sf_link
Update
So I tried the anwer kindly posted by Neven Subotic's:
// this array will store forms and their questions
let formAndQuestions = [];
let formIds = ["https://docs.google.com/forms/d/e/1FAIpQLSfzocEm6IEDKVzVGOlg8ijysWZyAvQur0NheJb_I_xozgKusA/viewform?usp=sf_link",
"https://docs.google.com/forms/d/e/1FAIpQLScrm0ZTrvlONf5MX37N93H_FajNzfbNy9ZtitX-Vq9PPuLPHA/viewform?usp=sf_link",
"https://docs.google.com/forms/d/e/1FAIpQLSeolFSh3OyS_XpX1lRIJP-8CH8WG0X0hL98SM9d85LqC22Bow/viewform?usp=sf_link"]
formIds.forEach( formId => {
const form = FormApp.openById( formId );
// lets get the name
const formName = form.getTitle();
// first we get all items
const allItemsInThisForm = form.getItems();
// then we get filter out anything that is not a questions
const allQuestionsInThisForm = allItemsInThisForm.filter( item => {
return isThisItemAQuestion( item )
});
// now we store them in our object
formAndQuestions.push( {
formId: formId,
formName: formName,
questions: allQuestionsInThisForm
})
});
// this function is used to only get the itemTypes you want
// see reference for more information
function isThisItemAQuestion( item ){
const itemType = item.getType();
const validQuestionItemTypes = [ FormApp.ItemType.TEXT, "add others here" ]
let isValid = false;
validQuestionItemsTypes.forEach( validItemType => {
if( itemType == validItemType ) {
isValid = true;
}
});
return isValid
}
Unfortunately I obtain the following error message with the following details Exception: No item with the given ID could be found, or you do not have permission to access it. (line 9, "const form = FormApp.openById( formId );"). I don't understand. As you can see in the gif, I can open these links, so I should have the permission to access them isn't it?
I also tried Ruben's ideas with:
// this array will store forms and their questions
let formAndQuestions = [];
let formIds = ["https://docs.google.com/forms/d/e/1FAIpQLSfzocEm6IEDKVzVGOlg8ijysWZyAvQur0NheJb_I_xozgKusA/viewform?usp=sf_link"]//,
//"https://docs.google.com/forms/d/e/1FAIpQLScrm0ZTrvlONf5MX37N93H_FajNzfbNy9ZtitX-Vq9PPuLPHA/viewform?usp=sf_link",
//"https://docs.google.com/forms/d/e/1FAIpQLSeolFSh3OyS_XpX1lRIJP-8CH8WG0X0hL98SM9d85LqC22Bow/viewform?usp=sf_link"]
function scrapeForms(){
formIds.forEach( formId => {
// The code below logs the HTML code of the Google home page.
var response = UrlFetchApp.fetch(formId);
results = response.getElementsByClassName("freebirdFormviewerViewItemsItemItemTitleContainer");
Logger.log(results.getContentText())
});
}
But got back:
TypeError: response.getElementsByClassName is not a function (ligne 13, fichier "Code")

According to What is this Javascript "require"? require is not part of the standard JavaScript an AFAIK it's not supported by Google Apps Script.
By the other hand, the error message can't be easily solved as Google Apps Script Chrome V8 engine doesn't support async functions. Related Is google apps script synchronous?
If you will be using Google Apps Script, and you are the form owner or a form editor, instead of trying to web scraping a Google Form use the Forms Service of Google Apps Script. For this you will need the form ../edit URLs instead of the ../viewform URLs. On the official docs there is a quickstart that might help you https://developers.google.com/apps-script/quickstart/forms.
You could use openByUrl to "open" a form. It will not be actually opened in your web browser, it will be opened on the server side. Then you could use getItems to get all the questions, sections, images, videos, etc.
If you aren't the form owner or a form editor then you should use UrlFetchApp service and somehow parse the web page source code of each form based on the position of the questions. Related question: Google Sheets: How to import the following data?
Also, if the form has several sections you should do a post request to emulate clicking on the next button in order to get the second and following sections. There are more "also if the form has..." but I will stop here as the main part of question was already answered, I think.

You first want to get all the forms, so place those in an array:
const formIds = ["someId", "anotherId", "andSoOn"]
Then, lets use the FormApp to get the form and all items. Items can be of different types, see documentation.
// this array will store forms and their questions
let formAndQuestions = [];
formIds.forEach( formId => {
const form = FormApp.openById( formId );
// lets get the name
const formName = form.getTitle();
// first we get all items
const allItemsInThisForm = form.getItems();
// then we get filter out anything that is not a questions
const allQuestionsInThisForm = allItemsInThisForm.filter( item => {
return isThisItemAQuestion( item )
});
// now we store them in our object
formAndQuestions.push( {
formId: formId,
formName: formName,
questions: allQuestionsInThisForm
}
});
// this function is used to only get the itemTypes you want
// see reference for more information
function isThisItemAQuestion( item ){
const itemType = item.getType();
const validQuestionItemTypes = [ FormApp.ItemType.TEXT, "add others here" ]
let isValid = false;
validQuestionItemsTypes.forEach( validItemType => {
if( itemType == validItemType ) {
isValid = true;
}
});
return isValid
}
Then you can initially log out the results and see what it looks like:
Logger.log( formAndQuestions )
Item Types

Setting up a Papa Parse progress bar with Web workers

I'm working on a CSV parsing web application, which collects data and then uses it to draw a plot graph. So far it works nicely, but unfortunately it takes some time to parse the CSV files with papaparse, even though they are only about 3MB.
So it would be nice to have some kind of progress shown, when "papa" is working. I could go for the cheap hidden div, showing "I'm working", but would prefer the use of <progress>.
Unfortunately the bar just gets updated AFTER papa has finished its work. So I tried to get into webworkers and use a worker file to calculate progress and also setting worker: true in Papa Parses configuration. Still no avail.
The used configuration (with step function) is as followed:
var papaConfig =
{
header: true,
dynamicTyping: true,
worker: true,
step: function (row) {
if (gotHeaders == false) {
for (k in row.data[0]) {
if (k != "Time" && k != "Date" && k != " Time" && k != " ") {
header.push(k);
var obj = {};
obj.label = k;
obj.data = [];
flotData.push(obj);
gotHeaders = true;
}
}
}
tempDate = row.data[0]["Date"];
tempTime = row.data[0][" Time"];
var tD = tempDate.split(".");
var tT = tempTime.split(":");
tT[0] = tT[0].replace(" ", "");
dateTime = new Date(tD[2], tD[1] - 1, tD[0], tT[0], tT[1], tT[2]);
var encoded = $.toJSON(row.data[0]);
for (j = 0; j < header.length; j++) {
var value = $.evalJSON(encoded)[header[j]]
flotData[j].data.push([dateTime, value]);
}
w.postMessage({ state: row.meta.cursor, size: size });
},
complete: Done,
}
Worker configuration on the main site:
var w = new Worker("js/workers.js");
w.onmessage = function (event) {
$("#progBar").val(event.data);
};
and the called worker is:
onmessage = function(e) {
var progress = e.data.state;
var size = e.data.size;
var newPercent = Math.round(progress / size * 100);
postMessage(newPercent);
}
The progress bar is updated, but only after the CSV file is parsed and the site is set up with data, so the worker is called, but the answer is handled after parsing. Papa Parse seems to be called in a worker, too. Or so it seems if checking the calls in the browsers debugging tools, but still the site is unresponsive, until all data shows up.
Can anyone point me to what I have done wrong, or where to adjust the code, to get a working progress bar? I guess this would also deepen my understanding of web workers.

You could use the FileReader API to read the file as text, split the string by "\n" and then count the length of the returned array. This is then your size variable for the calculation of percentage.
You can then pass the file string to Papa (you do not need to reread directly from the file) and pass the number of rows (the size variable) to your worker. (I am unfamiliar with workers and so am unsure how you do this.)
Obviously this only accurately works if there are no embedded line breaks inside the csv file (e.g. where a string is spread over several lines with line breaks) as these will count as extra rows, so you will not make it to 100%. Not a fatal error, but may look strange to the user if it always seems to finish before 100%.
Here is some sample code to give you ideas.
var size = 0;
function loadFile(){
var files = document.getElementById("file").files; //load file from file input
var file = files[0];
var reader = new FileReader();
reader.readAsText(file);
reader.onload = function(event){
var csv = event.target.result; //the string version of your csv.
var csvArray = csv.split("\n");
size = csvArray.length;
console.log(size); //returns the number of rows in your file.
Papa.parse(csv, papaConfig); //Send the csv string to Papa for parsing.
};
}

I haven't used Papa Parse with workers before, but a few things pop up after playing with it for a bit:
It does not seem to expect you to interact directly with the worker
It expects you to either want the entire final result, or the individual items
Using a web worker makes providing a JS Fiddle infeasible, but here's some HTML that demonstrates the second point:
<html>
<head>
<script src="papaparse.js"></script>
</head>
<body>
<div id="step">
</div>
<div id="result">
</div>
<script type="application/javascript">
var papaConfig = {
header: true,
worker: true,
step: function (row) {
var stepDiv = document.getElementById('step');
stepDiv.appendChild(document.createTextNode('Step received: ' + JSON.stringify(row)));
stepDiv.appendChild(document.createElement('hr'));
},
complete: function (result) {
var resultDiv = document.getElementById('result');
resultDiv.appendChild(document.createElement('hr'));
resultDiv.appendChild(document.createTextNode('Complete received: ' + JSON.stringify(result)))
resultDiv.appendChild(document.createElement('hr'));
}
};
var data = 'Column 1,Column 2,Column 3,Column 4 \n\
1-1,1-2,1-3,1-4 \n\
2-1,2-2,2-3,2-4 \n\
3-1,3-2,3-3,3-4 \n\
4,5,6,7';
Papa.parse(data, papaConfig);
</script>
</body>
</html>
If you run this locally, you'll see you get a line for each of the four rows of the CSV data, but the call to the complete callback gets undefined. Something like:
Step received: {"data":[{"Column 1":"1-1",...
Step received: {"data":[{"Column 1":"2-1",...
Step received: {"data":[{"Column 1":"3-1",...
Step received: {"data":[{"Column 1":"4","...
Complete received: undefined
However if you remove or comment out the step function, you will get a single line for all four results:
Complete received: {"data":[{"Column 1":"1-1",...
Note also that Papa Parse uses a streaming concept to support the step callback regardless of using a worker or not. This means you won't know how many items you are parsing directly, so calculating the percent complete is not possible unless you can find the length of items separately.

Inserting into Collection after Promises in a Meteor Method

I'm using this Gumroad-API npm package in order to fetch data from an external service (Gumroad). Unfortunately, it seems to use a .then() construct which can get a little unwieldy as you will find out below:
This is my meteor method:
Meteor.methods({
fetchGumroadData: () => {
const Gumroad = Meteor.npmRequire('gumroad-api');
let gumroad = new Gumroad({ token: Meteor.settings.gumroadAccessKey });
let before = "2099-12-04";
let after = "2014-12-04";
let page = 1;
let sales = [];
// Recursively defined to continue fetching the next page if it exists
let doThisAfterResponse = (response) => {
sales.push(response.sales);
if (response.next_page_url) {
page = page + 1;
gumroad.listSales(after, before, page).then(doThisAfterResponse);
} else {
let finalArray = R.unnest(sales);
console.log('result array length: ' + finalArray.length);
Meteor.call('insertSales', finalArray);
console.log('FINISHED');
}
}
gumroad.listSales(after, before, page).then(doThisAfterResponse); // run
}
});
Since the NPM package exposes the Gumorad API using something like this:
gumroad.listSales(after, before, page).then(callback)
I decided to do it recursively in order to grab all pages of data.
Let me try to re-cap what is happening here:
The journey starts on the last line of the code shown above.
The initial page is fetched, and doThisAfterResponse() is run for the first time.
We first dump the returned data into our sales array, and then we check if the response has given us a link to the next page (as an indication as to whether or not we're on the final page).
If so, we increment our page count and we make the API call again with the same function to handle the response again.
If not, this means we're at our final page. Now it's time to format the data using R.unnest and finally insert the finalArray of data into our database.
But a funny thing happens here. The entire execution halts at the Meteor.call() and I don't even get an error output to the server logs.
I even tried switching out the Meteor.call() for a simple: Sales.insert({text: 'testing'}) but the exact same behaviour is observed.
What I really need to do is to fetch the information and then store it into the database on the server. How can I make that happen?
EDIT: Please also see this other (much more simplified) SO question I made:
Calling a Meteor Method inside a Promise Callback [Halting w/o Error]

I ended up ditching the NPM package and writing my own API call. I could never figure out how to make my call inside the .then(). Here's the code:
fetchGumroadData: () => {
let sales = [];
const fetchData = (page = 1) => {
let options = {
data: {
access_token: Meteor.settings.gumroadAccessKey,
before: '2099-12-04',
after: '2014-12-04',
page: page,
}
};
HTTP.call('GET', 'https://api.gumroad.com/v2/sales', options, (err,res) => {
if (err) { // API call failed
console.log(err);
throw err;
} else { // API call successful
sales.push(...res.data.sales);
res.data.next_page_url ? fetchData(page + 1) : Meteor.call('addSalesFromAPI', sales);
}
});
};
fetchData(); // run the function to fetch data recursively
}

creating a drop down list in javascript from a text file

Is this possible? If so, I could really use some help on this. I'm very new to JavaScript and thus hardly know any of the syntactical specifications, nor proper methodology.
Some functions I wrote in an external file.
var base = base || {};
base.requestAjax = function () {
try{
return new XMLHttpRequest();
} catch (e) {
try {
return new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
return new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) {
alert("Something is wrong with your browser! Please ensure that you have javascript functionality enabled");
return false;
}
}
}
}
base.onReadyStateChange = function(ajaxRequest, formName, formArray, method, controller) {
var methodVerified = verifyMethod(method);
if (!methodVerified) {
throw new Exception("Error: please make sure the method passed matches that of \'GET\' or \'POST\'.");
}
for (input in formArray) {
document.formName.input.value = ajaxRequest.responseText;
}
ajaxRequest.open(method, controller, true);
file.send(null);
}
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
if (file.readyState != 4) {
throw new Exception("Error: text file ready state not equal to 4!");
}
if (file.status != 200) {
throw new Exception("Error: text file status not equal to 200!");
}
var lines = file.responseText.split("\n");
document.writeln("<select>");
for(line in lines) {
document.writeln("<option>" + line + "</option>");
}
document.writeln("</select>");
file.send(null);
}
base.verifyMethod = function(method) {
var methodLower = method.toString().toLowerCase();
switch(methodLower) {
case "get":
case "post":
return true;
default:
return false;
}
}
The html code which attempts to implement it
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" type="text/css" href="../includes/css/adsmanager.css" />
<script type="text/javascript" src="../includes/js/base.js">
<!--
function createCountriesList() {
var file = base.requestAjax;
var path = "../includes/_notes/countries.txt";
base.writeDropDownList(file, "GET", path);
}
//-->
</script>
</head>
<body>
<form name='adsManager'>
<button type="text" value="test" onclick="createCountriesList()" />
</form>
</body>
</html>
I plan to use the functions for other things, so I figured I'd create a namespace. I found this as a reference and based most of my model off of it.
Thanks to all who can help.

What is your point of failure? Is your Javascript sending the Ajax request and receiving the response?
Is lines getting data in this line of your code?
var lines = file.responseText.split("\n");
If you are getting this far, iterate through lines and add options like this:
var select = document.getElementById('id');
select.options.add(new Option(lines[i]));
In your writeDropDownList method, I made a few changes:
added a method that will be called when your Ajax call has completed. In that method, you should check the response string and add the options to your select box
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
file.onreadystatechange = requestComplete;
file.send(null);
}
requestComplete()
{
if (file.readyState == 4)
{
if(file.readyState == 200)
{
var lines = file.responseText.split("\n");
//foreach loop to populate select
}
}
}
In your code, you are checking and using files.responseText before you have even sent the Ajax request at files.send(null)
EDIT:
Some more comments regarding your code:
In the createCountriesList function, you create file and assign
it a value by calling requestAjax. You then pass it to
writeDropDownList function, where you assign it a value again by
calling requestAjax. You see that this is redundant? There is no
need to create file in createCountriesList and pass it as an
argument. Create it just once in writeDropDownList.
in writeDropDownList you call upload.requestAjax(). What is
upload. I don't see you initializing upload anywhere. Do you mean
to call base.requestAjax()?
you have a function base.OnReadyStateChange but at no point are you
telling your AJAX request to call that function when state changes. See the code I posted above. The function I added called
requestComplete does that, and I tell the AJAX request to call it using file.onreadystatechange = requestComplete;
You set method to GET, yet you are not passing any GET values in your URL
in file.open(method, path, true);, path is supposed to be the URL of the script the AJAX request will call. You have set path to ../includes/_notes/countries.txt. An AJAX request cannot call a .txt file since they do not execute.
I just had a look at the source of your code, and it is all sorts of broken. Please do not use it.
What is countries.txt ? Are you attempting to populate a dropdown with a list of all countries, or some countries depending on user's input?
If the former, there is no need for Javascript / AJAX. You need to add PHP code in your html to populate the select box.
If the latter, your AJAX request should be sending the user input as a GET variable.

Some comments:
Very nice code; it's readable and looks neat.
I'd use a different name than base as namespace - for my liking, the word is too "common" . Chances are that you're going to define a variable base somewhere and it will break.
To create HTML from JavaScript, first create a small example in pure HTML to see how it should look like. After that, create a script which produces the same HTML using document.createElement().
Look at frameworks like jQuery or Prototype because they make many boring tasks much more simple.