In my google sheet I have some data in it. Later I published my sheet as .csv format.
Pls find my sheet here, below are the Cell Values
C1 = tdsyltt = 'ఈరోజు ( బుధవారం ) క్విజ్ సిలబస్' ;
C2 = tdsyl = 'హబక్కూకు 1 & యాకోబు 2, 3' ;
C4 = document.getElementById("tdsyltt).innerHTML = tdsyltt ;
C5 = document.getElementById("tdsyl").innerHTML = tdsyl ;
And using my published URL I have developed a web app
code.gs
const doGet = _ => ContentService.createTextOutput(UrlFetchApp.fetch("https://docs.google.com/spreadsheets/d/e/2PACX-1vReY-tDEwKYjTiSjsfAN42qjFUwMv_OD3_64bFdGrgL-2p3otc13elLcCq3pkb5xqhTA-bW3QXobpqh/pub?gid=1861615717&single=true&range=c1:c5&output=csv").getContentText()).setMimeType(ContentService.MimeType.JAVASCRIPT);
In the web app output 1st line is ok,
but for 2nd, 3rd and 4th line there are Extra 2 Apostrophes coming at the starting and ending of the lines.
Here is My Web App
How to fix this ..?
Modification points:
In this case, how about directly retrieving the values from Spreadsheet using Spreadsheet service instead of UrlFetchApp.fetch("https://docs.google.com/spreadsheets/d/e/2PACX-1vReY-tDEwKYjTiSjsfAN42qjFUwMv_OD3_64bFdGrgL-2p3otc13elLcCq3pkb5xqhTA-bW3QXobpqh/pub?gid=1861615717&single=true&range=c1:c5&output=csv").getContentText()? I thought that the reason of your current issue might be due to exporting the Spreadsheet as CSV data. When the Spreadsheet service is used, the values can be retrieved.
In your previous question, I said "In this answer, your Spreadsheet is used. Of course, you can directly set the script in Web Apps.". Ref In your this question, I thought that this can be used.
In your sample Spreadsheet, document.getElementById("tdsyltt).innerHTML = tdsyltt ; is required to be document.getElementById("tdsyltt").innerHTML = tdsyltt ;. Please be careful about this.
When these points are reflected in your script, how about the following modification?
Modified script:
const doGet = _ => {
const spreadsheetId = "###"; // Please set your Spreadsheet ID.
const ss = SpreadsheetApp.openById(spreadsheetId);
const sheet = ss.getSheetByName("SYLLABUSC");
const html = sheet.getRange("C1:C6").getDisplayValues().filter(([c]) => c).join("\n");
// console.log(html); // When you directly run this function with the script editor, you can see the created value in the log.
return ContentService.createTextOutput(html).setMimeType(ContentService.MimeType.JAVASCRIPT);
}
From your Spreadsheet, in this sample script, I used "C1:C6" of "SYLLABUSC" sheet. So, please modify this for your actual situation.
Note:
When you modified the Google Apps Script of Web Apps, please modify the deployment as a new version. By this, the modified script is reflected in Web Apps. Please be careful about this.
You can see the detail of this in my report "Redeploying Web Apps without Changing URL of Web Apps for new IDE (Author: me)".
Try a way to solve this by changing the spreadsheet Publishing Format from csv to tsv then it will work.
Code.gs
const doGet = _ => ContentService.createTextOutput(UrlFetchApp.fetch("https://docs.google.com/spreadsheets/d/e/2PACX-1vReY-tDEwKYjTiSjsfAN42qjFUwMv_OD3_64bFdGrgL-2p3otc13elLcCq3pkb5xqhTA-bW3QXobpqh/pub?gid=1861615717&single=true&range=c1:c5&output=tsv").getContentText()).setMimeType(ContentService.MimeType.JAVASCRIPT);
I'm by no means what I would call a "developer" but dabble quite a bit. I'm working on some Apps Script code to query an API and push the results into SQL. I have most of the bits working but I've noticed that while I'm debugging in the Apps Script editor, when I step into the following line of code, the editor throws the "could not connect to server message at the top.
var response = UrlFetchApp.fetch(clientApiURL,options);
var resultSet = JSON.parse(response.getContentText()); <-- this is the line that is crashing the IDE
Anyone know how to better debug this? When I'm not debugging it, the code seems to behave and function properly. But with this API, not all objects are formatted the same way, so I like to use the debugger to inspect them. I can do that when the editor crashes.
Any help/insight would be super appreciated. I've also pasted below the value of response.getContentText()
{"result":{"lead":[{"id":"332","accountID":null,"ownerID":null,"companyName":"","title":null,"firstName":"RYAN","lastName":"CAVANAUGH","street":null,"city":null,"country":null,"state":null,"zipcode":null,"emailAddress":"email#here.com","website":null,"phoneNumber":null,"officePhoneNumber":null,"phoneNumberExtension":null,"mobilePhoneNumber":null,"faxNumber":null,"description":null,"campaignID":"789934082","trackingID":"202003_5e6fa18a87853a69eb306910","industry":null,"active":"1","isQualified":"1","isContact":"1","isCustomer":"1","status":"4","updateTimestamp":"2020-05-08 20:24:48","createTimestamp":"2020-05-03 20:23:29","leadScoreWeighted":"23","leadScore":"26","isUnsubscribed":"0","leadStatus":"customer","persona":"","product_5e554b933fb5b":""}]},"error":null,"id":"5222020","callCount":"215","queryLimit":"50000"}
This will reproduce the error:
function test(){
var obj={"result":{"lead":[{"id":"332","accountID":null,"ownerID":null,"companyName":"","title":null,"firstName":"RYAN","lastName":"CAVANAUGH","street":null,"city":null,"country":null,"state":null,"zipcode":null,"emailAddress":"email#here.com","website":null,"phoneNumber":null,"officePhoneNumber":null,"phoneNumberExtension":null,"mobilePhoneNumber":null,"faxNumber":null,"description":null,"campaignID":"789934082","trackingID":"202003_5e6fa18a87853a69eb306910","industry":null,"active":"1","isQualified":"1","isContact":"1","isCustomer":"1","status":"4","updateTimestamp":"2020-05-08 20:24:48","createTimestamp":"2020-05-03 20:23:29","leadScoreWeighted":"23","leadScore":"26","isUnsubscribed":"0","leadStatus":"customer","persona":"","product_5e554b933fb5b":""}]},"error":null,"id":"5222020","callCount":"215","queryLimit":"50000"}
var resultSet = JSON.parse(obj);
}
Taking a look at the problem:
function test(){
var obj={"result":{"lead":[{"id":"332","accountID":null,"ownerID":null,"companyName":"","title":null,"firstName":"RYAN","lastName":"CAVANAUGH","street":null,"city":null,"country":null,"state":null,"zipcode":null,"emailAddress":"email#here.com","website":null,"phoneNumber":null,"officePhoneNumber":null,"phoneNumberExtension":null,"mobilePhoneNumber":null,"faxNumber":null,"description":null,"campaignID":"789934082","trackingID":"202003_5e6fa18a87853a69eb306910","industry":null,"active":"1","isQualified":"1","isContact":"1","isCustomer":"1","status":"4","updateTimestamp":"2020-05-08 20:24:48","createTimestamp":"2020-05-03 20:23:29","leadScoreWeighted":"23","leadScore":"26","isUnsubscribed":"0","leadStatus":"customer","persona":"","product_5e554b933fb5b":""}]},"error":null,"id":"5222020","callCount":"215","queryLimit":"50000"}
var resultSet = JSON.parse(JSON.stringify(obj));
}
but so what the point of the parse is to return an object from a string not from object.
But I do see the problem. 'Cannot connect to the server'
I found that this does seem to work know:
function test(){
var obj='{"result":{"lead":[{"id":"332","accountID":"","ownerID":"","companyName":"","title":"","firstName":"RYAN","lastName":"CAVANAUGH","street":"","city":"","country":"","state":"","zipcode":"","emailAddress":"email#here.com","website":"","phoneNumber":"","officePhoneNumber":"","phoneNumberExtension":"","mobilePhoneNumber":"","faxNumber":"","description":"","campaignID":"789934082","trackingID":"202003_5e6fa18a87853a69eb306910","industry":"","active":"1","isQualified":"1","isContact":"1","isCustomer":"1","status":"4","updateTimestamp":"2020-05-08 20:24:48","createTimestamp":"2020-05-03 20:23:29","leadScoreWeighted":"23","leadScore":"26","isUnsubscribed":"0","leadStatus":"customer","persona":"","product_5e554b933fb5b":""}]},"error":"","id":"5222020","callCount":"215","queryLimit":"50000"}';
var resultSet = JSON.parse(obj);
var end="is near";//I just put this here to have a place to stop with debugger running
}
I replaced all of the null s with "".
I am trying to scrape data from an interactive map (looking to get crime data for a county). I am using R (rvest) and trying to use phantomjs too. I'm new to web scraping so I am not really understanding how all the elements work together (trying to get there).
The problem I believe I am having is that after I run the phantomjs and upload the html using R's rvest package, I end up with more scripts and no clear data in the html. My code is below.
writeLines("var url = 'http://www.google.com';
var page = new WebPage();
var fs = require('fs');
page.open(url, function (status) {
just_wait();
});
function just_wait() {
setTimeout(function() {
fs.write('cool.html', page.content, 'w');
phantom.exit();
}, 2500);
}
", con = "scrape.js")
A function that takes in the url that I want to scrape
s_scrape <- function(url = "https://gis.adacounty.id.gov/apps/crimemapper/",
js_path = "scrape.js",
phantompath = "/Users/alihoop/Documents/phantomjs/bin/phantomjs"){
# this section will replace the url in scrape.js to whatever you want
lines <- readLines(js_path)
lines[1] <- paste0("var url ='", url ,"';")
writeLines(lines, js_path)
command = paste(phantompath, js_path, sep = " ")
system(command)
}
Execute the js_scrape() function and get a html file saved as "cool.html"
js_scrape()
Where I am not understanding what to do next is the below R code:
map_data <- read_html('cool.html') %>%
html_nodes('script')
The output I get in the HTML via phantomjs is just scripts again. Looking for help on how to proceed when faced (in my mind) is javascript nested in javascript scripts(?)
Thank you!
This site uses javascript to make queries to the server. One solution is to reproduce the rest request and read the returning JSON file directly. This avoids the need to use Phantomjs.
From the developer tools options from your browser and looking through the xhr files, you will find a file(s) named "query" with a link similar to: "https://gisapi.adacounty.id.gov/arcgis/rest/services/CrimeMapper/CrimeMapperWAB/FeatureServer/11/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=1000"
Read this JSON response directly and convert to a list with the use of the jsonlite package:
library(jsonlite)
output<-jsonlite::fromJSON("https://gisapi.adacounty.id.gov/arcgis/rest/services/CrimeMapper/CrimeMapperWAB/FeatureServer/11/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=1000")
output$features
Find the first number in the link, (11 in this case) "FeatureServer/11/query?f=json". This number will determine which crime to query the server with. I found, it can take a value from 0 to 11. Enter 0 for arson, 4 for drugs, 11 for vandalism, etc.
I am experimenting with iMacros to automate as task that Firefox will do. I simply want to save the current page with the MAFF extension. The JavaScript that the iMacros forum has lead me to, is this:
// I stuck these variable in just to try something.
var doc = "http://www.traderjoes.com";
var file = "C:\\Export\\Test.maff";
var format = "MAFF";
// I stuck these variable in just to try something.
var MafObjects = {};
Components.utils.import("resource://maf/modules/mafObjects.jsm",
MafObjects);
var jobListener = {
onJobComplete: function(aJob, aResult) {
if (!Components.isSuccessCode(aResult)) {
// An error occurred
} else {
// The save operation completed successfully
}
},
onJobProgressChange: function(aJob, aWebProgress, aRequest,
aCurSelfProgress,
aMaxSelfProgress,
aCurTotalProgress,
aMaxTotalProgress) { },
onStatusChange: function(aWebProgress, aRequest, aStatus,
aMessage) { }
};
var saveJob = new MafObjects.SaveJob(jobListener);
saveJob.addJobFromDocument(doc, file, format);
saveJob.start();
I was only getting an error on line 26 because this was sample code. With the little JavaScript I know I tried to add some variables on the lines before the code starts. The thing is that when I try to search for syntax example for the method .addJobFromDocument I don’t find much, just like two results. Is this a method of JavaScript? Usually with things from the DOM you will get a great deal of information on them.
Does anybody know a way of automating the save of MAFF of the current open tab in Firefox and then closing the browser? iMacros was something I came to and glad to see it features but really I just want to automate from a command line the saving of a URL as a MAFF archive The doc (that I got from iMacros forum) also had these code snippets but I don’t have much idea how to use them. Thanks
var fileUri = Components.
classes["#mozilla.org/network/io-service;1"].
getService(Components.interfaces.nsIIOService).
newFileURI(file);
var persistObject = new MafObjects.MafArchivePersist(null, format);
persistObject.saveDocument(doc, fileUri);
Also:
var doc = gBrowser.contentDocument;
var file = Components.classes["#mozilla.org/file/local;1"].
createInstance(Components.interfaces.nsILocalFile);
file.initWithPath("C:\\My Documents\\Test.maff");
var format = "TypeMAFF";
I want to use Google Feed API from a server (Node.js). I have already installed the googleapis module. My code is:
// parts omitted
var googleapis = require('googleapis');
// parts omitted
googleapis.discover('feeds').execute(function(err, client) {
var feed = new google.feeds.Feed('http://rss.lemonde.fr/c/205/f/3050/index.rss');
});
// parts omitted
But Node.js console tells me that "google is not defined". Any idea of the problem and solution?
to access Google Feed API using Node.js, you should try the google-feed-api module as explained here:
https://www.npmjs.org/package/google-feed-api
Hope it helps!
Edit:
I tried this with your URL and worked fine:
var gfeed = require('google-feed-api');
var feed = new gfeed.Feed('http://rss.lemonde.fr/c/205/f/3050/index.rss');
feed.listItems(function(items){
console.log(items);
});
It's because google is literally not defined. I don't know very much about that module, but I think that instead of using the google var you should use client , because that's what the execute function returns.
So the code would be:
// parts omitted
var googleapis = require('googleapis');
// parts omitted
googleapis.discover('feeds').execute(function(err, client) {
var feed = new client.feeds.Feed('http://rss.lemonde.fr/c/205/f/3050/index.rss');
});
// parts omitted