We Keep Coding

How to write (SetValues) to a Google Sheet using a filtered forEach loop?

I've been trying for hours to make the following Google Apps Script work. What it needs to do, is send emails (from an html-template) to anyone that:
has a complete Event Schedule (which is completed if they have been
assigned to at least 4 events, which is counted in column Q);
has NOT been sent an email earlier (which is kept track of in column
R);
The script keeps track of errors in column S, i.e. if there's no email address provided.
It appears it only works:
if I comment out
data = data.filter(function(r){ return r[17] == true & r[16] > 3});
or if I comment out
ws.getRange("S3:S" + ws.getLastRow()).setValues(errors);
ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces);
How can I get this script to work properly?
A copy of the Google Sheet I'm referring to is this one:
https://docs.google.com/spreadsheets/d/1sbOlvLVVfiQMWxNZmtCLuizci2cQB9Kfd8tYz64gjP0/edit?usp=sharing
This is my code so far:
function SendEmail(){
var voornaam = 3;
var achternaam = 4;
var email = 5;
var event1 = 9;
var event2 = 10;
var event3 = 11;
var event4 = 12;
var event5 = 13;
var event6 = 14;
var event7 = 15;
var emailTemp = HtmlService.createTemplateFromFile("email");
var ws = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Events Day 1");
var datum = ws.getRange(1,3).getValue();
var spreker = ws.getRange(1,6).getValue();
var data = ws.getRange("A3:R" + ws.getLastRow()).getValues();
data = data.filter(function(r){ return r[17] == false && r[16] > 3}); //Either this needs to be commented out...
let errors = [];
let mailSucces = [];
data.forEach(function(row){
try{
emailTemp.voornaam = row[voornaam];
emailTemp.email = row[email];
emailTemp.datum = datum;
emailTemp.spreker = spreker;
emailTemp.event1 = row[event1];
emailTemp.event2 = row[event2];
emailTemp.event3 = row[event3];
emailTemp.event4 = row[event4];
emailTemp.event5 = row[event5];
emailTemp.event6 = row[event6];
emailTemp.event7 = row[event7];
var htmlMessage = emailTemp.evaluate().getContent();
GmailApp.sendEmail(
row[email],
"Here you go! Your personal schedule for the event of " + datum,
"Your emailprogramm doesn't support html.",
{
name: "Event Organisation Team", htmlBody: htmlMessage, replyTo: "info#fakeemail.com"
});
errors.push([""]);
mailSucces.push(["TRUE"]);
}
catch(err){
errors.push(["Error: no message sent."]);
mailSucces.push(["False"]);
}
}); //close forEach
ws.getRange("S3:S" + ws.getLastRow()).setValues(errors); //or this and the next line need to be commented out.
ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces);
}
Edit I have been trying and thinking en trying... but still haven't found out how to make it work. But I also got understanding of why it's not working; I just don't know how to get it fixed.
Let me elaborate on the problem a bit more:
The problem is, that within the forEach loop the range is a filtered variant of the data, pulled from the spreadsheet with getValues. Therefore, writing data back with ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces); results in mismatched checkmarks in te spreadsheet.
So, somehow I need to put the range of the previous used filter data = data.filter(function(r){ return r[17] == false & r[16] > 3}); in a variable...? I guess?
Furthermore, I don't think it's wise to use setValue within the loop, because (from what I understand from my searching on the topic) this results in a slow script, because every loop the script makes an API call to write in the spreadsheet. Hence the errors.push and mailSucces.push, and my attempt to do a setValue at the end, after the loop is finished.
Can someone help me to finish this problem?

The problem is different size of the range you write to and data you are writing in.
Try replacing:
ws.getRange("S3:S" + ws.getLastRow()).setValues(errors);
ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces);
With:
ws.getRange(3, 19, errors.length, 1).setValues(errors);
ws.getRange(3, 18, mailSucces.length, 1).setValues(mailSucces);
You should use this variation of getRange
https://developers.google.com/apps-script/reference/spreadsheet/sheet#getrangerow,-column,-numrows,-numcolumns
Your data has non-fixed number of rows and fixed number of columns (1). In general case your data will be matrix of X rows and Y columns. For that purpose you can make it completely dynamic:
sheet.getRange(startRow, startColumn, data.length, data[0].length)
Just make sure data.length is > 0 before you do this, otherwise data[0].length will break.
Edit:
I started writing a comment but it got too long. There are couple of things that may go wrong with sending emails. First thing I noticed is that you use & in filter, but in AppsScript/JavaScript/C-like-languages, you should use && for logical AND. Now the email: you only detect the code break with the catch block. At this point you don't know why the code breaks it could be anything. With GmailApp I recommend you to use createDraft while developing, then when all ok replace it with sendEmail for the final version, both functions have the exact same parameters, thank you Google devs ;-).
To find out the exact problem you should get the error message on break and display it. err.stack should tell you pretty much everything:
catch(err){
Logger.log(err.stack); // Added
errors.push(["Error: no message sent."]);
mailSucces.push(["False"]);
}
Run the sendEmail function from the code editor and you should see the Log for each catch(err) pass.

Adding Formula from JavaScript to CSV to be imported to Google Sheet results in multiple columns

I have been trying to add a formula shown below to a single column in Google sheets using JavaScript, but it is resulting in different columns, due to the comma.
=IF(A2="VALID", B2, 0)
Currently, the code I am using to generate the CSV is below:
var CSVRowDemo = [];
var CSVDemo = 'data:text/csv;charset=utf-8,';
CSVRowDemo.push(['VALID/INVALID', 'Value', 'Check'])
let index = 2;
let check = '"' + '=IF(A'+index+'=\"VALID\"", B'+index+', 0)'.replace(/"/g, '""') + '"';
CSVRowDemo.push(['VALID', '100', check])
CSVRowDemo.forEach(function(rowArray) {
let row = rowArray.join(',');
CSVDemo += row + '\r\n';
});
encodedUri = encodeURI(CSVDemo);
link = document.createElement("a");
link.setAttribute("href", encodedUri);
link.setAttribute("download", "Basic.csv");
document.body.appendChild(link);
link.click();
Expected to see it like this. (This is how it is rendered in LibreOffice Calc.)
Rather saw it like this in Google Sheets.
The closest, which comes to answer this is from this answer.
I am trying to use only Javascript instead of NodeJS or other libraries.
Any help is appreciated.

Although I'm not sure about the method for putting the created CSV data to Google Spreadsheet, can you test the the following modification?
From:
let check = '"' + '=IF(A'+index+'=\"VALID\"", B'+index+', 0)'.replace(/"/g, '""') + '"';
To:
let check = `"=IF(A${index}=""VALID"", B${index}, 0)"`;
or
let check = '"=IF(A'+index+'=""VALID"", B'+index+', 0)"';
In my environment, I could confirm that the CSV data with the following modification can be imported to the Google Spreadsheet.

Google Sheets conditional combine, clear all cells in range, Loop. VBA to JS

I'm translating VBA into a Google Sheets script, but am a loss on how to get past this loop through all rows in a single column hurdle.
I have a column which contains an #OrdernNumber, a -State code, a First name, and sometimes a Last name, followed by an empty cell. I need to combine the First and Last names into the First name cell, and then clear the Last name cell below. This is illustrated in the screenshot below:
Column C and D in the screenshot reveal the pattern I used to construct my original VBA code:
Sub WorkingCombineAndClearLoop()
Dim Rngcell As Range
For Each Rngcell In Range("B1:B100")
'if first character is #
If left(Rngcell.Value, 1) <> "#" _
'and if first character is -
And left(Rngcell.Value, 1) <> "-" _
'and if cell is not blank then
And Rngcell.Value <> "" Then
'combine left cell
Rngcell.Value = Rngcell.Offset(0, -1).Value _
'with bottom left cell
& " " & Rngcell.Offset(1, -1).Value
'then clear below cell
Rngcell.Offset(1, 0).ClearContents
End If
Next
End Sub
Following is my annotated JavaScript code. Because I can't get the loop working yet, I've had to change the code logic a bit:
function workingCombineAndClearNoLoop() {
//get active spreadsheet, sheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
//determine row count
var numRows = SpreadsheetApp.getActiveSheet().getRange("A:A").getLastRow();
//if A1's first character is not # AND
if (s.getRange('A1').getValue().substring(0,1) === "#" &&
//if A2's first character is not ' AND
s.getRange('A1').offset(1, 0).getValue().substring(0,1) === "-" ) {
//Set the value of A2 to...
s.getRange('A1').offset(2, 1).setValue(
//A2 cell content + " " + A2 cell content (concatenate)
s.getRange('A1').offset(2, 1).getValue() + " " +
s.getRange('A1').offset(3, 1).getValue())
//and then clear A2
s.getRange('A1').offset(3, 1).clearContent()
};
}
The following is where everything falls apart - the loop. I've only left the bare bones in this final copy of the code:
function notWorkingCombineAndClearWithLoop() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
var numRows = SpreadsheetApp.getActiveSheet().getRange("A:A").getLastRow();
//loop
//how to replace 'A1' with A[i]?, so that it loops through all cells in the column?
for (var i = 0; i < 25; i++) {
if (s.getRange('A1').getValue().substring(0,1) === "#" &&
s.getRange('A1').offset(1, 0).getValue().substring(0,1) === "-" ) {
s.getRange('A1').offset(2, 1).setValue(
s.getRange('A1').offset(2, 1).getValue() + " " +
s.getRange('A1').offset(3, 1).getValue())
s.getRange('A1').offset(3, 1).clearContent()
}
};
}
Note: I've completed Codacademy's JS course and several Google Sheets tutorials, and am pretty good at VBA, but I'm still swimming in circles with this. I've tried to follow many loop examples, using variables in place of A1. Because I can't find any working solution, I've left the non-functioning A1 as a simple placeholder.

Short answer
//how to replace 'A1' with A[i]?, so that it loops through all cells
//in the column?
Use
'A' + i
Explanation
getRange() has several forms, one of them use A1 notation reference. This is used in concordance with the Google Apps Script included in the question.
Example
The main intention of below code is to show how to use the A1 notation to loop through the cells of a column in Google Apps Script.
There was some errors in the indexes. They were corrected.
Also changed the fourth line to reduce the rows to loop through and added a couple of lines.
function editedNotWorkingCombineAndClearWithLoop() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
var numRows = SpreadsheetApp.getActiveSheet().getLastRow();
/*
* use variables to reduce number of calls to Apps Script services
* and improve readability
*/
var range, substring1, substring2;
//loop
for (var i = 1; i <= numRows; i++) {
//Here is the magic
range = s.getRange('A' + i );
substring1 = range.getValue().substring(0,1);
substring2 = range.offset(1, 0).getValue().substring(0,1);
if ( substring1 === '#' && substring2 === '-' ) {
range.offset(0, 1).setValue(range.getValue());
range.offset(1, 1).setValue(range.offset(1, 0).getValue());
range.offset(2, 1).setValue(
range.offset(2, 0).getValue()
+ " "
+ range.offset(3, 0).getValue()
)
}
}
}

Google Spreadsheet App Script will not wait for results to load

I have a small script and what I'm trying to do is to write one value from 'Sheet 1' to 'Sheet 2'. Wait for the results to load and compare the cells to see if it is above 10% or not. I have some =importhtml functions in the spreadsheet and it takes along time to load. I've tried sleep, utilities sleep, and flush. None have been working, maybe because I might be putting it in the wrong area..
function compareCells() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var listSheet = ss.getSheetByName('Stocks');
var dataSheet = ss.getSheetByName('Summary');
var listSheetLastRow = listSheet.getLastRow();
var currRow = 1;
for (i = 1; i <= listSheetLastRow; i++) {
if (listSheet.getRange(1, 3).getValue() == 'asd') {
var ticker = listSheet.getRange(currRow, 1).getValue();
dataSheet.getRange(5, 4).setValue(ticker);
var value1 = dataSheet.getRange(15, 4).getValue();
var value2 = dataSheet.getRange(22, 4).getValue();
SpreadsheetApp.flush();
if (value1 > 0.10 && value2 > 0.10) {
listSheet.getRange(currRow, 8).setValue('True');
listSheet.getRange(currRow, 9).setValue(value1);
listSheet.getRange(currRow, 10).setValue(value2);
} else {
listSheet.getRange(currRow, 8).setValue('False');
}
} else {
Browser.msgBox('Script aborted');
return null;
}
currRow++;
}
}

If it is not important that you use the =IMPORTHTML() function in your sheet, the easiest way to do this will be to use UrlFetchApp within Apps Script. Getting the data this way will cause your script to block until the HTML response is returned. You can also create a time-based trigger so your data is always fresh, and the user will not have to wait for the URL fetch when looking at your sheet.
Once you get the HTML response, you can do all of the same processing you'd do in Sheet1 within your script. If that won't work because you have complex processing in Sheet1, you can:
use UrlFetchpApp.fetch('http://sample.com/data.html') to retrieve your data
write the data to Sheet1
call SpreadsheetApp.flush() to force the write and whatever subsequent processing
proceed as per your example above
By handling these steps sequentially in your script you guarantee that your later steps don't happen before the data is present.

I had a similar problem but came up with a solution which uses a while loop which forces the script to wait until at least 1 extra column or 1 extra row has been added. So for this to work the formula needs to add data to at least one extra cell other than the one containing the formula, and it needs to extend the sheet's data range (number of rows or columns), for example by adding the formula to the end of the sheet, which looks like what you are doing. Every 0.5 seconds for 10 seconds it checks if extra cells have been added.
dataSheet.getRange(5, 4).setValue(ticker);
var wait = 0;
var timebetween = 500;
var timeout = 10000;
var lastRow = dataSheet.getLastRow();
var lastColumn = dataSheet.getLastColumn();
while (dataSheet.getLastColumn() <= lastColumn && dataSheet.getLastRow() <= lastRow){
Utilities.sleep(timebetween);
wait += timebetween;
if (wait >= timeout){
Logger.log('ERROR: Source data for ' + ticker + ' still empty after ' + timeout.toString() + ' seconds.');
throw new Error('Source data for ' + ticker + ' still empty after ' + timeout.toString() + ' seconds.');
}
}

In case if you are getting these two values (
var value1 = dataSheet.getRange(15, 4).getValue();
var value2 = dataSheet.getRange(22, 4).getValue();
) after the =importhtml call, you have to add sleep function before these two lines of code.
You also can have a loop until you get some values into the range from =importhtml call and add some sleep in the loop. Also note that as of April 2014 the limitation of script runtime is 6 minutes.
I also found this link which might be helpful.
Hope that helps!

Updating a ScriptProperty to avoid retrieving duplicate Twitter statuses

I was interested in writing a twitter bot to help out some friends at a local ski resort. I found this tutorial from Amit Agarwal which gave me enough to get started (it did take me more than 5 minutes since I did a lot of modifying). I host the script on google docs.
FIRST I think this is javascript (my understanding is that google apps script uses javascript...) and when I have had problems with the code so far, google searches for javascript-such-and-such have been helpful, but if this is not actually javascript, please let me know so I can update the tag accordingly!
I have no prior experience with javascript, so I am pretty happy that it's actually working. But I want to see if I'm doing this right.
The start function initiates the trigger, which kicks off the fetchTweets() function every interval (30 minutes). In order to avoid duplicates (the first errors I encountered) & potentially being flagged as spam, I needed a way to ensure that I was not posting the same tweets over and over again. Within the start() function, the initial since_id value is assigned:
ScriptProperties.setProperty("SINCE_TWITTER_ID", "404251049889759234");
Within the fetchTweet() function, I think I am updating this property with the statement:
ScriptProperties.setProperty("SINCE_TWITTER_ID", lastID + '\n');
Is this a good way to do this? Or is there a better/more reliable way? And if so, how can I be sure it's updating the property? (I can check the log file and it seems to be doing it, so I probably just need to create a permanent text file for the logger).
Any help is greatly appreciated!!
/** A S I M P L E T W I T T E R B O T **/
/** ======================================= **/
/** Written by Amit Agarwal #labnol on 03/08/2013 **/
/** Modified by David Zemens #agnarchy on 11/21/2013 **/
/** Tutorial link: http://www.labnol.org/?p=27902 **/
/** Live demo at http://twitter.com/DearAssistant **/
/** Last updated on 09/07/2013 - Twitter API Fix **/
function start() {
Logger.log("start!" + '\n')
// REPLACE THESE DUMMY VALUES
// https://script.google.com/macros/d/18DGYaa-jbaAK9rEv0HZ2cMcWjFGgkvVcvr6TfksMNbbu2Brk3gZeZ46R/edit
var TWITTER_CONSUMER_KEY = "___REDACTED___";
var TWITTER_CONSUMER_SECRET = "___REDACTED___";
var TWITTER_HANDLE = "___REDACTED___";
var SEARCH_QUERY = "___REDACTED___" + TWITTER_HANDLE;
// Store variables
ScriptProperties.setProperty("TWITTER_CONSUMER_KEY", TWITTER_CONSUMER_KEY);
ScriptProperties.setProperty("TWITTER_CONSUMER_SECRET", TWITTER_CONSUMER_SECRET);
ScriptProperties.setProperty("TWITTER_HANDLE", TWITTER_HANDLE);
ScriptProperties.setProperty("SEARCH_QUERY", SEARCH_QUERY);
ScriptProperties.setProperty("SINCE_TWITTER_ID", "404251049889759234");
// Delete exiting triggers, if any
var triggers = ScriptApp.getScriptTriggers();
for(var i=0; i < triggers.length; i++) {
ScriptApp.deleteTrigger(triggers[i]);
}
// Setup trigger to read Tweets every 2 hours
ScriptApp.newTrigger("fetchTweets")
.timeBased()
.everyMinutes(30)
//.everyHours(2)
.create();
}
function oAuth() {
//Authentication
var oauthConfig = UrlFetchApp.addOAuthService("twitter");
oauthConfig.setAccessTokenUrl("https://api.twitter.com/oauth/access_token");
oauthConfig.setRequestTokenUrl("https://api.twitter.com/oauth/request_token");
oauthConfig.setAuthorizationUrl("https://api.twitter.com/oauth/authorize");
oauthConfig.setConsumerKey(ScriptProperties.getProperty("TWITTER_CONSUMER_KEY"));
oauthConfig.setConsumerSecret(ScriptProperties.getProperty("TWITTER_CONSUMER_SECRET"));
}
function fetchTweets() {
oAuth();
// I put this line in to monitor whether the property is getting "stored" so as to avoid
// reading in duplicate tweets.
Logger.log("Getting tweets since " + ScriptProperties.getProperty("SINCE_TWITTER_ID"))
var twitter_handle = ScriptProperties.getProperty("TWITTER_HANDLE");
var search_query = ScriptProperties.getProperty("SEARCH_QUERY")
Logger.log("searching tweets to " + search_query + '\n');
// form the base URL
// restrict to a certain radius ---:
//var search = "https://api.twitter.com/1.1/search/tweets.json?count=5&geocode=42.827934,-83.564306,75mi&include_entities=false&result_type=recent&q=";
// unrestricted radius:
var search = "https://api.twitter.com/1.1/search/tweets.json?count=5&include_entities=false&result_type=recent&q=";
search = search + encodeString(search_query) + "&since_id=" + ScriptProperties.getProperty("SINCE_TWITTER_ID");
var options =
{
"method": "get",
"oAuthServiceName":"twitter",
"oAuthUseToken":"always"
};
try {
var result = UrlFetchApp.fetch(search, options);
var lastID = ScriptProperties.getProperty("SINCE_TWITTER_ID");
if (result.getResponseCode() === 200) {
var data = Utilities.jsonParse(result.getContentText());
if (data) {
var tweets = data.statuses;
//Logger.log(data.statuses);
for (var i=tweets.length-1; i>=0; i--) {
// Make sure this is a NEW tweet
if (tweets[i].id > ScriptProperties.getProperty("SINCE_TWITTER_ID")) {
lastID = (tweets[i].id_str);
var answer = tweets[i].text.replace(new RegExp("\#" + twitter_handle, "ig"), "").replace(twitter_handle, "");
// I find this TRY block may be necessary since a failure to send one of the tweets
// may abort the rest of the loop.
try {
Logger.log("found >> " + tweets[i].text)
Logger.log("converted >> " + answer + '\n');
sendTweet(tweets[i].user.screen_name, tweets[i].id_str, answer.substring(0,140));
// Update the script property to avoid duplicates.
ScriptProperties.setProperty("SINCE_TWITTER_ID", lastID);
Logger.log("sent to #" + tweets[i].user.screen_name + '\n');
} catch (e) {
Logger.log(e.toString() + '\n');
}
}
}
}
}
} catch (e) {
Logger.log(e.toString() + '\n');
}
Logger.log("Last used tweet.id: " + lastID + + "\n")
}
function sendTweet(user, reply_id, tweet) {
var options =
{
"method": "POST",
"oAuthServiceName":"twitter",
"oAuthUseToken":"always"
};
var status = "https://api.twitter.com/1.1/statuses/update.json";
status = status + "?status=" + encodeString("RT #" + user + " " + tweet + " - Thanks\!");
status = status + "&in_reply_to_status_id=" + reply_id;
try {
var result = UrlFetchApp.fetch(status, options);
Logger.log("JSON result = " + result.getContentText() + '\n');
}
catch (e) {
Logger.log(e.toString() + '\n');
}
}
// Thank you +Martin Hawksey - you are awesome
function encodeString (q) {
// Update: 09/06/2013
// Google Apps Script is having issues storing oAuth tokens with the Twitter API 1.1 due to some encoding issues.
// Henc this workaround to remove all the problematic characters from the status message.
var str = q.replace(/\(/g,'{').replace(/\)/g,'}').replace(/\[/g,'{').replace(/\]/g,'}').replace(/\!/g, '|').replace(/\*/g, 'x').replace(/\'/g, '');
return encodeURIComponent(str);
// var str = encodeURIComponent(q);
// str = str.replace(/!/g,'%21');
// str = str.replace(/\*/g,'%2A');
// str = str.replace(/\(/g,'%28');
// str = str.replace(/\)/g,'%29');
// str = str.replace(/'/g,'%27');
// return str;
}

When you use ScriptProperties.setProperty("KEY", "VALUE");, internally Script Properties will overwrite a duplicate key (i.e., if an old Property has the same key, your new one will replace it). So in your case, since you are using the same identifier for the key (SINCE_TWITTER_ID), it will replace any previous Script Property that is that key.
Furthermore, you can view Script Properties via File -> Project properties -> Project properties (tab). Imo Google didn't name that very well. User properties as specific to Google users. Script properties as specific to the Script Project you are working under.
Also, it probably isn't a good idea to include \n in your value when you set the property. That will lead to all sorts of bugs down the road, because you'll have to compare with something like the following:
var valToCompare = "My value\n";
instead of:
var valToCompare = "My value";
because the value in SINCE_TWITTER_ID will actually be "some value\n" after you call your fetchTweet() function.
Of course, one seems more logical I think, unless you really need the line breaks (in which case you should be using them somewhere else, for this application).

Its ok like that thou I dont know why you are adding \n at fhe end. Might confuse other code. You can see script properties in the script's file menu+ properties