I am having trouble to understand why this takes so long to execute and whether there is a solution to this. It requires 20 seconds to execute the .setValues() line. The grid is about 2000 (row) by 100 (col) cells.
Here is a link to the test speed sheet for testing material, that is like my data-set.
function test() {
var active = SpreadsheetApp.getActiveSpreadsheet();
var logs_sheet = active.getSheetByName("Logs");
logs_sheet.appendRow([new Date(), "", "", "STARTING EXECUTION"]);
var test_sheet = active.getSheetByName("Test");
var test_data = test_sheet.getRange(1, 1, test_sheet.getLastRow(), test_sheet.getLastColumn()).getValues();
var combine = test_data.slice();
combine[1].splice(3, 1, new Date());
combine.splice(2017, 1);
combine.splice(0, 0, combine[1]);
logs_sheet.appendRow([new Date(), "", "", "finished combine"]);
test_sheet.getRange(1, 1, combine.length, combine[0].length).setValues(combine);
logs_sheet.appendRow([new Date(), "", "", "FINISHED EXECUTION"]);
}
What's more is that setValues() appears to update the sheet quite fast, but then it continues running something invisible before moving on.
Here is a possible solution / workaround I came up with.
Writing the array to a Google Doc for temporary storage only takes 0.06 seconds, rather than 20 seconds storing it in Sheets!
I assume that this can be quite a large array then, but I have not tested the limits of this workaround. I did however try to open the Google Doc and that is not really possible. But there is no need to open it.
var doc_database = DocumentApp.openById(" some doc ID here ").getBody();
var old_array = JSON.parse(doc_database.getText());
doc_database.setText('');
// run old_array calcs and updates here
doc_database.setText(JSON.stringify( ... new array here .... ));
hope this hack helps others.
Best
PAV
Related
I've been trying for hours to make the following Google Apps Script work. What it needs to do, is send emails (from an html-template) to anyone that:
has a complete Event Schedule (which is completed if they have been
assigned to at least 4 events, which is counted in column Q);
has NOT been sent an email earlier (which is kept track of in column
R);
The script keeps track of errors in column S, i.e. if there's no email address provided.
It appears it only works:
if I comment out
data = data.filter(function(r){ return r[17] == true & r[16] > 3});
or if I comment out
ws.getRange("S3:S" + ws.getLastRow()).setValues(errors);
ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces);
How can I get this script to work properly?
A copy of the Google Sheet I'm referring to is this one:
https://docs.google.com/spreadsheets/d/1sbOlvLVVfiQMWxNZmtCLuizci2cQB9Kfd8tYz64gjP0/edit?usp=sharing
This is my code so far:
function SendEmail(){
var voornaam = 3;
var achternaam = 4;
var email = 5;
var event1 = 9;
var event2 = 10;
var event3 = 11;
var event4 = 12;
var event5 = 13;
var event6 = 14;
var event7 = 15;
var emailTemp = HtmlService.createTemplateFromFile("email");
var ws = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Events Day 1");
var datum = ws.getRange(1,3).getValue();
var spreker = ws.getRange(1,6).getValue();
var data = ws.getRange("A3:R" + ws.getLastRow()).getValues();
data = data.filter(function(r){ return r[17] == false && r[16] > 3}); //Either this needs to be commented out...
let errors = [];
let mailSucces = [];
data.forEach(function(row){
try{
emailTemp.voornaam = row[voornaam];
emailTemp.email = row[email];
emailTemp.datum = datum;
emailTemp.spreker = spreker;
emailTemp.event1 = row[event1];
emailTemp.event2 = row[event2];
emailTemp.event3 = row[event3];
emailTemp.event4 = row[event4];
emailTemp.event5 = row[event5];
emailTemp.event6 = row[event6];
emailTemp.event7 = row[event7];
var htmlMessage = emailTemp.evaluate().getContent();
GmailApp.sendEmail(
row[email],
"Here you go! Your personal schedule for the event of " + datum,
"Your emailprogramm doesn't support html.",
{
name: "Event Organisation Team", htmlBody: htmlMessage, replyTo: "info#fakeemail.com"
});
errors.push([""]);
mailSucces.push(["TRUE"]);
}
catch(err){
errors.push(["Error: no message sent."]);
mailSucces.push(["False"]);
}
}); //close forEach
ws.getRange("S3:S" + ws.getLastRow()).setValues(errors); //or this and the next line need to be commented out.
ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces);
}
Edit I have been trying and thinking en trying... but still haven't found out how to make it work. But I also got understanding of why it's not working; I just don't know how to get it fixed.
Let me elaborate on the problem a bit more:
The problem is, that within the forEach loop the range is a filtered variant of the data, pulled from the spreadsheet with getValues. Therefore, writing data back with ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces); results in mismatched checkmarks in te spreadsheet.
So, somehow I need to put the range of the previous used filter data = data.filter(function(r){ return r[17] == false & r[16] > 3}); in a variable...? I guess?
Furthermore, I don't think it's wise to use setValue within the loop, because (from what I understand from my searching on the topic) this results in a slow script, because every loop the script makes an API call to write in the spreadsheet. Hence the errors.push and mailSucces.push, and my attempt to do a setValue at the end, after the loop is finished.
Can someone help me to finish this problem?
The problem is different size of the range you write to and data you are writing in.
Try replacing:
ws.getRange("S3:S" + ws.getLastRow()).setValues(errors);
ws.getRange("R3:R" + ws.getLastRow()).setValues(mailSucces);
With:
ws.getRange(3, 19, errors.length, 1).setValues(errors);
ws.getRange(3, 18, mailSucces.length, 1).setValues(mailSucces);
You should use this variation of getRange
https://developers.google.com/apps-script/reference/spreadsheet/sheet#getrangerow,-column,-numrows,-numcolumns
Your data has non-fixed number of rows and fixed number of columns (1). In general case your data will be matrix of X rows and Y columns. For that purpose you can make it completely dynamic:
sheet.getRange(startRow, startColumn, data.length, data[0].length)
Just make sure data.length is > 0 before you do this, otherwise data[0].length will break.
Edit:
I started writing a comment but it got too long. There are couple of things that may go wrong with sending emails. First thing I noticed is that you use & in filter, but in AppsScript/JavaScript/C-like-languages, you should use && for logical AND. Now the email: you only detect the code break with the catch block. At this point you don't know why the code breaks it could be anything. With GmailApp I recommend you to use createDraft while developing, then when all ok replace it with sendEmail for the final version, both functions have the exact same parameters, thank you Google devs ;-).
To find out the exact problem you should get the error message on break and display it. err.stack should tell you pretty much everything:
catch(err){
Logger.log(err.stack); // Added
errors.push(["Error: no message sent."]);
mailSucces.push(["False"]);
}
Run the sendEmail function from the code editor and you should see the Log for each catch(err) pass.
I've just written my first google apps scripts, ported from VBA, which formats a column of customer order information (thanks to you all of your direction).
Description:
The code identifies state codes by their - prefix, then combines the following first name with a last name (if it exists). It then writes "Order complete" where the last name would have been. Finally, it inserts a necessary blank cell if there is no gap between the orders (see image below).
Problem:
The issue is processing time. It cannot handle longer columns of data. I am warned that
Method Range.getValue is heavily used by the script.
Existing Optimizations:
Per the responses to this question, I've tried to keep as many variables outside the loop as possible, and also improved my if statements. #MuhammadGelbana suggests calling the Range.getValue method just once and moving around with its value...but I don't understand how this would/could work.
Code:
function format() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
var lastRow = s.getRange("A:A").getLastRow();
var row, range1, cellValue, dash, offset1, offset2, offset3;
//loop through all cells in column A
for (row = 0; row < lastRow; row++) {
range1 = s.getRange(row + 1, 1);
//if cell substring is number, skip it
//because substring cannot process numbers
cellValue = range1.getValue();
if (typeof cellValue === 'number') {continue;};
dash = cellValue.substring(0, 1);
offset1 = range1.offset(1, 0).getValue();
offset2 = range1.offset(2, 0).getValue();
offset3 = range1.offset(3, 0).getValue();
//if -, then merge offset cells 1 and 2
//and enter "Order complete" in offset cell 2.
if (dash === "-") {
range1.offset(1, 0).setValue(offset1 + " " + offset2);
//Translate
range1.offset(2, 0).setValue("Order complete");
};
//The real slow part...
//if - and offset 3 is not blank, then INSERT CELL
if (dash === "-" && offset3) {
//select from three rows down to last
//move selection one more row down (down 4 rows total)
s.getRange(row + 1, 1, lastRow).offset(3, 0).moveTo(range1.offset(4, 0));
};
};
}
Formatting Update:
For guidance on formatting the output with font or background colors, check this follow-up question here. Hopefully you can benefit from the advice these pros gave me :)
Issue:
Usage of .getValue() and .setValue() in a loop resulting in increased processing time.
Documentation excerpts:
Minimize calls to services:
Anything you can accomplish within Google Apps Script itself will be much faster than making calls that need to fetch data from Google's servers or an external server, such as requests to Spreadsheets, Docs, Sites, Translate, UrlFetch, and so on.
Look ahead caching:
Google Apps Script already has some built-in optimization, such as using look-ahead caching to retrieve what a script is likely to get and write caching to save what is likely to be set.
Minimize "number" of read/writes:
You can write scripts to take maximum advantage of the built-in caching, by minimizing the number of reads and writes.
Avoid alternating read/write:
Alternating read and write commands is slow
Use arrays:
To speed up a script, read all data into an array with one command, perform any operations on the data in the array, and write the data out with one command.
Slow script example:
/**
* Really Slow script example
* Get values from A1:D2
* Set values to A3:D4
*/
function slowScriptLikeVBA(){
const ss = SpreadsheetApp.getActive();
const sh = ss.getActiveSheet();
//get A1:D2 and set it 2 rows down
for(var row = 1; row <= 2; row++){
for(var col = 1; col <= 4; col++){
var sourceCellRange = sh.getRange(row, col, 1, 1);
var targetCellRange = sh.getRange(row + 2, col, 1, 1);
var sourceCellValue = sourceCellRange.getValue();//1 read call per loop
targetCellRange.setValue(sourceCellValue);//1 write call per loop
}
}
}
Notice that two calls are made per loop(Spreadsheet ss, Sheet sh and range calls are excluded. Only including the expensive get/set value calls). There are two loops; 8 read calls and 8 write calls are made in this example for a simple copy paste of 2x4 array.
In addition, Notice that read and write calls alternated making "look-ahead" caching ineffective.
Total calls to services: 16
Time taken: ~5+ seconds
Fast script example:
/**
* Fast script example
* Get values from A1:D2
* Set values to A3:D4
*/
function fastScript(){
const ss = SpreadsheetApp.getActive();
const sh = ss.getActiveSheet();
//get A1:D2 and set it 2 rows down
var sourceRange = sh.getRange("A1:D2");
var targetRange = sh.getRange("A3:D4");
var sourceValues = sourceRange.getValues();//1 read call in total
//modify `sourceValues` if needed
//sourceValues looks like this two dimensional array:
//[//outer array containing rows array
// ["A1","B1","C1",D1], //row1(inner) array containing column element values
// ["A2","B2","C2",D2],
//]
//#see https://stackoverflow.com/questions/63720612
targetRange.setValues(sourceValues);//1 write call in total
}
Total calls to services: 2
Time taken: ~0.2 seconds
References:
Best practices
What does the range method getValues() return and setValues() accept?
Using methods like .getValue() and .moveTo() can be very expensive on execution time. An alternative approach is to use a batch operation where you get all the column values and iterate across the data reshaping as required before writing to the sheet in one call. When you run your script you may have noticed the following warning:
The script uses a method which is considered expensive. Each
invocation generates a time consuming call to a remote server. That
may have critical impact on the execution time of the script,
especially on large data. If performance is an issue for the script,
you should consider using another method, e.g. Range.getValues().
Using .getValues() and .setValues() your script can be rewritten as:
function format() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getActiveSheet();
var lastRow = s.getLastRow(); // more efficient way to get last row
var row;
var data = s.getRange("A:A").getValues(); // gets a [][] of all values in the column
var output = []; // we are going to build a [][] to output result
//loop through all cells in column A
for (row = 0; row < lastRow; row++) {
var cellValue = data[row][0];
var dash = false;
if (typeof cellValue === 'string') {
dash = cellValue.substring(0, 1);
} else { // if a number copy to our output array
output.push([cellValue]);
}
// if a dash
if (dash === "-") {
var name = (data[(row+1)][0]+" "+data[(row+2)][0]).trim(); // build name
output.push([cellValue]); // add row -state
output.push([name]); // add row name
output.push(["Order complete"]); // row order complete
output.push([""]); // add blank row
row++; // jump an extra row to speed things up
}
}
s.clear(); // clear all existing data on sheet
// if you need other data in sheet then could
// s.deleteColumn(1);
// s.insertColumns(1);
// set the values we've made in our output [][] array
s.getRange(1, 1, output.length).setValues(output);
}
Testing your script with 20 rows of data revealed it took 4.415 seconds to execute, the above code completes in 0.019 seconds
I'm using this script to build statistics on each of my coworkers and it includes 15 pivot tables that frequently need to have their filters updated for different time periods. I have a lot of coworkers on this sheet, so the script for 15 tables per person ends up taking way too long and times out.
I am including one table, but the code would have 15 iterations of this per person (times 20+ people).
How can I prevent it from timing out? Either through a work around or cleaning up my code to be more efficient.
function MassUpdateofFilters() {
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange('Andrew!A1').activate();
var sourceData = spreadsheet.getRange('Batches!1:997');
var pivotTable = spreadsheet.getRange('A1').createPivotTable(sourceData);
pivotTable.setValuesDisplayOrientation(SpreadsheetApp.Dimension.ROWS);
var pivotValue = pivotTable.addPivotValue(3, SpreadsheetApp.PivotTableSummarizeFunction.SUM);
var pivotGroup = pivotTable.addRowGroup(5);
pivotGroup.showTotals(false)
.sortDescending()
.sortBy(pivotTable.getPivotValues()[0], []);
pivotGroup = pivotTable.addRowGroup(4);
pivotGroup.showTotals(false);
var criteria = SpreadsheetApp.newFilterCriteria()
.setVisibleValues(['Andrew'])
.build();
pivotTable.addFilter(5, criteria);
criteria = SpreadsheetApp.newFilterCriteria()
.setVisibleValues([ '1', '2', '3', '5', '6', '7'])
.build();
pivotTable.addFilter(16, criteria);
criteria = SpreadsheetApp.newFilterCriteria()
.setVisibleValues(['April'])
.build();
pivotTable.addFilter(15, criteria);
criteria = SpreadsheetApp.newFilterCriteria()
.setVisibleValues(['2019'])
.build();
pivotTable.addFilter(17, criteria);
};
Google has a guide to best practices here: https://developers.google.com/apps-script/guides/support/best-practices
I see that you are calling the SpreadsheetApp service seven times. This is slow. It would be better to call it once to get the data, then call it again to write the data, (if at all possible). Also, could you use the cache service to pull the data from the spreadsheet once, and manipulate copies of it over and over for each separate user?
https://developers.google.com/apps-script/reference/cache/cache
I would first recommend going through and the best practice.
That being said I did have to run a script across a few thousand sheets and developed this little bit of code to help with timeout issues. It runs a loop that times itself out, and you can set a trigger to run it ever 5 or 10 minutes. After a few hours, it will do the trick. Keep in mind that Google will eventually time you out if you do this too much, and you may have to wait a day or so till you can run it again.
function changeAll() {
var startd = new Date();
var diff = 0;
var start = Number(PropertiesService.getScriptProperties().getProperty("START"));
Logger.log(start);
for(var i = start; i < YOUR_MAX && diff < 280; i++){
//MAKE CHANGES HERE **************************************************************************
//END CHANGE AREA ****************************************************************************
//Change time and counter for next run
PropertiesService.getScriptProperties().setProperty("START", i);
var now = new Date();
var difdate = (now.getTime() - startd.getTime()) * 0.001;
diff = difdate;
Logger.log("DIFF: " + diff);
}
}
Goal: I'm trying to create a behavior tracker for four classes in Google Spreadsheets. The tracker has nine sheets: Class7A, Class7B, Class8A, Class8B, and Mon-Fri summary sheets. The goal was for each ClassXX sheet to have behavior tracking information for an entire week, but for the default view to show only the current day's information.
Attempts: During initial workup (with only the Class7A sheet created), I got this to work using a modification of the script found here (Thank you Jacob Jan Tuinstra!): Optimize Google Script for Hiding Columns
I modified it to check the value in the third row of each column (which held a 1 for Monday, 2 for Tuesday, etc), and if it did not match the numerical equivalent for the day of the week (var d = new Date(); var n = d.getDay();), then it would hide that column. This process was somewhat slow - I'm assuming because of the iterating through each column - but it worked.
Quite excited, I went ahead and added the rest of the sheets, and tried again - but the code as written, seems to affect only the current sheet. I tried modifying it by replacing var sheet = ss.getSheets()[0]; with for script that iterated through the columns, until i>4 (I've since lost that piece of code), with no luck.
Deciding to go back and try adapting the original version of the script to instead explicitly run multiple times for each named sheet, I found the that script no longer seems to work at all. I get various version of "cannot find XX function in sheet" or "cannot find XX function in Range."
Source: A shared version (with student info scrubbed) can be found here: https://docs.google.com/spreadsheets/d/1OMq4a4_Gh_xyNk_IRy-mwJn5Hq36RXmdAzTzx7dGii0/edit?usp=sharing (editing is on).
Stretch Goal: Ultimately, I need to get this to reliably show only the current day's columns (either through preset ranges (same for each sheet), or the 1-5 values), and I need it to do so for all four ClassXX sheets, but not the summary pages (and preferably more quickly than the iterations). If necessary, I can remove the summary pages and set them up externally, but that's not my first preference. I would deeply appreciate any help with this; so far my attempts have seemed to only take me backwards.
Thanks!
Current code:
function onOpen() {
// get active spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
// create menu
var menu = [
{name: "Show Today Only", functionName: "hideColumn"},
{name: "Show All Days", functionName: "showColumn"},
{name: "Clear Week - WARNING will delete all data", functionName: "clearWeek"}
];
// add to menu
ss.addMenu("Show Days", menu);
}
var d = new Date();
var n = d.getDay();
function hideColumn() {
// get active spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
// get first sheet
var sheet = ss.getSheets()[0];
// get data
var data = sheet.getDataRange();
// get number of columns
var lastCol = data.getLastColumn()+1;
Logger.log(lastCol);
// itterate through columns
for(var i=1; i<lastCol; i++) {
if(data.getCell(2, i).getValue() != n) {
sheet.hideColumns(i);
}
}
}
function showColumn() {
// get active spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
// get first sheet
var sheet = ss.getSheets()[0];
// get data
var data = sheet.getDataRange();
// get number of columns
var lastCol = data.getLastColumn();
// show all columns
sheet.showColumns(1, lastCol);
}
I cannot recreate the problem of the script not working at all, it's working fine for Class7A so that part is working fine.
So let's look at the two other problems:
Applying this to all Sheets
Speeding up the script
First let's create some globals we use in both functions
var d = new Date();
var n = d.getDay();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheetNames = ss.getSheets().map(function(sheet) {return sheet.getName();});
var classSheets = sheetNames.filter(function(sheetName) {return sheetName.match("Class")});
Now we can iterate over classSheets and get the sheet by name and hide columns in each.
However hiding each individual column is very slow.
The sheet is built very structured, every week has 12 columns (except for friday which doesn't have the grey bar), so we can just calculate the ranges we want to hide.
function hideColumn() {
classSheets.map(function(sheetName){
var sheet = ss.getSheetByName(sheetName);
if (n == 1) {
// Hide everything after the first three columns + Monday
sheet.hideColumns(3 + 11, 12 * 4);
} else if (n == 5) {
// Hide everything to the left except the leftmost three columns
sheet.hideColumns(3, 4 * 12);
} else {
// Hide everything left of the current day
sheet.hideColumns(3, (n - 1) * 12);
// Hide everything after the current day
sheet.hideColumns(3 + n * 12, (5 - n) * 12 - 1);
}
});
}
Lastly we can shorten showColumn
function showColumn() {
classSheets.map(function(sheetName){
var sheet = ss.getSheetByName(sheetName);
var lastCol = sheet.getLastColumn();
sheet.showColumns(1, lastCol);
});
}
I'm stuck with this one and hope that anyone can help me.
The script picks a word and records the elapsed time (reaction time) between word appearance and keypress. My idea was to have one function that picks the word, set it as interval, and have this interval broken by keypress. The attributes "stim" and "type" are for further use and I got problems if I had them picked more than once.
Here it is on fiddle: Fiddle of my problem
It does not, however, work quite correct. The default reaction time (2000ms), should only be recorded when there is no keypress. But as it is now, its recorded each time.
Js code:
var teststim = [{
stim: "A",
type: "letter"
}, {
stim: "B",
type: "letter"
}, {
stim: "1",
type: "integer"
}, {
stim: "2",
type: "integer"
}];
var RT = [];
var Type = [];
var Stim = [];
var displayword = function () {
stuff = teststim[Math.floor((Math.random() * teststim.length))];
$("#present").fadeOut(1000, function () {
$("#present").text(stuff.stim).fadeIn();
t1 = (new Date()).getTime();
});
reac = 2000;
};
timing = setInterval(displayword, 2000);
$(document).keypress(function (e) {
clearInterval(timing);
var t2 = (new Date()).getTime();
reac = t2 - t1;
t1 = t2;
timing = setInterval(displayword, 2000);
});
If I understood your question correctly, your mistake is here:
reac = 2000; RT.push(reac); $("#RT").val(RT);
You push and show reaction time as soon as you show word to type.
Here is your fiddle fixed.
Key is to first show the word and then start counting and not do it concurrently.
By the way there is still a little problem with reseting t1 in wrong place, so if you hit the keypress before fadeOut, but after default reaction time has fired, you'll get result like "2200".
You can fix it by, for example, putting it out of fadeOut: fiddle
To make it also change every time you press a key you can modify it like that: fiddle