I'm writing a script that's going to look through a monthly report and create sheets for each store for a company we do work for and copy data for each to the new sheets. Currently the issue I'm running into is that we have two days of data and 171 lines is taking my script 369.261 seconds to run and it is failing to finish.
function menuItem1() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet1 = ss.getSheetByName("All Stores");
var data = sheet1.getDataRange().getValues();
var CurStore;
var stores = [];
var target_sheet;
var last_row;
var source_range
var target_range;
var first_row = sheet1.getRange("A" + 1 +":I" + 1);
//assign first store number into initial index of array
CurStore = data[1][6].toString();
//add 0 to the string so that all store numbers are four digits.
while (CurStore.length < 4) {CurStore = "0" + CurStore;}
stores[0] = CurStore;
// traverse through every row and add all unique store numbers to the array
for (var row = 2; row <= data.length; row++) {
CurStore = data[row-1][6].toString();
while (CurStore.length < 4) {
CurStore = "0" + CurStore;
}
if (stores.indexOf(CurStore) == -1) {
stores.push(CurStore.toString());
}
}
// sort the store numbers into numerical order
stores.sort();
// traverse through the stores array, creating a sheet for each store, set the master sheet as the active so we can copy values, insert first row (this is for column labels), traverse though every row and when the unique store is found,
// we take the whole row and paste it onto it's newly created sheet
// at the end push a notification to the user letting them know the report is finished.
for (var i = stores.length -1; i >= 0; i--) {
ss.insertSheet(stores[i].toString());
ss.setActiveSheet(sheet1);
target_sheet = ss.getSheetByName(stores[i].toString());
last_row = target_sheet.getLastRow();
target_range = target_sheet.getRange("A"+(last_row+1)+":G"+(last_row+1));
first_row.copyTo(target_range);
for (var row = 2; row <= data.length; row++) {
CurStore = data[row-1][6].toString();
while (CurStore.length < 4) {
CurStore = "0" + CurStore;
}
if (stores[i] == CurStore) {
source_range = sheet1.getRange("A" + row +":I" + row);
last_row = target_sheet.getLastRow();
target_range = target_sheet.getRange("A"+(last_row+1)+":G"+(last_row+1));
source_range.copyTo(target_range);
}
}
for (var j = 1; j <= 9; j++) {
target_sheet.autoResizeColumn(j);
}
}
Browser.msgBox("The report has been finished.");
}
Any help would be greatly appreciated as I'm still relatively new at using this, and I'm sure there are plenty of ways to speed this up, if not, I'll end up finding a way to break down the function to divide up the execution. If need be, I can also provide some sample data if need be.
Thanks in advance.
The problem is calling SpreadsheepApp lib related methods like getRange() in each iteration. As stated here:
Using JavaScript operations within your script is considerably faster
than calling other services. Anything you can accomplish within Google
Apps Script itself will be much faster than making calls that need to
fetch data from Google's servers or an external server, such as
requests to Spreadsheets, Docs, Sites, Translate, UrlFetch, and so on.
Your scripts will run faster if you can find ways to minimize the
calls the scripts make to those services.
I ran into the same situation and, instead of doing something like for(i=0;i<data.length;i++), I ended up dividing the data.length into 3 separate functions and ran them manually each time one of them ended.
Same as you, I had a complex report to automate and this was the only solution.
Related
I'm using Google Apps Script to write a script to edit Google Sheets for a Mailing List. I'd like it to run through all rows and delete any rows with 'BOUNCED' 'ERROR' or 'NO_RECIPIENT' in a specific cell.
The problem I'm having is the For Loop uses brackets [ ] to designate the rows and columns, which indexes the first row at 0. The deleteRows() action uses curved parenthesis, which indexes the first row at 1. For this reason, I'm having trouble deleting the correct row.
If I program deleteRow(i), it deletes the row following the one being tested by the For loop. If I program deleteRow(i+1), it deletes the correct row the first time, but subsequently deletes the following row. See my code below:
function cleanUp() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var data = sheet.getDataRange().getValues();
for ( var i = 1; i < 30; i++) {
if (data[i][9] === 'ERROR' || data[i][9] === 'BOUNCED' || data[i][9] === 'NO_RECIPIENT') {
sheet.deleteRow(i+1);
}
}
}
Once a row is deleted the rows below it change it's position. One way to avoid this problem is to do the loop in reverse order.
In other words, instead of
for(var i = 1; i < 30; i++)
Use
for(var i = 29 ; i > 0; i--)
I manage Google Sheet lists that sometimes exceed 10,000 rows. For sheets with rows up to around 5,000, the remove duplicates function noted below works finely. But for anything above 5,000, I receive the 'Exceeded maximum execution time' error. I would be grateful for some instruction on how to make the code more efficient such that it could run smoothly even for sheets with 10k+ rows.
function removeDuplicates() {
var sheet = SpreadsheetApp.getActiveSheet();
var data = sheet.getDataRange().getValues();
var newData = new Array();
for(i in data){
var row = data[i];
var duplicate = false;
for(j in newData){
if(row.join() == newData[j].join()){
duplicate = true;
}
}
if(!duplicate){
newData.push(row);
}
}
sheet.clearContents();
sheet.getRange(1, 1, newData.length, newData[0].length).setValues(newData);
}
There are a couple of things that are making your code slow. Let's look at your two for loops:
for (i in data) {
var row = data[i];
var duplicate = false;
for (j in newData){
if (row.join() == newData[j].join()) {
duplicate = true;
}
}
if (!duplicate) {
newData.push(row);
}
}
On the face of it, you're doing the right things: For every row in the original data, check if the new data already has a matching row. If it doesn't, add the row to the new data. In the process, however, you're doing a lot of extra work.
Consider, for example, the fact that at any given time, a row in data will have no more than one matching row in newData. But in your inner for loop, after you find that one match, it still continues checking the rest of the rows in newData. The solution to this would be to add a break; after duplicate = true; to stop iterating.
Consider also that for any given j, the value of newData[j].join() will always be the same. Suppose you have 100 rows in data, and no duplicates (the worst case). By the time your function finishes, you'll have calculated newData[0].join() 99 times, newData[1].join() 98 times... all in all you'll have done almost 5,000 calculations to get the same 99 values. A solution to this is memoization, whereby you store the result of a calculation in order to avoid doing the same calculation again later.
Even if you make those two changes, though, your code's time complexity is still O(n²). If you have 100 rows of data, in the worst case the inner loop will run 4,950 times. For 10,000 rows that number is around 50 million.
However, we can do this is O(n) time instead, if we get rid of the inner loop and reformulate the outer loop like so:
var seen = {};
for (var i in data) {
var row = data[i];
var key = row.join();
if (key in seen) {
continue;
}
seen[key] = true;
newData.push(row);
}
Here, instead of checking every row of newData for a row matching row in every iteration, we store every row we've seen so far as a key in the object seen. Then in each iteration we just have to check if seen has a key matching row, an operation we can do in nearly constant time, or O(1).1
As a complete function, here's what it looks like:
function removeDuplicates_() {
const startTime = new Date();
const sheet = SpreadsheetApp.getActiveSheet();
const data = sheet.getDataRange().getValues();
const numRows = data.length;
const newData = [];
const seen = {};
for (var i = 0, row, key; i < numRows && (row = data[i]); i++) {
key = JSON.stringify(row);
if (key in seen) {
continue;
}
seen[key] = true;
newData.push(row);
}
sheet.clearContents();
sheet.getRange(1, 1, newData.length, newData[0].length).setValues(newData);
// Show summary
const secs = (new Date() - startTime) / 1000;
SpreadsheetApp.getActiveSpreadsheet().toast(
Utilities.formatString('Processed %d rows in %.2f seconds (%.1f rows/sec); %d deleted',
numRows, secs, numRows / secs, numRows - newData.length),
'Remove duplicates', -1);
}
function onOpen() {
SpreadsheetApp.getActive().addMenu('Scripts', [
{ name: 'Remove duplicates', functionName: 'removeDuplicates_' }
]);
}
You'll see that instead of using row.join() this code uses JSON.stringify(row), because row.join() is fragile (['a,b', 'c'].join() == ['a', 'b,c'].join(), for example). JSON.stringify isn't free, but it's a good compromise for our purposes.
In my tests this processes a simple spreadsheet with 50,000 rows and 2 columns in a little over 8 seconds, or around 6,000 rows per second.
I'm working in Google apps script and seem to have screwed up one of my for loops. I'm sure that I am missing something trivial here, but I can't seem to spot it.
Code Snippet:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheets = ss.getSheets();
var lastRow = sheets[3].getLastRow();
var zw = sheets[3].getRange(2, 1, lastRow - 1, 26).getValues();
for (var j = 0; j < zw.length; ++j) {
if (zw[j][9] === 'Yes') {
var masterEmail = [];
var firstLetterLastName = [];
var first2Letter = [];
var masterEmail.push(zw[j][22]);
var firstLetterLastName.push(zw[j][1].charAt(0).toLowerCase());
var first2Letter.push(zw[j][1].charAt(0).toLowerCase() + zw[j][1].charAt(1).toLowerCase());
//The rest of the function follows...
}
}
What's Not Working:
The for loop doesn't increment. When running the code in a debugger, var j stays at a value of 0.0, and the rest of the function only runs based of off the values in the 0 position of zw.
What I need it to do (AKA - How I thought I had written it:)
The ZW variable is holding a 2 dimensional array of cell values from a Google sheet. I'm looping through that, checking the 9th value of each array entry for a string of "Yes" and then running the rest of the function (for each column with a "Yes") if the condition is true.
I thought I had this working before, but recently had to restructure and optimize some things. Now I'm starting to think I may need to rethink things and use a different loop method. Can anyone educate me?
Edit: Here's a bit more context as requested:
function menuItem1() {
var ui = SpreadsheetApp.getUi();
var response = ui.alert('Are you sure you want to send emails?', ui.ButtonSet.YES_NO);
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheets = ss.getSheets();
var lastRow = sheets[3].getLastRow();
var zw = sheets[3].getRange(2, 1, lastRow - 1, 26).getValues();
if (response === ui.Button.YES) {
for (var j = 0; j < zw.length; j++) {
if (zw[j][9] === 'Yes') {
var firstLetterLastName = [];
firstLetterLastName.push(zw[j][1].charAt(0).toLowerCase());
//Other Stuff....
}
}
}
}
I have a menu item attached to a simple onOpen, that calls menuItem1(). Calling the function prompts the user with a warning that they are about to send emails, then goes about getting data to assign email addresses based on the contents of the sheets. firstLetterLastName is an example.
I'm still not getting the loop to function, is it because I have it between two if statements? (Here is a link to the sheet)
Indeed it is quite trivial. You have mixed up your increment. You wrote
for (var j = 0; j < zw.length; ++j)
which means that you do 1 + i (and we know that at the start i = 0 which means your value will always be 1) instead of using the usual
for (var j = 0; j < zw.length; j++)
which would mean that you do i + 1 and update i, so you will get the expected 0 + 1 1 + 1 etc
EDIT:
First, I recommend instead of something like
if (responseMir === ui.Button.YES) {
// Your For loop
doing
if (responseMir !== ui.Button.YES) {
return
}
and in a similar fashion in the for loop
if (zw[j][9] !== 'Yes') {
break
}
It mostly helps increase readability by not including large blocks of code under a single if, when all you want to do is to stop execution.
Your for loop gets broken because of the mistake here:
teacherEmailMir.push(selValsMir[j][7]);
So your loop will go over once. However on the next itteration, you try to push selValsMir[1][7] which does not exist. Note that each itteration you have var selValsMir = []; inside the loop, which means that for every j selValsMir will always be an empty array. So with the following line
selValsMir.push([zw[j][0], zw[j][1], zw[j][2], zw[j][3], zw[j][4], zw[j][5], zw[j][7], zw[j][22], zw[j][23], zw[j][24]]);
your array will always have selValsMir.lenght = 1 and selValsMir[0].length = 10. So obviously trying to access anything from selValsMir[1] will throw you an error and stop the script right there.
I also recommend looking over the if statements that look at the first and first 2 letters of the name as I believe you can accomplish the same with less code. Always try to streamline. Consider using switch() where you end up using a lot of else if
So I am loading csv files from a server an inserting js function calls that create tables/sheets with jquery.sheet. Everything works thus far but when I put functions into the list they do not calculate.
The sheets (simplified)data object for the td has this before I modify anything:
Object {td: x.fn.x.init[1], dependencies: Array[0], formula: "", cellType: null, value: "=A2+B2+C2"…}
When I set the formula value it changes to:
Object {td: x.fn.x.init[1], dependencies: Array[0], formula: "=A2+B2+C2", cellType: null, value: "=A2+B2+C2"…}
So I understand how to set formula and value but what i wish to do is trigger an event to auto calculate a cell hopefully based on an "X,Y" co-ordinate, or find out if I am taking the wrong approach.
I dont know if this helps but when I go to edit a cell it will appear as ==A2+B2+C2 not =A2+B2+C2
I would supply my code but because of the C# asp and js interaction it is not short I don't think it would help.
Solved:
Two things are essential to load a formula from a csv file to a jquery.sheet instance and then have it calculate. First is to manually set the objects ["formula"] property, while leaving off the '=' in the beginning because it adds its own. then you must trigger the "calc(s,true)" function with s as the sheets index and in my case I put true as the second parameter because I believe it forces calculations on cells with a function.
var sheets = jQuery.sheet.instance[0];
for (var s = 0 ; s < names.length; s++) {
var sheet = sheets.spreadsheets[s];
for (var k = 1; k < sheet.length; k++) {
var row = sheet[k];
for (var j = 1; j < row.length; j++) {
var col = row[j];
//alert(cell.value);
if (col.value.startsWith("=")) {
col["formula"] = col.value.substring(1, col.value.length);
}
}
}
sheets.calc(s, true);
}
If a better way is found please let me know. I do not think this is very scalable as it is O(n^3).
I am changing the content of about 5000 HTML tags and as I read here doing 5000+ html rendering changes is slow and it is better just to redraw the HTML once,
Therefor I have created a function that loads the entire HTML into a JavaScript string and then goes through the text(looking for a label tag), changes the content of the tags and eventually redraws the HTML once.
With high hopes, this was also a failure and takes around 30 seconds on a 1000 tags test.
My function basically reverse counts all the visible label DIVs on the screen and adds numbering to them,
Here's the code, what am I doing wrong please? (here's an isolated jsfiddle: http://jsfiddle.net/tzvish/eLbTy/8/)
// Goes through all visible entries on the page and updates their count value
function updateCountLabels() {
var entries = document.getElementsByName('entryWell');
var entriesId = document.getElementsByName('entryId');
var entriesHtml = document.getElementById('resultContainer').innerHTML;
var visibleEntries = new Array();
var countEntries = 0 , pointer = 0;
// Create a list of all visible arrays:
for (i = 0; i < entries.length; i++) {
if ($(entries[i]).is(":visible")) {
countEntries++;
visibleEntries.push(entriesId[i].value);
}
}
// Update html of all visible arrays with numbering:
for (i = 0; i < visibleEntries.length; i++) {
currentId = visibleEntries[i];
// Get the position of insertion for that label element:
elementHtml = '<div id="entryCount'+currentId+'" class="entryCountLabel label label-info">';
pointer = entriesHtml.indexOf(elementHtml) + elementHtml.length;
pointerEnd = entriesHtml.indexOf("</div>",pointer);
entriesHtml = entriesHtml.substring(0, pointer) + countEntries + entriesHtml.substring(pointerEnd);
countEntries--;
}
// apply the new modified HTML:
document.getElementById('resultContainer').innerHTML = entriesHtml;
}
From here:
Loops are frequently found to be bottlenecks in JavaScript. To make a loop the most efficient, reverse the order in which you process the items so that the control condition compares the iterator to zero. This is far faster than comparing a value to a nonzero number and significantly speeds up array processing. If there are a large number of required iterations, you may also want to consider using Duff’s Device to speed up execution.
So change:
for (i = 0; i < entries.length; i++)
To
for (i = entries.length - 1; i ==0 ; i--)
Also notice following paragraph from mentioned link:
Be careful when using HTMLCollection objects. Each time a property is accessed on one of these objects, it requires a query of the DOM for matching nodes. This is an expensive operation that can be avoided by accessing HTMLCollection properties only when necessary and storing frequently used values (such as the length property) in local variables.
So change:
for (i = entries.length - 1; i == 0; i--)
To:
var len = entries.length;
for (i = len - 1; i ==0 ; i--)