I am using for loops to search through large sheets (approximately 4500 rows).
Most of them look something like this:
function lookup(value) {
var tables=SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Name");
var columnvalues = tables.getRange(1, 1,tables.getLastRow()).getValues();
var searchResult = columnvalues.findIndex(value); //Row Index - 1
}
Array.prototype.findIndex = function(search){
if(search == "") return false;
for (var i=0; i<this.length; i++)
if (this[i].toString().indexOf(search) > -1 ) return i;
return -1;
}
The app script currently runs relatively slowly. I am looking for a way to either speed up my current code or for another search method. I've been thinking about using the google spreadsheet lookup functions (index-match, vLookup) but I haven't found a way to access those functions in apps script. Any thoughts?
currently you wont find a way to do this faster with apps script. you have to get the entire column and search one by one like you are already doing.
the alternative you mention using sheet formulas can be flacky as you might find that the cell formulas dont inmediately update results as you change values.
the only case i can see this being sped up is if you need to make multiple search calls. in that case it will be faster to pass search terms as an array and search them all at once in the single loop.
As you are searching a single column something a bit simpler might be a bit faster.
function lookup(value) {
var searchResult = false;
if(value == "") return;
var tables=SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Name");
var columnvalues = tables.getRange(1, 1,tables.getLastRow()).getValues();
for (var i=0; i<columnvalues.length; i++) {
if(columnvalues[i][0] == value) {
searchResult = i; //Row Index - 1
break;
};
}
The fastest way that I have found of searching through an array is using the .filter() method of an array.
var yourData = range.getValues();
var results = yourData.filter(function(val) {
return val[0] == search ||
val[0].includes(search);
});
//now you have a 2d array of results
Modify that such that it is the range you want(sorry, on mobile) and that is the fastest way that I found. Avoiding the For-Loop is the best. You could also use a .forEach()
Related
Goal: Run through the columns of a 2D array (comes from an Excel file with uneven column lengths) and put the entries that exist into their own array.
What I did: The length of the longest column is 90 entries, which is the second column in the Excel file, and the shortest is 30, which is the first column. I set up a for loop to go through each column and a while loop to go through each entry while it exists and append it to a new array.
Original(ish) Code:
//read in Excel file into 2D array called "myExcel"
var columnNames = ["shortest", "longest", "irrelevant"];
shortArray = [];
longArray = [];
irrArray = [];
var s
for (var i = 0; i < columnNames.length; i++) {
var columnName = columnNames[i];
s = 0;
while (myExcel[s][columnName]) {
if ((columnName === "shortest")) {
var row = myExcel[s][columnName];
shortArray.append(row);
s++;
} else if ((columnName === "longest")) {
var row = myExcel[s][columnName];
longArray.append(row);
s++;
} else if ((columnName === "irrelevant")) {
var row = myExcel[s][columnName];
irrArray.append(row);
s++;
}
}
}
Problem: It's only half working. It makes it through the first column (30 rows) just fine--it stops when myExcel[s][columnName] no longer exists (when columnName = "shortest" and after s = 29). Then, it makes it all the way through columnName = "longest" and s = 89 before giving me the error "TypeError: Cannot read property 'longest' of undefined". I'm assuming it's because it's trying to go through row 90, which doesn't exist. But I thought that's where my while loop would stop.
What I've Tried:
do while loop
//blah
do {
//blah
} while (myExcel[s][columnName]);
Added additional while loop condition
//blah
while ((myExcel[s][columnName]) && s<myExcel.length) {
//blah
}
Using typeof
//blah
while (typeof (myExcel[s][columnName]) === 'string') { //also used this with !=='undefined' and ==='string' when I added a number to the end of each row in the Excel sheet
//blah
}
And basically every combination of these (and probably much more I'm forgetting). I'm sure it's an easy fix, but I've spent days trying to figure it out so I guess I have to ask for help at this point. I'm also a MATLAB person and recently had to learn both Python and Javascript because of COVID, so it could possibly be a language switch issue (although I don't think so because I've been googling and messing with this for days). Any help would be very appreciated!
In your while loop change the check to,
while(myExcel[s] && myExcel[s][columnName] ) {
If you are writing modern Js, then you could simply optional chain it like so, while(myExcel[s]?.[columnName])
The thing is, you are trying to traverse inside an outer array. But you first need to check if outer array exists and then go check the inner array.
I don't fully understand you approach, but I think you're looking for this:
var shortArray = [];
var longArray = [];
var irrArray = [];
for(let row of myExcel){
if(!row) continue; // not sure if this check is necessary.
if(row.shortest) shortArray.append(row.shortest);
if(row.longest) longArray.append(row.longest);
if(row.irrelevant) irrArray.append(row.irrelevant);
}
I understand how to go about tasks using loops, recursion is kind of a mystery to me, but from what I understand in certain cases it can save a ton of time if looping through a lot of data.
I created the following function to loop through a large(ish) data set.
var quotes = require('./quotes.js');
//Pulls in the exported function from quotes.js
var exportedQuotes = quotes.allQuotes();
var allAuthors = exportedQuotes.author;
//Create an empty key value object, we use these to coerce unique values to an array
var uniqs = {};
//I create this object to hold all the authors and their quotes
var fullQuote = {};
//Create an object with only unique authors
for(var i = 0; i < allAuthors.length ; i++){
fullQuote[allAuthors[i]] = null;
}
//Coerce unique authors from javascript object into an array
var uniqAuthors = Object.keys(uniqs);
var quoteCount = exportedQuotes.author.length;
var iterativeSolution = function(){
for(var i = 0; i < Object.keys(fullQuote).length; i++){
for(var j = 0; j < exportedQuotes.author.length; j++){
//If the author in the unique list is equal to the author in the duplicate list
if(Object.keys(fullQuote)[i] == exportedQuotes.author[j]){
//if an author has not had a quote attributed to its name
if(fullQuote[exportedQuotes.author[j]] == null){
//assign the author an array with the current quote at the 0 index
fullQuote[exportedQuotes.author[j]] = [exportedQuotes.quote[j]]
} else {
//if an author already has a quote assigned to its name then just add the current quote to the authors quote list
fullQuote[exportedQuotes.author[j]].push(exportedQuotes.quote[j])
}
}
}
}
}
I don't currently have the skills to do analyze this, but, I'm wondering if there is a case for recursion to save the time it takes to get through all the loops. And if there is a case for recursion what does it look like for nested loops in javascript, specifically when creating key value objects recursively?
There may be a slight misunderstanding about what recursion is: recursion does not save time. It's just a different way of doing the same traversal. It generally a little easier to read, and depending on the problem, will map to certain algorithms better. However, one of the first things we do when we need to start optimizing code for speed is to remove recursion, turning them back into loops, and then even "unrolling" loops, making code much uglier, but fast, in the process. Recursion vs plain loops is almost always a matter of taste. One looks nicer, but that's hardly the only quality we should judge code on.
And also: just because it sounds like I'm advocating against using it, doesn't mean you shouldn't just try it: take that code, put it in a new file, rewrite that file so that it uses recursion. Doing so lets you compare your code. Which one is faster? Which is easier to read? Now you know something about how (your) code behaves, and you'll have learned something valuable.
Also don't sell yourself short: if you wrote this code, you know how it works, so you know how to analyze it enough to rewrite it.
Algorithms makes code fast or slow, not recursion. Some quite fast algorithms can use recursion, but that's a whole different story. Many algorithms can be written as both with recursion, and without recursion.
However, your code has a big problem. Notice how many times you call this code?
Object.keys(fullQuote)
You are re-computing the value of that many many times in your code. Don't do that. Just call it once and store in a variable, like the following:
var uniqAuthors = Object.keys(uniqs);
var uniqFullQuote = Object.keys(fullQuote);
var quoteCount = exportedQuotes.author.length;
//Loop through all quotes and assign all quotes to a unique author::Each author has many quotes
for(var i = 0; i < uniqFullQuote.length; i++){
for(var j = 0; j < exportedQuotes.author.length; j++){
//If the author in the unique list is equal to the author in the duplicate list
if(uniqFullQuote[i] == exportedQuotes.author[j]){
//if an author has not had a quote attributed to its name
if(fullQuote[exportedQuotes.author[j]] == null){
//assign the author an array with the current quote at the 0 index
fullQuote[exportedQuotes.author[j]] = [exportedQuotes.quote[j]]
} else {
//if an author already has a quote assigned to its name then just add the current quote to the authors quote list
fullQuote[exportedQuotes.author[j]].push(exportedQuotes.quote[j])
}
}
}
}
You don't have to iterate Object.keys(fullQuote).
var quotes = require('./quotes.js'),
exportedQuotes = quotes.allQuotes(),
allAuthors = exportedQuotes.author,
fullQuote = Object.create(null);
for(var i=0; i < allAuthors.length; ++i)
(fullQuote[allAuthors[i]] = fullQuote[allAuthors[i]] || [])
.push(exportedQuotes.quote[i])
I don't recommend recursion. It won't improve the asymptotic cost, and in JS calling functions is a bit expensive.
I got really curious and created a recursive solution just to see how it works. Then timed it, my iterative solution took 53 seconds to run, while my recursive solution took 1 millisecond to run. The iterative approach can obviously be tweaked based on the answers provided below, to run faster, but a recursive approach forced me to think in a "leaner" manner when creating my function.
var exportedQuotes = quotes.allQuotes();
var allAuthors = exportedQuotes.author;
var n = allAuthors.length
var fullQuote = {};
var recursiveSolution = function(arrayLength) {
//base case
if(arrayLength <= 1){
if(fullQuote[exportedQuotes.author[0]] == null){
fullQuote[exportedQuotes.author[0]] = [exportedQuotes.quote[0]];
}else{
fullQuote[exportedQuotes.author[0]].push(exportedQuotes.quote[0])
}
return;
};
//recursive step
if(fullQuote[exportedQuotes.author[arrayLength]] == null){
fullQuote[exportedQuotes.author[arrayLength]] = [exportedQuotes.quote[arrayLength]];
}else{
fullQuote[exportedQuotes.author[arrayLength]].push(exportedQuotes.quote[arrayLength])
}
newLength = arrayLength - 1;
return recursiveSolution(newLength);
}
////////Timing functions
var timeIteration = function(){
console.time(iterativeSolution);
iterativeSolution(); // run whatever needs to be timed in between the statements
return console.timeEnd(iterativeSolution);
}
var timeRecursive = function(){
console.time(recursiveSolution(n));
recursiveSolution(n); // run whatever needs to be timed in between the statements
return console.timeEnd(recursiveSolution(n));
}
I recently asked a question about how to add items to a Google Form from a Google Spreadsheet. And it works great. Instead of using FormApp.create(), though, I'll have to use .openByUrl() because the ID has to stay the same. The problem is that if I run my script again, it'll open the existing form (great) and then append more items to the existing form.
This behaviour makes perfect sense but is not quite what I want. So I thought I'd just remove all existing items before I add new ones from my spreadsheet. I consulted the Google dev site for Form Services and feel like I should have all the pieces. I can't quite put them together, though.
I am now doing this following:
var form = FormApp.openByUrl('https://docs.google.com/forms/d/.../edit');
var items = form.getItems();
for (var i in items) {
form.deleteItem(i);
}
However, that'll give me an out of range error. Can someone point me in the right direction?
The problem is with how you're iterating over the array.
Try this:
var form = FormApp.openByUrl('https://docs.google.com/forms/d/.../edit');
var items = form.getItems();
for (var i=0; i<items.length; i++) {
form.deleteItem(i);
}
function clearForm(){
var items = form.getItems();
while(items.length > 0){
form.deleteItem(items.pop());
}
}
This worked for me when I ran into the same issue:
for (var i=0; i<items.length; i++) {
if (items[i] != null){
form.deleteItem(i);
}
}
Start by deleting the last item and repeat it until all items are deleted. This could be done by a reverse for loop:
function deleteAllItems(){
var form = FormApp.openById(/*put here your form id*/);
var items = form.getItems();
var end = items.length - 1;
for(var i = end ; i >= 0; i--){
form.deleteItem(i);
}
}
Another alternative is avoid of a variable index by using 0, so the first item will be deleted, no matter if a regular or a reverse loop is used. Note: This was already mentioned in a comment to another answer.
I also ran into the same problem. This one worked for me:
function deleteItems(){
var form = FormApp.openById('ID');
var items = form.getItems();
items.forEach(function(e){form.deleteItem(e)})
}
var form = FormApp.openByUrl('https://docs.google.com/forms/.../edit');
var items = form.getItems();
while(items.length > 0)
{
form.deleteItem(items.pop());
}
This works for me.
When you are checking the length of a variable=form.getItems() in a loop, its going to through some error because the length of that is not changing and the loops end up being infinite and throughing error.
So, heres my solution to the problem:
for(;form.getItems().length>0;)
{
form.deleteItem(0);
}
I ran into the same problem. However, I have fixed it by iterating in the reverse order.
var form=FormApp.openByUrl('form url here');
var Items=form.getItems();
var len=Items.length;
for (var i=Items.length-1;i>2;i--){ //Delete every item except first three items
form.deleteItem(i)
}
There are many options for looping over all form items and removing each, the most succinct being:
With Chrome V8 runtime
form.getItems().forEach(form.deleteItem)
Without Chrome V8 runtime
for each (var item in form.getItems()) {
form.deleteItem(item);
}
EDIT: Pete provided a really good solution that works when the fields contain numbers, however I need to be able to sort strings too - any ideas?
I'm trying to write a javascript sorting algorithm that will sort a table based on the column clicked - I know this is semi-reinventing the wheel but the design is too complex for me to try and insert some other plugin etc.
Some columns are text, some columns are numbers.
Clicking a column calls: sort(X,Y). X is the column number so we know which cells to compare for the sort. Y is the mode, i.e. ascending or descending.
The code for the sort function is:
function sort(field, mode) {
var tabrows = 0;
$(".data tr").each(function() { if($(this).hasClass("hdr")) { } else {tabrows++;} });
var swapped;
do {
swapped = false;
for (var i=0;i< tabrows; i++) {
var j = i + 3;
var k = i + 4;
var row1 = $(".data tr:nth-child("+j+")");
var row2 = $(".data tr:nth-child("+k+")");
var field1 = row1.find("td:eq("+field+")").text();
var field2 = row2.find("td:eq("+field+")").text();
if(shouldswap(field1, field2, mode)) {
swaprows(row1, row2);
swapped = true;
}
}
} while (swapped);
}
The shouldswap function is as follows:
function shouldswap(field1, field2,mode) {
if(field1 > field2) {
if(mode==1) {
return true;
} else {
return false;
}
}
return false;
}
Code for swaprows function:
function swaprows(row1, row2) {
row2.insertBefore(row1);
}
Can anyone see why this would cause the browser to freeze/lockup. I've been working on this for quite a while so I think a fresh pair of eyes may point out something silly! Any help is appreciated :)
The problem might be that you're calling the jQuery constructor a bunch of times and doing heavy operations on it (e.g. using .find() with complex selectors). Therefore, your function is just slow and that's probably the issue.
The good news is that JavaScript has a native implementation of QuickSort (a very fast sorting function) that will probably take care of your needs. When combined with a reduction in expensive calls, your code should end up being enormously more efficient. I'd change your code to look like this:
var sortByField = function(field, mode) {
var numExp = /^-?\d*\.?\d+$/;
var $rows = $(".data tr:not(.hdr)"), $table = $(".data");
$rows.each(function () {
this.fieldVal = $(this).find("td:eq("+field+")").text();
if(numExp.test(this.fieldVal)) { //if field is numeric, convert it to a number
this.fieldVal = +this.fieldVal;
}
}).sort(function (a, b) {
if (mode === 1) {
return (a.fieldVal > b.fieldVal) ? -1 : 1;
}
return (a.fieldVal < b.fieldVal) ? -1 : 1;
}).detach().each(function () {
$(this).appendTo($table);
});
};
This won't work well with multiple tables on one page (because it assumes everything is on the same table). So if you want to do that, you should pass in the table or table selector as a parameter. But that's an easy fix to make. You can see my solution in action here:
http://jsfiddle.net/r8wtK/ (updated)
It should be far more efficient than your code and should reduce "freezing" by quite a bit (ore even entirely).
UPDATE:
The OP noted that some fields may contain strings. Doing a string comparison on numbers is bad because it returns a lexicographical ordering (e.g. "10" < "2"). So I added a test to see if the data appear to be numeric before doing the sort.
Could it be that you're adding 3 and 4 to i in order to get your row indices? So when i gets to (tabrows-1), it appears that it will be trying to access rows with index of (tabrows+2) and (tabrows+3). If I understand your logic correctly, these are out of bounds, so row1, row2, field1 and field2 will be empty. Therefore, if you're in mode==1, I think this will make it so that your algorithm attempts to swap these two non-existent rows and keeps comparing for infinity. Does this make sense, or am I misunderstanding your logic?
If that's the case, I think you just need to change your for loop to:
for (var i=0;i< tabrows-4; i++) {
// your code
}
What is the purpose of adding 3 to j and 4 to k anyway? Do you have 3 rows of data at the top that you don't want to compare?
Situation
I'm currently writing a javascript widget that displays a random quote into a html element. the quotes are stored in a javascript array as well as how many times they've been displayed into the html element. A quote to be displayed cannot be the same quote as was previously displayed. Furthermore the chance for a quote to be selected is based on it's previous occurences in the html element. ( less occurrences should result in a higher chance compared to the other quotes to be selected for display.
Current solution
I've currently made it work ( with my severely lacking javascript knowledge ) by using a lot of looping through various arrays. while this currently works ( !! ) I find this solution rather expensive for what I want to achieve.
What I'm looking for
Alternative methods of removing an array element from an array, currently looping through the entire array to find the element I want removed and copy all other elements into a new array
Alternative method of calculating and selecting a element from an array based on it's occurence
Anything else you notice I should / could do different while still enforcing the stated business rules under Situation
The Code
var quoteElement = $("div#Quotes > q"),
quotes = [[" AAAAAAAAAAAA ", 1],
[" BBBBBBBBBBBB ", 1],
[" CCCCCCCCCCCC ", 1],
[" DDDDDDDDDDDD ", 1]],
fadeTimer = 600,
displayNewQuote = function () {
var currentQuote = quoteElement.text();
var eligibleQuotes = new Array();
var exclusionFound = false;
for (var i = 0; i < quotes.length; i++) {
var iteratedQuote = quotes[i];
if (exclusionFound === false) {
if (currentQuote == iteratedQuote[0].toString())
exclusionFound = true;
else
eligibleQuotes.push(iteratedQuote);
} else
eligibleQuotes.push(iteratedQuote);
}
eligibleQuotes.sort( function (current, next) {
return current[1] - next[1];
} );
var calculatePoint = eligibleQuotes[0][1];
var occurenceRelation = new Array();
var relationSum = 0;
for (var i = 0; i < eligibleQuotes.length; i++) {
if (i == 0)
occurenceRelation[i] = 1 / ((calculatePoint / calculatePoint) + (calculatePoint / eligibleQuotes[i+1][1]));
else
occurenceRelation[i] = occurenceRelation[0] * (calculatePoint / eligibleQuotes[i][1]);
relationSum = relationSum + (occurenceRelation[i] * 100);
}
var generatedNumber = Math.floor(relationSum * Math.random());
var newQuote;
for (var i = 0; i < occurenceRelation.length; i++) {
if (occurenceRelation[i] <= generatedNumber) {
newQuote = eligibleQuotes[i][0].toString();
i = occurenceRelation.length;
}
}
for (var i = 0; i < quotes.length; i++) {
var iteratedQuote = quotes[i][0].toString();
if (iteratedQuote == newQuote) {
quotes[i][1]++;
i = quotes.length;
}
}
quoteElement.stop(true, true)
.fadeOut(fadeTimer);
setTimeout( function () {
quoteElement.html(newQuote)
.fadeIn(fadeTimer);
}, fadeTimer);
}
if (quotes.length > 1)
setInterval(displayNewQuote, 10000);
Alternatives considered
Always chose the array element with the lowest occurence.
Decided against this as this would / could possibly reveal a too obvious pattern in the animation
combine several for loops to reduce the workload
Decided against this as this would make the code to esoteric, I'd probably wouldn't understand the code anymore next week
jsFiddle reference
http://jsfiddle.net/P5rk3/
Update
Rewrote my function with the techniques mentioned, while I fear that these techniques still loop through the entire array to find it's requirements, at least my code looks cleaner : )
References used after reading the answers here:
http://www.tutorialspoint.com/javascript/array_map.htm
http://www.tutorialspoint.com/javascript/array_filter.htm
http://api.jquery.com/jQuery.each/
I suggest array functions that are mostly supported (and easily added if not):
[].splice(index, howManyToDelete); // you can alternatively add extra parameters to slot into the place of deletion
[].indexOf(elementToSearchFor);
[].filter(function(){});
Other useful functions include forEach and map.
I agree that combining all the work into one giant loop is ugly (and not always possible), and you gain little by doing it, so readability is definitely the winner. Although you shouldn't need too many loops with these array functions.
The answer that you want:
Create an integer array that stores the number of uses of every quote. Also, a global variable Tot with the total number of quotes already used (i.e., the sum of that integer array). Find also Mean, as Tot / number of quotes.
Chose a random number between 0 and Tot - 1.
For each quote, add Mean * 2 - the number of uses(*1). When you get that that value has exceeded the random number generated, select that quote.
In case that quote is the one currently displayed, either select the next or the previous quote or just repeat the process.
The real answer:
Use a random quote, at the very maximum repeat if the quote is duplicated. The data usages are going to be lost when the user reloads/leaves the page. And, no matter how cleverly have you chosen them, most users do not care.
(*1) Check for limits, i.e. that the first or last quota will be eligible with this formula.
Alternative methods of removing an array element from an array
With ES5's Array.filter() method:
Array.prototype.without = function(v) {
return this.filter(function(x) {
return v !== x;
});
};
given an array a, a.without(v) will return a copy of a without the element v in it.
less occurrences should result in a higher chance compared to the other quotes to be selected for display
You shouldn't mess with chance - as my mathematician other-half says, "chance doesn't have a memory".
What you're suggesting is akin to the idea that numbers in the lottery that haven't come up yet must be "overdue" and therefore more likely to appear. It simply isn't true.
You can write functions that explicitly define what you're trying to do with the loop.
Your first loop is a filter.
Your second loop is a map + some side effect.
I don't know about the other loops, they're weird :P
A filter is something like:
function filter(array, condition) {
var i = 0, new_array = [];
for (; i < array.length; i += 1) {
if (condition(array[i], i)) {
new_array.push(array[i]);
}
}
return new_array;
}
var numbers = [1,2,3,4,5,6,7,8,9];
var even_numbers = filter(numbers, function (number, index) {
return number % 2 === 0;
});
alert(even_numbers); // [2,4,6,8]
You can't avoid the loop, but you can add more semantics to the code by making a function that explains what you're doing.
If, for some reason, you are not comfortable with splice or filter methods, there is a nice (outdated, but still working) method by John Resig: http://ejohn.org/blog/javascript-array-remove/