I'm making a custom function in google sheets which counts the number of occurrences of all items in a given array in a given range of cells.
The way I'm given to understand google sheets functions work are that the range you give is turned into a two-dimensional array of the items in the cells. So range A4:B5 would be transmitted to the function as
[[the contents of A4, the contents of B4],
[the contents of A5, the contents of B5]
The next input is a list of the items to check for in those cells. From what I could find online, arrays are given in google sheets by using brackets like these {}. the function I created is given below. I have never used javascript before but I know other languages and I just googled how to use for loops and if statements to create the function, so I'm certain the error is due to something simple that I don't know about or missed.
function count_if_in_set(range, given_list) {
let counter = 0;
for (dim_1 of range) {
for (dim_2 of dim_1) {
for (item of given_list) {
if (item == dim_2) {
counter += 1
}
}
}
}
return counter
}
When I try to use this function in google sheets with the following input: =count_if_in_set(Z30:Z33, {1}), I receive the following error: TypeError: given_list is not iterable (line 5).
The contents of cells Z30 to Z33 are the integers 1, 2, 3, 3 which should be given to the function as the following 2-dimensional array: [[1], [2], [3], [3]]
The problem is that the list [1] is not iterable. I have 2 hypotheses as to why this is:
I coded something wrong because I'm very new to Javascript
The input {1} is not transmitted to a list when google sheets gives it to the function
To check if it was the former, I went through all the aspects of my function. I first checked if you have to declare the type of variable it was when you created the function, but according to what I saw when I googled it you don't. I then changed all my for (a of b) to for (let a of b) but that did nothing to help, and after that I was stuck.
To try and solve it in the case it was a problem with giving the code an array, I tried changing my input from =count_if_in_set(Z30:Z33, {1}) to =count_if_in_set(Z30:Z33, [1]), but that threw up a formula parse error so I knew that wasn't it, and I tried changing the input to =count_if_in_set(Z30:Z33, (1)) but that returned the same error. And after that I was stuck and had no more ideas.
You can get the same result with a plain vanilla spreadsheet formula, like this:
=arrayformula( countif(Z30:Z43, { 1, 2, 3 }) )
To get just the grand total, use this:
=arrayformula( sum( countif(Z30:Z43, { 1, 2, 3 }) ) )
To count how many cells have a text string that contains one of the search keys, use this:
=arrayformula( sum( countif( Z30:Z43, "*" & { "a", "b" } & "*" ) ) )
If you need to use a custom function for some reason, try something this to get started:
function count_if_in_set(values, given_list) {
let counter = 0;
values.map(row => row.map(value =>
counter += (given_list.indexOf(value) !== -1)
));
return counter;
}
This is really an anti-pattern, because the map result is not used for anything. People would tend to use Array.reduce(), but the map-map pattern may be easier to follow, and it is the one typically employed in custom functions that most often do not aggregate the result but return exactly one value per argument value.
Some of the best resources for learning Google Apps Script include the Beginner's Guide, the New Apps Script Editor guide, the Fundamentals of Apps Script with Google Sheets codelab, the Extending Google Sheets page, javascript.info, Mozilla Developer Network and Apps Script at Stack Overflow.
Try this:
Just looking for the number of 1,2,3,4,5,6,7,8 or 9 in the selected range and return the item and count
function checkforitems(a, b) {
let obj = {pA:[]};
Logger.log(a);
Logger.log(b);
let arr = b[0];//b enters as a 2d array with a single element
//collect counts with a pivot table
a.forEach(r => {
r.forEach(c => {
let index = arr.indexOf(c);
if(~index) {
if(!obj.hasOwnProperty(arr[index])) {
obj[arr[index]]=1;
obj.pA.push(arr[index]);//collect elements as an array
} else {
obj[arr[index]]+=1;
}
}
});
});
let l = '';
//obj.pA.sort((x,y) => x - y);//if searching for numbers you can use this to sort them before displaying them
obj.pA.forEach(e => {
l += `${e}-${obj[e]}\n`;
});
return l;
}
My Test Sheet:
COL1
COL2
COL3
COL4
COL5
COL6
COL7
COL8
COL9
COL10
1
17
8
10
2
7
4
19
12
11
8
13
7
1
6
14
8
19
15
1
17
15
15
6
7
3
3
17
8
12
8
2
17
9
9
7
15
16
19
11
14
11
19
0
15
4
16
11
1
11
1
3
3
19
3
1
5
4
3
16
10
8
8
2
17
18
0
1
17
6
1
0
10
18
12
16
11
4
7
13
10
18
6
12
12
5
3
11
9
5
13
2
2
8
5
4
8
12
18
2
0
18
18
18
17
4
6
14
8
8
1
11
12
1
15
17
18
3
0
6
19
5
17
11
12
9
12
1
6
15
12
5
7
1
14
9
4
4
18
12
3
1
11
8
11
9
17
6
12
5
11
12
16
5
5
5
6
12
3
5
16
0
18
14
8
4
16
0
10
0
15
13
4
17
14
10
9
9
2
4
13
12
11
15
12
18
0
8
19
19
3
1
0
3
1
16
18
6
1
2
My formula (L22):
=checkforitems(A2:J21,{1,2,3,4,5,6,7,8,9})
returned result:
1-16
8-14
2-8
7-6
4-11
6-10
3-12
9-8
5-11
Test Sheet With Results:
To anyone looking for a way to make that function work:
I did the same thing but changed it slightly so the second input was a range of cells which contained the range I wanted to search through
I'm writing a webpage where you can upload text files and then do some analysis on the file.
the text file is formatted like this:
0 1 475
1 2 437
2 3 553
3 4 500
4 5 612
5 6 491
6 7 444
7 8 544
8 9 491
9 10 595
'*' is only used to make a list on stack-overflow not actually in the text file
I only need to extract the third column(three digits)
and place the values in a variable that I can later use for my analysis.
How can I do this? I can't seem to find an answer with the javascript.
thanks
Here is a way of doing it:
document.querySelector('#fileInput').addEventListener('change', (e) => {
readFile(e.target.files[0]);
});
function readFile(file) {
const reader = new FileReader();
reader.readAsText(file);
reader.onload = function() {
const values = reader.result.split('\n').map(line => (+line.split(' ')[2]));
console.log(values);
};
}
const fileContent = `0 1 475
1 2 437
2 3 553
3 4 500
4 5 612
5 6 491
6 7 444
7 8 544
8 9 491
9 10 595`;
const blob = new Blob([fileContent], {type: 'text/plain'});
readFile(blob);
<input id="fileInput" type="file" onchange="readFile(this.files[0])">
In this example I've used a Blob to imitate a file but you can also use the <input type="file" /> to test the function with a real file.
What this does is to use a FileReader to read a file as text and then parse the content of the file by creating an array that has each line of text in the file as an element (reader.result.split('\n')) and then mapping that array to only keep the last number. That is achieved by splitting the line on every white space character and converting to a number and keeping only the third element from the resulting array (+line.split(' ')[2], the [2] selects the third element and + converts that element to a number).
You could do something like this, where the map function transforms each string into an integer:
const input = ["0 1 475",
"1 2 437",
"2 3 553",
"3 4 500",
"4 5 612",
"5 6 491",
"6 7 444",
"7 8 544",
"8 9 491",
"9 10 595"];
const output = input.map((str) => parseInt(str.split(" ")[2]))
console.log(output);
You can use regular expression:
let str= `0 1 475
1 2 437
2 3 553
3 4 500
4 5 612
5 6 491
6 7 444
7 8 544
8 9 491
9 10 595`
let re=/\d{3}$/gm
console.log(str.match(re))
I want to scrape historical results of South African LOTTO draws (especially Total Pool Size, Total Sales, etc.) from the South African National Lottery website. By default one sees links to results for the last ten draws, or one can select a date range to pull up a larger set of links to draws (which will still display only ten per page).
Hovering in the browser over a link e.g. 'LOTTO DRAW 2012' we see javascript:void(); so it is clear that the draw results will be rendered using Javascript. Reading advice on an R Web Scraping Cheat Sheet, I realized that I needed to open Google Chrome Developer tools, then open Network tab, and then click the link to the draw 'LOTTO DRAW 2012'. When I did so, I could see that this url is being called with an initiator
When I right-click on the initiator and select 'Copy Response', I can see the data I need inside a 'drawDetails' object in what appears to be JSON code.
{"code":200,"message":"OK","data":{"drawDetails":{"drawNumber":"2012","drawDate":"2020\/04\/11","nextDrawDate":"2020\/04\/15","ball1":"48","ball2":"6","ball3":"43","ball4":"41","ball5":"25","ball6":"45","bonusBall":"38","div1Winners":"1","div1Payout":"10546013.8","div2Winners":"0","div2Payout":"0","div3Winners":"28","div3Payout":"7676.4","div4Winners":"62","div4Payout":"2751.4","div5Winners":"1389","div5Payout":"206.3","div6Winners":"1872","div6Payout":"133","div7Winners":"28003","div7Payout":"50","div8Winners":"20651","div8Payout":"20","rolloverAmount":"0","rolloverNumber":"0","totalPrizePool":"13280236.5","totalSales":"11610950","estimatedJackpot":"2000000","guaranteedJackpot":"0","drawMachine":"RNG2","ballSet":"RNG","status":"published","winners":52006,"millionairs":1,"gpwinners":"52006","wcwinners":"0","ncwinners":"0","ecwinners":"0","mpwinners":"0","lpwinners":"0","fswinners":"0","kznwinners":"0","nwwinners":"0"},"totalWinnerRecord":{"lottoMillionairs":28716702,"lottoWinners":337285646,"ithubaMillionairs":135763,"ithubaWinners":305615802}},"videoData":[{"id":"1049","listid":"1","parentid":"1","videosource":"youtube","videoid":"chHfFxVi9QI","imageurl":"","title":"LOTTO, LOTTO PLUS 1 AND LOTTO PLUS 2 DRAW 2012 (11 APRIL 2020)","description":"","custom_imageurl":"","custom_title":"","custom_description":"","specialparams":"","lastupdate":"0000-00-00 00:00:00","allowupdates":"1","status":"0","isvideo":"1","link":"https:\/\/www.youtube.com\/watch?v=chHfFxVi9QI","ordering":"10001","publisheddate":"2020-04-11 20:06:17","duration":"182","rating_average":"0","rating_max":"0","rating_min":"0","rating_numRaters":"0","statistics_favoriteCount":"0","statistics_viewCount":"329","keywords":"","startsecond":"0","endsecond":"0","likes":"6","dislikes":"0","commentcount":"0","channel_username":"","channel_title":"","channel_subscribers":"9880","channel_subscribed":"0","channel_location":"","channel_commentcount":"0","channel_viewcount":"0","channel_videocount":"1061","channel_description":"","channel_totaluploadviews":"0","alias":"lotto-lotto-plus-1-and-lotto-plus-2-draw-2012-11-april-2020","rawdata":"","datalink":"https:\/\/www.googleapis.com\/youtube\/v3\/videos?id=chHfFxVi9QI&part=id,snippet,contentDetails,statistics&key=AIzaSyC1Xvk2GUdb_N3UiFtjsgZ-uMviJ_8MFZI"}]}
It is a POST type request, and so I tried to follow this answer, but cannot find onclick values indicating the data submitted with the form. Moreover, the request URL for 'LOTTO DRAW 2012' is identical to that for 'LOTTO DRAW 2011', so there is no unique identifier for the particular draw being passed with the URL itself. Thus it is not clear to me how the unique request for the results of a particular draw is made.
Hence, the smaller question is, given a particular LOTTO draw number or draw date, how does one find out the unique identifier that is used to make the POST request for the data pertaining to that draw specifically?
The larger question is, if one is able to obtain such unique identifiers for all the historical draws, how can one generate the JSON drawDetails object for all the historical draws in turn, or otherwise complete the scraping operation?
You are right - the contents on the page are updated by javascript via an ajax request. The server returns a json string in response to an http POST request. With POST requests, the server's response is determined not only by the url you request, but by the body of the message you send to the server. In this case, your body is a simple form with 3 fields: gameName, which is always LOTTO, isAjax which is always true, and drawNumber, which is the field you want to vary.
If you are using httr, you specify these fields as a named list in the body parameter of the POST function.
Once you have the response for each draw, you will want to parse the json into an R-friendly format such as a list or data frame using a library such as jsonlite. From looking at the structure of this particular json, it makes most sense to extract the component $data$drawDetailsand make that a one-row dataframe. This will allow you to bind several draws together into a single data frame.
Here is a function that does all that for you:
lotto_details <- function(draw_numbers)
{
do.call("rbind", lapply(draw_numbers, function(x)
{
res <- httr::POST(paste0("https://www.nationallottery.co.za/index.php",
"?task=results.redirectPageURL&",
"Itemid=265&option=com_weaver&",
"controller=lotto-history"),
body = list(gameName = "LOTTO", drawNumber = x, isAjax = "true"))
as.data.frame(jsonlite::fromJSON(httr::content(res, "text"))$data$drawDetails)
}))
}
Which you use like this:
lotto_details(2009:2012)
#> drawNumber drawDate nextDrawDate ball1 ball2 ball3 ball4 ball5 ball6
#> 1 2009 2020/04/01 2020/04/04 51 15 7 32 42 45
#> 2 2010 2020/04/04 2020/04/08 43 4 21 24 10 3
#> 3 2011 2020/04/08 2020/04/11 42 43 8 18 2 29
#> 4 2012 2020/04/11 2020/04/15 48 6 43 41 25 45
#> bonusBall div1Winners div1Payout div2Winners div2Payout div3Winners
#> 1 1 0 0 0 0 21
#> 2 22 0 0 0 0 31
#> 3 34 0 0 0 0 21
#> 4 38 1 10546013.8 0 0 28
#> div3Payout div4Winners div4Payout div5Winners div5Payout div6Winners
#> 1 8455.3 60 2348.7 1252 189 1786
#> 2 6004.3 71 2080.6 1808 137.3 2352
#> 3 8584.5 60 2384.6 1405 171.1 2079
#> 4 7676.4 62 2751.4 1389 206.3 1872
#> div6Payout div7Winners div7Payout div8Winners div8Payout rolloverAmount
#> 1 115.2 24664 50 19711 20 3809758.17
#> 2 91.7 35790 50 25981 20 5966533.86
#> 3 100.5 27674 50 21895 20 8055430.87
#> 4 133 28003 50 20651 20 0
#> rolloverNumber totalPrizePool totalSales estimatedJackpot
#> 1 2 6198036.67 9879655 6000000
#> 2 3 9073426.56 11696905 8000000
#> 3 4 10649716.37 10406895 10000000
#> 4 0 13280236.5 11610950 2000000
#> guaranteedJackpot drawMachine ballSet status winners millionairs
#> 1 0 RNG2 RNG published 47494 0
#> 2 0 RNG2 RNG published 66033 0
#> 3 0 RNG2 RNG published 53134 0
#> 4 0 RNG2 RNG published 52006 1
#> gpwinners wcwinners ncwinners ecwinners mpwinners lpwinners fswinners
#> 1 47494 0 0 0 0 0 0
#> 2 66033 0 0 0 0 0 0
#> 3 53134 0 0 0 0 0 0
#> 4 52006 0 0 0 0 0 0
#> kznwinners nwwinners
#> 1 0 0
#> 2 0 0
#> 3 0 0
#> 4 0 0
Created on 2020-04-13 by the reprex package (v0.3.0)
The question already has a satisfactory answer (see above) that I've accepted. I simultaneously arrived at a nearly identical solution; I add it here only because it explicitly covers the full range of available draw numbers and will automatically detect the most recent draw number so that the code can be run 'as is' in the future, provided the National Lottery website design remains the same.
theurl <- "https://www.nationallottery.co.za/index.php?task=results.redirectPageURL&Itemid=265&option=com_weaver&controller=lotto-history"
x <- rvest::html_text(xml2::read_html(theurl))
preceding_string <- "LOTTO, LOTTO PLUS 1 AND LOTTO PLUS 2 DRAW "
drawnums <- as.integer(vapply(gregexpr(preceding_string, x)[[1]] + nchar(preceding_string),
function(k) substr(x, start = k, stop = k + 3), NA_character_))
drawnumrange <- 1506:max(drawnums)
response <- lapply(drawnumrange, function(d) httr::POST(url = theurl,
body = list(gameName = "LOTTO", drawNumber = as.character(d), isAjax =
"true"), encode = "form"))
jsondat <- lapply(response, function(r) jsonlite::parse_json(r)$data$drawDetails)
lottotable <- as.data.frame(do.call(rbind, jsondat))
numericcols <- c(1, 4:32, 36:37)
lottotable[numericcols] <- sapply(lottotable[numericcols], as.numeric)
xlsx::write.xlsx2(lottotable[1:37], "lottotable.xlsx", row.names = FALSE)
I do my calculations like this:
117^196
I get:
177
Now what I want to do is to get 117 back so I need to make a replace
(replace)^196 = 117
Whats the opposite operation from the xor operator?
The opposite of xor is xor :). If you xor something twice (a^b)^b == a.
This is relatively easy to show. For each bit:
1 ^ 1 = 0
1 ^ 0 = 1
0 ^ 1 = 1
0 ^ 0 = 0
Doing this on any pair of numbers a,b, it's easy to see that
a^b xor'd by either a or b yields the other (xor a yields b, and vice versa)
1 2 filter result
0^0^0 = 0
0^1^0 = 1
0^1^1 = 0
1^0^0 = 1
1^0^1 = 0
1^1^1 = 1
it's just xor it self.
like +'s opposite is -
xor's opposite is xor
Just use the result that you got: 177
117 ^ 196 = 177 | () ^ 196
117 ^ 196 ^ 196 = 177 ^ 196 | self-inverse
117 ^ 0 = 177 ^ 196 | neutral element
117 = 177 ^ 196
XOR has three important properties. It is
associative
commutative
self-inverse
This means that a value is its own inverse:
a^a = 0
Since it is also both commutative and associative, you can rearrange and xor-expression containing an event amount of the same operands like this:
a^O^b^c^O^d = O^O^a^b^c^d = 0^a^b^c^d = a^b^c^d
You could say that operands that appear an even amount of time "cancel each other out".