We Keep Coding

Creating large array/data from seed/key

firstly the language Im writing in is node (javascript) but really Im looking for the computer science behind it, and how to actually do it, not just the code.
Basically what I have is a 2,000 x 2,000 two dimensional array (what I mean by that is that every entry in the 2,000 entry long array has its own 2,000 entries). Inside this array I have values 0, 1, 2 3, etc. They are spaced out different, with different rarities as to how common each appears. What I want to do is generate this array based on a key, idc how long the key/seed is, just a reasonable length that can get the job done. I want the same key to generate the same array if its the same key and a different one if its a different key. Basically take a key and generate longer data from it but for no recognizable patterns to appear in this data.
My thoughts on this is to have a key that is a decimal of some sort I multiply against a bunch of constants to get a location in the array, but tbh I really have no Idea where to start. Essentially its like how minecraft takes a seed and turns it into map and the same seed will again generate an identical map.

Any random number generator (RNG) that can be seeded will give the same series of random values for a given seed & should have no determinable pattern. Unfortunately, the default RND for javascript is not seedable; as per this SE post, you will need to write your own or use a someone else's.
Once you have a seedable RNG, for each entry, first get a random value & then convert the random value into the desired output value. There's a number of different ways to do the conversion; if you only have a few, I would do something like this (assumes random_value is between 0 & 1):
if(rand_value <= 70){
output_value = 1;
}
else if(rand_value <= 90){
output_value = 2;
}
else if(rand_value <= 97){
output_value = 3;
}
else {
output_value = 4
}
This gives a 70%, 20%, 7% & 3% to get a 1,2,3 or 4 respectively; adjust the values as needed. Note: if you have many output values you should edit your question to reflect this, as there are cleaner ways to solve this than a giant if else block.

Best ways to get random items from an array in javascript

Considering the performance, what's the best way to get random subset from an array?
Say we get an array with 90000 items, I wanna get 10000 random items from it.
One approach I'm thinking about is to get a random index from 0 to array.length and then remove the selected one from the original array by using Array.prototype.splice. Then get the next random item from the rest.
But the splice method will rearrange the index of all the items after the one we just selected and move them forward on step. Doesn't it affect the performance?
Items may duplicates, but what we select should not. Say we've selected index 0, then we should only look up the rest 1~89999.

If you want a subset of the shuffled array, you do not need to shuffle the whole array. You can stop the classic fisher-yates shuffle when you have drawn your 10000 items, leaving the other 80000 indices untouched.

I would first randomize the whole array then splice of a 10000 items.
How to randomize (shuffle) a JavaScript array?
Explains a good way to randomize a array in javascript

A reservoir sampling algorithm can do this.
Here's an attempt at implementing Knuth's "Algorithm S" from TAOCP Volume 2 Section 3.4.2:
function sample(source, size) {
var chosen = 0,
srcLen = source.length,
result = new Array(size);
for (var seen = 0; chosen < size; seen++) {
var remainingInput = srcLen - seen,
remainingOutput = size - chosen;
if (remainingInput*Math.random() < remainingOutput) {
result[chosen++] = source[seen];
}
}
return result;
}
Basically it makes one pass over the input array, choosing or skipping items based on a function of a random number, the number of items remaining in the input, and the number of items remaining to be required in the output.
There are three potential problems with this code: 1. I may have mucked it up, 2. Knuth calls for a random number "between zero and one" and I'm not sure if this means the [0, 1) interval JavaScript provides or the fully closed or fully open interval, 3. it's vulnerable to PRNG bias.
The performance characteristics should be very good. It's O(srcLen). Most of the time we finish before going through the entire input. The input is accessed in order, which is a good thing if you are running your code on a computer that has a cache. We don't even waste any time reading or writing elements that don't ultimately end up in the output.
This version doesn't modify the input array. It is possible to write an in-place version, which might save some memory, but it probably wouldn't be much faster.

Random Selections from Arrays

I am creating a game in Unity. I'm in the planning stage of it right now, but I'm trying to work out a problem I've come to. The game involves randomly selected objects from three different categories falling and the player has to catch the particular objects in particular bins.
So here's what needs to happen:
One or two of the arrays must be randomly chosen, one or two of the objects within that particular array must be chosen, no more than four objects can fall at once, the different objects must fall from different places and fall at different times.
Now I have a clip of code that I got from another project I did that's written in JavaScript (which is what I've been using, but I could also do it in Boo or C++) that solves part of the last point. It chooses a random location along the x access and then has the object fall until y=0, and then it resets.
function Update()
{
transform.position.y -= 50 * Time.deltaTime;
if(transform.position.y < 0)
{
transform.position.y = 50;
transform.position.x = Random.Range(0,60);
transform.position.z = -16;
}
}
I'm going to rewrite part of it to say that it will reset after it hits a particular collider, yields for a short time period, and find then a new random and drop that instead. But what I'm having problems with is the actual randomizing of the objects. I have six objects in each of the three arrays, and I've looked for codes where something is chosen from an array by numerical value, but nothing about randomly choosing one of the arrays and then choosing something within the random array. Neither have I found anything about the random selection in JavaScript, Boo, or C++.
Any information on this code would be helpful, thanks in advance!

To select one object at random from one of three arrays at random, you better work with an array of array. You then will need to generate two random numbers and store them as indexes to the array of arrays.
so instead of three different arrays, initialize a single array
var a = [];
a.push([1,2,3]);
a.push([10,20]);
a.push([100,200,300,400]);
and then
var i = Math.floor(Math.random()*a.length);
var j = Math.floor(Math.random()*a[i].length);
var o = a[i][j];

Getting the index of an object in an ordered list in Firebase

I'm building a leaderboard using Firebase. The player's position in the leaderboard is tracked using Firebase's priority system.
At some point in my program's execution, I need to know what position a given user is at in the leaderboard. I might have thousands of users, so iterating through all of them to find an object with the same ID (thus giving me the index) isn't really an option.
Is there a more performant way to determine the index of an object in an ordered list in Firebase?
edit: I'm trying to figure out the following:
/
---- leaderboard
--------user4 {...}
--------user1 {...}
--------user3 {...} <- what is the index of user3, given a snapshot of user3?
--------...

If you are processing tens or hundreds of elements and don't mind taking a bandwidth hit, see Katos answer.
If you're processing thounsands of records, you'll need to follow an approach outlined in principle in pperrin's answer. The following answer details that.
Step 1: setup Flashlight to index your leaderboard with ElasticSearch
Flashlight is a convenient node script that syncs elasticsearch with Firebase data.
Read about how to set it up here.
Step 2: modify Flashlight to allow you to pass query options to ElasticSearch
as of this writing, Flashlight gives you no way to tell ElasticSearch you're only interested in the number of documents matched and not the documents themselves.
I've submitted this pull request which uses a simple one-line fix to add this functionality. If it isn't closed by the time you read this answer, simply make the change in your copy/fork of flashlight manually.
Step 3: Perform the query!
This is the query I sent via Firebase:
{
index: 'firebase',
type: 'allTime',
query: {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"range": {
"points": {
"gte": minPoints
}
}
}
}
},
options: {
"search_type": "count"
}
};
Replace points with the name of the field tracking points for your users, and minPoints with the number of points of the user whose rank you are interested in.
The response will look something like:
{
max_score: 0,
total: 2
}
total is the number of users who have the same or greater number of points -- in other words, the user's rank!

Since Firebase stores object, not arrays, the elements do not have an "index" in the list--JavaScript and by extension JSON objects are inherently unordered. As explained in Ordered Docs and demonstrated in the leaderboard example, you accomplish ordering by using priorities.
A set operation:
var ref = new Firebase('URL/leaderboard');
ref.child('user1').setPriority( newPosition /*score?*/ );
A read operation:
var ref = new Firebase('URL/leaderboard');
ref.child('user1').once('value', function(snap) {
console.log('user1 is at position', snap.getPriority());
});

To get the info you want, at some point a process is going to have to enumerate the nodes to count them. So the question is then where/when the couting takes place.
Using .count() in the client will mean it is done every time it is needed, it will be pretty accurate, but procesing/traffic heavy.
If you keep a separate index of the count it will need regular refreshing, or constant updating (each insert causeing a shuffling up of the remaining entries).
Depending on the distribution and volume of your data I would be tempted to go with a background process that just updates(/rebuilds) the index every (say) ten or twenty additions. And indexes every (say) 10 positions.
"Leaderboard",$UserId = priority=$score
...
"Rank",'10' = $UserId,priority=$score
"Rank",'20' = $UserId,priority=$score
...
From a score you get the rank within ten and then using a startat/endat/count on your "Leaderboard" get it down to the unit.
If your background process is monitoring the updates to the leaderboard, it could be more inteligent about its updates to the index either updating only as requried.

I know this is an old question, but I just wanted to share my solutions for future reference. First of all, the Firebase ecosystem has changed quite a bit, and I'm assuming the current best practices (i.e. Firestore and serverless functions). I personally considered these solutions while building a real application, and ended up picking the scheduled approximated ranks.
Live ranks (most up-to-date, but expensive)
When preparing a user leaderboard I make a few assumptions:
The leaderboard ranks users based on a number which I'll call 'score' from now on
New users rank lowest on the leaderboard, so upon user creation, their rank is set to the total user count (with a Firebase function, which sets the rank, but also increases the 'total user' counter by 1).
Scores can only increase (with a few adaptations decreasing scores can also be supported).
Deleted users keep a 'ghost' spot on the leaderboard.
Whenever a user gets to increase their score, a Firebase function responds to this change by querying all surpassed users (whose score is >= the user's old score but < the user's new score) and have their rank decreased by 1. The user's own rank is increased by the size of the before-mentioned query.
The rank is now immediately available on client reads. However, the ranking updates inside of the proposed functions are fairly read- and write-heavy. The exact number of operations depends greatly on your application, but for my personal application a great frequency of score changes and relative closeness of scores rendered this approach too inefficient. I'm curious if anyone has found a more efficient (live) alternative.
Scheduled ranks (simplest, but expensive and periodic)
Schedule a Firebase function to simply sort the entire user collection by ascending score and write back the rank for each (in a batch update). This process can be repeated daily, or more frequent/infrequent depending on your application. For N users, the function always makes N reads and N writes.
Scheduled approximated ranks (cheapest, but non-precise and periodic)
As an alternative for the 'Scheduled ranks' option, I would suggest an approximation technique: instead of writing each user's exact rank upon for each scheduled update, the collection of users (still sorted as before) is simply split into M chunks of equal size and the scores that bound these chunks are written to a separate 'stats' collection.
So, for example: if we use M = 3 for simplicity and we read 60 users sorted by ascending score, we have three chunks of 20 users. For each of the (still sorted chunks) we get the score of the last (lowest score of chunk) and the first user (highest score of chunk) (i.e. the range that contains all scores of that chunk). Let's say that the chunk with the lowest scores has scores ranging from 20-120, the second chunk has scores from 130-180 and the chunk with the highest scores has scores 200-350. We now simply write these ranges to a 'stats' collection (the write-count is reduced to 1, no matter how many users!).
Upon rank retrieval, the user simply reads the most recent 'stats' document and approximates their percentile rank by comparing the ranges with their own score. Of course it is possible that a user scores higher than the greatest score or lower than the lowest score from the previous 'stats' update, but I would just consider them belonging to the highest scoring group and the lowest scoring group respectively.
In my own application I used M = 20 and could therefore show the user percentile ranks by 5% accuracy, and estimate even within that range using linear interpolation (for example, if the user score is 450 and falls into the 40%-45%-chunk ranging from 439-474, we estimate the user's percentile rank to be 40 + (450 - 439) / (474 - 439) * 5 = 41.57...%).
If you want to get real fancy you can also estimate exact percentile ranks by fitting your expected score distribution (e.g. normal distribution) to the measured ranges.
Note: all users DO need to read the 'stats' document to approximate their rank. However, in most applications not all users actually view the statistics (as they are either not active daily or just not interested in the stats). Personally, I also used the 'stats' document (named differently) for storing other DB values that are shared among users, so this document is already retrieved anyways. Besides that, reads are 3x cheaper than writes. Worst case scenario is 2N reads and 1 write.

What are the chances that JavaScript Math.Random() will create the same number twice in a row?

Is this correct? using - http://en.wikipedia.org/wiki/Binomial_probability
Looks like values are from .0000000000000000 to .9999999999999999
Probability of happening twice = p^2 = (1/9999999999999999)^2 = 1.0 e-32
I think I am missing something here?
Also, how does being a pseudo random number generator change this calculation?
Thank You.

In an ideal world Math.random() would be absolutely random, with one output being completely independent from another, which (assuming p=the probability of any given number being produced) results in a probably of p^2 for any value being repeated immediately after another (as others have already said).
In practice people want Math.random to be fast which means pseudo-random number generators are used by the engines. There are many different kinds of PRNG but the most basic is a linear congruential generator, which is basically a function along the lines of:
s(n + 1) = some_prime * s(n) + some_value mod some_other_prime
If such a generator is used then you won't see a value repeated until you've called random() some_other_prime times. You're guaranteed of that.
Relatively recently however it's become apparent that this kind of behaviour (coupled with seeding the PRNGs with the current time) could be used for some forms tracking have led to browsers doing a number of things that mean you can't assume anything about subsequent random() calls.

I think the probability of getting two numbers in a row is 1 divided by the range of the generator, assuming that it has a good distribution.
The reason for this is that the first number can be anything, and the second number needs to just be that number again, which means we don't care about the first number at all. The probability of getting the same number twice in a row is the same as the probability of getting any particular number once.
Getting some particular number twice in a row, e.g. two 0.5s in a row, would be p^2; however, if you just care about any number twice in a row, it's just p.

If the numbers were truly random, you'd expect them, indeed, to appear with probability 1/p, so twice that would be 1/p^2.
The value for p is not exactly the one you have though, because the numbers are being represented internally as binary. Figure out how many bits of mantissa the numbers have in javascript and use that for your combinatoric count.
The "pseudorandom" part is more interesting, because the properties of pseudorandom number generators vary. Knuth does some lovely work with that in Seminumerical Algorithms, but basically most usual PN generators have at least some spectral distributiuon. Cryptograp0hic PN generators are generally stronger.
Update: The amount of time shouldn't be significant. Whether it's a millisecond or a year, as long as you don't update the state The probabilities will stay the same.

The probability that you would get 2 given numbers is (1/p)^2, but the probability that you get 2 of same numbers (any) is 1/p. That is because the first number can be anything, and the second just needs to match that.

You can kind of find out, just let it run a few days :)
var last = 0.1;
var count = 0 | 0;
function rand(){
++count;
var num = Math.random();
if(num === last){
console.log('count: '+count+' num: '+num);
}
last = num;
}
while(true) rand();