Is it possible to seed the random number generator (Math.random) in JavaScript?
No, it is not possible to seed Math.random(). The ECMAScript specification is intentionally vague on the subject, providing no means for seeding nor require that browsers even use the same algorithm. So such a function must be externally provided, which thankfully isn't too difficult.
I've implemented a number of good, short and fast Pseudorandom number generator (PRNG) functions in plain JavaScript. All of them can be seeded and provide high quality numbers. These are not intended for security purposes--if you need a seedable CSPRNG, look into ISAAC.
First of all, take care to initialize your PRNGs properly. To keep things simple, the generators below have no built-in seed generating procedure, but accept one or more 32-bit numbers as the initial seed state of the PRNG. Similar or sparse seeds (e.g. a simple seed of 1 and 2) have low entropy, and can cause correlations or other randomness quality issues, sometimes resulting in the output having similar properties (such as randomly generated levels being similar). To avoid this, it is best practice to initialize PRNGs with a well-distributed, high entropy seed and/or advancing past the first 15 or so numbers.
There are many ways to do this, but here are two methods. Firstly, hash functions are very good at generating seeds from short strings. A good hash function will generate very different results even when two strings are similar, so you don't have to put much thought into the string. Here's an example hash function:
function cyrb128(str) {
let h1 = 1779033703, h2 = 3144134277,
h3 = 1013904242, h4 = 2773480762;
for (let i = 0, k; i < str.length; i++) {
k = str.charCodeAt(i);
h1 = h2 ^ Math.imul(h1 ^ k, 597399067);
h2 = h3 ^ Math.imul(h2 ^ k, 2869860233);
h3 = h4 ^ Math.imul(h3 ^ k, 951274213);
h4 = h1 ^ Math.imul(h4 ^ k, 2716044179);
}
h1 = Math.imul(h3 ^ (h1 >>> 18), 597399067);
h2 = Math.imul(h4 ^ (h2 >>> 22), 2869860233);
h3 = Math.imul(h1 ^ (h3 >>> 17), 951274213);
h4 = Math.imul(h2 ^ (h4 >>> 19), 2716044179);
return [(h1^h2^h3^h4)>>>0, (h2^h1)>>>0, (h3^h1)>>>0, (h4^h1)>>>0];
}
Calling cyrb128 will produce a 128-bit hash value from a string which can be used to seed a PRNG. Here's how you might use it:
// Create cyrb128 state:
var seed = cyrb128("apples");
// Four 32-bit component hashes provide the seed for sfc32.
var rand = sfc32(seed[0], seed[1], seed[2], seed[3]);
// Only one 32-bit component hash is needed for mulberry32.
var rand = mulberry32(seed[0]);
// Obtain sequential random numbers like so:
rand();
rand();
Note: If you want a slightly more robust 128-bit hash, consider MurmurHash3_x86_128, it's more thorough, but intended for use with large arrays.
Alternatively, simply choose some dummy data to pad the seed with, and advance the generator beforehand a few times (12-20 iterations) to mix the initial state thoroughly. This has the benefit of being simpler, and is often used in reference implementations of PRNGs, but it does limit the number of initial states:
var seed = 1337 ^ 0xDEADBEEF; // 32-bit seed with optional XOR value
// Pad seed with Phi, Pi and E.
// https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number
var rand = sfc32(0x9E3779B9, 0x243F6A88, 0xB7E15162, seed);
for (var i = 0; i < 15; i++) rand();
Note: the output of these PRNG functions produce a positive 32-bit number (0 to 232-1) which is then converted to a floating-point number between 0-1 (0 inclusive, 1 exclusive) equivalent to Math.random(), if you want random numbers of a specific range, read this article on MDN. If you only want the raw bits, simply remove the final division operation.
JavaScript numbers can only represent whole integers up to 53-bit resolution. And when using bitwise operations, this is reduced to 32. Modern PRNGs in other languages often use 64-bit operations, which require shims when porting to JS that can drastically reduce performance. The algorithms here only use 32-bit operations, as it is directly compatible with JS.
Now, onward to the the generators. (I maintain the full list with references and license info here)
sfc32 (Simple Fast Counter)
sfc32 is part of the PractRand random number testing suite (which it passes of course). sfc32 has a 128-bit state and is very fast in JS.
function sfc32(a, b, c, d) {
return function() {
a >>>= 0; b >>>= 0; c >>>= 0; d >>>= 0;
var t = (a + b) | 0;
a = b ^ b >>> 9;
b = c + (c << 3) | 0;
c = (c << 21 | c >>> 11);
d = d + 1 | 0;
t = t + d | 0;
c = c + t | 0;
return (t >>> 0) / 4294967296;
}
}
You may wonder what the | 0 and >>>= 0 are for. These are essentially 32-bit integer casts, used for performance optimizations. Number in JS are basically floats, but during bitwise operations, they switch into a 32-bit integer mode. This mode is processed faster by JS interpreters, but any multiplication or addition will cause it to switch back to a float, resulting in a performance hit.
Mulberry32
Mulberry32 is a simple generator with a 32-bit state, but is extremely fast and has good quality randomness (author states it passes all tests of gjrand testing suite and has a full 232 period, but I haven't verified).
function mulberry32(a) {
return function() {
var t = a += 0x6D2B79F5;
t = Math.imul(t ^ t >>> 15, t | 1);
t ^= t + Math.imul(t ^ t >>> 7, t | 61);
return ((t ^ t >>> 14) >>> 0) / 4294967296;
}
}
I would recommend this if you just need a simple but decent PRNG and don't need billions of random numbers (see Birthday problem).
xoshiro128**
As of May 2018, xoshiro128** is the new member of the Xorshift family, by Vigna & Blackman (professor Vigna was also responsible for the Xorshift128+ algorithm powering most Math.random implementations under the hood). It is the fastest generator that offers a 128-bit state.
function xoshiro128ss(a, b, c, d) {
return function() {
var t = b << 9, r = a * 5; r = (r << 7 | r >>> 25) * 9;
c ^= a; d ^= b;
b ^= c; a ^= d; c ^= t;
d = d << 11 | d >>> 21;
return (r >>> 0) / 4294967296;
}
}
The authors claim it passes randomness tests well (albeit with caveats). Other researchers have pointed out that it fails some tests in TestU01 (particularly LinearComp and BinaryRank). In practice, it should not cause issues when floats are used (such as in these implementations), but may cause issues if relying on the raw lowest order bit.
JSF (Jenkins' Small Fast)
This is JSF or 'smallprng' by Bob Jenkins (2007), who also made ISAAC and SpookyHash. It passes PractRand tests and should be quite fast, although not as fast as sfc32.
function jsf32(a, b, c, d) {
return function() {
a |= 0; b |= 0; c |= 0; d |= 0;
var t = a - (b << 27 | b >>> 5) | 0;
a = b ^ (c << 17 | c >>> 15);
b = c + d | 0;
c = d + t | 0;
d = a + t | 0;
return (d >>> 0) / 4294967296;
}
}
No, it is not possible to seed Math.random(), but it's fairly easy to write your own generator, or better yet, use an existing one.
Check out: this related question.
Also, see David Bau's blog for more information on seeding.
NOTE: Despite (or rather, because of) succinctness and apparent elegance, this algorithm is by no means a high-quality one in terms of randomness. Look for e.g. those listed in this answer for better results.
(Originally adapted from a clever idea presented in a comment to another answer.)
var seed = 1;
function random() {
var x = Math.sin(seed++) * 10000;
return x - Math.floor(x);
}
You can set seed to be any number, just avoid zero (or any multiple of Math.PI).
The elegance of this solution, in my opinion, comes from the lack of any "magic" numbers (besides 10000, which represents about the minimum amount of digits you must throw away to avoid odd patterns - see results with values 10, 100, 1000). Brevity is also nice.
It's a bit slower than Math.random() (by a factor of 2 or 3), but I believe it's about as fast as any other solution written in JavaScript.
No, but here's a simple pseudorandom generator, an implementation of Multiply-with-carry I adapted from Wikipedia (has been removed since):
var m_w = 123456789;
var m_z = 987654321;
var mask = 0xffffffff;
// Takes any integer
function seed(i) {
m_w = (123456789 + i) & mask;
m_z = (987654321 - i) & mask;
}
// Returns number between 0 (inclusive) and 1.0 (exclusive),
// just like Math.random().
function random()
{
m_z = (36969 * (m_z & 65535) + (m_z >> 16)) & mask;
m_w = (18000 * (m_w & 65535) + (m_w >> 16)) & mask;
var result = ((m_z << 16) + (m_w & 65535)) >>> 0;
result /= 4294967296;
return result;
}
Antti Sykäri's algorithm is nice and short. I initially made a variation that replaced JavaScript's Math.random when you call Math.seed(s), but then Jason commented that returning the function would be better:
Math.seed = function(s) {
return function() {
s = Math.sin(s) * 10000; return s - Math.floor(s);
};
};
// usage:
var random1 = Math.seed(42);
var random2 = Math.seed(random1());
Math.random = Math.seed(random2());
This gives you another functionality that JavaScript doesn't have: multiple independent random generators. That is especially important if you want to have multiple repeatable simulations running at the same time.
Please see Pierre L'Ecuyer's work going back to the late 1980s and early 1990s. There are others as well. Creating a (pseudo) random number generator on your own, if you are not an expert, is pretty dangerous, because there is a high likelihood of either the results not being statistically random or in having a small period. Pierre (and others) have put together some good (pseudo) random number generators that are easy to implement. I use one of his LFSR generators.
https://www.iro.umontreal.ca/~lecuyer/myftp/papers/handstat.pdf
Combining some of the previous answers, this is the seedable random function you are looking for:
Math.seed = function(s) {
var mask = 0xffffffff;
var m_w = (123456789 + s) & mask;
var m_z = (987654321 - s) & mask;
return function() {
m_z = (36969 * (m_z & 65535) + (m_z >>> 16)) & mask;
m_w = (18000 * (m_w & 65535) + (m_w >>> 16)) & mask;
var result = ((m_z << 16) + (m_w & 65535)) >>> 0;
result /= 4294967296;
return result;
}
}
var myRandomFunction = Math.seed(1234);
var randomNumber = myRandomFunction();
It's not possible to seed the builtin Math.random function, but it is possible to implement a high quality RNG in Javascript with very little code.
Javascript numbers are 64-bit floating point precision, which can represent all positive integers less than 2^53. This puts a hard limit to our arithmetic, but within these limits you can still pick parameters for a high quality Lehmer / LCG random number generator.
function RNG(seed) {
var m = 2**35 - 31
var a = 185852
var s = seed % m
return function () {
return (s = s * a % m) / m
}
}
Math.random = RNG(Date.now())
If you want even higher quality random numbers, at the cost of being ~10 times slower, you can use BigInt for the arithmetic and pick parameters where m is just able to fit in a double.
function RNG(seed) {
var m_as_number = 2**53 - 111
var m = 2n**53n - 111n
var a = 5667072534355537n
var s = BigInt(seed) % m
return function () {
return Number(s = s * a % m) / m_as_number
}
}
See this paper by Pierre l'Ecuyer for the parameters used in the above implementations:
https://www.ams.org/journals/mcom/1999-68-225/S0025-5718-99-00996-5/S0025-5718-99-00996-5.pdf
And whatever you do, avoid all the other answers here that use Math.sin!
To write your own pseudo random generator is quite simple.
The suggestion of Dave Scotese is useful but, as pointed out by others, it is not quite uniformly distributed.
However, it is not because of the integer arguments of sin. It's simply because of the range of sin, which happens to be a one dimensional projection of a circle. If you would take the angle of the circle instead it would be uniform.
So instead of sin(x) use arg(exp(i * x)) / (2 * PI).
If you don't like the linear order, mix it a bit up with xor. The actual factor doesn't matter that much either.
To generate n pseudo random numbers one could use the code:
function psora(k, n) {
var r = Math.PI * (k ^ n)
return r - Math.floor(r)
}
n = 42; for(k = 0; k < n; k++) console.log(psora(k, n))
Please also note that you cannot use pseudo random sequences when real entropy is needed.
Many people who need a seedable random-number generator in Javascript these days are using David Bau's seedrandom module.
Math.random no, but the ran library solves this. It has almost all distributions you can imagine and supports seeded random number generation. Example:
ran.core.seed(0)
myDist = new ran.Dist.Uniform(0, 1)
samples = myDist.sample(1000)
Here's the adopted version of Jenkins hash, borrowed from here
export function createDeterministicRandom(): () => number {
let seed = 0x2F6E2B1;
return function() {
// Robert Jenkins’ 32 bit integer hash function
seed = ((seed + 0x7ED55D16) + (seed << 12)) & 0xFFFFFFFF;
seed = ((seed ^ 0xC761C23C) ^ (seed >>> 19)) & 0xFFFFFFFF;
seed = ((seed + 0x165667B1) + (seed << 5)) & 0xFFFFFFFF;
seed = ((seed + 0xD3A2646C) ^ (seed << 9)) & 0xFFFFFFFF;
seed = ((seed + 0xFD7046C5) + (seed << 3)) & 0xFFFFFFFF;
seed = ((seed ^ 0xB55A4F09) ^ (seed >>> 16)) & 0xFFFFFFFF;
return (seed & 0xFFFFFFF) / 0x10000000;
};
}
You can use it like this:
const deterministicRandom = createDeterministicRandom()
deterministicRandom()
// => 0.9872818551957607
deterministicRandom()
// => 0.34880331158638
No, like they said it is not possible to seed Math.random()
but you can install external package which make provision for that. i used these package which can be install using these command
npm i random-seed
the example is gotten from the package documentation.
var seed = 'Hello World',
rand1 = require('random-seed').create(seed),
rand2 = require('random-seed').create(seed);
console.log(rand1(100), rand2(100));
follow the link for documentation https://www.npmjs.com/package/random-seed
SIN(id + seed) is a very interesting replacement for RANDOM functions that cannot be seeded like SQLite:
https://stackoverflow.com/a/75089040/7776828
Most of the answers here produce biased results. So here's a tested function based on seedrandom library from github:
!function(f,a,c){var s,l=256,p="random",d=c.pow(l,6),g=c.pow(2,52),y=2*g,h=l-1;function n(n,t,r){function e(){for(var n=u.g(6),t=d,r=0;n<g;)n=(n+r)*l,t*=l,r=u.g(1);for(;y<=n;)n/=2,t/=2,r>>>=1;return(n+r)/t}var o=[],i=j(function n(t,r){var e,o=[],i=typeof t;if(r&&"object"==i)for(e in t)try{o.push(n(t[e],r-1))}catch(n){}return o.length?o:"string"==i?t:t+"\0"}((t=1==t?{entropy:!0}:t||{}).entropy?[n,S(a)]:null==n?function(){try{var n;return s&&(n=s.randomBytes)?n=n(l):(n=new Uint8Array(l),(f.crypto||f.msCrypto).getRandomValues(n)),S(n)}catch(n){var t=f.navigator,r=t&&t.plugins;return[+new Date,f,r,f.screen,S(a)]}}():n,3),o),u=new m(o);return e.int32=function(){return 0|u.g(4)},e.quick=function(){return u.g(4)/4294967296},e.double=e,j(S(u.S),a),(t.pass||r||function(n,t,r,e){return e&&(e.S&&v(e,u),n.state=function(){return v(u,{})}),r?(c[p]=n,t):n})(e,i,"global"in t?t.global:this==c,t.state)}function m(n){var t,r=n.length,u=this,e=0,o=u.i=u.j=0,i=u.S=[];for(r||(n=[r++]);e<l;)i[e]=e++;for(e=0;e<l;e++)i[e]=i[o=h&o+n[e%r]+(t=i[e])],i[o]=t;(u.g=function(n){for(var t,r=0,e=u.i,o=u.j,i=u.S;n--;)t=i[e=h&e+1],r=r*l+i[h&(i[e]=i[o=h&o+t])+(i[o]=t)];return u.i=e,u.j=o,r})(l)}function v(n,t){return t.i=n.i,t.j=n.j,t.S=n.S.slice(),t}function j(n,t){for(var r,e=n+"",o=0;o<e.length;)t[h&o]=h&(r^=19*t[h&o])+e.charCodeAt(o++);return S(t)}function S(n){return String.fromCharCode.apply(0,n)}if(j(c.random(),a),"object"==typeof module&&module.exports){module.exports=n;try{s=require("crypto")}catch(n){}}else"function"==typeof define&&define.amd?define(function(){return n}):c["seed"+p]=n}("undefined"!=typeof self?self:this,[],Math);
function randIntWithSeed(seed, max=1) {
/* returns a random number between [0,max] including zero and max
seed can be either string or integer */
return Math.round(new Math.seedrandom('seed' + seed)()) * max
}
test for true randomness of this code: https://es6console.com/kkjkgur2/
There are plenty of good answers here but I had a similar issue with the additional requirement that I would like portability between Java's random number generator and whatever I ended up using in JavaScript.
I found the java-random package
These two pieces of code had identical output assuming the seed is the same:
Java:
Random randomGenerator = new Random(seed);
int randomInt;
for (int i=0; i<10; i++) {
randomInt = randomGenerator.nextInt(50);
System.out.println(randomInt);
}
JavaScript:
let Random = require('java-random');
let rng = new Random(seed);
for (let i=0; i<10; i++) {
let val = rng.nextInt(50);
console.log(val);
}
I have written a function that returns a seeded random number, it uses Math.sin to have a long random number and uses the seed to pick numbers from that.
Use :
seedRandom("k9]:2#", 15)
it will return your seeded number
the first parameter is any string value ; your seed.
the second parameter is how many digits will return.
function seedRandom(inputSeed, lengthOfNumber){
var output = "";
var seed = inputSeed.toString();
var newSeed = 0;
var characterArray = ['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','y','x','z','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','U','R','S','T','U','V','W','X','Y','Z','!','#','#','$','%','^','&','*','(',')',' ','[','{',']','}','|',';',':',"'",',','<','.','>','/','?','`','~','-','_','=','+'];
var longNum = "";
var counter = 0;
var accumulator = 0;
for(var i = 0; i < seed.length; i++){
var a = seed.length - (i+1);
for(var x = 0; x < characterArray.length; x++){
var tempX = x.toString();
var lastDigit = tempX.charAt(tempX.length-1);
var xOutput = parseInt(lastDigit);
addToSeed(characterArray[x], xOutput, a, i);
}
}
function addToSeed(character, value, a, i){
if(seed.charAt(i) === character){newSeed = newSeed + value * Math.pow(10, a)}
}
newSeed = newSeed.toString();
var copy = newSeed;
for(var i=0; i<lengthOfNumber*9; i++){
newSeed = newSeed + copy;
var x = Math.sin(20982+(i)) * 10000;
var y = Math.floor((x - Math.floor(x))*10);
longNum = longNum + y.toString()
}
for(var i=0; i<lengthOfNumber; i++){
output = output + longNum.charAt(accumulator);
counter++;
accumulator = accumulator + parseInt(newSeed.charAt(counter));
}
return(output)
}
A simple approach for a fixed seed:
function fixedrandom(p){
const seed = 43758.5453123;
return (Math.abs(Math.sin(p)) * seed)%1;
}
In PHP, there is function srand(seed) which generate fixed random value for particular seed.
But, in JS, there is no such inbuilt function.
However, we can write simple and short function.
Step 1: Choose some Seed (Fix Number).
var seed = 100;
Number should be Positive Integer and greater than 1, further explanation in Step 2.
Step 2: Perform Math.sin() function on Seed, it will give sin value of that number. Store this value in variable x.
var x;
x = Math.sin(seed); // Will Return Fractional Value between -1 & 1 (ex. 0.4059..)
sin() method returns a Fractional value between -1 and 1.And we don't need Negative value, therefore, in first step choose number greater than 1.
Step 3: Returned Value is a Fractional value between -1 and 1. So mulitply this value with 10 for making it more than 1.
x = x * 10; // 10 for Single Digit Number
Step 4: Multiply the value with 10 for additional digits
x = x * 10; // Will Give value between 10 and 99 OR
x = x * 100; // Will Give value between 100 and 999
Multiply as per requirement of digits.
The result will be in decimal.
Step 5: Remove value after Decimal Point by Math's Round (Math.round()) Method.
x = Math.round(x); // This will give Integer Value.
Step 6: Turn Negative Values into Positive (if any) by Math.abs method
x = Math.abs(x); // Convert Negative Values into Positive(if any)
Explanation End.Final Code
var seed = 111; // Any Number greater than 1
var digit = 10 // 1 => single digit, 10 => 2 Digits, 100 => 3 Digits and so. (Multiple of 10)
var x; // Initialize the Value to store the result
x = Math.sin(seed); // Perform Mathematical Sin Method on Seed.
x = x * 10; // Convert that number into integer
x = x * digit; // Number of Digits to be included
x = Math.round(x); // Remove Decimals
x = Math.abs(x); // Convert Negative Number into Positive
Clean and Optimized Functional Code
function random_seed(seed, digit = 1) {
var x = Math.abs(Math.round(Math.sin(seed++) * 10 * digit));
return x;
}
Then Call this function using
random_seed(any_number, number_of_digits)any_number is must and should be greater than 1.number_of_digits is optional parameter and if nothing passed, 1 Digit will return.
random_seed(555); // 1 Digit
random_seed(234, 1); // 1 Digit
random_seed(7895656, 1000); // 4 Digit
For a number between 0 and 100.
Number.parseInt(Math.floor(Math.random() * 100))
I am looking to implement an O(1) lookup for binary quadkeys
I know that there is no true hashtable for javascript, instead using objects and o(1) lookup with their properties, but the issue there is that the keys are always converted to strings.
I suspect to have over a 10 million entries in memory, and if I have to rely on keys being strings and with the average quadkey string being 11.5 characters that would equate to (10 million entries * 11.5 length * 2 bytes) = 230,000,000 bytes or 230 MB.
Compared to storing as int64 (10 million entries * 8 bytes) = 80,000,000 bytes or 80 MB
I know that int64 isn't supported natively with javascript, but there are libraries that can assist with that in terms of doing the bitwise operations i'd want.
Now, even though there exists libraries that can deal with int64, they are ultimately objects that are not truly representing 8 bytes, so I believe I cannot use any int64 library in a hashtable, instead I was considering using 2-deep hash table with two int32. The first key would be the first 4 bytes, and the 2nd key being the last 4 bytes. It's not as ideal as 1 operation lookup to find a value, but 2 operations which is still good enough.
However, I feel this is not worth it if all keys are stored as strings, or the fact that all numbers are double-precision floating-point format numbers (8 bytes) so it each hash entry would then take up 16 bytes (two "int32" numbers).
My questions are:
1. If you store a number as the key to a property, will it take up 8
bytes of memory or will it convert to string and take up
length*2bytes?
Is there a way to encode binary quadkeys into the double-precision
floating-point format number standard that javascript has adopted,
even if the number makes no sense , the underlying bits serve a
purpose (unique identification of a construct).
PS: I am marking this with nodejs as there may exist libraries that could assist in my end goal
Edit 1:
Seems 1 is possible with Map() and node 0.12.x+
As far as number 2 I was able to use a int64 lib (bytebuffer) and convert a 64int to a buffer.
I wanted to just use the buffer as the key to Map() but it would not let me as the buffer was internally an object, which each instance acts as a new key to Map().
So I considered turning the buffer back into native type, a 64bit double.
Using readDoubleBE I read the buffer as a double, which represents my 64int binary and successfully lets me use it in a map and allows for O(1) lookup.
var ByteBuffer = require("bytebuffer");
var lookup = new Map();
var startStr = "123123738232777";
for(var i = 1; i <= 100000; i++) {
var longStr = startStr + i.toString();
var longVal = new ByteBuffer.Long.fromValue(longStr);
var buffer = new ByteBuffer().writeUint64(longVal).flip().toBuffer();
var doubleRepresentation = buffer.readDoubleBE();
lookup.set(doubleRepresentation, longStr);
}
console.log(exists("12312373823277744234")); // true
console.log(exists("123123738232777442341232332322")); // false
function exists(longStr) {
var longVal = new ByteBuffer.Long.fromValue(longStr);
var doubleRepresentation = new ByteBuffer().writeUint64(longVal).flip().toBuffer().readDoubleBE();
return lookup.has(doubleRepresentation);
}
The code is sloppy and there are probably shortcuts I am missing, so any suggestions/hints are welcomed.
Ideally I wanted to take advantage of bytebuffer's varints so that I can have even more savings with memory but I wasn't sure if that is possible in a Map, because I wasn't able to use a buffer as a key.
Edit 2:
Using memwatch-next I was able to see that max heap size was 497962856 bytes with this method with Set() whereas using a string in Set() was 1111082864 bytes. That is about 500MB vs 1GB, which is a far cry from 80mb and 230mb, not sure where the extra memory use is coming from. It's worth nothing for these memory tests I used Set over Map that way it should only store unique keys in the data structure. (Using Set as just a existence checker, where Map would serve as a lookup)
Because your keys are integers (and are unique) you could just use them as array indices. However, JS arrays are limited to contain max entries bounded by 32 or 64 bit integer depending on your platform.
To overcome this, you could use your two-step approach, but without using objects and their string hashes. You can store it something like this
store[0010][1000] = 'VALUE' - the item with binary key 00101000 is stored under index 0010 in first array and index 1000 in child array
In decimal, you're dealing with store[2][8] = 'VALUE', which is equivalent to doing store[40] = 'VALUE' in 64bit space
You get yourself a tree with all the properties you want:
You can easily look up by key (in two steps)
Your keys are integers
You're dealing with 32bit integers (or less, depending on how you chunk it)
The doubling in memory from Map to Set in your version of node comes from a bad implementation. Well, not "bad" per se just not fit for millions of entries. The simpler handling of Set gets bought with memory. There's no free lunch, as always, sorry.
Why do they use so much in general? They are supposed to handle any object and the method to be able to handle all of the possible varieties is really expensive. It is possible to optimize if all you have is of one kind but you have to check for it and in 99,99% of all cases it's not worth the hassle because the maps and sets and arrays are short, a couple of thousand entries at most. To be bland: developer time is expensive and better spend elsewhere. I could add: it's open source, do it yourself! but I know it's way easier said than done ;-)
You need to roll it yourself. You can use a Uint32Array for that and build a hash-table around it.
The Bing-Maps are encoded with strings of base-4 digits (at most 23) according to MS and the Quad-key description. Using the encoding of the latter (haven't read the former, so it might be wrong in the details) we can put it into two 32-bit integers:
function quadToInts(quad, zoom){
var high,low, qlen, i, c;
high = 0>>>0;
low = 0>>>0
zoom = zoom>>>0;
// checks & balances omitted!
qlen = quad.length;
for(i = 0; i < 16 && i < qlen ;i++){
c = parseInt(quad.charAt(i));
high |= c << (32-(i*2 + 2));
}
// max = 23 characters (says MS)
for(i = 0; i < 8 && i < qlen - 16 ;i++){
c = parseInt(quad.charAt(16 + i));
low |= c << (32-(i*2 + 2));
}
low |= zoom;
return [high,low];
}
And back
// obligatory https://graphics.stanford.edu/~seander/bithacks.html
function rev(v){
var s = 32;
var mask = (~0)>>>0;
while ((s >>>= 1) > 0) {
mask ^= (mask << s)>>>0;
v = ((v >>> s) & mask) | ((v << s) & (~mask)>>>0);
}
return v;
}
function intsToQuad(k1,k2){
var r, high, low, zoom, c, mask;
r = "";
mask = 0x3; // 0b11
high = k1>>>0;
low = k2>>>0;
zoom = low & (0x20 - 1);
low ^= zoom;
high = rev(high);
for(var i = 0;i<16;i++){
c = high & mask;
c = (c<<1 | c>>>1) & mask;
r += c.toString(10);
high >>>= 2;
}
low = rev(low);
for(var i = 0;i<16;i++){
c = low & mask;
c = (c<<1 | c>>>1) & mask;
r += c.toString(10);
low >>>= 2;
}
return [r,zoom];
}
(All quick hacks, please check before use! And the C&P devil might have had its hand in here, too)
A rough sketch for a hash-table base on the following hash-function
// shamelessly stolen from http://www.burtleburtle.net/bob/c/lookup3.c
function hashword(k1, // high word of 64 bit value
k2, // low word of 64 bit value
seed // the seed
){
var a,b,c;
var rot = function(x,k){
return (((x)<<(k)) | ((x)>>>(32-(k))));
};
/* Set up the internal state */
a = b = c = 0xdeadbeef + ((2<<2)>>>0) + seed>>>0;
if(arguments.length === 2){
seed = k1^k2;
}
b+=k2;
a+=k1;
c ^= b; c -= rot(b,14)>>>0;
a ^= c; a -= rot(c,11)>>>0;
b ^= a; b -= rot(a,25)>>>0;
c ^= b; c -= rot(b,16)>>>0;
a ^= c; a -= rot(c,4)>>>0;
b ^= a; b -= rot(a,14)>>>0;
c ^= b; c -= rot(b,24)>>>0;
return c;
}
function hashsize(N){
var highbit = function(n) {
var r = 0 >>> 0;
var m = n >>> 0;
while (m >>>= 1) {
r++;
}
return r;
};
return (1<<(highbit(N)+1))>>>0;
}
function hashmask(N){
return (hashsize(N)-1)>>>0;
}
And the (rather incomplete) code for table handling
/*
Room for 8-byte (64-bit) entries
Table pos. Array pos.
0 0 high, low
1 2 high, low
2 4 high, lowl
...
n n*2 high, low
*/
function HashTable(N){
var buf;
if(!N)
return null;
N = (N+1) * 2;
buf = new ArrayBuffer(hashsize(N) * 8);
this.table = new Uint32Array(buf);
this.mask = hashmask(N);
this.length = this.table.length;
}
HashTable.prototype.set = function(s,z){
var hash, quad, entry, check, i;
entry = null;
quad = quadToInts(s,z);
hash = hashword(quad[0],quad[1]);
entry = hash & this.mask;
check = entry * 2;
if(this.table[check] != 0 || this.table[check + 1] != 0){
//handle collisions here
console.log("collision in SET found")
return null;
} else {
this.table[check] = quad[0];
this.table[check + 1] = quad[1];
}
return entry;
};
HashTable.prototype.exists = function(s,z){
var hash, quad, entry, check, i;
entry = null;
quad = quadToInts(s,z);
hash = hashword(quad[0],quad[1]);
entry = hash & this.mask;
check = entry * 2;
if(this.table[check] != 0 || this.table[check + 1] != 0){
return entry;
}
return -1;
};
HashTable.prototype.get = function(index){
var entry = [0,0];
if(index > this.length)
return null;
entry[0] = this.table[index * 2];
entry[1] = this.table[index * 2 + 1];
return entry;
};
// short test
var ht = new HashTable(100);
ht.set("01231031020112310",17);
ht.set("11231031020112311",12);
ht.set("21231033020112310",1);
ht.set("31231031220112311321312",23);
var s = "";
for(var i=0;i<ht.table.length;i+=2){
if(ht.table[i] !== 0){
var e = intsToQuad(ht.table[i],ht.table[i+1]);
s += e[0] +", " +e[1] + "\n";
}
}
console.log(s)
Collisions should be rare, so a couple of short standard arrays would do to catch them. To handle it you need to add another byte to the 8 bytes for the two integers representing the Quad or, better, set the second integer to all ones (won't happen with a Quad) and the first to the position(s) in the collision array(s).
To add payload is a bit more complicated because you have only a fixed length to do so.
I have set the size of the table to the next higher power of two. That can be too much or even waaay too much and you might be tempted to adapt it, so be aware that the masking doesn't work as expected anymore you need to do a modulo instead.