We Keep Coding

Storing a Minecraft-like map as a JavaScript array

I'm writing a map editor that converts a 3D space into JavaScript arrays so that it can be exported to a JSON file.
Each map will have a 2D plane acting as a ground layer (the user will have to specify X and Y size), then to add height, the user can place blocks on top of this 2D plane, following a X & Y grid (similar to Minecraft).
My idea was to have an array for each Z layer, and fill it with the information about which blocks are placed there. Because the X and Y sizes of the map must be specified, a simple array should do the trick, as to read the map you would simply loop for each Z layer array and fill the map with its contents, which would be another array. Creating rows defined by the X and Y size of the ground layer.
I know you can fill arrays like layer[165] = grassBlock
after you declare them, Which would make everything before index 165 empty and thus save space. But in a JSON format, wouldn't that array have 164 zeroes or nulls before it reaches this index?
Is this even the most efficient way to store a 3D space? I'm trying to minimize map size and speed up load time as much as possible.

If you only have block/empty then a single bit is sufficient and you can use a single array Javascript typed array for the matrix.
Assuming size of the matrix is X, Y and Z then the conversion from coordinates (x, y, z) to array index could be:
index = (x*Y + y)*Z + z;
then the map could be stored as a single Uint8Array object initialized with length (X*Y*Z + 7) >> 3 (each of the bytes will give you 8 bits but you need to round up).
To read/write a single bit you can finally use
bit = (matrix[index >> 3] >> (index & 7)) & 1; // Read element
matrix[index >> 3] |= 1 << (index & 7); // Set element to 1
matrix[index >> 3] &= ~(1 << (index & 7)); // Clear element to 0
If instead you need to store a logical ID and there are no more than 256 distinct values (including "empty") then a single byte per element is enough. The index computation is as above but you can use as size X*Y*Z and then simply read/write element with matrix[index].
If more than 256 but less than 65537 distinct values are needed then a Uint16Array can be used.
If most of the elements do not carry specific data except the class (e.g. they're just "air", "land", "water") and only a small percentage require much more then may be a byte map with a value for "other" and then just a dictionary mapping (x,y,z) to data only for "other" blocks is a resonable approach (very simple code, still fast access and update).
Note that while Javascript has data types to store binary data efficiently unfortunately JSON doesn't provide a type to send/receive arbitrary bytes (not text) over the network and you'll need to convert to and load from base64 encoding or something similar (if you want to use JSON).

Getting bits from Javascript Uint8ClampedArray

I have a node server running, and in the server I generate and store into Redis a bunch of bits representing colours on a canvas. Every four bits of the stored bits represents a 4 bit colour. (Ex. If I store 001001101011111, then the colours I'm interested in are 0010, 0110, 1011, and 1111).
var byteArr = new ArrayBuffer(360000);
redis_client.set("board", byteArr);
redis_client.setbit("board", 360000, 0);
// There is some garbage at the beginning of the stored value, so zeroing them out.
for(var i = 0; i < 160; i++){
redis_client.setbit("board", i, 0);
}
When a client connects to a server, I grab this string from Redis, and send it through a Websocket:
wss.on('connection', function(ws) {
redis_client.get("board", function (error, result) {
var initial_send = {"initial_send":true, "board":result};
ws.send(JSON.stringify(initial_send));
});
On the client side, I read the board like so:
socket.onmessage = function (event) {
var o = JSON.parse(event.data);
board = o["board"];
var clampedBoard = new Uint8ClampedArray(board.length);
for(var i = 0; i < board.length; i++){
clampedBoard = board[i];
}
}
At this point, the length of the board is 45000, I believe this is because in Javascript, the smallest TypedArray constructor only allows 1 byte units. So because my initial ArrayBuffer was 360000 in size, when I receive it in the client, it is of size 360000/8.
This is where I'm having issues. At this point if I get clampedBoard[0], it should give me the first 8 bits (the first two colours I care about), and I can do clampedBoard[0]>>4, and clampedBoard[0]&15 to get me those two colours, and I can then
look them up in a map where the keys are 0000, 0001, etc,.
But that isn't what is happening.
Here's what I've tried, and what I know:
Printing values out on client-side: console.log(clampedBoard[0]) gives back a [] looking null/undefined character on Chrome's console.
On the server side, when initializing byteArr and clearing the first 160 values to 0, I manually set the first 8 bits to '00111111', which is the binary representation of the ASCII character '?'.
When console.logging clampBoard[0] on the client side, I get the same [] null/undefined character, but when console.logging board[0], it prints out a '?'. I'm not sure of why this is so.
And when attempting to look up in my map of colours by doing clampedBoard[0]>>4, it always defaults to the key in the dictionary which represents 0, even though it should be 0011.
If there is any more information I can provide, please let me know.

First of all, you can do:
board = o["board"];
var clampedBoard = new Uint8ClampedArray(board);
So you won't need the value initialization loop.
Also, the way you were passing it to the object, the "stringification" will convert it to an string instead of array. In order to get an array, you'll need to create a Node.js' Buffer out of the ArrayBuffer, then convert it into a native array, as so:
var view = Buffer.from(result);
var initial_send = {"initial_send":true, "board":[...view]};

Fast way to make RGB array into RGBA array in Javascript

An emulator I am working with internally stores a 1-dimensional framebuffer of RGB values. However, HTML5 canvas uses RGBA values when calling putImageData. In order to display the framebuffer, I currently loop through the RGB array and create a new RGBA array, in a manner similar to this.
This seems suboptimal. There has been much written on performing canvas draws quickly, but I'm still lost on how to improve my application performance. Is there any way to more quickly translate this RGB array to an RGBA array? The alpha channel will always be fully opaque. Also, is there any way to interface with a canvas so that it takes an array of RGB, not RGBA, values?

There's no way to use plain RGB, but the loop in that code could be optimised somewhat by removing repeated calculations, array deferences, etc.
In general you shouldn't use ctx.getImageData to obtain the destination buffer - you don't normally care what values are already there and should use ctx.createImageData instead. If at all possible, re-use the same raw buffer for every frame.
However, since you want to preset the alpha values to 0xff (they default to 0x00) and only need to do so once, it seems to be much most efficient to just fill the canvas and then fetch the raw values with getImageData.
ctx.fillStyle = '#ffffff'; // implicit alpha of 1
ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
dest = ctx.getImageData(0, 0).data
and then for each frame for can just leave the alpha byte untouched:
var n = 4 * w * h;
var s = 0, d = 0;
while (d < n) {
dest[d++] = src[s++];
dest[d++] = src[s++];
dest[d++] = src[s++];
d++; // skip the alpha byte
}
You could also experiment with "loop unrolling" (i.e. repeating that four line block multiple times within the while loop) although results will vary across browsers.
Since it's very likely that your total number of pixels will be a multiple of four, just repeat the block another three times and then the while will only be evaluated for every four pixel copies.

Both ctx.createImageData and ctx.getImageData will create a buffer, the later (get) will be slower since it has also to copy the buffer.
This jsperf : http://jsperf.com/drawing-pixels-to-data
confirms that we have a like 33% slowdown on Chrome, and 16 times slower on Firefox (FFF seems to byte-copy when Chrome copy with 32 or 64 bits move).
i'll just recall that you can handle typed array of different types, and even create a view on the buffer (image.data.buffer).
So this may allow you to write the bytes 4 by 4.
var dest = ctx.createImageData(width, height);
var dest32 = new Int32Array(dest.data.buffer);
var i = 0, j=0, last = 3*width*height;
while (i<last) {
dest32[j] = src[i]<<24 + src[i+1] << 16
+ src[i+2] << 8 + 255;
i+=3;
j++;
}
You will see in this jsperf test i made that it is faster to
write using 32 bits integers :
http://jsperf.com/rgb-to-rgba-conversion-with-typed-arrays
notice that there is a big issue in those tests : since this test is
awfull in terms of garbage creation, accuracy is so-so.
Still after many launch, we see that we have around 50%
gain on write 4 vs write 1.
Edit : it might be worth to see if reading the source with a DataView wouldn't speed things up.
but the input array has to be a buffer (or have a buffer property like a Uint8Array).
(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays/DataView)
do not hesitate to update the fiddle with such a try.
Edit 2 :
I don't understand i re-ran the test and now write 4 is slower : ??? and after, faster again : -------
Anyway you have great interest in keeping the dest32 buffer under your hand and not
create a new one each time anyway, so since this test measure the Int32Array creation, it does not correspond to your use case.

Bias in randomizing normally distributed numbers (javascript)

I’m having problems generating normally distributed random numbers (mu=0 sigma=1)
using JavaScript.
I’ve tried Box-Muller's method and ziggurat, but the mean of the generated series of numbers comes out as 0.0015 or -0.0018 — very far from zero!! Over 500,000 randomly generated numbers this is a big issue. It should be close to zero, something like 0.000000000001.
I cannot figure out whether it’s a method problem, or whether JavaScript’s built-in Math.random() generates not exactly uniformly distributed numbers.
Has someone found similar problems?
Here you can find the ziggurat function:
http://www.filosophy.org/post/35/normaldistributed_random_values_in_javascript_using_the_ziggurat_algorithm/
And below is the code for the Box-Muller:
function rnd_bmt() {
var x = 0, y = 0, rds, c;
// Get two random numbers from -1 to 1.
// If the radius is zero or greater than 1, throw them out and pick two
// new ones. Rejection sampling throws away about 20% of the pairs.
do {
x = Math.random()*2-1;
y = Math.random()*2-1;
rds = x*x + y*y;
}
while (rds === 0 || rds > 1)
// This magic is the Box-Muller Transform
c = Math.sqrt(-2*Math.log(rds)/rds);
// It always creates a pair of numbers. I'll return them in an array.
// This function is quite efficient so don't be afraid to throw one away
// if you don't need both.
return [x*c, y*c];
}

If you generate n independent normal random variables, the standard deviation of the mean will be sigma / sqrt(n).
In your case n = 500000 and sigma = 1 so the standard error of the mean is approximately 1 / 707 = 0.0014. The 95% confidence interval, given 0 mean, would be around twice this or (-0.0028, 0.0028). Your sample means are well within this range.
Your expectation of obtaining 0.000000000001 (1e-12) is not mathematically grounded. To get within that range of accuracy, you would need to generate about 10^24 samples. At 10,000 samples per second that would still take 3 quadrillon years to do...this is precisely why it's good to avoid computing things by simulation if possible.
On the other hand, your algorithm does seem to be implemented correctly :)

how can website javascript code using same input data compute a different result depending on visitor?

The javascript code snippet below is on a website. Recently I've observed the code can produce different results for a visitor on the website than I obtain from my computer using the exact same data that the visitor input to the website. This seems to be visitor dependent (some visitors are fine). I've tried several computers/operating systems in my office, and they all produce the same (correct) results as each other, derived from the visitor's input data in question.
Part of the results (not shown below) provided by the website is a plot of the user's entered data, which I observe is always correct, so I know the visitor's input data they entered into the website (from which the javascript computation uses to compute a result) are interpreted correctly by their machine (I can see the plot the user receives (generated by auto-PDF email), and the same data entered in my computer produces the exact same plot for the entered data; just the results derived from this data are different).
In one case I analyzed, the visitor's incorrect data, mysteriously, was always a factor of 1.3 lower than the correct result. It doesn't seem like a rounding error or difference in 32b vs 64b OS.
Any ideas what could be causing such a thing? Is the code below not robust for all versions of javascript, or could different javascript versions product different results (seems hard to believe, but I'm using some fancy math below, maybe one of the mat functions is antiquated). Unfortunately I don't have access to a machine producing incorrect data to troubleshoot. I also don't know anything about the machine/OS/platform used by visitors (could be anything). Any ideas appreciated. I'm not that experienced with javascript (it could be something obvious below).
Thanks in advance.
function calculate(){
var fc=document.abcform.CF.value*1;
var of = new Array(20);
var pn = new Array(20);
var pj = new Array(19);
var cbox = new Array(20);
var alpha;
var con;
var segment;
var subttl=0;
of[0]=document.abcform.OS1.value*1; pn[0]=document.abcform.abc1.value*1;
of[1]=document.abcform.OS2.value*1; pn[1]=document.abcform.abc2.value*1;
of[2]=document.abcform.OS3.value*1; pn[2]=document.abcform.abc3.value*1;
of[3]=document.abcform.OS4.value*1; pn[3]=document.abcform.abc4.value*1;
of[4]=document.abcform.OS5.value*1; pn[4]=document.abcform.abc5.value*1;
of[5]=document.abcform.OS6.value*1; pn[5]=document.abcform.abc6.value*1;
of[6]=document.abcform.OS7.value*1; pn[6]=document.abcform.abc7.value*1;
of[7]=document.abcform.OS8.value*1; pn[7]=document.abcform.abc8.value*1;
of[8]=document.abcform.OS9.value*1; pn[8]=document.abcform.abc9.value*1;
of[9]=document.abcform.OS10.value*1; pn[9]=document.abcform.abc10.value*1;
of[10]=document.abcform.OS11.value*1; pn[10]=document.abcform.abc11.value*1;
of[11]=document.abcform.OS12.value*1; pn[11]=document.abcform.abc12.value*1;
of[12]=document.abcform.OS13.value*1; pn[12]=document.abcform.abc13.value*1;
of[13]=document.abcform.OS14.value*1; pn[13]=document.abcform.abc14.value*1;
of[14]=document.abcform.OS15.value*1; pn[14]=document.abcform.abc15.value*1;
of[15]=document.abcform.OS16.value*1; pn[15]=document.abcform.abc16.value*1;
of[16]=document.abcform.OS17.value*1; pn[16]=document.abcform.abc17.value*1;
of[17]=document.abcform.OS18.value*1; pn[17]=document.abcform.abc18.value*1;
of[18]=document.abcform.OS19.value*1; pn[18]=document.abcform.abc19.value*1;
of[19]=document.abcform.OS20.value*1; pn[19]=document.abcform.abc20.value*1;
cbox[0]=document.abcform.c1.checked; cbox[1]=document.abcform.c2.checked; cbox[2]=document.abcform.c3.checked;
cbox[3]=document.abcform.c4.checked; cbox[4]=document.abcform.c5.checked; cbox[5]=document.abcform.c6.checked;
cbox[6]=document.abcform.c7.checked; cbox[7]=document.abcform.c8.checked; cbox[8]=document.abcform.c9.checked;
cbox[9]=document.abcform.c10.checked; cbox[10]=document.abcform.c11.checked; cbox[11]=document.abcform.c12.checked;
cbox[12]=document.abcform.c13.checked; cbox[13]=document.abcform.c14.checked; cbox[14]=document.abcform.c15.checked;
cbox[15]=document.abcform.c16.checked; cbox[16]=document.abcform.c17.checked; cbox[17]=document.abcform.c18.checked;
cbox[18]=document.abcform.c19.checked; cbox[19]=document.abcform.c20.checked;
for (var i = 0; i <= 18; i++) { pj[i] = '' }
for (var j = 1; j <= 19; j++){
if (j == 1 || cbox[j]) {
alpha = (pn[j-1] - pn[j])/(10*(Math.LOG10E*Math.log(of[j]/of[j-1])));
con = (Math.pow(of[j-1],alpha))*(Math.pow(10,0.1*pn[j-1]));
if ((alpha <= (1 + 1e-14)) && (alpha >= (1 - 1e-14))) {
segment = con*Math.log(of[j]/of[j-1]); }
else { segment = (con/(1-alpha))*(Math.pow(of[j],1-alpha)-Math.pow(of[j-1],1-alpha)); }
pj[j-1] = round(1E12*(Math.sqrt(2*segment))/(2*Math.PI*fc));
subttl = subttl + Math.pow(pj[j-1],2);
} else {break;}
}
document.abcform.pj1.value=pj[0]; document.abcform.pj2.value=pj[1]; document.abcform.pj3.value=pj[2];
document.abcform.pj4.value=pj[3]; document.abcform.pj5.value=pj[4]; document.abcform.pj6.value=pj[5];
document.abcform.pj7.value=pj[6]; document.abcform.pj8.value=pj[7]; document.abcform.pj9.value=pj[8];
document.abcform.pj10.value=pj[9]; document.abcform.pj11.value=pj[10]; document.abcform.pj12.value=pj[11];
document.abcform.pj13.value=pj[12]; document.abcform.pj14.value=pj[13]; document.abcform.pj15.value=pj[14];
document.abcform.pj16.value=pj[15]; document.abcform.pj17.value=pj[16]; document.abcform.pj18.value=pj[17];
document.abcform.pj19.value=pj[18];
document.abcform.tj.value=round(Math.sqrt(subttl));
}
function round(x) { return Math.round(x*100000)/100000; }

Doesn't seem like a rounding error or
difference in 32b vs 64b OS.
Why not... this is exactly what it seems like. Maybe even a 16bit machine for that matter. You are doing lots of crazy floating point operations -- they typically perform lots of truncation and rounding.
Option two: The log or pow or sqrt functions are implemented differently on different browsers. Solution -- implement your own and see if you get the same "different" results.

Definitely sounds like rounding errors to me, and it doesn't matter what the underlying elements are (64 bit, 32 bit, implementations of pow, etc).
Consider:
alpha = (pn[j-1] - pn[j])/(10*(Math.LOG10E*Math.log(of[j]/of[j-1])));
(Math.pow(of[j-1],alpha))*(Math.pow(10,0.1*pn[j-1]))
That right there can have different results, since it's all floating math. You'll have to either use ints, or something like BigNumber.

In cases like that printf debugging is sometimes helpful - print out all the intermediate values (like alpha) for all the iterations and then compare the outputs to see where they are different. Gives you a starting point.

After debugging, it turns out this is related to a loss of precision by converting to scientific notation in code not shown above. Thus, it's a non-javascript issue (e.g. user error). Thanks for all of the insight above though.