I'm writing a map editor that converts a 3D space into JavaScript arrays so that it can be exported to a JSON file.
Each map will have a 2D plane acting as a ground layer (the user will have to specify X and Y size), then to add height, the user can place blocks on top of this 2D plane, following a X & Y grid (similar to Minecraft).
My idea was to have an array for each Z layer, and fill it with the information about which blocks are placed there. Because the X and Y sizes of the map must be specified, a simple array should do the trick, as to read the map you would simply loop for each Z layer array and fill the map with its contents, which would be another array. Creating rows defined by the X and Y size of the ground layer.
I know you can fill arrays like layer[165] = grassBlock
after you declare them, Which would make everything before index 165 empty and thus save space. But in a JSON format, wouldn't that array have 164 zeroes or nulls before it reaches this index?
Is this even the most efficient way to store a 3D space? I'm trying to minimize map size and speed up load time as much as possible.
If you only have block/empty then a single bit is sufficient and you can use a single array Javascript typed array for the matrix.
Assuming size of the matrix is X, Y and Z then the conversion from coordinates (x, y, z) to array index could be:
index = (x*Y + y)*Z + z;
then the map could be stored as a single Uint8Array object initialized with length (X*Y*Z + 7) >> 3 (each of the bytes will give you 8 bits but you need to round up).
To read/write a single bit you can finally use
bit = (matrix[index >> 3] >> (index & 7)) & 1; // Read element
matrix[index >> 3] |= 1 << (index & 7); // Set element to 1
matrix[index >> 3] &= ~(1 << (index & 7)); // Clear element to 0
If instead you need to store a logical ID and there are no more than 256 distinct values (including "empty") then a single byte per element is enough. The index computation is as above but you can use as size X*Y*Z and then simply read/write element with matrix[index].
If more than 256 but less than 65537 distinct values are needed then a Uint16Array can be used.
If most of the elements do not carry specific data except the class (e.g. they're just "air", "land", "water") and only a small percentage require much more then may be a byte map with a value for "other" and then just a dictionary mapping (x,y,z) to data only for "other" blocks is a resonable approach (very simple code, still fast access and update).
Note that while Javascript has data types to store binary data efficiently unfortunately JSON doesn't provide a type to send/receive arbitrary bytes (not text) over the network and you'll need to convert to and load from base64 encoding or something similar (if you want to use JSON).
Related
I was trying to solve a 2D matrix problem, and stumble upon the matrix-array convert formula:
r * c matrix convert to an array => matrix[x][y] => arr[x * c + y]
an array convert to r * c matrix => arr[x] =>matrix[x / c][x % c]
But I can't seem to understand why it works like this. Can someone explain how the conversion works? I mean why the conversions manipulate the columns(c)? And why using subtraction and remainder([x / c][x % c])? And why x * c + y?
I was solving the problem in JavaScript.
First have a look at "Row Major Vs Column Major Order: 2D arrays access in programming languages" on Computing Science Stack Overflow as an insight to the main ways an array can be serialized:
Row order stores all the values for a row contiguously in serialized output, starting with first row, before proceeding to the next row.
Column order stores all the values for a column in serialized output, starting with the first column before proceeding to the next.
Formula Confusion
The quoted formula creates output in row order because it multiplies one of the indexes by the number of columns, not the number of rows, which means index x is being used as the row index. This is nonsensical in a spread sheet context where "x" means column and "y" means row.
JavaScript Formula for row order storage of a row/column matrix
matrix[row][column] is serialized to arr[row * c + col]
arr[index] is restored to matrix[ Math.floor(index/c), index % c]
where calculating the row in de-serialization requires truncation of the division result since there is no integral division operator in JavaScript to match that used in the quoted formula.
Typical JavaScript Application
imageData store canvas pixel data in row order:
Pixels then proceed from left to right, then downward, throughout the [serialized] array.
It's not that you can't use x and y variable names to access image pixel data, but bear in mind x is a column index, and y is the row number from the top.
I have two very simple functions in python that calculate the size of a struct (C struct, numpy struct, etc) given the range of numbers you want to store. So, if you wanted to store numbers from 0 to 8389798, 8389798 would be the value you feed the function:
def ss(value):
nbits = 8
while value > 2**nbits:
nbits += 8
return nbits * value
def mss(value):
total_size = 0
max_bits = [(0,0)] # (bits for 1 row in struct, max rows in struct)
while value > 2 ** max_bits[-1][0] :
total_size += max_bits[-1][0] * max_bits[-1][1]
value -= max_bits[-1][1]
new_struct_bits = max_bits[-1][0]+8
max_bits.append( (new_struct_bits,2**new_struct_bits) )
total_size += max_bits[-1][0] * value
#print max_bits
return total_size
ss is for a single struct where you need as many bytes in the first row to store "1" as you would in the last row to store "8389798". However, this is not as space efficient as breaking your struct up into structs of 1 byte, 2 bytes, 3 bytes, etc, up until N bytes needed to store your value. This is what mss calculates.
So now i want to see how much more efficient mss is over ss for a range of values - that range being 1 to, say, 100 billion. That's much to much data to save and plot, and it's totally unnecessary to do so in the first place. Far better to take the plot window, and for every value of X that has a pixel in that window, calculate the value of y [which is ss(x) - mss(x)].
This sort of interactive graph is really the only way i can think of to look at the relationship between mss and ss. Does anyone know how i should plot such a graph? I'm willing to use a JavaScript solution because I can rewrite the python to that, as well as use "solutions" like Excel, R, Wolfram, if they offer a way to do interactive/generated graphs.
Suppose if you are given a bunch of points in (x,y) values and you need to generate points by linearly interpolate between the 2 nearest values in the x axis, what is the fastest implementation to do so?
I searched around but I was unable to find a satisfactory answer, I feel its because I wasnt searching for the right words.
For example, if I was given (0,0) (0.5 , 1) (1, 0.5), then I want to get a value at 0.7; it would be (0.7-0.5)/(1-0.5) * (0.5-1) + 1; but what data structure would allow me to find the 2 nearest key values to interpolate in between? Is a simple linear search/ binary search if I have many key values the best I could do?
The way I usually implement O(1) interpolation is by means of an additional data structure, which I call IntervalSelector that in time O(1) will give the two surrounding values of the sequence that have to be interpolated.
An IntervalSelector is a class that, when given a sequence of n abscissas builds and remembers a table that will map any given value of x to the index i such that sequence[i] <= x < sequence[i+1] in time O(1).
Note: In what follows arrays are 1 based.
The algorithm that builds the table proceeds as follow:
Find delta to be the minimum distance between two consecutive elements in the input sequence of abscissas.
Set count := (b-a)/delta + 1, where a and b are respectively the first and last of the (ascending) sequence and / stands for the integer quotient of the division.
Define table to be an Array of count elements.
For i between 1 and n set table[(sequence[j]-a)/delta + 1] := j.
Repeat every entry of table visited in 4 to the unvisited positions that come right after it.
On output, table maps j to i if (j-1)*d <= sequence[i] - a < j*d.
Here is an example:
Since elements 3rd and 4th are the closest ones, we divide the interval in subintervals of this smallest length. Now, we remember in the table the positions of the left end of each of these deta-intervals. Later on, when an input x is given, we compute the delta-interval of such x as (x-a)/delta + 1 and use the table to deduce the corresponding interval in the sequence. If x falls to the left of the ith sequence element, we choose the (i-1)th.
More precisely:
Given any input x between a and b calculate j := (x-a)/delta + 1 and i := table[j]. If x < sequence[i] put i := i - 1. Then, the index i satisfies sequence[i] <= x < sequence[i+1]; otherwise the distance between these two consecutive elements would be smaller than delta, which is not.
Remark: Be aware that if the minimum distance delta between consecutive elements in sequence is too small the table will have too many entries. The simple description I've presented here ignores these pathological cases, which require additional work.
Yes, a simple binary search should do well and will typically suffice.
If you need to get better, you might try interpolation search (has nothing to do with your value interpolation).
If your points are distributed at fixed intervals (like in your example, 0 0.5 1), you can also simply store the values in an array and access them in constant time via their index.
As far as I understand topojson.presimplify(JSON) in D3 adds Z coordinate to each point in the input topojson shape based on its significance, which then allows to use it for the dynamic simplification like in http://bl.ocks.org/mbostock/6245977
This method topojson.presimplify() takes quite a long time to execute on complicated maps, especially in Firefox which makes the browser unresponsive for few seconds.
Can it be baked directly into the topojson file via the command line as it is done with projections:
topojson --projection 'd3.geo.mercator().translate([0,0]).scale(1)' -o cartesian.topo.json spherical.topo.json
I found a workaround for this which is not completely as simple as I wanted but still achieves the same result.
After the topojson.presimplify(data) is called, data already holds the pre simplified geometry with added Z axis values.
Then I convert it to the JSON string and manually copy it to a new file with JSON.stringify(data)
Nevertheless these conversion to a JSON string has a problem with Infinity values which often occur for Z and with JSON.stringify method are converted to null. Also when there is a value for Z coordinate it is usually too precise and writing all decimal points takes too much space.
For that reason before converting data to a JSON string I trim the numbers:
// Simplifying the map
topojson.presimplify(data);
// Changing Infinity values to 0, limiting decimal points
var arcs = data.arcs;
for(var i1 = arcs.length; i1--;) {
var arc = arcs[i1];
for(var i2 = arc.length; i2--;) {
var v = arc[i2][2];
if(v === Infinity) arc[i2][2] = 0;
else {
arc[i2][2] = M.round(v * 1e9)/1e9;
}
}
}
This makes Infinity values to appear as exactly 0 and other values are trimmed to 9 decimal points which is enough for dynamic simplification to work properly.
Since such string is too long to easily print it for copying to the new json file it is much easier to store it in the localStorage of the browser:
localStorage.setItem(<object name>, JSON.stringify(data))
Then in Safari or Chrome open the developer console and in the tab Resources -> Local Storage -> <Website URL> the stored object can be found, copied and then pasted into a text editor.
Usually it is pasted as a <key> <value> pair, so one needs to remove from the beginning of the pasted string so that it starts from {.
Since Infinity values have been converted to 0, in the dynamic simplification function it should be taken into account so that points with Z = 0 are treated as Z = Infinity and are always plotted with any simplification area:
point: function(x, y, z) {
if (z===0 || z >= simplificationArea) {
this.stream.point(x, y);
}
}
An emulator I am working with internally stores a 1-dimensional framebuffer of RGB values. However, HTML5 canvas uses RGBA values when calling putImageData. In order to display the framebuffer, I currently loop through the RGB array and create a new RGBA array, in a manner similar to this.
This seems suboptimal. There has been much written on performing canvas draws quickly, but I'm still lost on how to improve my application performance. Is there any way to more quickly translate this RGB array to an RGBA array? The alpha channel will always be fully opaque. Also, is there any way to interface with a canvas so that it takes an array of RGB, not RGBA, values?
There's no way to use plain RGB, but the loop in that code could be optimised somewhat by removing repeated calculations, array deferences, etc.
In general you shouldn't use ctx.getImageData to obtain the destination buffer - you don't normally care what values are already there and should use ctx.createImageData instead. If at all possible, re-use the same raw buffer for every frame.
However, since you want to preset the alpha values to 0xff (they default to 0x00) and only need to do so once, it seems to be much most efficient to just fill the canvas and then fetch the raw values with getImageData.
ctx.fillStyle = '#ffffff'; // implicit alpha of 1
ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
dest = ctx.getImageData(0, 0).data
and then for each frame for can just leave the alpha byte untouched:
var n = 4 * w * h;
var s = 0, d = 0;
while (d < n) {
dest[d++] = src[s++];
dest[d++] = src[s++];
dest[d++] = src[s++];
d++; // skip the alpha byte
}
You could also experiment with "loop unrolling" (i.e. repeating that four line block multiple times within the while loop) although results will vary across browsers.
Since it's very likely that your total number of pixels will be a multiple of four, just repeat the block another three times and then the while will only be evaluated for every four pixel copies.
Both ctx.createImageData and ctx.getImageData will create a buffer, the later (get) will be slower since it has also to copy the buffer.
This jsperf : http://jsperf.com/drawing-pixels-to-data
confirms that we have a like 33% slowdown on Chrome, and 16 times slower on Firefox (FFF seems to byte-copy when Chrome copy with 32 or 64 bits move).
i'll just recall that you can handle typed array of different types, and even create a view on the buffer (image.data.buffer).
So this may allow you to write the bytes 4 by 4.
var dest = ctx.createImageData(width, height);
var dest32 = new Int32Array(dest.data.buffer);
var i = 0, j=0, last = 3*width*height;
while (i<last) {
dest32[j] = src[i]<<24 + src[i+1] << 16
+ src[i+2] << 8 + 255;
i+=3;
j++;
}
You will see in this jsperf test i made that it is faster to
write using 32 bits integers :
http://jsperf.com/rgb-to-rgba-conversion-with-typed-arrays
notice that there is a big issue in those tests : since this test is
awfull in terms of garbage creation, accuracy is so-so.
Still after many launch, we see that we have around 50%
gain on write 4 vs write 1.
Edit : it might be worth to see if reading the source with a DataView wouldn't speed things up.
but the input array has to be a buffer (or have a buffer property like a Uint8Array).
(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays/DataView)
do not hesitate to update the fiddle with such a try.
Edit 2 :
I don't understand i re-ran the test and now write 4 is slower : ??? and after, faster again : -------
Anyway you have great interest in keeping the dest32 buffer under your hand and not
create a new one each time anyway, so since this test measure the Int32Array creation, it does not correspond to your use case.