How to pass canvas imageData to emscripten c++ program without copying it? - javascript

I have Image data of the canvas:
myImage = ctx.getImageData(0, 0, 640, 480);
I figured out, that i can create new Uint8Array and use set() to copy imagedata. This is working example:
var numBytes = width * height * 4;
var ptr= Module._malloc(numBytes);
var heapBytes= new Uint8Array(Module.HEAPU8.buffer, ptr, numBytes);
heapBytes.set(new Uint8Array(myImage.data));
_processImage(heapBytes.byteOffset, width, height);
myImage.data.set(heapBytes);
But, unfortunately every .set() operation is far more slower than processing image, and the code above is slower than JS implementation!
So, I want to process image without copying it. I can successfuly read and write the data directly to the heap this way:
Module.HEAPU8.set(myImage.data, myImage.data.byteOffset);
_processImage(myImage.data.byteOffset, width, height);
myImage.data.set(new Uint8ClampedArray(Module.HEAPU8.buffer , myImage.data.byteOffset , numBytes));
It's faster, but still the first .set() takes 17ms to execute.
The c++ function prototype is:
extern "C" {
int processImage(unsigned char *buffer, int width, int height)
{
}
}
Is there any way to pass the array to C++ without using set()? Just telling the c++ where the data is in memory, and allow to modify it?

Just telling the c++ where the data is in memory, and allow to modify it?
As of v1.34.12, Emscripten has a SPLIT_MEMORY option, where you can tell Emscripten to use an existing buffer as part of its memory space that is split up into evenly sized chunks
You could, for example, get the buffer from the canvas
var existingBuffer = myImage.data.buffer;
var bufferSize = existingBuffer.byteLength; // Must be equal to SPLIT_MEMORY
and then, modifying the example from the explanation of split memory, tell Emscripten to use this buffer as part of its memory space
var chunkIndex = 2; // For example
allocateSplitChunk(chunkIndex, existingBuffer);
and then pass a pointer to the chunk to your C++ function.
var pointerToImageInGlobalMemorySpace = chunkIndex * bufferSize;
_processImage(pointerToImageInGlobalMemorySpace, width, height);
However there are problems and limitations
The Emscripten memory space must be split into chunks exactly the size of the canvas image data buffer.
There are apparently serious performance implications for all of the Emscripten-compiled code, which might make this perform worse than your original code.

Related

How do you convert a 32bit RGB png with encoded height values to 16 bit png using browser side javascript?

I am trying to create a 16bit heightmap png from a 32bit Rgb encoded height value png.
According to mapbox I can decode their png pixel values to height values in meters with this formula. height = -10000 + ((R * 256 * 256 + G * 256 + B) * 0.1)
I am new to understanding this type of information and have tried to figure it out myself but I need some help.
I found this program called image-js, docs for this program are here . Note I am using this in the browser not with node.
I think it has the abilities that I need. However, I am really not sure how to get the data from the original image and create a new 16bit png with the new height values calculated from the above formula.
If you look in the docs under image I can set many properties such as
bitDepth and set that to 16
getPixelsArray() this function will give me an array of pixels values in the form of r,g,b. I think I can then run each pixel through the above formula to get the height. After that how do I turn this height data back into a 16bit grayscale heightmap png?
Thanks!
Example images.
32bit Rgb encoded height value png
16 bit heightmap png. You have to zoom in to see the image changes
Update:
Thanks to traktor for the suggestion. I wanted to add some information to my question.
I don't necessarily need to use image-js. My main goal is to get a 16 bit heightmap from a 32 bit RGB. So any other browser side javscript method would be fine. I also am using browser side createWritable(); stream to write file to disk.
I have a method used from this nice module https://github.com/colkassad/terrain-rgb-height however I cannot get the browser side version of pngjs to work correctly. I think there is a difference between browser side stream reader/writers and node reader/writers that make it not work.
Thanks!
Looking through the source code for Image-js package on GitHub turns up that the options object used in calls to the library is verified in source file kind.js which enumerates supported options properties in the assignment statement:
const { components, alpha, bitDepth, colorModel } = definition;
However, many options are defaulted using kind option property, which itself defaults to "RGBA".
Knowing more about option properties allows using them (I couldn't find options documentation outside of the code).
I would suggest
Create an Image-js image from the 32bit encoded height png. Omit a kind property to use the default 32bit PNG pixel model of 3 color channels plus one alpha channel.
Convert the image data (held as a typed array in the image object's data property) to a single dimensional array of height values using the MapBox conversion algorithm. The layout of the (image.data) typed array appears to be the same as that used for ImageData
Create a new image-js image using
new Image(width, height, decodedHeightArray, {kind="GREY", bitDepth:16})
where decodedHeightArray is the array prepared in the previous step.
Just adding my code to what tracktor's answer provided
Thank you very much traktor
This worked perfectly. For anyone who wants the code
I am using image-js to encode and decode the image
let file_handleSixteen = await dir_handle.getFileHandle(sixteen_file_name, {create: true})
let writableSixteen = await file_handleSixteen.createWritable();
async function convert16(arrayBuff) {
let image = await Image.load(arrayBuff);
let width = image.width
let height = image.height
let decodedHeightArray = []
let pixelsArray = image.getPixelsArray()
for (const pixel of pixelsArray) {
let r = pixel[0]
let g = pixel[1]
let b = pixel[2]
let height = getHeightFromRgb(r, g, b);
decodedHeightArray.push(height)
}
let newImage = new Image(width, height, decodedHeightArray, {kind: "GREY", bitDepth: 16})
return newImage.toBlob()
}
function getHeightFromRgb(r, g, b) {
return -10000 + ((r * 256 * 256 + g * 256 + b) * 0.1);
}
const file = await file_handleRgb.getFile();
let imageBuffer = await file.arrayBuffer()
let convertedArray = await convert16(imageBuffer)
await writableSixteen.write(convertedArray)
await writableSixteen.close();
I am also using the browser streams api to write the file to disk. Note the filestreams writable.write only accepts certain values one of them being a Blob that is why converted it to Blob before passing it to the write method

Resizing images using createjs / easeljs

I'd like to dynamically downsize some images on my canvas using createjs, and then store the smaller images to be displayed when zooming out of the canvas for performance reasons. Right now, I'm using the following code:
var bitmap = createjs.Bitmap('somefile.png');
// wait for bitmap to load (using preload.js etc.)
var oc = document.createElement('canvas');
var octx = oc.getContext('2d');
oc.width = bitmap.image.width*0.5;
oc.height = bitmap.image.height*0.5;
octx.drawImage(bitmap.image, 0, 0, oc.width, oc.height);
var dataUrl = oc.toDataURL('image/png'); // very expensive
var smallBitmap = new createjs.Bitmap(dataUrl);
This works, but:
The toDataURL operation is very expensive when converting to image/png and too slow to use in practice (and I can't convert to the faster image/jpeg due to the insufficient quality of the output for all settings I tried)
Surely there must be a way to downsize the image without having to resort to separate canvas code, and then do a conversion manually to draw onto the createjs Bitmap object??
I've also tried:
octx.drawImage(bitmap.image, 0, 0, oc.width, oc.height);
var smallBitmap = new createjs.Bitmap(oc);
But although very fast, this doesn't seem to actually work (and in any case I'm having to create a separate canvas element every time to facilitate this.)
I'm wondering if there is a way that I can use drawImage to draw a downsampled version of the bitmap into a createjs Bitmap instance directly without having to go via a separate canvas object or do a conversion to string?
If I understand correctly, internally this is how the createjs cache property works (i.e. uses drawImage internally to write into the DisplayObject) but I'm unable to figure out how use it myself.
You have tagged this post with createjs and easeljs, but your examples show plain Canvas context usage for scaling.
You can use the scale parameter on Bitmap.cache() to get the result you want, then reuse the cacheCanvas as necessary.
// This will create a half-size cache (50%)
// But scale it back up for you when it displays on the stage
var bmp = new createjs.Bitmap(img);
bmp.cache(0, 0, img.width, img.height, 0.5);
// Pull out the generated cache and use it in a new Bitmap
// This will display at the new scaled size.
var bmp2 = new createjs.Bitmap(bmp.cacheCanvas);
// Un-cache the first one to reset it if you want
bmp.uncache();
Here is a fiddle to see it in action: http://jsfiddle.net/lannymcnie/ofdsyn7g/
Note that caching just uses another canvas with a drawImage to scale it down. I definitely would stay away from toDataURL, as it not performant at all.

How to draw on an HTML5 Canvas, pixel-by-pixel

Suppose that I have a 900x900 HTML5 Canvas element.
I have a function called computeRow that accepts, as a parameter, the number of a row on the grid and returns an array of 900 numbers. Each number represents a number between 0 and 200. There is an array called colors that contains an array of strings like rgb(0,20,20), for example.
Basically, what I'm saying is that I have a function that tells pixel-by-pixel, what color each pixel in a given row on the canvas is supposed to be. Running this function many times, I can compute a color for every pixel on the canvas.
The process of running computeRow 900 times takes about 0.5 seconds.
However, the drawing of the image takes much longer than that.
What I've done is I've written a function called drawRow that takes an array of 900 numbers as the input and draws them on the canvas. drawRow takes lots longer to run than computeRow! How can I fix this?
drawRow is dead simple. It looks like this:
function drawRow(rowNumber, result /* array */) {
var plot, context, columnNumber, color;
plot = document.getElementById('plot');
context = plot.getContext('2d');
// Iterate over the results for each column in the row, coloring a single pixel on
// the canvas the correct color for each one.
for(columnNumber = 0; columnNumber < width; columnNumber++) {
color = colors[result[columnNumber]];
context.fillStyle = color;
context.fillRect(columnNumber, rowNumber, 1, 1);
}
}
I'm not sure exactly what you are trying to do, so I apologize if I am wrong.
If you are trying to write a color to each pixel on the canvas, this is how you would do it:
var ctx = document.getElementById('plot').getContext('2d');
var imgdata = ctx.getImageData(0,0, 640, 480);
var imgdatalen = imgdata.data.length;
for(var i=0;i<imgdatalen/4;i++){ //iterate over every pixel in the canvas
imgdata.data[4*i] = 255; // RED (0-255)
imgdata.data[4*i+1] = 0; // GREEN (0-255)
imgdata.data[4*i+2] = 0; // BLUE (0-255)
imgdata.data[4*i+3] = 255; // APLHA (0-255)
}
ctx.putImageData(imgdata,0,0);
This is a lot faster than drawing a rectangle for every pixel. The only thing you would need to do is separate you color into rgba() values.
If you read the color values as strings from an array for each pixel it does not really matter what technique you use as the bottleneck would be that part right there.
For each pixel the cost is split on (roughly) these steps:
Look up array (really a node/linked list in JavaScript)
Get string
Pass string to fillStyle
Parse string (internally) into color value
Ready to draw a single pixel
These are very costly operations performance-wise. To get it more efficient you need to convert that color array into something else than an array with strings ahead of the drawing operations.
You can do this several ways:
If the array comes from a server try to format the array as a blob / typed array instead before sending it. This way you can copy the content of the returned array almost as-is to the canvas' pixel buffer.
Use a web workers to parse the array and pass it back as a transferable object which you them copy into the canvas' buffer. This can be copied directly to the canvas - or do it the other way around, transfer the pixel buffer to worker, fill there and return.
Sort the array by color values and update the colors by color groups. This way you can use fillStyle or calculate the color into an Uint32 value which you copy to the canvas using a Uint32 buffer view. This does not work well if the colors are very spread but works ok if the colors represent a small palette.
If you're stuck with the format of the colors then the second option is what I would recommend primarily depending on the size. It makes your code asynchronous so this is an aspect you need to deal with as well (ie. callbacks when operations are done).
You can of course just parse the array on the same thread and find a way to camouflage it a bit for the user in case it creates a noticeable delay (900x900 shouldn't be that big of a deal even for a slower computer).
If you convert the array convert it into unsigned 32 bit values and store the result in a Typed Array. This way you can iterate your canvas pixel buffer using Uint32's instead which is much faster than using byte-per-byte approach.
fillRect is meant to be used for just that - filling an area with a single color, not pixel by pixel. If you do pixel by pixel, it is bound to be slower as you are CPU bound. You can check it by observing the CPU load in these cases. The code will become more performant if
A separate image is created with the required image data filled in. You can use a worker thread to fill this image in the background. An example of using worker threads is available in the blog post at http://gpupowered.org/node/11
Then, blit the image into the 2d context you want using context.drawImage(image, dx, dy).

Fast way to make RGB array into RGBA array in Javascript

An emulator I am working with internally stores a 1-dimensional framebuffer of RGB values. However, HTML5 canvas uses RGBA values when calling putImageData. In order to display the framebuffer, I currently loop through the RGB array and create a new RGBA array, in a manner similar to this.
This seems suboptimal. There has been much written on performing canvas draws quickly, but I'm still lost on how to improve my application performance. Is there any way to more quickly translate this RGB array to an RGBA array? The alpha channel will always be fully opaque. Also, is there any way to interface with a canvas so that it takes an array of RGB, not RGBA, values?
There's no way to use plain RGB, but the loop in that code could be optimised somewhat by removing repeated calculations, array deferences, etc.
In general you shouldn't use ctx.getImageData to obtain the destination buffer - you don't normally care what values are already there and should use ctx.createImageData instead. If at all possible, re-use the same raw buffer for every frame.
However, since you want to preset the alpha values to 0xff (they default to 0x00) and only need to do so once, it seems to be much most efficient to just fill the canvas and then fetch the raw values with getImageData.
ctx.fillStyle = '#ffffff'; // implicit alpha of 1
ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
dest = ctx.getImageData(0, 0).data
and then for each frame for can just leave the alpha byte untouched:
var n = 4 * w * h;
var s = 0, d = 0;
while (d < n) {
dest[d++] = src[s++];
dest[d++] = src[s++];
dest[d++] = src[s++];
d++; // skip the alpha byte
}
You could also experiment with "loop unrolling" (i.e. repeating that four line block multiple times within the while loop) although results will vary across browsers.
Since it's very likely that your total number of pixels will be a multiple of four, just repeat the block another three times and then the while will only be evaluated for every four pixel copies.
Both ctx.createImageData and ctx.getImageData will create a buffer, the later (get) will be slower since it has also to copy the buffer.
This jsperf : http://jsperf.com/drawing-pixels-to-data
confirms that we have a like 33% slowdown on Chrome, and 16 times slower on Firefox (FFF seems to byte-copy when Chrome copy with 32 or 64 bits move).
i'll just recall that you can handle typed array of different types, and even create a view on the buffer (image.data.buffer).
So this may allow you to write the bytes 4 by 4.
var dest = ctx.createImageData(width, height);
var dest32 = new Int32Array(dest.data.buffer);
var i = 0, j=0, last = 3*width*height;
while (i<last) {
dest32[j] = src[i]<<24 + src[i+1] << 16
+ src[i+2] << 8 + 255;
i+=3;
j++;
}
You will see in this jsperf test i made that it is faster to
write using 32 bits integers :
http://jsperf.com/rgb-to-rgba-conversion-with-typed-arrays
notice that there is a big issue in those tests : since this test is
awfull in terms of garbage creation, accuracy is so-so.
Still after many launch, we see that we have around 50%
gain on write 4 vs write 1.
Edit : it might be worth to see if reading the source with a DataView wouldn't speed things up.
but the input array has to be a buffer (or have a buffer property like a Uint8Array).
(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays/DataView)
do not hesitate to update the fiddle with such a try.
Edit 2 :
I don't understand i re-ran the test and now write 4 is slower : ??? and after, faster again : -------
Anyway you have great interest in keeping the dest32 buffer under your hand and not
create a new one each time anyway, so since this test measure the Int32Array creation, it does not correspond to your use case.

understanding javascript variable handling and referening especially canvas ImageArray and array assigning

consider this code:
var deSaturated = deSaturate(greyscaleCtx.getImageData(0, 0, canvasWidth, canvasHeight));
imageData comes from getImageData canvas function.
function deSaturate (imageData) {
var theData = imageData.data;
var dataLength = theData.length;
var i = dataLength-1;
var lightLevel;
// Iterate through each pixel, desaturating it
while ( i >= 0) {
// To find the desaturated value, average the brightness of the red, green, and blue values
theData[i] = theData[i+1] = theData[i+2] = (theData[i] + theData[i + 1] + theData[i + 2]) / 3;
// Fully opaque
theData[i+3] = 255;
// returning an average intensity of all pixels. Used for calibrating sensitivity based on room light level.
lightLevel += theData[i]; //combining the light level in the samefunction
i -= 4;
}
imageData.data = theData; //bring back theData into imageData.data - do I really need this?
var r = [lightLevel/dataLength,imageData]
return r;
}
during the writing and optimizing of this code I found out I don't really understand how js is treating for example "theData" variable. is working with it just a short way to reference imageData.data in which case I don't need the following code in the end:
imageData.data = theData
but then do I pay in degraded performance ( a lot of DOM I/O)?
or is doing theData = imageData.data actually copying the original array (represented as Uint8ClampedArray) and then I have to reassign the modified data to imageData.data.
I guess this is basic javascript, but I found contradictory code examples in MDN and other developer resources and I would really like to understand this properly.
thanks for the help!
Just ran a quick test:
var idata = ctx.getImageData(0,0,300,300);
var data = idata.data;
for(var i=0;i<data.length;i++){
data[i]=0;
}
ctx.putImageData(idata,0,0);
And that properly blanks out part of the screen as expected. However without putImageData nothing will happen. So changing the data object, whether stored in a different variable or not, will be reflected in that imageData object. However this will not affect the canvas until putImageData has been called.
So, yes, you can remove that final assignment and it will work as desired.
However I will warn that it is not a valid assumption that it is a Uint8ClampedArray. Yes, that is how Chrome handles it (last I checked), and it is indeed what the official specification uses. However some browsers have no notion of Uint8ClampedArray, while still supporting canvas through the now deprecated CanvasPixelArray.
So all you are guaranteed to get is something with the some array-like interface. I had to learn this the hard way when I tried to cache interesting features of image data by creating a new Uint8ClampedArray, which failed in some browsers.
See: https://developer.mozilla.org/en-US/docs/DOM/CanvasPixelArray
In javascript, assigning either an array or an object just assigns a reference to that array or object - it does not make a copy of the data. A copy is only made if you physically create a new array and copy the data over or call some function that is designed to do that for you.
So, if imageData.data is an array, then assigning it to theData just makes a shortcut for referring to the same data. It does not make a new copy of the data. Thus, after modifying the data pointed to by theData, you don't have to assign it back to imageData.data because there is only one copy of the data and both theData and imageData.data point already point to that same copy of the data.
So, in direct answer to your question, this line is unnecessary:
imageData.data = theData;

Categories