We Keep Coding

HTML5 Blending modes - layers in paint program

I'm creating html5 paint application and I'm currently working on blending layers. I'm wondering which of the approaches would be the best (fastest and gimp/photoshop like) to build in such program. My layers (canvases) are stacked.
Changing blend mode by CSS3 property (probably very fast - blending directly on graphics card)
Having hidden canvases (layers) and one canvas to show flattened image to user. So we draws on these hidden canvases and there is some mechanism which taking each of hidden canvas and draw it to user viewable canvas (probably slower but each context.drawImage(...) is optimized and computed on graphics card)
Hidden canvases (layers) are truly virtual. There is no hidden canvases elements at all. Instead there is some structure in memory which imitate canvas. These structure just saving user actions performed on this virtual layer. Then when repainting is required, user canvas is reconstructed by taking each operations from each virtual layers and paint it (true paint). Operations must be correctly ordered is such virtual layer structure (this is probably slow but may be faster than 2nd approach(we don't wasting time to draw anything on layer, just storing options how we will draw on real layer)?)
Blending by WebGL(probably fast)
Compute manually each pixel and show score to user (super slow?)
I'm using second approach but i'm not really sure it's the best (especially for more layers). I was wondering do you know any other approach, maybe you would suggest what would be better to implement blending operations to make it work as in Gimp/Adobe Photoshop

My way to implement this was to have multiple canvases, each of them store it's own blending mode. But it's just virual canvas (not drawable for user). Then there is 33 times per second timer (made by RequestAnimationFrame) which grab all this canvases, flatten them and draw it to viewport. For each layer I perform blend mode change on viewport canvas. It works perfect since drawImage(...) is computed on GPU and it's really fast.
I think code is fairly easy to understand and to apply in own solutions:
Process(lm: dl.LayerManager) {
var canvasViewPort = lm.GetWorkspaceViewPort();
var virtualLayers = lm.GetAllNonTemporary();
canvasViewPort.Save();
canvasViewPort.Clear();
canvasViewPort.SetBlendMode(virtualLayers[0].GetBlendMode());
canvasViewPort.ApplyBlendMode();
canvasViewPort.DrawImage(virtualLayers[0], 0, 0, canvasViewPort.Width(), canvasViewPort.Height(), 0, 0, canvasViewPort.Width(), canvasViewPort.Height());
for (var i = 1; i < virtualLayers.length; ++i) {
var layer = virtualLayers[i];
var shift = other.GetShift();
canvasViewPort.SetBlendMode(virtualLayers[i].GetBlendMode());
canvasViewPort.ApplyBlendMode();
canvasViewPort.DrawImage(layer, 0, 0, layer.Width(), layer.Height(), shift.x, shift.y, layer.Width(), layer.Height());
}
canvasViewPort.Restore();
}
Don't be affraid of this code. It maintains underlaying canvas, pure CanvasRenderingContext2D.
I will perform such optimization: I Won't redraw canvasViewPort until something changes on any virtual canvas.
Next optimization I would perform: Get selected layer, cache all before and all after current layer.
So it will looks like:
CurrentLayer: 3 (which might be affected by some tool)
[1]:L1
[2]:L2
[3]:L3
[4]:L4
[5]:L5
[6]:L6
Draw to temporary with appropiate blend mode [1, 2] to tmp1 and [4, 5, 6] to tmp2.
When something change on third layer, I just need to redraw tmp1 + [3] + tmp2.
So it's just only 2 iteration's. Super fast.

Reading pixeldata from images in JavaScript returns unexpected results for semi transparent pixels

I am reading RGBA data from a png file in JavaScript. To do this I draw the image on a canvas and use .getImageData. The data differs from what I expect:
Test image:
https://raw.githubusercontent.com/FlorianLudwig/imghash/master/test/transparent.png
The third pixel's RGBA in JS is: [9, 0, 0, 54] (tested in chrome and firefox) but it should be [12, 0, 0, 54] (at least it is what gimp and python pillow claim).
My guesses what could be wrong
drawImage does some compositing (setting ctx.globalCompositeOperation to copy does not make any difference)
there is some color mangament going on at load time of the image
Example Code
http://codepen.io/anon/pen/EaemZw
Select the test image
check console output

1) Compositing
The more draw operations applied the more rounding errors will occur over time. However, with an initial draw this won't matter.
copy mode doesn't affect it as you discovered. This is just ignoring the existing background data. source-over do use alpha value of background, but as there is none initially it will have the same effect.
The main problem with this though is that there is general problems with alpha and rounding errors due to pre-multiplied alpha values which needs to be converted internally.
2) Color management
Browsers do support various levels of color management. This includes the following data from the PNG file:
sRGB chunk stating how the PNG conforms to the sRGB standard (relative, absolute etc.)
iCCP chunk which contains an ICC profile including gamma, that if the browser support ICC and also this version of ICC (typical versions are version 2 and 4).
If no sRGB or iCCP profile are found, it will look for gAMA chunk which contains a number representing the gamma.
When these data are applied it will alter the original bitmap. Gamma will not affect the lowest nor the highest values, but the mid-range will have a noticeable change. Values that are still altered will be altered due to color management and conversion errors (1.).
For example, even if the file does not have a gamma chunk (or a "file gamma"), which is the case for your test file, or its value is 1, a display gamma will still be applied (except in IE). For windows this would be 2.2 and for Mac typically 1.8.
And this happens before the image is returned to be used in the code with canvas.
Alternatively I would suggest taking a look at my pngtoy (it's free/MIT) which I created for this type of scenario, to enable developers to get a raw unaltered bitmap from a PNG (just see notes/status as it is currently still in alpha). I supplied a code example below which also reads the third pixel giving 12 for red channel as expected.
Test using pngtoy
This will show the raw unaltered bitmap value.
var out = document.querySelector("output"),
png = new PngToy();
png.fetch("https://i.imgur.com/oX1kom7.png").then(
function() {png.decode().then(show)}
);
function show(bmp) {
var data = bmp.bitmap;
out.innerHTML += data[8] + "," + data[9] + "," + data[10] + "," + data[11];
}
<script src="https://rawgit.com/epistemex/pngtoy/master/tests/Promise.js"></script>
<script src="https://rawgit.com/epistemex/pngtoy/master/pngtoy.min.js"></script>
<output>Loading external scripts (pngtoy.min.js + Promise polyfill for IE...)<br></output>

Your guess#2 is correct.
Browsers are given freedom by the spec to do Alpha & Gamma adjustments to incoming images.

How does 2D drawing frameworks such as Pixi.js make canvas drawing faster?

I found a bunnymark for Javascript canvas here.
Now of course, I understand their default renderer is using webGL but I am only interested in the native 2D context performance for now. I disabled webGL on firefox and after spawning 16500 bunnies, the counter showed a FPS of 25. I decided to wrote my own little very simple rendering loop to see how much overhead Pixi added. To my surprise, I only got a FPS of 20.
My roughly equivalent JSFiddle.
So I decided to take a look into their source here and it doesn't appear to be that the magic is in their rendering code:
do
{
transform = displayObject.worldTransform;
...
if(displayObject instanceof PIXI.Sprite)
{
var frame = displayObject.texture.frame;
if(frame)
{
context.globalAlpha = displayObject.worldAlpha;
context.setTransform(transform[0], transform[3], transform[1], transform[4], transform[2], transform[5]);
context.drawImage(displayObject.texture.baseTexture.source,
frame.x,
frame.y,
frame.width,
frame.height,
(displayObject.anchor.x) * -frame.width,
(displayObject.anchor.y) * -frame.height,
frame.width,
frame.height);
}
}
Curiously, it seems they are using a linked list for their rendering loop and a profile on both app shows that while my version allocates the same amount of cpu time per frame, their implementation shows cpu usage in spikes.
My knowledge ends here unfortunately and I am curious if anyone can shed some light on whats going on.

I think, in my opinion, that it boils down to how "compilable" (cache-able) the code is. Chrome and Firefox uses two different JavaScript "compilers"/engines as we know which optimizes and caching code differently.
Canvas operations
Using transform versus direct coordinates should not have an impact as setting a transform merely updates the matrix which is in any case is used with what-ever is in it.
The type of position values can affect performance though, float versus integer values, but as both your fiddle and PIXI seem to use floats only this is not the key here.
So here I don't think canvas is the cause of the difference.
Variable and property caching
(I got unintentionally too focused on the prototypal aspect in the first version of this answer. The essence I was trying to get at was mainly object traversing, so here the following text is re-worded a bit -)
PIXI uses object properties as the fiddle but these custom objects in PIXI are smaller in size so the traversing of the object tree takes less time compared to what it takes to traverse a larger object such as canvas or image (a property such as width would also be at the end of this object).
It's a well known classic optimization trick to cache variables due to this very reason (traverse time). The effect is less today as the engines has become smarter, especially V8 in Chrome which seem to be able to predict/cache this better internally, while in Firefox it seem to still have a some impact not to cache these variables in code.
Does it matter performance-wise? For short operations very little, but drawing 16,500 bunnies onto canvas is demanding and do gain a benefit from doing this (in FF) so any micro-optimization do actually count in situations such as this.
Demos
I prototyped the "renderer" to get even closer to PIXI as well as caching the object properties. This gave a performance burst in Firefox:
http://jsfiddle.net/AbdiasSoftware/2Dbys/8/
I used a slow computer (to scale the impact) which ran your fiddle at about 5 FPS. After caching the values it ran at 6-7 fps which is more than 20% increase on this computer showing it do have an effect. On a computer with a larger CPU instruction cache and so forth the effect may be less, but it's there as this is related to the FF engine itself (disclaimer: I am not claiming this to be a scientific test however, only a pointer :-) ).
/// cache object properties
var lastTime = 0,
w = canvas.width,
h = canvas.height,
iw = image.width,
ih = image.height;
This next version caches these variables as properties on an object (itself) to show that also this improves performance compared to using large global objects directly - result about the same as above:
http://jsfiddle.net/AbdiasSoftware/2Dbys/9/
var RENDER = function () {
this.width = canvas.width;
this.height = canvas.height;
this.imageWidth = image.width;
this.imageHeight = image.height;
}
In conclusion
I am certain based on the results and previous experience that PIXI can run the code faster due to using custom small-sized objects rather than getting the properties directly from large objects (elements) such as canvas and image.
The FF engine seem not yet to be as "smart" as the V8 engine in regard to object traversing of tree and branches so caching variables do have an impact in FF which comes to display when the demand is high (such as when drawing 16,500 bunnies per "frame").

One difference I noticed between your version and Pixi's is this:
You render image at certain coordinates by passing x/y straight to drawImage function:
drawImage(img, x, y, ...);
..whereas Pixi translates entire canvas context, and then draws image at 0/0 (of already shifted context):
setTransform(1, 0, 0, 1, x, y);
drawImage(img, 0, 0, ...);
They also pass more arguments to drawImage; arguments that control "destination rectangle" — dx, dy, dw, dh.
I suspected this is where speed difference hides. However, changing your test to use same "technique" doesn't really make things better.
But there's something else...
I clocked bunnies to 5000, disabled WebGL, and Pixi actually performs worse than the custom fiddle version.
I get ~27 FPS on Pixi:
and ~32-35 FPS on Fiddle:
This is all on Chrome 33.0.1712.4 dev, Mac OS X.

I'd suspect that this is some canvas compositing issue. Canvas is transparent by default, so the page background needs to be combined with the canvas contents...
I found this in their source...
// update the background color
if (this.view.style.backgroundColor != stage.backgroundColorString &&
!this.transparent) {
this.view.style.backgroundColor = stage.backgroundColorString;
}
Maybe they set the canvas to be opaque for this demo (the fiddle doesn't really work for me, seems like most of the bunnys jump out with an extremely large dt most of the time)?
I don't think it's an object property access timing / compilability thing: The point is valid, but I don't think it can explain that much of a difference.

Performance issue with off-screen canvas in Firefox

I'm having the following problem, and I believe I must be doing something stupid, but I can't find it. (This is in Firefox 8)
I'm taking a large sprite file, making a new small canvas to hold just one tile of that sprite, and then using the most basic overload of drawImage to draw this isolated tile in hundreds of places in my "screen" canvas.
I'm doing this instead of simply using the last overload of drawImage, that takes only a piece of the larger sprite file. By avoiding this clipping, in Chrome, I get a performance increase of about 10%, to my surprise. However, in Firefox, frame rate drops from 300 to 17 FPS.
So, what I'm seeing essentially is that drawing "from image to canvas" is about 20 times faster than drawing "from canvas to canvas" in Firefox.
Is this correct? I haven't found any information on this particular case, but this is what I see on my tests.
Also, this the code I'm using. Am I doing anything incredibly stupid?
function Test5() {
var imgSprite = $('imgSprite');
var tileCanvas = document.createElement("canvas");
tileCanvas.width = 64; tileCanvas.height = 31;
var tileCtx = tileCanvas.getContext("2d");
tileCtx.drawImage(imgSprite, 0, 0, 64, 31, 0, 0, 64, 31);
//--------------------------------------
var ctx = getContext('mainScreen');
ctx.fillStyle = '#fff';
time(function() { // time will run this function many times and time it
ctx.fillRect(0,0, 1200,900);
var x=0, y=0, row = 0;
for (var i=1; i < 1000; i++) {
ctx.drawImage(tileCanvas, x,y); // <-- this is the line I care about
// some simple code to calculate new x/y
}
}, 1000, "Test5", 'Drawing from an individual tile canvas, instead of a section of big sprite');
}
If, instead of
ctx.drawImage(tileCanvas, x,y);
I do :
ctx.drawImage(imgSprite, 0, 0, 64, 31, x, y, 64, 31);
That's 20 times faster
Am I missing something here?
EDIT: After asking this I made a little page for myself to test several different things in different platforms and see what was the best way to do things in each.
http://www.crystalgears.com/isoengine/jstests3-canvas.html
I'm sorry that code is horrifically ugly, it was a quick hack, not meant to be seen by others, or even myself in a few days.

It's hard to be sure without profiling (esp. with your particular images and maybe your particular graphics card+driver), but you may be hitting https://bugzilla.mozilla.org/show_bug.cgi?id=705559
Certainly that bug should cause drawImage with <canvas> argument to be somewhat slower; though 20x slower is very surprising (hence the "may").

HTML5 canvas: Same code outputs different results in different browsers

In a simple canvas test I created for performance and quality measurement purposes, a canvas is painted with randomised colors and images during an unlimited period.
A sample is shown here: http://litterific.com/minisite/
Warning: Only open this in Opera or Chrome, the script is pretty heavy can hang up on slow computers, don't leave the script running while you are getting coffee ;)) It is just a rough prototype and did not optimize it.
What I noticed here is that the results as painted by the script (js/asset.js) are different in various browsers. Especially in Opera there is much more "green" in the painting than in Chrome
code is found here: http://litterific.com/minisite/js/asset.js
My question is:
How this is caused. Different random seeds? Different rounding or different color behavior in Opera?
Note: It is exactly the same script in both browsers, so perhaps you could have a look at it in both Chrome and Opera.

It's not random numbers causing the problems, it's "funny" pixel data. Here's the change:
for (i = 0, n = pixels.data.length; i < n; i += 4){
pixels.data[i + 0] = Math.max(0, Math.min(255, Math.floor(r * f)));
pixels.data[i + 1] = Math.max(0, Math.min(255, Math.floor(g * f)));
pixels.data[i + 2] = 0;
pixels.data[i + 3] = pixels.data[i + 3];
}
If you ensure that the pixel values are integers in the right range, Opera works fine.
Oh also, and this is probably obvious, it goes a lot faster if you hoist those multiplications out of the loop.

As you guessed, Math.random starts with a different seed in each case. There is unfortunately no way to provide a fixed seed to the Math.random function. If you really need that, you will have to find one or implement it yourself.
I have noticed that different canvas implementations do vary slightly when drawing partially opaque objects, but that is a minor issue compared to your differing random sequences!
Btw, your script does produce nice looking output :)