When using the canvas.getContext('2d') to load the png file which has a transparent part, it looks exactly the same as the png file itself. But when loading by canvas.getContext('webgl'), it will display as white in the transparent part. And then if you add discard in the shader, it will be better but still not perfect as the png file. How to fix this issue?
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, this.img);
void main() {
vec4 color = texture2D(u_image, v_texCoord);
if(color.a < 0.5) {
discard;
}
gl_FragColor = color;
}
It sounds like you may need to activate blending.
gl.enable(gl.BLEND);
And then set the blending function to work with pre-multiplied alpha (the default)
gl.blendFunc(gl.ONE, gl.ONE_MINUS_SRC_ALPHA);
Transparency is actually kind of complicated
There is
#1 what the canvas itself needs
The default canvas wants premultiplied alpha. In other words it wants you to provide RGBA values where RGB has been multiplied by A.
You can set the canvas so it does not expect premultiplied alpha when creating the webgl context by passing in premultipledAlpha: false as in
const gl = someCanvas.getContext('webgl', {premultipliedAlpha: false});
Note: IIRC This doesn't work on iOS.
#2 what format you load the images
The default for loading images in WebGL is unpremultiplied alpha. In other words if the image has a pixel that is
255, 128, 64, 128 RGBA
It will be loaded exactly like that (*)
You can tell WebGL to premultiply for you when loading an image by setting
gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true);
Before calling gl.texImage2D.
Now that same pixel will above will end up being
128, 64, 16, 128 RGBA
Each of RGB has been multiplied by A (A above is 128 where 128 represents 128/255 or 0.5019607843137255)
#3 what you write out in your shaders
If you loaded un-premultiplied data you might choose to premultiply in your shader
gl_FragColor = vec4(someColor.rgb * someColor.a, someColor.a);
#4 how you blend
If you want to blend what you are drawing into what has already been drawn then you need to turn on blending
gl.enable(gl.BLEND);
But you also need to set how the blending happens. There are multiple functions that affect blending. The most common one to use is gl.blendFunc which sets how the src pixel (the one generated by your shader) and the dst pixel (the one being drawn on top of in the canvas) are affected before being combined. The 2 most common settings are
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA); // unpremultiplied alpha
and
gl.blendFunc(gl.ONE, gl.ONE_MINUS_SRC_ALPHA); // premultiplied alpha
The first argument is how to multiply the src pixel. Above we are either multiplying by the alpha of the src (SRC_ALPHA) or by 1 (ONE). The second argument is how to multiply the dst pixel. ONE_MINUS_SRC_ALPHA is exactly what it says (1 - alpha)
How you put all these together is up to you.
This article and This one somewhat cover these issues
(*) Images may have color conversion applied.
Related
Originally I was working with 2D Textures and everything was fine because we are allowed to use UNPACK_FLIP_Y_WEBGL. However, in WebGL2 we are now able to utilize 3D Textures. The problem is that my output is still upside-down and now the UNPACK_FLIP_Y_WEBGL is flagged as illegal for 3D Textures.
Does any one know how to fix this?
Below is some sample code that is similar to the config setup I have for loading the 3D Texture (if I was loading a 2D texture then this would work as long as I have the pixelStorei set to UNPACK_FLIP_Y_WEBGL).
texImage3D: [
gl.TEXTURE_3D, // target
0, // mip level
gl.RGB8, // sized (internal) format
size, // width
size, // height
size, // depth
0, // border
gl.RGB, // base format
gl.UNSIGNED_BYTE, // type
data, // Uint8Array color look up table
]
texParameteri3D: [
[gl.TEXTURE_3D, gl.TEXTURE_BASE_LEVEL, 0],
[gl.TEXTURE_3D, gl.TEXTURE_MAX_LEVEL, 0],
[gl.TEXTURE_3D, gl.TEXTURE_MAG_FILTER, gl.LINEAR],
[gl.TEXTURE_3D, gl.TEXTURE_MIN_FILTER, gl.LINEAR],
]
pixelStorei3D: [gl.UNPACK_ALIGNMENT, 1]
gl.getParameter(gl.MAX_VERTEX_UNIFORM_VECTORS);
gl.getParameter(gl.MAX_FRAGMENT_UNIFORM_VECTORS);
Found out that in WebGL tiles are rendered from bottom-left positioning. Meaning we need to flip the Y value. So in the vertex shader file we can just add some math to it to fix this. I don't know why the UNPACK_FLIP_Y_WEBGL was made to do this and then not supported for 3D Textures when the fix and what that UNPACK does is super simple. You just need to do 1 - y in the vertex tile position value.
#version 300 es
in vec2 tile_position;
in vec4 vertex_position;
out vec2 vertex_tile_position;
void main() {
vertex_tile_position = vec2(tile_position.s, 1.0 - tile_position.t);
gl_Position = vec4(vertex_position);
}
I am developing a pose recognition webapp using mediapipe pose library (https://google.github.io/mediapipe/solutions/pose.html).
I am using the segmentationMask to find some specific points of the human body that satisfy a constraint (the value in the n-th pixel must be > 0.1).
I'am able to do this evaluation in python. The library returns the segmentation mask as a matrix with the same width and height as the input image, and contains values in [0.0, 1.0] where 1.0 and 0.0 indicate high certainty of a “human” and “background” pixel respectively. So I can iterate over the matrix and I am able to find the point that satisfy the constraint.
I am trying to do the same thing in javascript, but I have a problem. The The javascript version of the library does not return a matrix but returns an ImageBitmap used by the html canvas to draw the mask.
The problem is that with ImageBitmap I cannot access every point of the matrix and I am not able to find the points I am interested in.
Is there a way to transform the javascript segmentationMask ImageBitmap in order be similar to the segmenationMask of the python versione library or at least retrive the same informations (I need the values included in this range [0.0, 1.0] for every pixel of the image).
Thank you all.
There is unfortunately no direct way to get an ImageData from an ImageBitmap, but you can drawImage() this ImageBitmap on a clear canvas and then call ctx.getImageData(0, 0, canvas.width, canvas.height) to retrieve an ImageData where you'll get access to all the pixels data.
The confidence will be stored in the Alpha channel (every fourth item in imageData.data) as a value between 0 and 255.
function onResults(results) {
canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
canvasCtx.drawImage(results.segmentationMask, 0, 0,
canvasElement.width, canvasElement.height);
const imgData = canvasCtx.getImageData(0, 0, canvasElement.width, canvasElement.height);
let i = 0;
for (let y = 0; y<imgData.height; y++) {
for (let x = 0; x<imgData.width; x++) {
const confidence = imgData.data[i + 3];
// do something with confidence here
i++;
}
}
}
And since you're gonna read a lot from that context, don't forget to pass the willReadFrequently option when you get it.
As a fiddle since StackSnippets won't allow the use of the camera.
Note that depending on what you do you may want to colorize this image from red to black using globalCompositeOperation and treat the data as an Uint32Array where the confidence would be expressed between 0 and 0xFF000000.
In short
I would like to read a single pixel value from a WebGL 2 depth texture in JavaScript. Is this at all possible?
The scenario
I am rendering a scene in WebGL 2. The renderer is given a depth texture to which it writes the depth buffer. This depth texture is used in post processing shaders and the like, so it is available to us.
However, I need to read my single pixel value in JavaScript, not from within a shader. If this had been a normal RGB texture, I would do
function readPixel(x, y, texture, outputBuffer) {
const frameBuffer = gl.createFramebuffer();
gl.bindFramebuffer( gl.FRAMEBUFFER, frameBuffer );
gl.framebufferTexture2D( gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, texture, 0 );
gl.readPixels(x, y, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, outputBuffer);
}
This will write the pixel at x, y into outputBuffer.
However, is it at all possible to do the same with a depth texture? If I just pass a depth texture to my function above, the output buffer only has zeros, and I receive a WebGL warning GL_INVALID_FRAMEBUFFER_OPERATION: Framebuffer is incomplete.. Checking the framebuffer state reveals FRAMEBUFFER_INCOMPLETE_ATTACHMENT.
Naturally, the depth texture is not an RGBA texture, but is there some other values we can give it to get our depth value, or is it impossible?
Motivation
I am aware of that this question has been asked some number of times on StackOverflow and elsewhere in some form of another, but there is always some variation making it confusing for me to get a straight-up yes or no answer to the question in the form I ask it here. In addition, many questions and sources are very old, WebGL 1 only, with some mentions of webgl_depth_texture making a difference etc etc.
If the answer is no, I'd welcome any suggestions for how else to easily obtain this depth pixel. As this operation is not done for every frame, I value simplicity over performance. The use case is picking, and classical ray intersection is not feasible. (I also know that I can encode a scalar depth value into and out of an RGB pixel, but I need to be able to access the pixel from within the js code in the first place.)
I'd welcome any insights.
There is no possibility WebGL 2.0 is based on OpenGL ES 3.0.
In OpenGL ES 3.2 Specification - 4.3.2 Reading Pixels is clearly specified:
[...] The second is an implementation-chosen format from among those defined
in table 3.2, excluding formats DEPTH_COMPONENT and DEPTH_STENCIL [...]
I have been playing with GPUComputationRenderer on a modified version of this three.js example which modifies the velocity of interacting boids using GPU shaders to hold, read and manipulate boid position and velocity data.
I have got to a stage where I can put GPU computed data (predicted collision times) into the texture buffer using the shader. But now I want to read some of that texture data inside the main javascript animation script (to find the earliest collision).
Here is the relevant code in the render function (which is called on each animation pass)
//... GPU calculations as per original THREE.js example
gpuCompute.compute(); //... gpuCompute is the gpu computation renderer.
birdUniforms.texturePosition.value = gpuCompute.getCurrentRenderTarget( positionVariable ).texture;
birdUniforms.textureVelocity.value = gpuCompute.getCurrentRenderTarget( velocityVariable ).texture;
var xTexture = birdUniforms.texturePosition.value;//... my variable, OK.
//... From http://zhangwenli.com/blog/2015/06/20/read-from-shader-texture-with-threejs/
//... but note that this reads from the main THREE.js renderer NOT from the gpuCompute renderer.
//var pixelBuffer = new Uint8Array(canvas.width * canvas.height * 4);
//var gl = renderer.getContext();
//gl.readPixels(0, 0, canvas.width, canvas.height, gl.RGBA, gl.UNSIGNED_BYTE, pixelBuffer);
var pixelBuffer = new Uint8Array( WIDTH * WIDTH * 4); //... OK.
//var gl = gpuCompute.getContext();//... no getContext function!!!
//... from Nick Whaley here: http://stackoverflow.com/questions/13475209/three-js-get-data-from-three-webglrendertarget
//WebGLRenderer.readRenderTargetPixels ( renderTarget, x, y, width, height, buffer )
gpuCompute.readRenderTargetPixels ( xTexture, 0, 0, WIDTH, WIDTH, pixelBuffer ); //... readRenderTargetPixels is not a function!
As shown in the code I was "wanting" the gpuCompute renderer object to provide functions such as .getContext() or readRenderTargetPixels() but they do not exist for gpuCompute.
EDIT:
Then I tried adding the following code:-
//... the WebGLRenderer code is included in THREE.js build
myWebglRenderer = new THREE.WebGLRenderer();
var myRenderTarget = gpuCompute.getCurrentRenderTarget( positionVariable );
myWebglRenderer.readRenderTargetPixels (
myRenderTarget, 0, 0, WIDTH, WIDTH, pixelBuffer );
This executes OK but pixelBuffer remains entirely full of zeroes instead of the desired position coordinate values.
Please can anybody suggest how I might read the texture data into a pixel buffer? (preferably in THREE.js/plain javascript because I am ignorant of WebGL).
This answer is out of date. See link at bottom
The short answer is it won't be easy. In WebGL 1.0 there is no easy way to read pixels from floating point textures which is what GPUComputationRenderer uses.
If you really want to read back the data you'll need to render the GPUComputationRenderer floating point texture into an 8bit RGBA texture doing some kind of encoding from 32bit floats to 8bit textures. You can then read that back in JavaScript and look at the values.
See WebGL Read pixels from floating point render target
Sorry for the long delay. I've not logged in in SO for a long time.
In the example of water with tennis balls,
https://threejs.org/examples/?q=water#webgl_gpgpu_water
The height of the water at the balls positions is read back from the GPU.
An integer 4-component texture is used to give a 1-component float texture.
The texture has 4x1 pixels, where the first one is the height and the other 2 are the normal of the water surface (the last pixel is not used)
This texture is computed and read back for each one of the tennis balls, and in CPU the ball physics is performed.
I'm trying to learn how to take advantage of gpu possibilities for threejs and webgl stuff so im just analysing code to get some patterns, methods how things are done and I need some code explanation.
I found this example: One million particles, which seems to be the easiest one involving calculations made in shaders and spit back out.
So from what I have figured out:
- Data for velocity and position of particles are kept in textures passed to shaders to perform calculations there, and get them back for update
Particles are created randomly on the plane no more than the texture size ?
for (var i = 0; i < 1000000; i++) {
particles.vertices.push(new THREE.Vector3((i % texSize)/texSize,
Math.floor(i/texSize)/texSize , 0))
;
}
I don't see any particles position updates? How is the data from shaders retrieved and updates each particle?
pick()
only passes the mouse position to calculate the direction of particles movement?
why are there 2 buffers? and 8 (4 pairs of fragment and vector) shaders? Is only the one for calculating velocity and position not enough?
how does the shader update the texture? I just see reading from it not writing to it?
Thanks in advance for any explanations!
How the heck have they done that:
In this post, I'll explain how this results get computed nearly solely on the gpu via WebGL/Three.js - it might look a bit sloppy as I'm using integrated graphics of an Intel i7 4770k:
Introduction:
Simple idea to keep everything intra-gpu: Each particle's state will be represented by one texture pixel color value. One Million particles will result in 1024x1024 pixel textures, one to hold the current position and another one that holds the velocities of those particles.
Nobody ever forbid to abuse the RGB color values of a texture for completely different data of 0...255 universe. You basically have 32-bit (R + G + B + alpha) per texture pixel for whatever you want to save in GPU memory. (One might even use multiple texture pixels if he needs to store more data per particle/object).
They basically used multiple shaders in a sequential order. From the source code, one can identify these steps of their processing pipeline:
Randomize particles (ignored in this answer) ('randShader')
Determine each particles velocity by its distance to mouse location ('velShader')
Based on velocity, move each particle accordingly ('posShader')
Display the screen ('dispShader')**
.
Step 2: Determining Velocity per particle:
They call a draw process on 1 Million points which's output will be saved as a texture. In the vertex shader each fragment gets 2 additional varyings named "vUv", which basically determine the x and y pixel positions inside the textures used in the process.
Next step is its fragment shader, as only this shader can output (as RGB values into the framebuffer, which gets converted to a texture buffer afterwards - all happening inside gpu memory only). You can see in the id="velFrag" fragment shader, that it gets an input variable called uniform vec3 targetPos;. Those uniforms are set cheaply with each frame from the CPU, because they are shared among all instances and don't involve large memory transfers. (containing the mouse coordinate, in -1.00f to +1.00f universe probably - they probably also update mouse coords once every FEW frames, to lower cpu usage).
Whats going on here? Well, that shader calculates the distance of that particle to the mouse coordinate and depending on that it alter that particles velocity - the velocity also holds information about the particles flight direction. Note: this velocity step also makes particles gain momentum and keep flying/overshooting mouse position, depending on gray value.
.
Step 3: Updating positions per particle:
So far each particle got a velocity and an previous position. Those two values will get processed into a new position, again being outputted as a texture - this time into the positionTexture. Until the whole frame got rendered (into default framebuffer)and then marked as the new texture, the old positionTexture remains unchanged and can get read with ease:
In id="posFrag" fragment shader, they read from both textures (posTexture and velTexture) and process this data into a new position. They output the x and y position coordinates into the colors of that texture (as red and green values).
.
Step 4: Prime time (=output)
To output the results, they probably took again a million points/vertexes and gave it the positionTexture as an input. Then the vertex shader sets the position of each point by reading the texture's RGB value at location x,y (passed as vertex attributes).
// From <script type="x-shader/x-vertex" id="dispVert">
vec3 mvPosition = texture2D(posTex, vec2(x, y)).rgb;
gl_PointSize = 1.0;
gl_Position = projectionMatrix * modelViewMatrix * vec4(mvPosition,1.0);
In the display fragment shader, they only need to set a color (note the low alpha, causing it to allow 20 particles to stack up to fully light up a pixel).
// From <script type="x-shader/x-fragment" id="dispFrag">
gl_FragColor = vec4(vec3(0.5, 1.0, 0.1), 0.05);
.
I hope this made it clear how this little demo works :-) I am not the author of that demo, though. Just noticed this answer actually became a super duper detailed one - fly through the thick keywords to get the short version.