The basic idea is that a pointer is returned and then an image will be displayed using the pointer after creating a Vec at each video frame:
JavaScript:
var ptr = exports.alloc(size);
let bytes = new Uint8ClampedArray(exports.memory.buffer, ptr, size);
let image = new ImageData(bytes, 320, 240);
function tick() {
requestAnimationFrame(tick);
ctx.drawImage(video, 0, 0, width, height);
let imageData = ctx.getImageData(0, 0, width, height).data;
bytes.set(imageData);
exports.create_vector();
ctx.putImageData(img, 0, 0);
}
Rust:
#[no_mangle]
pub extern "C" fn alloc(capacity: usize) -> *mut c_void {
let mut buf = Vec::with_capacity(capacity);
let ptr = buf.as_mut_ptr();
mem::forget(buf);
return ptr as *mut c_void;
}
#[no_mangle]
pub extern "C" fn create_vector() {
let _: Vec<u8> = Vec::with_capacity(320 * 240);
}
Here is the actual code.
Here are the error messages:
Chrome:
Uncaught TypeError: Cannot perform %TypedArray%.prototype.set on a detached ArrayBuffer
at Uint8ClampedArray.set (<anonymous>)
at tick
Safari:
This webpage was reloaded because a problem occurred
Mozilla:
DOMException: "An attempt was made to use an object that is not, or is no longer, usable"
TypeError: Underlying ArrayBuffer has been detached from the view
The main culprits seem to be:
let bytes = new Uint8ClampedArray(exports.memory.buffer, ptr, size);
// ...
exports.create_vector();
// ...
i.e., the browser crashes when I try to use ptr again after calling exports.create_vector.
What is going wrong here? Is there a solution?
When you allocate a new buffer and there is not enough memory allocated by the WASM "process" for it, the browser will allocate a new, larger buffer and will copy all data into it from the old location. It's quite transparent inside the WASM interpreter, but all pointers passed to JavaScript (with which you construct Uint8ClampedArray) get invalidated. There are two solutions to my knowledge:
Allocate enough memory in the WASM "process" at initialization and immediately deallocate it so pointers will not get invalidated so long as memory usage is lower than the selected size.
Keep track of buffers passed to JavaScript and "renew" pointers on such errors.
Related
I am trying to use the posenet MobileNetV1 network in an electron app. I want to be able to read image from file system (it does not matter if it is a png or jpg), and run it through the network.
What have I done so far:
I am using the following modules:
import * as posenet from '#tensorflow-models/posenet';
var imageToBase64 = require('image-to-base64');
var toUint8Array = require('base64-to-uint8array')
And initializing the network with:
var net = posenet.load();
In order to read image I am converting it to base64 than to Uint8Array, than I am using them to create an object {data: bytes, width: width, height: height}, which is fitting in the definition of ImageData.
Everything is running but the results in percentages are very low:
{
score: 0.002851587634615819,
keypoints: [
{ score: 0.0007664567674510181, part: 'nose', position: [Object] },
{
score: 0.0010295170359313488,
part: 'leftEye',
position: [Object]
},
{
score: 0.0006740405224263668,
part: 'rightEye',
position: [Object]
},
Notice that in the future I intend to build this app so modules like Canvas are no good since it does not build well.
If someone could give me a working poc it would be great since I am working on that for a very long time.
electron has two separated contexts; one that can be considered as a server side context called the main context and the renderer context in which the browser and its scripts are called. Though the question is not precise enough, it is trying to execute posenet in the main context of electron which can be compared as if one is trying to run this code in nodejs
posenet in the main renderer
const data = Buffer.from(base64str, 'base64')
const t = tf.node.decodeImage(data)
const net = await posenet.load()
const poses = net.estimateMultiplePoses(t, {
flipHorizontal: false,
maxDetections: 2,
scoreThreshold: 0.6,
nmsRadius: 20})
})
// do whatever with the poses
posenet from a script executed by the browser
const im = new Image()
im.src = base64str
const net = await posenet.load()
im.onload = async() => {
const poses = await net.estimateMultiplePoses(im, {
flipHorizontal: false,
maxDetections: 2,
scoreThreshold: 0.6,
nmsRadius: 20})
})
// do whatever with the poses
}
Even though you copied the structure, maybe PoseNet is checking the if the Object is of a certain class which it will not be unless you actually create ImageData object and then set the fields. That is my guess as to why it doesn't like it.
Have you tried:
let clamped = Uint8ClampedArray.from(someframeBuffer);
let imageData = new ImageData(clamped, width, height);
It seems PoseNet accepts ImageData|HTMLImageElement|HTMLCanvasElement|HTMLVideoElement objects that can be passed to its prediction function.
i want to read back the data stored in the GL.bufferData array in javascript.
Here is my code
var TRIANGLE_VERTEX = geometryNode["triangle_buffer"];
GL.bindBuffer(GL.ARRAY_BUFFER, TRIANGLE_VERTEX);
GL.bufferData(GL.ARRAY_BUFFER,new Float32Array(vertices),GL.STATIC_DRAW);
is it possible in webgl to read back the bufferdata in GPU?
if possible then please explain me with a sample code.
How to know the memory size(filled and free) of the Gpu in webgl at run time and how to debug the shader code and data in GPU in webgl.
It is not directly possible to read the data back in WebGL1. (see below for WebGL2). This is limitation of OpenGL ES 2.0 on which WebGL is based.
There are some workarounds:
You could try to render that data to a texture then use readPixels to read the data.
You'd have to encode the data into bytes in your shader because readPixels in WebGL can only read bytes
You can wrap your WebGL to store the data yourself something like
var buffers = {};
var nextId = 1;
var targets = {};
function copyBuffer(buffer) {
// make a Uint8 view of buffer in case it's not already
var view = new Uint8Buffer(buffer.buffer);
// now copy it
return new UintBuffer(view);
}
gl.bindBuffer = function(oldBindBufferFn) {
return function(target, buffer) {
targets[target] = new Uint8Buffer(buffer.buffer);
oldBindBufferFn(target, buffer);
};
}(gl.bindBuffer.bind(gl));
gl.bufferData = function(oldBufferDataFn) {
return function(target, data, hint) {
var buffer = targets[target];
if (!buffer.id) {
buffer.id = nextId++;
}
buffers[buffer.id] = copyBuffer(data);
oldBufferDataFn(target, data, hint);
};
}(gl.bufferData.bind(gl)));
Now you can get the data with
data = buffers[someBuffer.id];
This is probably what the WebGL Inspector does
Note that there are a few issues with the code above. One it doesn't check for errors. Checking for errors would make it way slower but not checking for error will give you incorrect results if your code generates errors. A simple example
gl.bufferData(someBuffer, someData, 123456);
This would generate an INVALID_ENUM error and not update the data in someBuffer but our code isn't checking for errors so it would have put someData into buffers and if you read that data it wouldn't match what's in WebGL.
Note the code above is pseudo code. For example I didn't supply a wrapper for gl.bufferSubData.
WebGL2
In WebGL2 there is a function gl.getBufferSubData that will allow you to read the contents of a buffer. Note that WebGL2, even though it's based on OpenGL ES 3.0 does not support gl.mapBuffer because there is no performant and safe way to expose that function.
I'm using Emscripten to try to get an open source game to run in a browser. It compiles fine, loads all of its files any everything, but when I run it it get the following exception:
exception thrown: TypeError: surfData.colors32 is undefined,_SDL_FillRect#file:///home/misson20000/dev/js/game.js:9702:9
__ZN9Surface5ClearEhhh#file:///home/misson20000/dev/js/game.js:112026:3
...
_main#file:///home/misson20000/dev/js/game.js:10525:11
asm._main#file:///home/misson20000/dev/js/game.js:170793:10
callMain#file:///home/misson20000/dev/js/game.js:173065:15
doRun#file:///home/misson20000/dev/js/game.js:173122:7
run/<#file:///home/misson20000/dev/js/game.js:173134:7
The code that is calling SDL_FillRect (a simple clear function) follows:
SDL_FillRect(fSurface, NULL, MapColor(r, g, b));
MapColor is defined as
return SDL_MapRGB(fSurface->format, r, g, b);
Digging around in the source code for a bit reveals that the surface in question is a screen surface.
How can I made surfData.colors32 not be undefined?
The colors32 is used when you create SDL surface with SDL_HWPALETTE flag. To correctly use surface of this type you should call SDL_SetColors before SDL_FillRect. Take a look in src/library_sdl.js:
SDL_SetColors: function(surf, colors, firstColor, nColors) {
var surfData = SDL.surfaces[surf];
// we should create colors array
// only once cause client code
// often wants to change portion
// of palette not all palette.
if (!surfData.colors) {
var buffer = new ArrayBuffer(256 * 4); // RGBA, A is unused, but faster this way
surfData.colors = new Uint8Array(buffer);
surfData.colors32 = new Uint32Array(buffer);
}
//...
}
I have encountered a unknown to me anomaly in Javascript coding: I completely don't know why, but there are some functions that a compiler just won't return from, and it just 'sits' there, even though after the '}' bracket there are next lines of code.
Look at following example - I have a main function which contains calls to functions like initShaders(), initBuffers() or drawScene() one after another. And the inside of troublesome initShaders() looks like this:
<!DOCTYPE html>
<html>
<canvas id = 'c' width = '500' height = '500'></canvas>
<script src ='js/webgl-utils.js'></script>
<script src ='js/webgl-debug.js'></script>
<!-- MAIN WEBGL SCRIPT -->
<script>
var gl;
var offset;
function initWebGL() {
// Create space and try to create context for WebGL
var c = document.getElementById('c');
gl = null;
gl = c.getContext('experimental-webgl');
// Check if browser supports WebGL
if (!gl) {
alert("Unable to initialize WebGL. Your browser may not support it.");
}
// Continue only if WebGL is available and working
else {
// Javascript console debug tool
gl = WebGLDebugUtils.makeDebugContext(gl, throwOnGLError, logAndValidate);
offset = [1, 1];
// Choose background color
gl.clearColor(0, 0, 0, 1);
initShaders();
initBuffers();
drawScene();
}
}
var program;
function initShaders() {
program = gl.createProgram();
// Call this function when loadProgram function will load shaders
function initProgram(loadedProgram) {
program = loadedProgram;
gl.useProgram(program);
/*-- Specify ATTRIBUTES locations --*/
// Specify locations of variables passed to shader program
program.vertexPosAttrib = gl.getAttribLocation(program, 'aVertexPosition');
// Enable this attribute as an array
gl.enableVertexAttribArray(program.vertexPosAttrib);
program.offsetUniform = gl.getUniformLocation(program, 'uOffset');
// Set the format of an in-data
gl.vertexAttribPointer(program.vertexPosAttrib, vertexPosBuffer.itemSize, gl.FLOAT, false, 0 ,0);
/*-- Specify UNIFORMS locations --*/
// TODO exit this function...
};
// Load the shaders from the file
loadProgram('shaders/vshader-00.txt', 'shaders/fshader-00.txt', program, initProgram);
}
var vertexPosBuffer;
function initBuffers() {
vertexPosBuffer = generateQuad();
}
function drawScene() {
// Clear the scene before rendering
gl.clear(gl.COLOR_BUFFER_BIT);
/*-- Pass UNIFORMS to shader program --*/
gl.uniform2f(program.offsetUniform, offset[0], offset[1]);
// Draw some triangles with predefined amount of vertices
gl.drawArrays(gl.TRIANGLE_STRIP, 0, vertexPosBuffer.numItems);
}
initWebGL();
</script>
</html>
The thing is that after the command 'gl.vertexAttribPointer(...)' it just do nothing, while it should exit this function, go after loadProgram function and keep going. It works fine if I put initBuffers() at the end of loadProgram() and then call drawScene() inside innitBuffers(). But this is just unpractical. What is the cause of this issue?
EDIT: PS program variable has been globally declared earlier.
EDIT2: Pasted the whole code from the main file for better clarity.
In JavaScript, most things are non-blocking. the loadProgram call will return immediately. However by then the program is not loaded yet. When it is loaded, the callback initProgram will be called.
JavaScript is event driven and therefore the control flow may be a bit harder to follow.
If this is running in a browser, you may want to start a timer at the end of initProgram which should then call your 'main' function when it is fired. This mechanism would be somewhat similar to what is happening with loadProgram which calls initShaders when it is done loading the program.
I'm using some javascript to allow users to dynamically load a sketch on click to a canvas element using:
Processing.loadSketchFromSources('canvas_id', ['sketch.pde']);
If I call Processing.loadSketchFromSources(...) a second (or third...) time, it loads a second (or third...) .pde file onto the canvas, which is what I would expect.
I'd like for the user to be able to click another link to load a different sketch, effectively unloading the previous one. Is there a method I can call (or a technique I can use) to check if Processing has another sketch running, and if so, tell it to unload it first?
Is there some sort of Processing.unloadSketch() method I'm overlooking? I could simply drop the canvas DOM object and recreate it, but that (1) seems like using a hammer when I need a needle, and (2) it results in a screen-flicker that I'd like to avoid.
I'm no JS expert, but I've done my best to look through the processing.js source to see what other functions may exist, but I'm hitting a wall. I thought perhaps I could look at Processing.Sketches.length to see if something is loaded already, but simply pop'ing it off the array doesn't seem to work (didn't think it would).
I'm using ProcessingJS 1.3.6.
In case someone else comes looking for the solution, here's what I did that worked. Note that this was placed inside a closure (not included here for brevity) -- hence the this.launch = function(), blah blah blah... YMMV.
/**
* Launches a specific sketch. Assumes files are stored in
* the ./sketches subdirectory, and your canvas is named g_sketch_canvas
* #param {String} item The name of the file (no extension)
* #param {Array} sketchlist Array of sketches to choose from
* #returns true
* #type Boolean
*/
this.launch = function (item, sketchlist) {
var cvs = document.getElementById('g_sketch_canvas'),
ctx = cvs.getContext('2d');
if ($.inArray(item, sketchlist) !== -1) {
// Unload the Processing script
if (Processing.instances.length > 0) {
// There should only be one, so no need to loop
Processing.instances[0].exit();
// If you may have more than one, then use this loop:
for (i=0; i < Processing.instances.length; (i++)) {
// Processing.instances[i].exit();
//}
}
// Clear the context
ctx.setTransform(1, 0, 0, 1, 0, 0);
ctx.clearRect(0, 0, cvs.width, cvs.height);
// Now, load the new Processing script
Processing.loadSketchFromSources(cvs, ['sketches/' + item + '.pde']);
}
return true;
};
I'm not familiar with Processing.js, but the example code from the site has this:
var canvas = document.getElementById("canvas1");
// attaching the sketchProc function to the canvas
var p = new Processing(canvas, sketchProc);
// p.exit(); to detach it
So in your case, you'll want to keep a handle to the first instance when you create it:
var p1 = Processing.loadSketchFromSources('canvas_id', ['sketch.pde']);
When you're ready to "unload" and load a new sketch, I'm guessing (but don't know) that you'll need to clear the canvas yourself:
p1.exit();
var canvas = document.getElementById('canvas_id');
var context = canvas.getContext('2d');
context.clearRect(0, 0, canvas.width, canvas.height);
// Or context.fillRect(...) with white, or whatever clearing it means to you
Then, from the sound of things, you're free to attach another sketch:
var p2 = Processing.loadSketchFromSources('canvas_id', ['sketch2.pde']);
Again, I'm not actually familiar with that library, but this appears straightforward from the documentation.
As of processing.js 1.4.8, Andrew's accepted answer (and the other answers I've found in here) do not seem to work anymore.
This is what worked for me:
var pjs = Processing.getInstanceById('pjs');
if (typeof pjs !== "undefined") {
pjs.exit();
}
var canvas = document.getElementById('pjs')
new Processing(canvas, scriptText);
where pjs is the id of the canvas element where the scrips is being run.