We Keep Coding

NodeJS get pixel color from another window

How can you access the color of a pixel on the screen using NodeJS?
My use-case is for Windows, but a cross-platform solution would be ideal.
Here's an example in C# which could be helpful: C# get pixel color from another window

I ended up converting the linked C# answer to NodeJS by using node-ffi:
import ffi from "ffi";
import chroma from "chroma-js";
var user32 = new ffi.Library("user32", {
"GetDC": ["int32", ["int32"]],
"ReleaseDC": ["int32", ["int32", "int32"]],
});
var gdi32 = new ffi.Library("gdi32", {
"GetPixel": ["uint32", ["int32", "int32", "int32"]],
});
export function GetPixelColor(x: number, y: number) {
let hdc = user32.GetDC(0);
let pixel = gdi32.GetPixel(hdc, x, y);
user32.ReleaseDC(0, hdc);
let color = chroma(pixel & 0x000000FF, (pixel & 0x0000FF00) >> 8, (pixel & 0x00FF0000) >> 16);
return color.hex().substr(1); // to match with robotjs.getPixelColor() returns
}
To use:
let pixelColor = GetPixelColor(0, 0);
console.log("Color: #" + pixelColor);
UPDATE
Since then, I put together this library: https://github.com/Venryx/windows-ffi
It supports screenshotting of regions of the screen, rather than just individual pixels. This makes it much faster for reading image-data over an area, and is the solution I use nowadays.
Alternatives
windows-ss: https://github.com/Sxxov/windows-ss
screenshot-desktop: https://github.com/bencevans/screenshot-desktop
desktop-screenshot: https://github.com/johnvmt/node-desktop-screenshot
Note: Does not have a built-in way to access the raw pixel data. (you'd have to read and parse the file manually)
RobotJS: https://github.com/octalmage/robotjs
Warning: It currently causes crashes if you call getPixelColor when a UAC prompt is open. (see issue)

JavaScript ArrayBuffer "detaches" when constructing a new Rust Vec via WASM

The basic idea is that a pointer is returned and then an image will be displayed using the pointer after creating a Vec at each video frame:
JavaScript:
var ptr = exports.alloc(size);
let bytes = new Uint8ClampedArray(exports.memory.buffer, ptr, size);
let image = new ImageData(bytes, 320, 240);
function tick() {
requestAnimationFrame(tick);
ctx.drawImage(video, 0, 0, width, height);
let imageData = ctx.getImageData(0, 0, width, height).data;
bytes.set(imageData);
exports.create_vector();
ctx.putImageData(img, 0, 0);
}
Rust:
#[no_mangle]
pub extern "C" fn alloc(capacity: usize) -> *mut c_void {
let mut buf = Vec::with_capacity(capacity);
let ptr = buf.as_mut_ptr();
mem::forget(buf);
return ptr as *mut c_void;
}
#[no_mangle]
pub extern "C" fn create_vector() {
let _: Vec<u8> = Vec::with_capacity(320 * 240);
}
Here is the actual code.
Here are the error messages:
Chrome:
Uncaught TypeError: Cannot perform %TypedArray%.prototype.set on a detached ArrayBuffer
at Uint8ClampedArray.set (<anonymous>)
at tick
Safari:
This webpage was reloaded because a problem occurred
Mozilla:
DOMException: "An attempt was made to use an object that is not, or is no longer, usable"
TypeError: Underlying ArrayBuffer has been detached from the view
The main culprits seem to be:
let bytes = new Uint8ClampedArray(exports.memory.buffer, ptr, size);
// ...
exports.create_vector();
// ...
i.e., the browser crashes when I try to use ptr again after calling exports.create_vector.
What is going wrong here? Is there a solution?

When you allocate a new buffer and there is not enough memory allocated by the WASM "process" for it, the browser will allocate a new, larger buffer and will copy all data into it from the old location. It's quite transparent inside the WASM interpreter, but all pointers passed to JavaScript (with which you construct Uint8ClampedArray) get invalidated. There are two solutions to my knowledge:
Allocate enough memory in the WASM "process" at initialization and immediately deallocate it so pointers will not get invalidated so long as memory usage is lower than the selected size.
Keep track of buffers passed to JavaScript and "renew" pointers on such errors.

How to read webgl GL.bufferData in javascript

i want to read back the data stored in the GL.bufferData array in javascript.
Here is my code
var TRIANGLE_VERTEX = geometryNode["triangle_buffer"];
GL.bindBuffer(GL.ARRAY_BUFFER, TRIANGLE_VERTEX);
GL.bufferData(GL.ARRAY_BUFFER,new Float32Array(vertices),GL.STATIC_DRAW);
is it possible in webgl to read back the bufferdata in GPU?
if possible then please explain me with a sample code.
How to know the memory size(filled and free) of the Gpu in webgl at run time and how to debug the shader code and data in GPU in webgl.

It is not directly possible to read the data back in WebGL1. (see below for WebGL2). This is limitation of OpenGL ES 2.0 on which WebGL is based.
There are some workarounds:
You could try to render that data to a texture then use readPixels to read the data.
You'd have to encode the data into bytes in your shader because readPixels in WebGL can only read bytes
You can wrap your WebGL to store the data yourself something like
var buffers = {};
var nextId = 1;
var targets = {};
function copyBuffer(buffer) {
// make a Uint8 view of buffer in case it's not already
var view = new Uint8Buffer(buffer.buffer);
// now copy it
return new UintBuffer(view);
}
gl.bindBuffer = function(oldBindBufferFn) {
return function(target, buffer) {
targets[target] = new Uint8Buffer(buffer.buffer);
oldBindBufferFn(target, buffer);
};
}(gl.bindBuffer.bind(gl));
gl.bufferData = function(oldBufferDataFn) {
return function(target, data, hint) {
var buffer = targets[target];
if (!buffer.id) {
buffer.id = nextId++;
}
buffers[buffer.id] = copyBuffer(data);
oldBufferDataFn(target, data, hint);
};
}(gl.bufferData.bind(gl)));
Now you can get the data with
data = buffers[someBuffer.id];
This is probably what the WebGL Inspector does
Note that there are a few issues with the code above. One it doesn't check for errors. Checking for errors would make it way slower but not checking for error will give you incorrect results if your code generates errors. A simple example
gl.bufferData(someBuffer, someData, 123456);
This would generate an INVALID_ENUM error and not update the data in someBuffer but our code isn't checking for errors so it would have put someData into buffers and if you read that data it wouldn't match what's in WebGL.
Note the code above is pseudo code. For example I didn't supply a wrapper for gl.bufferSubData.
WebGL2
In WebGL2 there is a function gl.getBufferSubData that will allow you to read the contents of a buffer. Note that WebGL2, even though it's based on OpenGL ES 3.0 does not support gl.mapBuffer because there is no performant and safe way to expose that function.

Function calling HTML5 canvas routines doesn't exit as expected

I have encountered a unknown to me anomaly in Javascript coding: I completely don't know why, but there are some functions that a compiler just won't return from, and it just 'sits' there, even though after the '}' bracket there are next lines of code.
Look at following example - I have a main function which contains calls to functions like initShaders(), initBuffers() or drawScene() one after another. And the inside of troublesome initShaders() looks like this:
<!DOCTYPE html>
<html>
<canvas id = 'c' width = '500' height = '500'></canvas>
<script src ='js/webgl-utils.js'></script>
<script src ='js/webgl-debug.js'></script>
<!-- MAIN WEBGL SCRIPT -->
<script>
var gl;
var offset;
function initWebGL() {
// Create space and try to create context for WebGL
var c = document.getElementById('c');
gl = null;
gl = c.getContext('experimental-webgl');
// Check if browser supports WebGL
if (!gl) {
alert("Unable to initialize WebGL. Your browser may not support it.");
}
// Continue only if WebGL is available and working
else {
// Javascript console debug tool
gl = WebGLDebugUtils.makeDebugContext(gl, throwOnGLError, logAndValidate);
offset = [1, 1];
// Choose background color
gl.clearColor(0, 0, 0, 1);
initShaders();
initBuffers();
drawScene();
}
}
var program;
function initShaders() {
program = gl.createProgram();
// Call this function when loadProgram function will load shaders
function initProgram(loadedProgram) {
program = loadedProgram;
gl.useProgram(program);
/*-- Specify ATTRIBUTES locations --*/
// Specify locations of variables passed to shader program
program.vertexPosAttrib = gl.getAttribLocation(program, 'aVertexPosition');
// Enable this attribute as an array
gl.enableVertexAttribArray(program.vertexPosAttrib);
program.offsetUniform = gl.getUniformLocation(program, 'uOffset');
// Set the format of an in-data
gl.vertexAttribPointer(program.vertexPosAttrib, vertexPosBuffer.itemSize, gl.FLOAT, false, 0 ,0);
/*-- Specify UNIFORMS locations --*/
// TODO exit this function...
};
// Load the shaders from the file
loadProgram('shaders/vshader-00.txt', 'shaders/fshader-00.txt', program, initProgram);
}
var vertexPosBuffer;
function initBuffers() {
vertexPosBuffer = generateQuad();
}
function drawScene() {
// Clear the scene before rendering
gl.clear(gl.COLOR_BUFFER_BIT);
/*-- Pass UNIFORMS to shader program --*/
gl.uniform2f(program.offsetUniform, offset[0], offset[1]);
// Draw some triangles with predefined amount of vertices
gl.drawArrays(gl.TRIANGLE_STRIP, 0, vertexPosBuffer.numItems);
}
initWebGL();
</script>
</html>
The thing is that after the command 'gl.vertexAttribPointer(...)' it just do nothing, while it should exit this function, go after loadProgram function and keep going. It works fine if I put initBuffers() at the end of loadProgram() and then call drawScene() inside innitBuffers(). But this is just unpractical. What is the cause of this issue?
EDIT: PS program variable has been globally declared earlier.
EDIT2: Pasted the whole code from the main file for better clarity.

In JavaScript, most things are non-blocking. the loadProgram call will return immediately. However by then the program is not loaded yet. When it is loaded, the callback initProgram will be called.
JavaScript is event driven and therefore the control flow may be a bit harder to follow.
If this is running in a browser, you may want to start a timer at the end of initProgram which should then call your 'main' function when it is fired. This mechanism would be somewhat similar to what is happening with loadProgram which calls initShaders when it is done loading the program.

Dynamically "unload" a Processing JS sketch from canvas

I'm using some javascript to allow users to dynamically load a sketch on click to a canvas element using:
Processing.loadSketchFromSources('canvas_id', ['sketch.pde']);
If I call Processing.loadSketchFromSources(...) a second (or third...) time, it loads a second (or third...) .pde file onto the canvas, which is what I would expect.
I'd like for the user to be able to click another link to load a different sketch, effectively unloading the previous one. Is there a method I can call (or a technique I can use) to check if Processing has another sketch running, and if so, tell it to unload it first?
Is there some sort of Processing.unloadSketch() method I'm overlooking? I could simply drop the canvas DOM object and recreate it, but that (1) seems like using a hammer when I need a needle, and (2) it results in a screen-flicker that I'd like to avoid.
I'm no JS expert, but I've done my best to look through the processing.js source to see what other functions may exist, but I'm hitting a wall. I thought perhaps I could look at Processing.Sketches.length to see if something is loaded already, but simply pop'ing it off the array doesn't seem to work (didn't think it would).
I'm using ProcessingJS 1.3.6.

In case someone else comes looking for the solution, here's what I did that worked. Note that this was placed inside a closure (not included here for brevity) -- hence the this.launch = function(), blah blah blah... YMMV.
/**
* Launches a specific sketch. Assumes files are stored in
* the ./sketches subdirectory, and your canvas is named g_sketch_canvas
* #param {String} item The name of the file (no extension)
* #param {Array} sketchlist Array of sketches to choose from
* #returns true
* #type Boolean
*/
this.launch = function (item, sketchlist) {
var cvs = document.getElementById('g_sketch_canvas'),
ctx = cvs.getContext('2d');
if ($.inArray(item, sketchlist) !== -1) {
// Unload the Processing script
if (Processing.instances.length > 0) {
// There should only be one, so no need to loop
Processing.instances[0].exit();
// If you may have more than one, then use this loop:
for (i=0; i < Processing.instances.length; (i++)) {
// Processing.instances[i].exit();
//}
}
// Clear the context
ctx.setTransform(1, 0, 0, 1, 0, 0);
ctx.clearRect(0, 0, cvs.width, cvs.height);
// Now, load the new Processing script
Processing.loadSketchFromSources(cvs, ['sketches/' + item + '.pde']);
}
return true;
};

I'm not familiar with Processing.js, but the example code from the site has this:
var canvas = document.getElementById("canvas1");
// attaching the sketchProc function to the canvas
var p = new Processing(canvas, sketchProc);
// p.exit(); to detach it
So in your case, you'll want to keep a handle to the first instance when you create it:
var p1 = Processing.loadSketchFromSources('canvas_id', ['sketch.pde']);
When you're ready to "unload" and load a new sketch, I'm guessing (but don't know) that you'll need to clear the canvas yourself:
p1.exit();
var canvas = document.getElementById('canvas_id');
var context = canvas.getContext('2d');
context.clearRect(0, 0, canvas.width, canvas.height);
// Or context.fillRect(...) with white, or whatever clearing it means to you
Then, from the sound of things, you're free to attach another sketch:
var p2 = Processing.loadSketchFromSources('canvas_id', ['sketch2.pde']);
Again, I'm not actually familiar with that library, but this appears straightforward from the documentation.

As of processing.js 1.4.8, Andrew's accepted answer (and the other answers I've found in here) do not seem to work anymore.
This is what worked for me:
var pjs = Processing.getInstanceById('pjs');
if (typeof pjs !== "undefined") {
pjs.exit();
}
var canvas = document.getElementById('pjs')
new Processing(canvas, scriptText);
where pjs is the id of the canvas element where the scrips is being run.