I'm trying to use WebGL and would like to mix some different types into one buffer of bytes. I understand TypedArrays serve this purpose but it's not clear if I can mix types with them (OpenGL vertex data is often floats mixed with unsigned bytes or integers).
In my test I want to pack 2 floats into a UInt8Array using set(), but it appears to just place the 2 floats into the first 2 elements of the UInt8Array. I would expect this to fill the array of course since we have 8 bytes of data.
Is there anyway to achieve this in JavaScript or do I need to keep all my vertex data as floats?
src = new Float32Array(2); // 2 elements = 8 bytes
src[0] = 100;
src[1] = 200;
dest = new UInt8Array(8); // 8 elements = 8 bytes
dest.set(src, 0); // insert src at offset 0
dest = 100,200,0,0,0,0,0,0 (only the first 2 bytes are set)
You can mix types by making different views on the same buffer.
const asFloats = new Float32Array(2);
// create a uint8 view to the same buffer as the float32array
const asBytes = new Uint8Array(asFloats.buffer);
console.log(asFloats);
asBytes[3] = 123;
console.log(asFloats);
The way TypeArrays really work is there is something called an ArrayBuffer which is a certain number of bytes long. To view the bytes you need an ArrayBufferView of which there are various types Int8Array, Uint8Array, Int16Array, Uint16Array, Int32Array, Uint32Array, Float32Array, Float64Array.
You can create the ArrayBuffer from scratch.
const buffer = new ArrayBuffer(8);
const asFloats = new Float32Array(buffer);
asFloats[0] = 1.23;
asFloats[1] = 4.56;
console.log(asFloats);
Or you can do the more normal thing which is to create an ArrayBufferView of a specific type and it will create both the ArrayBufferView of that type and create the ArrayBuffer for it as well if you don't pass one into the constructor. You can then access that buffer from someArrayBufferView.buffer as shown in the first example above.
You can also assign a view an offset in the ArrayBuffer and a length to make it smaller than the ArrayBuffer. Example:
// make a 16byte ArrayBuffer and a Uint8Array (ArrayBufferView)
const asUint8 = new Uint8Array(16);
// make a 1 float long view in the same buffer
// that starts at byte 4 in that buffer
const byteOffset = 4;
const length = 1; // 1 float32
const asFloat = new Float32Array(asUint8.buffer, byteOffset, length);
// show the buffer is all 0s
console.log(asUint8);
// set the float
asFloat[0] = 12345.6789
// show the buffer is affected at byte 4
console.log(asUint8);
// set a float out of range of its length
asFloat[1] = -12345.6789; // this is effectively a no-op
// show the buffer is NOT affected at byte 8
console.log(asUint8);
So if you want to for example mix float positions and Uint8 colors for WebGL you might do something like
// we're going to have
// X,Y,Z,R,G,B,A, X,Y,Z,R,G,B,A, X,Y,Z,R,G,B,A,
// where X,Y,Z are float32
// and R,G,B,A are uint8
const sizeOfVertex = 3 * 4 + 4 * 1; // 3 float32s + 4 bytes
const numVerts = 3;
const asBytes = new Uint8Array(numVerts * sizeOfVertex);
const asFloats = new Float32Array(asBytes.buffer);
// set the positions and colors
const positions = [
-1, 1, 0,
0, -1, 0,
1, 1, 0,
];
const colors = [
255, 0, 0, 255,
0, 255, 0, 255,
0, 0, 255, 255,
];
{
const numComponents = 3;
const offset = 0; // in float32s
const stride = 4; // in float32s
copyToArray(positions, numComponents, offset, stride, asFloats);
}
{
const numComponents = 4;
const offset = 12; // in bytes
const stride = 16; // in bytes
copyToArray(colors, numComponents, offset, stride, asBytes);
}
console.log(asBytes);
console.log(asFloats);
function copyToArray(src, numComponents, offset, stride, dst) {
const strideDiff = stride - numComponents;
let srcNdx = 0;
let dstNdx = offset;
const numElements = src.length / numComponents;
if (numElements % 1) {
throw new Error("src does not have an even number of elements");
}
for (let elem = 0; elem < numElements; ++elem) {
for(let component = 0; component < numComponents; ++component) {
dst[dstNdx++] = src[srcNdx++];
}
dstNdx += strideDiff;
}
}
Related
I'm using threejs to render some models (gltf/glb). All of them are of different sizes, some are big and some are small. Now I want scale all of them to the same size.
if i use mesh.scale() that would scale the object relative to it's own size. Is there any way to achieve this without manually calculating each model's scale?
UPDATE:
Here's my code
function loadModels(points) {
// loader
const loader = new GLTFLoader();
const dl = new DRACOLoader();
dl.setDecoderPath("/scripts/decoder/");
loader.setDRACOLoader(dl);
let lengthRatios;
const meshes = [];
for (let i = 0; i < store.length; i++) {
loader.load(
store[i].model,
(gltf) => {
const mesh = gltf.scene;
const meshBounds = new THREE.Box3().setFromObject(mesh);
// Calculate side lengths of model1
const lengthMeshBounds = {
x: Math.abs(meshBounds.max.x - meshBounds.min.x),
y: Math.abs(meshBounds.max.y - meshBounds.min.y),
z: Math.abs(meshBounds.max.z - meshBounds.min.z),
};
if (lengthRatios) {
lengthRatios = [
lengthRatios[0] / lengthMeshBounds.x,
lengthRatios[1] / lengthMeshBounds.y,
lengthRatios[2] / lengthMeshBounds.z,
];
} else {
lengthRatios = [
lengthMeshBounds.x,
lengthMeshBounds.y,
lengthMeshBounds.z,
];
}
meshes.push(mesh);
if (meshes.length == store.length) {
addModels();
}
},
(xhr) => {
console.log((xhr.loaded / xhr.total) * 100 + "% loaded");
},
(error) => {
console.log("An error happened");
}
);
}
function addModels() {
// Select smallest ratio in order to contain the models within the scene
const minRation = Math.min(...lengthRatios);
for (let i = 0; i < meshes.length; i++) {
// Use smallest ratio to scale the model
meshes[i].scale.set(minRation, minRation, minRation);
// position the model/mesh
meshes[i].position.set(...points[i]);
// add it to the scene
scene.add(meshes[i]);
}
}
}
You should create a bounding box around each model.
//Creating the actual bounding boxes
mesh1Bounds = new THREE.Box3().setFromObject( model1 );
mesh2Bounds = new THREE.Box3().setFromObject( model2 );
// Calculate side lengths of model1
let lengthMesh1Bounds = {
x: Math.abs(mesh1Bounds.max.x - mesh1Bounds.min.x),
y: Math.abs(mesh1Bounds.max.y - mesh1Bounds.min.y),
z: Math.abs(mesh1Bounds.max.z - mesh1Bounds.min.z),
};
// Calculate side lengths of model2
let lengthMesh2Bounds = {
x: Math.abs(mesh2Bounds.max.x - mesh2Bounds.min.x),
y: Math.abs(mesh2Bounds.max.y - mesh2Bounds.min.y),
z: Math.abs(mesh2Bounds.max.z - mesh2Bounds.min.z),
};
// Calculate length ratios
let lengthRatios = [
(lengthMesh1Bounds.x / lengthMesh2Bounds.x),
(lengthMesh1Bounds.y / lengthMesh2Bounds.y),
(lengthMesh1Bounds.z / lengthMesh2Bounds.z),
];
// Select smallest ratio in order to contain the models within the scene
let minRatio = Math.min(...lengthRatios);
// Use smallest ratio to scale the model
model.scale.set(minRatio, minRatio, minRatio);
I want to generate random images using some random function into an Uint8Array in reactjs. Now I want to render it through tag. For example:
img = new Uint8Array(100 * 100 * 3); // want to create a 100 * 100 3 channel color image
//someRandomFunction(img);
let blob = new Blob( [ img ], { type: "image/png" } );
var urlCreator = window.URL || window.webkitURL;
const imageUrl = urlCreator.createObjectURL( blob );
Now I want to render this "imgUrl" in the img tag.
I have converted this data into a blob and created a URL to set the image source. However, no luck so far. It always shows some 0x0 empty image. Even when I have an all zero array, shouldn't it show a complete black image ?
Just to give a little bit more context, essentially, I am trying to copy the behavior of numpy and opencv from python. There we can create a numpy array and then show that image through opencv function like this:
img = np.random.randint([100, 100, 3], dtype=np.uint8)
cv2.imshow('image', img);
How can I achieve that in reactjs ?
Can anyone please help me out here ?
You can try with Uint8ClampedArray, ImageData and canvas instead of img.
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const arr = new Uint8ClampedArray(40000);
// Iterate through every pixel
for (let i = 0; i < arr.length; i += 4) {
arr[i + 0] = 0; // R value
arr[i + 1] = 190; // G value
arr[i + 2] = 0; // B value
arr[i + 3] = 255; // A value
}
// Initialize a new ImageData object
let imageData = new ImageData(arr, 200);
// Draw image data to the canvas
ctx.putImageData(imageData, 20, 20);
For more information you can see MDN ImageData and Uint8ClampedArray
I know that you can do
const buffer = new ArrayBuffer(16);
const dataView = new DataView(buffer);
dataView.setUint8(1, 4)
console.log(dataView.getUint8(1)); // 1
However, I would like to set an unsigned byte before the dataView deceleration line, so would it be possible to do this without having access to dataView hence would if be possible to set an unsigned byte of 4 at the byte offset 1 to the ArrayBuffer instead of using dataView.setUint8(1, 4)?
Or alternatively would it be to convert a DataView to an ArrayBuffer?
I think the important thing you're missing is that a DataView is just a View. So when you do dataView.setUint8(1, 4) you do modify the buffer. The dataView itself does not hold the data, just a reference to the buffer. So your code already does what you want. To get an ArrayBuffer of it just use the original buffer:
const buffer = new ArrayBuffer(16);
const dataView = new DataView(buffer);
dataView.setUint8(1, 4)
console.log(dataView.getUint8(1)); // 4
console.log(new Uint8Array(buffer)) // [ 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
A Note For Readers: This is a long question, but it needs a background to understand the question asked.
The color quantization technique is commonly used to get the dominant colors of an image.
One of the well-known libraries that do color quantization is Leptonica through the Modified Median Cut Quantization (MMCQ) and octree quantization (OQ)
Github's Color-thief by #lokesh is a very simple implementation in JavaScript of the MMCQ algorithm:
var colorThief = new ColorThief();
colorThief.getColor(sourceImage);
Technically, the image on a <img/> HTML element is backed on a <canvas/> element:
var CanvasImage = function (image) {
this.canvas = document.createElement('canvas');
this.context = this.canvas.getContext('2d');
document.body.appendChild(this.canvas);
this.width = this.canvas.width = image.width;
this.height = this.canvas.height = image.height;
this.context.drawImage(image, 0, 0, this.width, this.height);
};
And that is the problem with TVML, as we will see later on.
Another implementation I recently came to know was linked on this article Using imagemagick, awk and kmeans to find dominant colors in images that links to Using python to generate awesome linux desktop themes.
The author posted an article about Using python and k-means to find the dominant colors in images that was used there (sorry for all those links, but I'm following back my History...).
The author was super productive, and added a JavaScript version too that I'm posting here: Using JavaScript and k-means to find the dominant colors in images
In this case, we are generating the dominant colors of an image, not using the MMCQ (or OQ) algorithm, but K-Means.
The problem is that the image must be a as well:
<canvas id="canvas" style="display: none;" width="200" height="200"></canvas>
and then
function analyze(img_elem) {
var ctx = document.getElementById('canvas').getContext('2d')
, img = new Image();
img.onload = function() {
var results = document.getElementById('results');
results.innerHTML = 'Waiting...';
var colors = process_image(img, ctx)
, p1 = document.getElementById('c1')
, p2 = document.getElementById('c2')
, p3 = document.getElementById('c3');
p1.style.backgroundColor = colors[0];
p2.style.backgroundColor = colors[1];
p3.style.backgroundColor = colors[2];
results.innerHTML = 'Done';
}
img.src = img_elem.src;
}
This is because the Canvas has a getContext() method, that expose 2D image drawing APIs - see An introduction to the Canvas 2D API
This context ctx is passed to the image processing function
function process_image(img, ctx) {
var points = [];
ctx.drawImage(img, 0, 0, 200, 200);
data = ctx.getImageData(0, 0, 200, 200).data;
for (var i = 0, l = data.length; i < l; i += 4) {
var r = data[i]
, g = data[i+1]
, b = data[i+2];
points.push([r, g, b]);
}
var results = kmeans(points, 3, 1)
, hex = [];
for (var i = 0; i < results.length; i++) {
hex.push(rgbToHex(results[i][0]));
}
return hex;
}
So you can draw an image on the Canvas through the Context and get image data:
ctx.drawImage(img, 0, 0, 200, 200);
data = ctx.getImageData(0, 0, 200, 200).data;
Another nice solution is in CoffeeScript, ColorTunes, but this is using a as well:
ColorTunes.getColorMap = function(canvas, sx, sy, w, h, nc) {
var index, indexBase, pdata, pixels, x, y, _i, _j, _ref, _ref1;
if (nc == null) {
nc = 8;
}
pdata = canvas.getContext("2d").getImageData(sx, sy, w, h).data;
pixels = [];
for (y = _i = sy, _ref = sy + h; _i < _ref; y = _i += 1) {
indexBase = y * w * 4;
for (x = _j = sx, _ref1 = sx + w; _j < _ref1; x = _j += 1) {
index = indexBase + (x * 4);
pixels.push([pdata[index], pdata[index + 1], pdata[index + 2]]);
}
}
return (new MMCQ).quantize(pixels, nc);
};
But, wait, we have no <canvas/> element in TVML!
Of course, there are native solutions like Objective-C ColorCube, DominantColor - this is using K-means
and the very nice and reusable ColorArt by #AaronBrethorst from CocoaControls.
Despite the fact that this could be used in a TVML application through a native to JavaScriptCore bridge - see How to bridge TVML/JavaScriptCore to UIKit/Objective-C (Swift)?
my aim is to make this work completely in TVJS and TVML.
The simplest MMCQ JavaScript implementation does not need a Canvas: see Basic Javascript port of the MMCQ (modified median cut quantization) by Nick Rabinowitz, but needs the RGB array of the image:
var cmap = MMCQ.quantize(pixelArray, colorCount);
that is taken from the HTML <canvas/> and that is the reason for it!
function createPalette(sourceImage, colorCount) {
// Create custom CanvasImage object
var image = new CanvasImage(sourceImage),
imageData = image.getImageData(),
pixels = imageData.data,
pixelCount = image.getPixelCount();
// Store the RGB values in an array format suitable for quantize function
var pixelArray = [];
for (var i = 0, offset, r, g, b, a; i < pixelCount; i++) {
offset = i * 4;
r = pixels[offset + 0];
g = pixels[offset + 1];
b = pixels[offset + 2];
a = pixels[offset + 3];
// If pixel is mostly opaque and not white
if (a >= 125) {
if (!(r > 250 && g > 250 && b > 250)) {
pixelArray.push([r, g, b]);
}
}
}
// Send array to quantize function which clusters values
// using median cut algorithm
var cmap = MMCQ.quantize(pixelArray, colorCount);
var palette = cmap.palette();
// Clean up
image.removeCanvas();
return palette;
}
[QUESTION]
How to generate the dominant colors of a RGB image without using the HTML5 <canvas/>, but in pure JavaScript from an image's ByteArray fetched with XMLHttpRequest?
[UPDATE]
I have posted this question to Color-Thief github repo, adapting the RGB array calculations to the latest codebase.
The solution I have tried was this
ColorThief.prototype.getPaletteNoCanvas = function(sourceImageURL, colorCount, quality, done) {
var xhr = new XMLHttpRequest();
xhr.open('GET', sourceImageURL, true);
xhr.responseType = 'arraybuffer';
xhr.onload = function(e) {
if (this.status == 200) {
var uInt8Array = new Uint8Array(this.response);
var i = uInt8Array.length;
var biStr = new Array(i);
while (i--)
{ biStr[i] = String.fromCharCode(uInt8Array[i]);
}
if (typeof colorCount === 'undefined') {
colorCount = 10;
}
if (typeof quality === 'undefined' || quality < 1) {
quality = 10;
}
var pixels = uInt8Array;
var pixelCount = 152 * 152 * 4 // this should be width*height*4
// Store the RGB values in an array format suitable for quantize function
var pixelArray = [];
for (var i = 0, offset, r, g, b, a; i < pixelCount; i = i + quality) {
offset = i * 4;
r = pixels[offset + 0];
g = pixels[offset + 1];
b = pixels[offset + 2];
a = pixels[offset + 3];
// If pixel is mostly opaque and not white
if (a >= 125) {
if (!(r > 250 && g > 250 && b > 250)) {
pixelArray.push([r, g, b]);
}
}
}
// Send array to quantize function which clusters values
// using median cut algorithm
var cmap = MMCQ.quantize(pixelArray, colorCount);
var palette = cmap? cmap.palette() : null;
done.apply(this,[ palette ])
} // 200
};
xhr.send();
}
but it does not gives back the right RGB colors array.
[UPDATE]
Thanks to all the suggestions I got it working. Now a full example is available on Github,
The canvas element is being used as a convenient way to decode the image into an RGBA array. You can also use pure JavaScript libraries to do the image decoding.
jpgjs is a JPEG decoder and pngjs is a PNG decoder. It looks like the JPEG decoder will work with TVJS as is. The PNG decoder, however, looks like it's made to work in a Node or web browser environment, so you might have to tweak that one a bit.
I'm trying to implement histogram RGB but my algorithm doesn't produce similarly look like surface as in graphics programs. For example image on this site:
OpenCV histogram
My version looks like:
As I understood it correctly, RGB Histogram just measuring how often each value occured in specific channel. So I implement it in such way:
public Process(layerManager: dl.LayerManager) {
var surface = layerManager.GetCurrent();
var components = new Uint8Array(1024);
surface.ForEachPixel((arr: number[], i: number): void => {
components[arr[i]] += 1;
components[arr[i + 1] + 256] += 1;
components[arr[i + 2] + 512] += 1;
components[arr[i + 3] + 768] += 1;
});
var histogram = layerManager.GetHistogram();
histogram.Clear();
var viewPort = layerManager.GetHistogramViewPort();
viewPort.Clear();
this.DrawColor(histogram, components, 0, new ut.Color(255, 0, 0, 255));
//histogram.SetBlendMode(ds.BlendMode.Overlay);
//this.DrawColor(histogram, components, 256, new ut.Color(0, 255, 0, 255));
//this.DrawColor(histogram, components, 512, new ut.Color(0, 0, 255, 255));
}
private DrawColor(surface: ds.ICanvas, components: Uint8Array, i: number, fillStyle: ut.Color) {
var point = new ut.Point(0, 255);
surface.BeginPath();
surface.FillStyle(fillStyle.R, fillStyle.G, fillStyle.B, fillStyle.A);
surface.RGBAStrokeStyle(fillStyle.R, fillStyle.G, fillStyle.B, fillStyle.A);
surface.LineWidth(1);
surface.MoveTo(point);
for (var j = i + 256; i < j; ++i) {
point = new ut.Point(point.X + 1, 255 - components[i]);
surface.ContinueLine(point);
}
surface.ClosePathAndStroke();
var viewPort = layerManager.GetHistogramViewPort();
viewPort.DrawImage(surface.Self<HTMLElement>(), 0, 0, 255, 255, 0, 0, viewPort.Width(), viewPort.Height());
}
Am I missing something?
You have a Uint8Array array to hold the results, but the most common RGB values are occurring more than 255 times. This causes an overflow and you end up seeing a histogram of the values modulo 256, which is effectively random for high values. That's why the left and middle parts of the graph (where values are less than 255) are correct, but the higher-valued areas are all over the place.
Use a larger data type to store the results, and normalize to the size of your output canvas before drawing.