how do getByteTimeDomain/FrequencyData() method work - javascript

I'm trying to develop pitch-detector using JavaScript Web Audio API. By googling, I've knew we perceive pitch by frequency so I found getByteFrequencyData() method. But I don't know how to use it correctly.
example.js
function draw() {
var img = new Image();
img.src="foo.jpg";
img.onload = function() {
ctx.drawImage(img, 0, 0);
var imgData=ctx.getImageData(0, 0, canvas.width, canvas.height);
var raster = imgData.data;
for (var i=0;i<raster.length;i++) {
if (i % 4 != 3) {
raster[i] = 255 - raster[i];
}
}
ctx.putImageData(imgData,0, 0);
}
}
As we see above, getImageData() returns very obvious, easy-to-access data. In contrast, What does a parameter "buffer" of getByteFrequencyData() save/represent/mean? How does it represent audio frequency data? How can I manipulate/handle it and develop my own program using these methods?
Thanks.

The spec entry for getByteFrequencyData tells you exactly what it is. The analyser node determines the frequency content in a set of bins where the value of each bin is the magnitude of that frequency component. getByteFrequencyData just converts that dB and then scales the values to the range of 0 to 255.
I generally recommend people to use getFloatFrequencyData() first because I think it's a bit easier to understand without having to deal with the scaling.

Related

Mediapipe pose SegmentationMask python javascript differences

I am developing a pose recognition webapp using mediapipe pose library (https://google.github.io/mediapipe/solutions/pose.html).
I am using the segmentationMask to find some specific points of the human body that satisfy a constraint (the value in the n-th pixel must be > 0.1).
I'am able to do this evaluation in python. The library returns the segmentation mask as a matrix with the same width and height as the input image, and contains values in [0.0, 1.0] where 1.0 and 0.0 indicate high certainty of a “human” and “background” pixel respectively. So I can iterate over the matrix and I am able to find the point that satisfy the constraint.
I am trying to do the same thing in javascript, but I have a problem. The The javascript version of the library does not return a matrix but returns an ImageBitmap used by the html canvas to draw the mask.
The problem is that with ImageBitmap I cannot access every point of the matrix and I am not able to find the points I am interested in.
Is there a way to transform the javascript segmentationMask ImageBitmap in order be similar to the segmenationMask of the python versione library or at least retrive the same informations (I need the values included in this range [0.0, 1.0] for every pixel of the image).
Thank you all.
There is unfortunately no direct way to get an ImageData from an ImageBitmap, but you can drawImage() this ImageBitmap on a clear canvas and then call ctx.getImageData(0, 0, canvas.width, canvas.height) to retrieve an ImageData where you'll get access to all the pixels data.
The confidence will be stored in the Alpha channel (every fourth item in imageData.data) as a value between 0 and 255.
function onResults(results) {
canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
canvasCtx.drawImage(results.segmentationMask, 0, 0,
canvasElement.width, canvasElement.height);
const imgData = canvasCtx.getImageData(0, 0, canvasElement.width, canvasElement.height);
let i = 0;
for (let y = 0; y<imgData.height; y++) {
for (let x = 0; x<imgData.width; x++) {
const confidence = imgData.data[i + 3];
// do something with confidence here
i++;
}
}
}
And since you're gonna read a lot from that context, don't forget to pass the willReadFrequently option when you get it.
As a fiddle since StackSnippets won't allow the use of the camera.
Note that depending on what you do you may want to colorize this image from red to black using globalCompositeOperation and treat the data as an Uint32Array where the confidence would be expressed between 0 and 0xFF000000.

Split stereo audio file Into AudioNodes for each channel

How can i split a stereo audio file (I'm currently working with a WAV, but I'm interested in how to do it for MP3 as well, if that's different) into left and right channels to feed into two separate Fast Fourier Transforms (FFT) from the P5.sound.js library.
I've written out what I think I need to be doing below in the code, but I haven't been able to find examples of anyone doing this through Google searches and all my layman's attempts are turning up nothing.
I'll share what I have below, but in all honesty, it's not much. Everything in question would go in the setup function where I've made a note:
//variable for the p5 sound object
var sound = null;
var playing = false;
function preload(){
sound = loadSound('assets/leftRight.wav');
}
function setup(){
createCanvas(windowWidth, windowHeight);
background(0);
// I need to do something here to split the audio and return a AudioNode for just
// the left stereo channel. I have a feeling it's something like
// feeding audio.getBlob() to a FileReader() and some manipulation and then converting
// the result of FileReader() to a web audio API source node and feeding that into
// fft.setInput() like justTheLeftChannel is below, but I'm not understanding how to work
// with javascript audio methods and createChannelSplitter() and the attempts I've made
// have just turned up nothing.
fft = new p5.FFT();
fft.setInput(justTheLeftChannel);
}
function draw(){
sound.pan(-1)
background(0);
push();
noFill();
stroke(255, 0, 0);
strokeWeight(2);
beginShape();
//calculate the waveform from the fft.
var wave = fft.waveform();
for (var i = 0; i < wave.length; i++){
//for each element of the waveform map it to screen
//coordinates and make a new vertex at the point.
var x = map(i, 0, wave.length, 0, width);
var y = map(wave[i], -1, 1, 0, height);
vertex(x, y);
}
endShape();
pop();
}
function mouseClicked(){
if (!playing){
sound.loop();
playing = true;
} else {
sound.stop();
playing = false;
}
}
Solution:
I'm not a p5.js expert, but I've worked with it enough that I figured there has to be a way to do this without the whole runaround of blobs / file reading. The docs aren't very helpful for complicated processing, so I dug around a little in the p5.Sound source code and this is what I came up with:
// left channel
sound.setBuffer([sound.buffer.getChannelData(0)]);
// right channel
sound.setBuffer([sound.buffer.getChannelData(1)]);
Here's a working example - clicking the canvas toggles between L/stereo/R audio playback and FFT visuals.
Explanation:
p5.SoundFile has a setBuffer method which can be used to modify the audio content of the sound file object in place. The function signature specifies that it accepts an array of buffer objects and if that array only has one item, it'll produce a mono source - which is already in the correct format to feed to the FFT! So how do we produce a buffer containing only one channel's data?
Throughout the source code there are examples of individual channel manipulation via sound.buffer.getChannelData(). I was wary of accessing undocumented properties at first, but it turns out that since p5.Sound uses the WebAudio API under the hood, this buffer is really just plain old WebAudio AudioBuffer, and the getChannelData method is well-documented.
The only downside of approach above is that setBuffer acts directly on the SoundFile so I'm loading the file again for each channel you want to separate, but I'm sure there's a workaround for that.
Happy splitting!

Resizing images using createjs / easeljs

I'd like to dynamically downsize some images on my canvas using createjs, and then store the smaller images to be displayed when zooming out of the canvas for performance reasons. Right now, I'm using the following code:
var bitmap = createjs.Bitmap('somefile.png');
// wait for bitmap to load (using preload.js etc.)
var oc = document.createElement('canvas');
var octx = oc.getContext('2d');
oc.width = bitmap.image.width*0.5;
oc.height = bitmap.image.height*0.5;
octx.drawImage(bitmap.image, 0, 0, oc.width, oc.height);
var dataUrl = oc.toDataURL('image/png'); // very expensive
var smallBitmap = new createjs.Bitmap(dataUrl);
This works, but:
The toDataURL operation is very expensive when converting to image/png and too slow to use in practice (and I can't convert to the faster image/jpeg due to the insufficient quality of the output for all settings I tried)
Surely there must be a way to downsize the image without having to resort to separate canvas code, and then do a conversion manually to draw onto the createjs Bitmap object??
I've also tried:
octx.drawImage(bitmap.image, 0, 0, oc.width, oc.height);
var smallBitmap = new createjs.Bitmap(oc);
But although very fast, this doesn't seem to actually work (and in any case I'm having to create a separate canvas element every time to facilitate this.)
I'm wondering if there is a way that I can use drawImage to draw a downsampled version of the bitmap into a createjs Bitmap instance directly without having to go via a separate canvas object or do a conversion to string?
If I understand correctly, internally this is how the createjs cache property works (i.e. uses drawImage internally to write into the DisplayObject) but I'm unable to figure out how use it myself.
You have tagged this post with createjs and easeljs, but your examples show plain Canvas context usage for scaling.
You can use the scale parameter on Bitmap.cache() to get the result you want, then reuse the cacheCanvas as necessary.
// This will create a half-size cache (50%)
// But scale it back up for you when it displays on the stage
var bmp = new createjs.Bitmap(img);
bmp.cache(0, 0, img.width, img.height, 0.5);
// Pull out the generated cache and use it in a new Bitmap
// This will display at the new scaled size.
var bmp2 = new createjs.Bitmap(bmp.cacheCanvas);
// Un-cache the first one to reset it if you want
bmp.uncache();
Here is a fiddle to see it in action: http://jsfiddle.net/lannymcnie/ofdsyn7g/
Note that caching just uses another canvas with a drawImage to scale it down. I definitely would stay away from toDataURL, as it not performant at all.

How to pass canvas imageData to emscripten c++ program without copying it?

I have Image data of the canvas:
myImage = ctx.getImageData(0, 0, 640, 480);
I figured out, that i can create new Uint8Array and use set() to copy imagedata. This is working example:
var numBytes = width * height * 4;
var ptr= Module._malloc(numBytes);
var heapBytes= new Uint8Array(Module.HEAPU8.buffer, ptr, numBytes);
heapBytes.set(new Uint8Array(myImage.data));
_processImage(heapBytes.byteOffset, width, height);
myImage.data.set(heapBytes);
But, unfortunately every .set() operation is far more slower than processing image, and the code above is slower than JS implementation!
So, I want to process image without copying it. I can successfuly read and write the data directly to the heap this way:
Module.HEAPU8.set(myImage.data, myImage.data.byteOffset);
_processImage(myImage.data.byteOffset, width, height);
myImage.data.set(new Uint8ClampedArray(Module.HEAPU8.buffer , myImage.data.byteOffset , numBytes));
It's faster, but still the first .set() takes 17ms to execute.
The c++ function prototype is:
extern "C" {
int processImage(unsigned char *buffer, int width, int height)
{
}
}
Is there any way to pass the array to C++ without using set()? Just telling the c++ where the data is in memory, and allow to modify it?
Just telling the c++ where the data is in memory, and allow to modify it?
As of v1.34.12, Emscripten has a SPLIT_MEMORY option, where you can tell Emscripten to use an existing buffer as part of its memory space that is split up into evenly sized chunks
You could, for example, get the buffer from the canvas
var existingBuffer = myImage.data.buffer;
var bufferSize = existingBuffer.byteLength; // Must be equal to SPLIT_MEMORY
and then, modifying the example from the explanation of split memory, tell Emscripten to use this buffer as part of its memory space
var chunkIndex = 2; // For example
allocateSplitChunk(chunkIndex, existingBuffer);
and then pass a pointer to the chunk to your C++ function.
var pointerToImageInGlobalMemorySpace = chunkIndex * bufferSize;
_processImage(pointerToImageInGlobalMemorySpace, width, height);
However there are problems and limitations
The Emscripten memory space must be split into chunks exactly the size of the canvas image data buffer.
There are apparently serious performance implications for all of the Emscripten-compiled code, which might make this perform worse than your original code.

Fast way to make RGB array into RGBA array in Javascript

An emulator I am working with internally stores a 1-dimensional framebuffer of RGB values. However, HTML5 canvas uses RGBA values when calling putImageData. In order to display the framebuffer, I currently loop through the RGB array and create a new RGBA array, in a manner similar to this.
This seems suboptimal. There has been much written on performing canvas draws quickly, but I'm still lost on how to improve my application performance. Is there any way to more quickly translate this RGB array to an RGBA array? The alpha channel will always be fully opaque. Also, is there any way to interface with a canvas so that it takes an array of RGB, not RGBA, values?
There's no way to use plain RGB, but the loop in that code could be optimised somewhat by removing repeated calculations, array deferences, etc.
In general you shouldn't use ctx.getImageData to obtain the destination buffer - you don't normally care what values are already there and should use ctx.createImageData instead. If at all possible, re-use the same raw buffer for every frame.
However, since you want to preset the alpha values to 0xff (they default to 0x00) and only need to do so once, it seems to be much most efficient to just fill the canvas and then fetch the raw values with getImageData.
ctx.fillStyle = '#ffffff'; // implicit alpha of 1
ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
dest = ctx.getImageData(0, 0).data
and then for each frame for can just leave the alpha byte untouched:
var n = 4 * w * h;
var s = 0, d = 0;
while (d < n) {
dest[d++] = src[s++];
dest[d++] = src[s++];
dest[d++] = src[s++];
d++; // skip the alpha byte
}
You could also experiment with "loop unrolling" (i.e. repeating that four line block multiple times within the while loop) although results will vary across browsers.
Since it's very likely that your total number of pixels will be a multiple of four, just repeat the block another three times and then the while will only be evaluated for every four pixel copies.
Both ctx.createImageData and ctx.getImageData will create a buffer, the later (get) will be slower since it has also to copy the buffer.
This jsperf : http://jsperf.com/drawing-pixels-to-data
confirms that we have a like 33% slowdown on Chrome, and 16 times slower on Firefox (FFF seems to byte-copy when Chrome copy with 32 or 64 bits move).
i'll just recall that you can handle typed array of different types, and even create a view on the buffer (image.data.buffer).
So this may allow you to write the bytes 4 by 4.
var dest = ctx.createImageData(width, height);
var dest32 = new Int32Array(dest.data.buffer);
var i = 0, j=0, last = 3*width*height;
while (i<last) {
dest32[j] = src[i]<<24 + src[i+1] << 16
+ src[i+2] << 8 + 255;
i+=3;
j++;
}
You will see in this jsperf test i made that it is faster to
write using 32 bits integers :
http://jsperf.com/rgb-to-rgba-conversion-with-typed-arrays
notice that there is a big issue in those tests : since this test is
awfull in terms of garbage creation, accuracy is so-so.
Still after many launch, we see that we have around 50%
gain on write 4 vs write 1.
Edit : it might be worth to see if reading the source with a DataView wouldn't speed things up.
but the input array has to be a buffer (or have a buffer property like a Uint8Array).
(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays/DataView)
do not hesitate to update the fiddle with such a try.
Edit 2 :
I don't understand i re-ran the test and now write 4 is slower : ??? and after, faster again : -------
Anyway you have great interest in keeping the dest32 buffer under your hand and not
create a new one each time anyway, so since this test measure the Int32Array creation, it does not correspond to your use case.

Categories