I'm working on a React Native app where I want to play an audio file and visualize it, I didn't find a suitable package for it and decided to make it myself.
I made everything but audio visualization. To visualizate a file I need some kind of library that will analyze audio for me and return numbers array. I will use each number of the array as a point on my future graph.
Let's imagine I have this package, ideally I would like to use like this:
const audioPath = somePackage.analyzeAudio(audio.url);
console.log(audioPath);
// Output: [0, 0, 1, 2, 5, 10, 8, 0]
In array [0, 0, 1, 2, 5, 10, 8, 0] I will understand that at the beginning the audio has no sound at all then it's getting louder and at the end it's silent again. Later I can use these numbers to plot a graph.
Is there a way to do it ?
I couldn't find anything useful to analyze an audio on client side so I decided to do it on server (nodejs) and then send parsed data to client.
I implemented it with help of this package => https://github.com/audiojs/web-audio-api.
This code helped me a lot => https://github.com/victordibia/beats/blob/master/beats.js
Related
I'm working on Mat in OpenCV. However, I need to manually calculate the Mat by myself. Is there is a way of accessing Mat likes 2D array?
const myMat = cv.matFromArray(cv, 3, 3, cv.CV_64F, [
[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
])
const newMat = someProcessThatReturnMat(myMat)
/* OpenCV in JS cannot access Mat like this */
const result = someProcess(newMat[1][2], newMat[2][0])
Thank you in advance
Updated: The problem is cv.matFromArray cannot convert 2D array to Mat. You have to use it as 1D array. That's why it never return the correct values. For example:
const myMat = cv.matFromArray(3, 3, cv.CV_64F, [1,2,3,4,5,6,7,8,9])
And then, you can access the value
const value = myMat.doubleAt(1, 2) // row 1, col 2
You need to use the doubleAt(y,x) method.
It's double because the mat's content is CV_64F, not because you want doubles.
You can also use .data64F to access a flat Float64Array of the Mat's data.
OpenCV.js is... rough. It originated from someone's Google Summer of Code and hasn't received significant care since. Documentation amounts to some tutorials; API docs seem to be missing entirely. The "Mat" interface emulated the at() method from C++, badly, instead of looking at numpy (python) or making this access feel "native" to javascript. Overloading the [] operator is possible using a Proxy but that was not implemented.
Here's an example: https://docs.opencv.org/4.x/de/d06/tutorial_js_basic_ops.html
Feel free to browse OpenCV's issues and maybe suggest some improvements.
I'm looking for a way get a console output from Google Chrome into my python program. I have a script coded in JS that finishes in around 1 second, but my python implementation (exactly same logic, etc., the only difference is that it's in python and not JS) takes about 15 seconds to run. Therefore I'm looking for a way to get the printout in Chrome console to my python program.
This is the current way I'm doing it:
Python program uses pyautogui to click and does what it needs to do inside to trigger the function running in JS.
JS completes the function in 1s and prints to console, something like:
(22) [6, 4, 4, 6, 0, 1, 1 2, 4, 4, 6, 4, 2, 4, 4, 6, 0, 0, 2, 4, 4, 6, 0]
I would like to find a way to get this output into python as I have another script that takes the output and does further calculations to it.
I've thought of using Selenium but that'd introduce too much overhead (probably requiring 3s+ for waiting for chrome to open up).
I am having hard time figuring out how to train the brain.js neural network with dynamic dataset. The GitHub documentation says the following: Each training pattern should have an input and an output, both of which can be either an array of numbers from 0 to 1 or a hash of numbers from 0 to 1
net.train([{input: [0, 0], output: [0]},
{input: [0, 1], output: [1]},
{input: [1, 0], output: [1]},
{input: [1, 1], output: [0]}]);
const output = net.run([1, 0]); // [0.987]
The problem is that I don't know beforehand how many elements are in the input array of my training data so I don't know how many {input: [0, 0], output: [0]} elements I need to pass to net.train().
For example:
How do I train the neural network if I have the following arrays without hardcoding the number of {input: [0, 0], output: [0]} elements.
var input1_array = [.1, .2, .3, .4, .5]
var input2_array = [.6, .7, .8, .9, .95]
var output1_array = [.2, .6, .8, .85, .95]
// the following doesn't work
net.train([input:[input1_array, input2_array], output:[output1_array]]);
I hope question is still valid.
First, I would start with blank array (i.e/ input1_array=[];), but you can't do it as in train you need to have some value. For that reason you can put some initial data like
var input1_array = [0]; if your network will have two inputs then var input1_array = [0,0];
If you will use two or more previous values as inputs you will need
var trainingdata= [{input: [0, 0], output: [0]},
{input: [0, 1], output: [1]},
] //(sorry for typos, but idea to have two entries)
Now you have some initial zeros to not have NULL going to you NN.
I was using prompt during testing and input did the following
trainingdata.push (input:[prompt("Enter next in value")],output:[prompt("Enter next out value")])
Depending on what you do with the network, you may need retrain it each time a new value comes (like time series prediction with LSTM) or just collect data (NN type).
In your example replace prompt with whatever source of data like something from page (getElementById) or some server data JSON parsed value.
I found one trick. Make function to push to array is length
You can do something like:
if (trainingData.length < maxtrainingdata) {
trainingData.push({input: [var1], output: [var2]});
}
else {
trainingData.shift();
trainingData.push({input: [var1], output: [
}
Im pretty new to AI but my idea here is, to make your dataset dynamic.
So let's say you have an AI that needs to learn how to answer to some questions correctly.
In the beginning the AI will just answer them randomly with no idea of what it is doing.
You need to make some testing environment for it to learn.
With this I mean create a piece of code in your AI that knows which answers are correct, every time it gets it right push that result to the dynamic dataset.
Let me give you another example.
If you were training some AI to escape a room.
You would need to make a piece of code when it escapes that pushes those results to the dataset so it can learn the best possible ways to do it.
Did you understand?
Obviously this operation is very expensive.
If you want to train it constantly, you would need some real powerful processing power.
The best way you can work around this issues is by saving iterations.
Imagine that every 100 results you reboot and re-train, that would help. Or if you don't need to actively train it, you can just save push the data into a file that can be re-trained every time the AI reboots.
Your AI would become smarter every time you reboot 🤩
I am attempting to implement a DraftJS editor that highlights words in a transcription while its recorded audio is playing (kind of like karaoke).
I receive the data in this format:
[
{
transcript: "This is the first block",
timestamps: [0, 1, 2.5, 3.2, 4.1, 5],
},
{
transcript: "This is the second block. Let's sync the audio with the words",
timestamps: [6, 7, 8.2, 9, 10, 11.3, 12, 13, 14, 15, 16, 17.2],
},
...
]
I then map this received data to ContentBlocks and initialize the editor's ContentState with them by using ContentState.createFromBlockArray(blocks)
It seems like the "DraftJS" way of storing the timestamp metadata would be to create an Entity for each word with its respective timestamp, and then scan through the currentContent as the audio plays and highlight entities up until the current elapsed time. But I am not sure if this is the right way to do this, as it doesn't seem performant for large transcriptions.
Note: the transcript needs to remain editable while maintaining this karaoke functionality
Any help or discussion is appreciated!
I ended up doing exactly what I described in the question: store timestamps in DraftJS entities. After a few more weeks with DraftJS it seems this is the correct way to do this.
In attempting to build a WebGL 3D library for myself (learning purposes mostly) I followed documentation that I found from various sources that stated that the TypedArray function set() (specifically for Float32Array), is supposed to be "as fast as" memcpy in C (obviously tongue in cheek), literally the fastest according to html5rocks. On appearances that seemed to be correct (no loop setup in javascript, disappearing into some uberfast typed array pure C nonsense, etc).
I took a gander at glMatrix (good job on it btw!), and noticed that he (author) stated that he unrolled all of the loops for speed. This is obviously something a javascript guru would do normally for as much speed as possible, but, based on my previous reading, I thought I had 1-up on this library, specifically, he created his lib to be functional with both arrays and typed arrays, thus I thought that I would get more speed by using set() since I was only interested in staying in TypedArray types.
To test my theory I set up this jsperf. Not only does set() comparatively lack speed, every other technique I tried (in the jsperf), beats it. It is the slowest by far.
Finally, my question: Why? I can theoretically understand a loop unrolling becoming highly optimized in spidermonkey or chrome V8 js-engines, but losing out to a for loop seems ridiculous (copy2 in jsperf), especially if its intent is theoretically to speed up copies due to the raw contiguous in memory data types (TypedArray). Either way it feels like the set() function is broken.
Is it my code? my browser? (I am using Firefox 24) or am I missing some other theory of optimization? Any help in understanding this contrary result to my thoughts and understandings would be incredibly helpful.
This is an old question, but there is a reason to use TypedArrays if you have a specific need to optimize some poorly performing code. The important thing to understand about TypedArray objects in JavaScript is that they are views which represent a range of bytes inside of an ArrayBuffer. The underlying ArrayBuffer actually represents the contiguous block of binary data to operate on, but we need a view in order to access and manipulate a window of that binary data.
Separate (or even overlapping) ranges in the same ArrayBuffer can be viewed by multiple different TypedArray objects. When you have two TypedArray objects that share the same ArrayBuffer, the set operation is extremely fast. This is because the machine is working with a contiguous block of memory.
Here's an example. We'll create an ArrayBuffer of 32 bytes, one length-16 Uint8Array to represent the first 16 bytes of the buffer, and another length-16 Uint8Array to represent the last 16 bytes:
var buffer = new ArrayBuffer(32);
var array1 = new Uint8Array(buffer, 0, 16);
var array2 = new Uint8Array(buffer, 16, 16);
Now we can initialize some values in the first half of the buffer:
for (var i = 0; i < 16; i++) array1[i] = i;
console.log(array1); // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
console.log(array2); // [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
And then very efficiently copy those 8 bytes into the second half of the buffer:
array2.set(array1);
console.log(array1); // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
console.log(array2); // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
We can confirm that the two arrays actually share the same buffer by looking at the buffer with another view. For example, we could use a length-8 Uint32Array that spans the entire 32 bytes of the buffer:
var array3 = new Uint32Array(buffer)
console.log(array3); // [50462976, 117835012, 185207048, 252579084,
// 50462976, 117835012, 185207048, 252579084]
I modified a JSPerf test I found to demonstrate the huge performance boost of a copy on the same buffer:
http://jsperf.com/typedarray-set-vs-loop/3
We get an order of magnitude better performance on Chrome and Firefox, and it's even much faster than taking a normal array of double length and copying the first half to the second half. But we have to consider the cycles/memory tradeoff here. As long as we have a reference to any single view of an ArrayBuffer, the rest of the buffer's data can not be garbage collected. An ArrayBuffer.transfer function is proposed for ES7 Harmony which would solve this problem by giving us the ability explicitly release memory without waiting for the garbage collector, as well as the ability to dynamically grow ArrayBuffers without necessarily copying.
Well set doesn't exactly have simple semantics like that, in V8 after doing some figuring out of what should be done it will essentially arrive at exactly the same loop that the other methods are directly doing in the first place.
Note that JavaScript is compiled into highly optimized machine code if you play your cards right (all the tests do that) so there should be no "worshipping" of some methods just because they are "native".
I've also been exploring how set() performs and I have to say that for smaller blocks (such as the 16 indices used by the original poster), set() is still around 5x slower than the comparable unrolled loop, even when operating on a contiguous block of memory.
I've adapted the original jsperf test here. I think its fair to say that for small block transfers such as this, set() simply can't compete with unrolled index assignment performance. For larger block transfers (as seen in sbking's test), set() does perform better but then it is competing with literally 1 million array index operations so it would seem bonkers to not be able to overcome this with a single instruction.
The contiguous buffer set() in my test does perform slightly better than the separate buffer set(), but again, at this size of transfer the performance benefit is marginal