Related
Since music speed and pitch is coupled together, if I speed up music, pitch is also increased. And conversely, if I slow down music, pitch is also decreased.
However, I saw that using granular synthesis, I can decouple speed and pitch. So, I'm currently trying hard to implement granular synthesis.
First of all, I think I succeeded in implementing double speed and half speed, while pitch is same. The code is as same as the following:
※ grain size is 2000. It means that I use 0.04ms of sound as one grain. (2000 samples * 1 s / 44100 samples = 0.04s = 40ms)
// get music
const $fileInput = document.createElement('input');
$fileInput.setAttribute('type', 'file');
document.body.appendChild($fileInput);
$fileInput.addEventListener('change', async (e) => {
const music = await $fileInput.files[0].arrayBuffer();
const actx = new (window.AudioContext || window.webkitAudioContext)({ latencyHint: 'playback', sampleRate: 44100 });
const audioData = await actx.decodeAudioData(music);
const original = audioData.getChannelData(0);
const arr = [];
const grainSize = 2000;
// Please choose one code out of double speed code or half speed code
// copy and paste audio processing code here
});
// double speed
// ex: [0,1,2,3, 4,5,6,7, 8] => [0,1, 4,5, 8] discard 2 items out of 4 items
for (let i = 0; i < original.length; i += grainSize) {
if (original[i + (grainSize / 2) - 1] !== undefined) {
for (let j = 0; j < grainSize / 2; j++) {
arr.push(original[i + j]);
}
} else {
for (let j = i; j < original.length; j++) {
arr.push(j);
}
}
}
// half speed
// ex: [0,1, 2,3, 4] => [0,1,0,0, 2,3,0,0, 4,0,0] add 'two' zeros after every 'two' items
for (let i = 0; i < original.length; i += grainSize) {
if (original[i + grainSize - 1] !== undefined) {
for (let j = 0; j < grainSize; j++) {
arr.push(original[i + j]);
}
} else {
for (let j = i; j < original.length; j++) {
arr.push(original[j]);
}
}
for (let j = 0; j < grainSize; j++) {
arr.push(0);
}
}
// play sound
const f32Arr = Float32Array.from(arr);
const audioBuffer = new AudioBuffer({ length: arr.length, numberOfChannels: 1, sampleRate: actx.sampleRate });
audioBuffer.copyToChannel(f32Arr, 0);
const absn = new AudioBufferSourceNode(actx, { buffer: audioBuffer });
absn.connect(actx.destination);
absn.start();
But the problem is, I totally have no idea how to implement pitch shifter (that is, different pitch, same speed).
As far as I think, same speed means same AudioBuffer size. Therefore, the only variable in my hand is grain size. But I seriously don't know what should I do. It would be greatly appreciated if you share some of your knowledge. Thank you very much!
To Phil Freihofner
Hello, thank you for the kind explanation. I tried your method. As far as I understand, your method is a process that does the following:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] // input data (10 samples)
→ [0, 2, 4, 6, 8] // double speed, 1 octave high (sampling interval: 2)
→ [0, 0, 2, 2, 4, 4, 6, 6, 8, 8] // change duration
The result sounds 1 octave high with same duration (successful pitch shifting). However, I don't know what should I do if I do sampling from the input data with sampling interval 1.5? What I mean is I have no idea how to make the length of [0, 1, 3, 4, 6, 7, 9] as the same length with the input data.
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] // input data
// from
[0, 1, 3, 4, 6, 7, 9] // sampling interval: 1.5
// to
[?, ?, ?, ?, ?, ?, ?, ?, ?, ?]
Meanwhile, I learned that pitch shift can be achieved by a way and as far as I understand, the way is as following:
[1] make each granule be started at the same position as original source
[2] play each granule with different speed.
In addition, I found that I can achieve pitch shifting and time stretching if I transform an input data like the following:
input data = [0, 1, 2, 3, 4, 5, 6, 7]
grain size = 4
<pitch shifting>
in case of p = 2
result = [0, 2, 0, 2, 4, 6, 4, 6] // sounds like one octave high
// If I remember correctly, [0, 2, 0, 0, 4, 6, 0, 0] is also fine
// (and it is more fit to the definition above ([1] and [2])
// but the sound was not good (stuttering).
// I found that [0, 2, 0, 2...] is better.
in case of p = 1.5
result = [0, 1, 3, 0, 4, 5, 7, 4]
in case of p = 0.5
result = [0, 0, 1, 1, 4, 4, 5, 5] // sounds like one octave low
<time stretching>
in case of speed = 2
result = [0, 1, 4, 5]
in case of speed = 1.2
result = [0, 1, 2, 4, 5, 6]
in case of speed = 0.5
result = [0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 4, 5, 6, 7]
// If I remember correctly, [0, 1, 2, 3, 0, 0, 0, 0...] is also fine
// but the sound was not good (stuttering)
in case of speed = 0.75
result = [0, 1, 2, 3, 0, 4, 5, 6, 7, 4]
Anyway, thank you for the answer.
I've not read your code close enough to comment on it specifically, but I can comment on the general theory of how pitch shifting is accomplished.
The granules are usually given volume envelopes, with a fade-in and fade-out. I've seen the Hann function (Hanning Window) mentioned as a possibility. Also, the granules are overlapped, with the windowing creating a cross-fade, in effect.
Let's say a granule is 2000 frames, but with the windowing. If you make a granule at every 1000 frames and play them back, overlapping, at the same spacing (every 1000 frames), you should hear the equivalent of the original sound.
Varying the playback distance between the overlapping granules is how the different time lengths of the sound are accomplished. For example, instead of playing a granule every 1000 frames, use 900 or 1100.
I'm pretty sure there are factors to take into consideration concerning the size and shape of the windowing and the range of possible intervals between the granules, but I am not up on them. My simple experiments with this have been with Java and mostly work, but with some artificiality creeping into the playback.
I think consulting at StackOverflow's Signal Processing site would be a good bet for getting more info on the specifics.
EDIT: I just realized that I misread your question! You were asking about changing the pitch while retaining the length of time over which the sounds play. I don't know if this is the "best" way, but I'd consider a plan of doing this in two steps. First, change the sound to the desired pitch. Then, alter the duration of the new sound to be that of the original sound.
The first step can be done with linear interpolation. I tried to explain how to do this in a previous question. For the second step, we break the transformed wave into granules.
However, I just noticed, Spektre has an additional answer on that post that does directly using what you ask, via using FFT. This is probably a better way, but I haven't tried implementing it myself.
EDIT 2, in response to the question added to the OP:
Given, PCM data for 10 frames as follows [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] (I'm using signed floats ranging from -1 to 1. You may have to scale this to convert to your format.)
To change the pitch of the playback to 1.5x (but also changes the length) we get the following data: [0, 0.15, 0.3, 0.45, 0.6, 0.75, 0.9]
The 0.15 is a value that is halfway between points 0.1 and 0.2, arrived at by linear interpolation. If the speed were 1.25x, the data points would be as follows: [0, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, ?? (depends on what follows 0.9)].
The linear intepolation calculation for index 1 in the sequence is as follows:
pitchShiftedAudioData1 = originalPCM1 * (1 - 0.25) + originalPCM2 * 0.75;
In other words, since we land at a point that is 0.25 of the way in between originalPCM1 and originalPCM2, the above calculates what that value would be if the data progressed linearly from 1 to 2.
After doing all this, there would still remain additional steps to form the pitch-shifted data into granules. One has to use a windowing function for each granule. If the window were only 10 frames long (far too short, but will illustrate), a possible window might be the following: [0.01, 0.15 , 0.5 , 0.85 , 1, 1, 0.85, 0.5, 0.15 , 0.01]. (In actuality, it should follow the Hann function.)
This is applied to the data from the different starting points, to create the granules, where N is the index in the array of the signal.
[ signal[N] * window[0], signal[N+1] * window1, signal[N+2] * window2, ..., signal[N+10] * window[10] ]
To create the new signal, the resulting granules are placed sequentially, overlapping, and summed. The relative placement of the granules (how close together or far apart) determines the timing. This is my naive understanding of a brute-force way to accomplish time-shifting, and I've had some OK, not great, results.
I hope this clarifies what I was attempting to describe somewhat!
If you aren't able to follow, please consider unchecking this as the answer. Others may participate that will be able to provide easier to understand information or corrections.
Time-shifting is pretty advanced, IMHO, so expect some complicated calculations and coding (unless someone has a tool to recommend).
I am working on my first full program with two weeks of programming under my belt, and have run into a road block I can't seem to figure out. I am making a connect 4 game, and have started by building the logic in JavaScript before pushing to the DOM. I have started to make it with cell objects made by a constructor, that are then pushed into a game object in the form of a 2D array. I have managed to create a function that makes the play each time, and changes the value of the cell at the lowest point of that column with a 2 day array. However, I am not sure how to get my check for wins function to operate.
So far my logic is that, for each point in the 2D array, you can check by row, by column, and by diagonals. I understand the logic of how to check for win, but I don't understand how to traverse through the arrays by row and column. In the example below, this.cellsArray is an array of cell objects in the Board Constructor. The array has 7 column arrays, with 6 rows each, as I flipped the typical row column logic to account for Connect Four's column based nature. However I can't access the array like this.cellsArray[col][row], as col and row aren't defined, and I'm not sure how to define an index value? Any help would be appreciated!
Connect 4
Example:
//array location is equal to an instance of this.cellsArray[col][row]
Board.prototype.checkRowRight = function (arrayLocation) {
if ((arrayLocation[i+1][i].value === arrayLocation.value) && (arrayLocation[i+2][i]=== arrayLocation.value) && (arrayLocation[i+3][i].value === arraylocation.value)){
this.winner = this.currentPlayer;
this.winnerFound = true;
console.log('Winner has been found!')
}
};
Referencing back to my logic found here and refactoring out the winning line detection code, this can easily be converted into Javascript as follows:
function chkLine(a,b,c,d) {
// Check first cell non-zero and all cells match
return ((a != 0) && (a ==b) && (a == c) && (a == d));
}
function chkWinner(bd) {
// Check down
for (r = 0; r < 3; r++)
for (c = 0; c < 7; c++)
if (chkLine(bd[r][c], bd[r+1][c], bd[r+2][c], bd[r+3][c]))
return bd[r][c];
// Check right
for (r = 0; r < 6; r++)
for (c = 0; c < 4; c++)
if (chkLine(bd[r][c], bd[r][c+1], bd[r][c+2], bd[r][c+3]))
return bd[r][c];
// Check down-right
for (r = 0; r < 3; r++)
for (c = 0; c < 4; c++)
if (chkLine(bd[r][c], bd[r+1][c+1], bd[r+2][c+2], bd[r+3][c+3]))
return bd[r][c];
// Check down-left
for (r = 3; r < 6; r++)
for (c = 0; c < 4; c++)
if (chkLine(bd[r][c], bd[r-1][c+1], bd[r-2][c+2], bd[r-3][c+3]))
return bd[r][c];
return 0;
}
And a test call:
x =[ [0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 1, 0, 0],
[0, 0, 0, 1, 1, 0, 0],
[0, 0, 1, 2, 2, 2, 0],
[0, 1, 2, 2, 1, 2, 0] ];
alert(chkWinner(x));
The chkWinner function will, when called with the board, return the first (and only, assuming each move changes only one cell and you're checking after every move) winning player.
The idea is to basically limit the checks to those that make sense. For example, when checking cells to the right (see the second loop), you only need to check each row 0-6 starting in each of the leftmost four columns 0-3.
That's because starting anywhere else would run off the right hand side of the board before finding a possible win. In other words, column sets {0,1,2,3}, {1,2,3,4}, {2,3,4,5} and {3,4,5,6} would be valid but {4,5,6,7} would not (the seven valid columns are 0-6).
This is an old thread but i'll throw my solution into the mix since this shows up as a top search result for "how to calculate connect4 win javascript"
I tackled this problem by using matrix addition.
Assume your game board is stored in memory as a 2D array like this:
[ [0, 0, 0, 0, 0, 0, 0],
[0, 0, Y, 0, 0, 0, 0],
[0, 0, Y, 0, 0, 0, 0],
[0, 0, R, 0, 0, 0, 0],
[0, 0, Y, 0, 0, 0, 0],
[0, 0, R, R, R, 0, 0] ];
On each "Coin Drop" you should call a function passing the x/y position of the coin.
THIS is where you calculate weather the user has won the game
let directionsMatrix = {
vertical: { south: [1, 0], north: [-1, 0] },
horizontal: { east: [0, 1], west: [0, -1] },
backward: { southEast: [1, 1], northWest: [-1, -1] },
forward: { southWest: [1, -1], northEast: [-1, 1] },
};
NOTE: "South" in matrix notation is [1,0], meaning "Down 1 cell, Right 0 cells"
Now we can loop through each Axis/Direction to check if there is 4 in a row.
const playerHasWon = (colnum, rowNum, playerColor, newGrid) => {
//For each [North/South, East/West, NorthEast/Northwest, SouthEast/Southwest]
for (let axis in directionsMatrix) {
// We difine this variable here so that "East" and "West" share the same count,
// This allows a coin to be dropped in a middle cell
let numMatches = 1;
// For each [North, South]
for (let direction in directionsMatrix[axis]) {
// Get X/Y co-ordinates of our dropped coin
let cellReference = [rowNum, colnum];
// Add co-ordinates of 1 cell in test direction (eg "North")
let testCell = newGrid[cellReference[0]][cellReference[1]];
// Count how many matching color cells are in that direction
while (testCell == playerColor) {
try {
// Add co-ordinates of 1 cell in test direction (eg "North")
cellReference[0] += directionsMatrix[axis][direction][0];
cellReference[1] += directionsMatrix[axis][direction][1];
testCell = newGrid[cellReference[0]][cellReference[1]];
// Test if cell is matching color
if (testCell == playerColor) {
numMatches += 1;
// If our count reaches 4, the player has won the game
if (numMatches >= 4) {
return true;
}
}
} catch (error) {
// Exceptions are to be expected here.
// We wrap this in a try/catch to ignore the array overflow exceptions
// console.error(error);
break;
}
}
// console.log(`direction: ${direction}, numMatches: ${numMatches}`);
// If our count reaches 4, the player has won the game
if (numMatches >= 4) {
return true;
}
}
}
// If we reach this statement: they have NOT won the game
return false;
};
Here's a link to the github repo if you wish to see the full code.
Here's a link to a live demo
I'm trying to implement histogram RGB but my algorithm doesn't produce similarly look like surface as in graphics programs. For example image on this site:
OpenCV histogram
My version looks like:
As I understood it correctly, RGB Histogram just measuring how often each value occured in specific channel. So I implement it in such way:
public Process(layerManager: dl.LayerManager) {
var surface = layerManager.GetCurrent();
var components = new Uint8Array(1024);
surface.ForEachPixel((arr: number[], i: number): void => {
components[arr[i]] += 1;
components[arr[i + 1] + 256] += 1;
components[arr[i + 2] + 512] += 1;
components[arr[i + 3] + 768] += 1;
});
var histogram = layerManager.GetHistogram();
histogram.Clear();
var viewPort = layerManager.GetHistogramViewPort();
viewPort.Clear();
this.DrawColor(histogram, components, 0, new ut.Color(255, 0, 0, 255));
//histogram.SetBlendMode(ds.BlendMode.Overlay);
//this.DrawColor(histogram, components, 256, new ut.Color(0, 255, 0, 255));
//this.DrawColor(histogram, components, 512, new ut.Color(0, 0, 255, 255));
}
private DrawColor(surface: ds.ICanvas, components: Uint8Array, i: number, fillStyle: ut.Color) {
var point = new ut.Point(0, 255);
surface.BeginPath();
surface.FillStyle(fillStyle.R, fillStyle.G, fillStyle.B, fillStyle.A);
surface.RGBAStrokeStyle(fillStyle.R, fillStyle.G, fillStyle.B, fillStyle.A);
surface.LineWidth(1);
surface.MoveTo(point);
for (var j = i + 256; i < j; ++i) {
point = new ut.Point(point.X + 1, 255 - components[i]);
surface.ContinueLine(point);
}
surface.ClosePathAndStroke();
var viewPort = layerManager.GetHistogramViewPort();
viewPort.DrawImage(surface.Self<HTMLElement>(), 0, 0, 255, 255, 0, 0, viewPort.Width(), viewPort.Height());
}
Am I missing something?
You have a Uint8Array array to hold the results, but the most common RGB values are occurring more than 255 times. This causes an overflow and you end up seeing a histogram of the values modulo 256, which is effectively random for high values. That's why the left and middle parts of the graph (where values are less than 255) are correct, but the higher-valued areas are all over the place.
Use a larger data type to store the results, and normalize to the size of your output canvas before drawing.
Been trying to sort this out for a few days and I am not sure if the CSS matrix is different from standard graphics matrices, or if I have something wrong (likely I have something wrong).
I am primarily trying to figure out how to rotate on the X and Y axis. When I use "transform: rotateX(2deg) rotateY(2deg) translate3d(0px, -100px, 0px);" and I use javascript to grab the matrix style, this is what I am able to output.
0.9993908270190958, -0.001217974870087876, -0.03487823687206265, 0,
0, 0.9993908270190958, -0.03489949670250097, 0,
0.03489949670250097, 0.03487823687206265, 0.9987820251299122, 0,
0, -99.93908270190957, 3.489949670250097, 1
But if I try to calculate the matrix using javascript (with 2 degrees on both X and Y) I get
0.9993908270190958, 0, -0.03489949670250097, 0,
-0.001217974870087876, 0.9993908270190958, -0.03487823687206265, 0,
0.03487823687206265, 0.03489949670250097, 0.9987820251299122, 0,
0.1217974870087876, -99.93908270190957, 3.487823687206265, 1
Now while several numbers are different in the second one, I believe one number is causing the problem. Note the numbers in row 1/column 2 and in row 2/column 1, for both matrices. The "-0.001217974870087876" looks to be switched. And if I understand how everything is calculated that is likely throwing off all the other numbers.
Here's the code I am using to create the second matrix
var basematrix = [
[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 1, 0],
[0, -100, 0, 1]
];
function RotateWorld(y, x)
{
var halfrot = Math.PI / 180;
var xcos = Math.cos(x * halfrot);
var xsin = Math.sin(x * halfrot);
var ycos = Math.cos(y * halfrot);
var ysin = Math.sin(y * halfrot);
var ymatrix = [
[ycos, 0, -ysin, 0],
[0, 1, 0, 0],
[ysin, 0, ycos, 0],
[0, 0, 0, 1]
];
var xmatrix = [
[1, 0, 0, 0],
[0, xcos, xsin, 0],
[0, -xsin, xcos, 0],
[0, 0, 0, 1]
];
var calcmatrix = MatrixMultiply(ymatrix, basematrix);
calcmatrix = MatrixMultiply(xmatrix, calcmatrix);
calcmatrix = TransMultiply(calcmatrix);
for (var i = 0; i < 4; i++)
{
for (var j = 0; j < 4; j++)
{
document.getElementById('info').innerHTML += calcmatrix[i][j] + ', ';
}
}
}
function MatrixMultiply(matrixa, matrixb)
{
var newmatrix = [];
for (var i = 0; i < 4; ++i)
{
newmatrix[i] = [];
for (var j = 0; j < 4; ++j)
{
newmatrix[i][j] = matrixa[i][0] * matrixb[0][j]
+ matrixa[i][1] * matrixb[1][j]
+ matrixa[i][2] * matrixb[2][j]
+ matrixa[i][3] * matrixb[3][j];
}
}
return newmatrix;
}
function TransMultiply(matrix)
{
var newmatrix = matrix;
var x = matrix[3][0];
var y = matrix[3][1];
var z = matrix[3][2];
var w = matrix[3][3];
newmatrix[3][0] = x * matrix[0][0] + y * matrix[1][0] + z * matrix[2][0];
newmatrix[3][1] = x * matrix[0][1] + y * matrix[1][1] + z * matrix[2][1];
newmatrix[3][2] = x * matrix[0][2] + y * matrix[1][2] + z * matrix[2][2];
newmatrix[3][3] = x * matrix[0][3] + y * matrix[1][3] + z * matrix[2][3] + newmatrix[3][3];
if (newmatrix[3][3] != 1 && newmatrix[3][3] != 0)
{
newmatrix[3][0] = x / w;
newmatrix[3][1] = y / w;
newmatrix[3][2] = z / w;
}
return newmatrix;
}
My code is a bit verbose as I am just trying to learn how to work with the CSS matrix. But hopefully someone can help me get that one number into the right place.
Edit
I hate to bump a post but I am running out of places to ask, so I am hoping a few more people will see it with a chance of getting an answer. I have tried every possible search to figure this out (unique questions don't get ranked very high in Google). I have probably read over 20 articles on working with matrices and they are yielding nothing. If I need to add more information please let me know. Also if there is a better place to ask let me know that as well. I would assume by now several people have looked at the code and the code must be ok, maybe my assumption that CSS is the culprit is a possibility, if so how does one track that down?
Take a look at this page, it explains how css 3dmatrix work. Also here you have an implementation in JS of CSSMatrix object, very similar to WebKitCSSMatrix which is already included in your (webkit) browser for your use.
You have a bug in your implementation of function TransMultiply(matrix) { .. }
var newmatrix = matrix;
That isn't cloning your matrix, that's setting newmatrix to refer to your original matrix! Anything using this method is going to have the original matrix and new matrix messed up. You might want to use a method that creates new 4x4 matricies, like:
function new4x4matrix(){
return [[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0]];
}
and then wherever you need a new matrix, do:
var newmatrix = new4x4matrix();
Edit: err, but you may actually need a clone method: fine.
function cloneMatrix(matrixa)
{
var newmatrix = [];
for (var i = 0; i < 4; ++i)
{
newmatrix[i] = [];
for (var j = 0; j < 4; ++j)
{
newmatrix[i][j] = matrixa[i][j];
}
}
return newmatrix;
}
and instead, for TransMultiply do:
var newmatrix = cloneMatrix(matrix);
This is my code:
var spriteSheet = new createjs.SpriteSheet({
images: ["images/mario2.png"],
frames: {width:24, height:33, regX: 0, regY: 0},
animations: {
walk_right: [5, 9],
walk_up: [10, 14],
walk_down: [15, 19],
walk_left: [0, 4]
}
});
Player = new createjs.BitmapAnimation(spriteSheet);
Player.name = "Mario";
Player.gotoAndStop("walk_right");
Player.speed = 6;
Player.x = 30;
Player.y = 330;
//Player object toevoegen aan de stage
stage.addChild(Player);
var monsterspritesheet = new createjs.SpriteSheet({
images: ["images/MonsterA.png"], //image to use
frames: {width: 64, height: 54, regX: 0, regY: 0},
animations: {
walk: [0, 9, "walk", 4],
idle: [10, 20, "idle", 4]
}
});
Monster1 = new createjs.BitmapAnimation(monsterspritesheet);
Monster1.name = "Monster";
Monster1.gotoAndStop("walk");
Monster1.speed = 6;
Monster1.x = 180;
Monster1.y = 330;
stage.addChild(Monster1);
stage.update();
For hitTest I use the following code:
var testpos = Player.globalToLocal(Monster1.x ,Monster1.y);
if (Player.hitTest(testpos.x, testpos.y) === true)
{
console.log("HIT");
}
THE PROBLEM:
When my player is below my monster, i get the message "HIT" but I would like to get it to work whenever my player hits the monster and not when it is below it.
I have been playing with regX and regY of both Player & Monster but it doesn't seem to work. Anyone knows the solution?
I think your issue here is that you need a CollisionDetection but hitTest() will only check for one pixel, which is in most cases used for mouse-interaction-detection and is perfect for it. However using it for detecting collisions between two bitmaps might need a little more work than just testing for one pixel.
I would suggest you to use my CollisionDetection class for bitmaps and bitmapAnimations, it works with bounding boxes or pixel-perfect and you basically just need to change one line.
You can check it out on Github: Bitmap Collision Detection for EaselJS
Description is on the github.com Page. (just in case you'll use it: let me know if you run into any bugs)