We Keep Coding

Finding a pattern in an array that is not always consistant

I have an ordered data set of decimal numbers. This data is always similar - but not always the same. The expected data is a few, 0 - 5 large numbers, followed by several (10 - 90) average numbers then follow by smaller numbers. There are cases where a large number may be mixed into the average numbers' See the following arrays.
let expectedData = [35.267,9.267,9.332,9.186,9.220,9.141,9.107,9.114,9.098,9.181,9.220,4.012,0.132];
let expectedData = [35.267,32.267,9.267,9.332,9.186,9.220,9.141,9.107,30.267,9.114,9.098,9.181,9.220,4.012,0.132];
I am trying to analyze the data by getting the average without high numbers on front and low numbers on back. The middle high/low are fine to keep in the average. I have a partial solution below. Right now I am sort of brute forcing it but the solution isn't perfect. On smaller datasets the first average calculation is influenced by the large number.
My question is: Is there a way to handle this type of problem, which is identifying patterns in an array of numbers?
My algorithm is:
Get an average of the array
Calculate an above/below average value
Remove front (n) elements that are above average
remove end elements that are below average
Recalculate average
In JavaScript I have: (this is partial leaving out below average)
let total= expectedData.reduce((rt,cur)=> {return rt+cur;}, 0);
let avg = total/expectedData.length;
let aboveAvg = avg*0.1+avg;
let remove = -1;
for(let k=0;k<expectedData.length;k++) {
if(expectedData[k] > aboveAvg) {
remove=k;
} else {
if(k==0) {
remove = -1;//no need to remove
}
//break because we don't want large values from middle removed.
break;
}
}
if(remove >= 0 ) {
//remove front above average
expectedData.splice(0,remove+1);
}
//remove belows
//recalculate average

I believe you are looking for some outlier detection Algorithm. There are already a bunch of questions related to this on Stack overflow.
However, each outlier detection algorithm has its own merits.
Here are a few of them
https://mathworld.wolfram.com/Outlier.html
High outliers are anything beyond the 3rd quartile + 1.5 * the inter-quartile range (IQR)
Low outliers are anything beneath the 1st quartile - 1.5 * IQR
Grubbs's test
You can check how it works for your expectations here
Apart from these 2, the is a comparison calculator here . You can visit this to use other Algorithms per your need.

I would have tried to get a sliding window coupled with an hysteresis / band filter in order to detect the high value peaks, first.
Then, when your sliding windows advance, you can add the previous first value (which is now the last of analyzed values) to the global sum, and add 1 to the number of total values.
When you encounter a peak (=something that causes the hysteresis to move or overflow the band filter), you either remove the values (may be costly), or better, you set the value to NaN so you can safely ignore it.
You should keep computing a sliding average within your sliding window in order to be able to auto-correct the hysteresis/band filter, so it will reject only the start values of a peak (the end values are the start values of the next one), but once values are stabilized to a new level, values will be kept again.
The size of the sliding window will set how much consecutive "stable" values are needed to be kept, or in other words how much UNstable values are rejected when you reach a new level.

For that, you can check the mode of the values (rounded) and then take all the numbers in a certain range around the mode. That range can be taken from the data itself, for example by taking the 10% of the max - min value. That helps you to filter your data. You can select the percent that fits your needs. Something like this:
let expectedData = [35.267,9.267,9.332,9.186,9.220,9.141,9.107,9.114,9.098,9.181,9.220,4.012,0.132];
expectedData.sort((a, b) => a - b);
/// Get the range of the data
const RANGE = expectedData[ expectedData.length - 1 ] - expectedData[0];
const WINDOW = 0.1; /// Window of selection 10% from left and right
/// Frequency of each number
let dist = expectedData.reduce((acc, e) => (acc[ Math.floor(e) ] = (acc[ Math.floor(e) ] || 0) + 1, acc), {});
let mode = +Object.entries(dist).sort((a, b) => b[1] - a[1])[0][0];
let newData = expectedData.filter(e => mode - RANGE * WINDOW <= e && e <= mode + RANGE * WINDOW);
console.log(newData);

Determine if a numerical value is within some margin of a non-specific integer

I am using various JavaScript Math functions and, due to floating point numbers, these functions occasionally return values that are either 0.000000001 larger or smaller than the correct integer answer. I am looking to set up an if-else statement for said functions that will return the correct integer answer should the value be within some small range of an integer (note that the non-specificity of said integer is of utmost importance).
So I am asking, is there a way, using JavaScript, to determine if the value returned from a math function (Math.cbrt() for example) is within some margin of a non-specific integer?
NOTE: I have tried using Number.EPSILON in a function which calculates the x-th root of a number like so
var index = $('#Index').val();
var radicand = $('#Radicand').val();
var powerXroot = Math.pow(radicand,(1/index))+(Number.EPSILON * Math.pow(radicand,(1/index)));
but it doesn't work for all indicies.

You can use Math.round() to get the nearest integer to the result. If this integer is within 0.000000001 of the result, replace the result with the integer instead.
Say, you have computed var x = ... and want to make it an integer if it's sufficiently close to one.
function roundIfAlmostInteger(x) {
if (Math.abs(x - Math.round(x)) < 0.000000001) {
x = Math.round(x);
}
return x;
}
Illustration:
x = 2.3 - 0.1 - 0.2; // now x is 1.9999999999999998
x = roundIfAlmostInteger(x); // now x is 2

Select optimal set of samples to approximate a curve with predetermined number of samples?

Background
I have a pet project that I love to overthink from time to time. The project has to do with an RC aircraft control input device. People familiar with that hobby are probably also familiar with what is known as "stick expo", which is a common feature of RC transmitters where the control sticks are either more or less sensitive near the neutral center position and become less or more sensitive as the stick moves closer to its minimum or maximum values.
I've read some papers that I don't fully understand. I clearly don't have the math background to solve this, so I'm hoping that perhaps one of you might.
Problem
I have decided to approximate the curve by taking a pre-determined number of samples and use linear interpolation to determine output values for any input values between the sample points. I'm trying to find a way to determine the most optimal set of sample points.
If you look at this example of a typical growth curve for this application, you will notice that some sections are more linear (straighter), and some are less linear (more curvy).
These samples are equally distant from each other, but they don't have to be. It would be smart to increase the sample density where there is more change and thereby increasing the resolution in the curvy segments by borrowing redundant points from the straight segments.
Is it possible to quantify the degree of error? If it is, then is it also possible to determine the optimal set of samples for a given function and a pre-determined number of samples?
Reference Code
Snippet from the class that uses a pre-calculated set of points to approximate an output value.
/* This makes the following assumptions:
* 1. The _points[] data member contians at least 2 defined Points.
* 2. All defined Points have x and y values between MIN_VALUE and MAX_VALUE.
* 3. The Points in the array are ordered by ascending values of x.
*/
int InterpolatedCurve::value( int x ) {
if( _points[0].x >= x ) { return _points[0].y; }
for( unsigned int i = 1; i < _point_count; i++ ) {
if( _points[i].x >= x ) {
return map(x, _points[i-1].x, _points[i].x,
_points[i-1].y, _points[i].y);
}
}
// This is an error condition that is not otherwise reported.
// It won't happen as long as the points are set up correctly.
return x;
}
// Example map function (borrowed from Arduino site)
long map( long x, long x1, long x2, long y1, long y2 ) {
return (x - x1) * (y2 - y1) / (x2 - x1) + y1;
}
Although my project is actually in C++, I'm using a Google spreadsheet to produce some numbers while I ponder this problem.
// x: Input value between -1 and 1
// s: Scaling factor for curve between 0 (linear) and 1 (maximum curve)
// c: Tunable constant
function expo_fn( x, s, c ) {
s = typeof s !== 'undefined' ? s : 1.0;
c = typeof c !== 'undefined' ? c : 4.0;
var k = c * ((c - 1.0) * s*s*s + s)/c + 1.0;
return ((k - 1.0) * x*x*x*x*x + x)/k;
};
The following creates a set of isometrically distributed (non-optimal) points between input values -1 and 1. These output values were expanded to integers between -16383 and 16383 for the above example spreadsheet. Factor is a value between 0 and 1 that determines the "curviness"--zero being a flat, linear curve and 1 being the least-linear curve I care to generate.
function Point( x, y ) {
this.x = x;
this.y = y;
};
function compute_points_iso( count, factor ) {
var points = [];
for( var i = 0; i < count; ++i ) {
var x = 2.0/(count - 1.0) * i - 1.0;
var y = expo_fn(x, factor);
points.push(new Point(x,y));
}
return points;
};
Relevant Academic Work
I have been studying this paper describing an algorithm for selecting significant data points, but my program doesn't quite work right yet. I will report back if I ever get this thing working.

The key here is to realize that you can bound the error on your linear interpolation in terms of the second derivative of the function. I.e. if we approximate f(x) \approx f(x_0) + f'(x_0)*(x-x_0), then the error in this approximation is less than abs[ 0.5*f''(x_0)(x-x_0)^2 ].
The outline of an iterative approach could look like this:
Construct an initial, e.g. uniformly spaced, grid
Compute the second derivative of the function on this grid.
Compute the bound on the error using the second-derivative and the inter-sample spacing
Move the samples closer together where the error is large; move them further apart where the error is small.
I'd expect this to be an iterative solution that loops on steps 2,3,4.
Most of the details are in step 4.
For a fixed number of sample points one could use the median of the error bounds to select
where finer/coarser sampling is required (i.e. those locations where the error is larger than the median error will have the sample points pulled closer together).
Let E_0 be this median of the error bounds; then we can, for each sample in the point, compute a new desired sample spacing (dx')^2=2*E_0/f''(x); then you'd need some logic to go through and change the grid spacing so that it is closer to these ideal spacings.
My answer is influenced by having used the "Self-Organizing Map" algorithm on data; this or related algorithms may be relevant for your problem. However, I can't recall ever
seeing a problem like yours where the goal is to make your estimates of the error uniform across the grid.

Bias in randomizing normally distributed numbers (javascript)

I’m having problems generating normally distributed random numbers (mu=0 sigma=1)
using JavaScript.
I’ve tried Box-Muller's method and ziggurat, but the mean of the generated series of numbers comes out as 0.0015 or -0.0018 — very far from zero!! Over 500,000 randomly generated numbers this is a big issue. It should be close to zero, something like 0.000000000001.
I cannot figure out whether it’s a method problem, or whether JavaScript’s built-in Math.random() generates not exactly uniformly distributed numbers.
Has someone found similar problems?
Here you can find the ziggurat function:
http://www.filosophy.org/post/35/normaldistributed_random_values_in_javascript_using_the_ziggurat_algorithm/
And below is the code for the Box-Muller:
function rnd_bmt() {
var x = 0, y = 0, rds, c;
// Get two random numbers from -1 to 1.
// If the radius is zero or greater than 1, throw them out and pick two
// new ones. Rejection sampling throws away about 20% of the pairs.
do {
x = Math.random()*2-1;
y = Math.random()*2-1;
rds = x*x + y*y;
}
while (rds === 0 || rds > 1)
// This magic is the Box-Muller Transform
c = Math.sqrt(-2*Math.log(rds)/rds);
// It always creates a pair of numbers. I'll return them in an array.
// This function is quite efficient so don't be afraid to throw one away
// if you don't need both.
return [x*c, y*c];
}

If you generate n independent normal random variables, the standard deviation of the mean will be sigma / sqrt(n).
In your case n = 500000 and sigma = 1 so the standard error of the mean is approximately 1 / 707 = 0.0014. The 95% confidence interval, given 0 mean, would be around twice this or (-0.0028, 0.0028). Your sample means are well within this range.
Your expectation of obtaining 0.000000000001 (1e-12) is not mathematically grounded. To get within that range of accuracy, you would need to generate about 10^24 samples. At 10,000 samples per second that would still take 3 quadrillon years to do...this is precisely why it's good to avoid computing things by simulation if possible.
On the other hand, your algorithm does seem to be implemented correctly :)

Searching/Sorting Multidimensional Arrays

I've created a multi-dimensional array based on the x/y coords of the perimeter of a circle. An object can be dragged along the arc (in javascript) and then 'dropped' anywhere on it. The problem is, I need to find the closest x and y coordinate to where the object is 'dropped.'
My current solution involves looping through an array and finding the closest value to x, and then looping again to find the y coordinate, but it doesn't seem very clean and there are problems with it.
Does anyone have any suggestions?
Thanks!

So, let's see. We assume a predefined set of (x, y) coordinates. You are given another point and have to find the nearest element of the array to that given point. I am going to assume "nearest" means the smallest Pythagorean or Euclidean distance from the given point to each of the other points.
The simplest algorithm is probably the best (if you want to look at others in Wikipedia, have at it). Since you didn't give us any code for the structure, I'm going to assume an array of objects, each object having an x and a y property, ditto for the given point.
var findNearestPoint = function (p, points) {
var minDist = Number.POSITIVE_INFINITY,
minPoint = -1,
i,
l,
curDist,
sqr = function(x) { return x * x; };
for (i = 0, l = points.length; i < l; i++) {
curDist = sqr(p.x - points[i].x) + sqr(p.y - points[i].y);
if (curDist < minDist) {
minDist = curDist;
minPoint = i;
}
}
return points[i];
};
(Untested, but you get the idea.)

If your arrays are created in sequential order (that is from smallest to greatest or greatest to smallest), you could use introduce a Binary Search Algorithm.
Get middle element of x array.
If x equals your value, stop and look for y, otherwise.
If x is lower, search in the lower half of the array (starting from step 1).
If x is higher, search in the upper half of the array (starting from step 1).
Then use the same formula on y. You might have to change to algorithm a bit to make it so it works with the closest matching element. Having not seen your array, I can't offer code to solve to problem.