We Keep Coding

Finding a pattern in an array that is not always consistant

I have an ordered data set of decimal numbers. This data is always similar - but not always the same. The expected data is a few, 0 - 5 large numbers, followed by several (10 - 90) average numbers then follow by smaller numbers. There are cases where a large number may be mixed into the average numbers' See the following arrays.
let expectedData = [35.267,9.267,9.332,9.186,9.220,9.141,9.107,9.114,9.098,9.181,9.220,4.012,0.132];
let expectedData = [35.267,32.267,9.267,9.332,9.186,9.220,9.141,9.107,30.267,9.114,9.098,9.181,9.220,4.012,0.132];
I am trying to analyze the data by getting the average without high numbers on front and low numbers on back. The middle high/low are fine to keep in the average. I have a partial solution below. Right now I am sort of brute forcing it but the solution isn't perfect. On smaller datasets the first average calculation is influenced by the large number.
My question is: Is there a way to handle this type of problem, which is identifying patterns in an array of numbers?
My algorithm is:
Get an average of the array
Calculate an above/below average value
Remove front (n) elements that are above average
remove end elements that are below average
Recalculate average
In JavaScript I have: (this is partial leaving out below average)
let total= expectedData.reduce((rt,cur)=> {return rt+cur;}, 0);
let avg = total/expectedData.length;
let aboveAvg = avg*0.1+avg;
let remove = -1;
for(let k=0;k<expectedData.length;k++) {
if(expectedData[k] > aboveAvg) {
remove=k;
} else {
if(k==0) {
remove = -1;//no need to remove
}
//break because we don't want large values from middle removed.
break;
}
}
if(remove >= 0 ) {
//remove front above average
expectedData.splice(0,remove+1);
}
//remove belows
//recalculate average

I believe you are looking for some outlier detection Algorithm. There are already a bunch of questions related to this on Stack overflow.
However, each outlier detection algorithm has its own merits.
Here are a few of them
https://mathworld.wolfram.com/Outlier.html
High outliers are anything beyond the 3rd quartile + 1.5 * the inter-quartile range (IQR)
Low outliers are anything beneath the 1st quartile - 1.5 * IQR
Grubbs's test
You can check how it works for your expectations here
Apart from these 2, the is a comparison calculator here . You can visit this to use other Algorithms per your need.

I would have tried to get a sliding window coupled with an hysteresis / band filter in order to detect the high value peaks, first.
Then, when your sliding windows advance, you can add the previous first value (which is now the last of analyzed values) to the global sum, and add 1 to the number of total values.
When you encounter a peak (=something that causes the hysteresis to move or overflow the band filter), you either remove the values (may be costly), or better, you set the value to NaN so you can safely ignore it.
You should keep computing a sliding average within your sliding window in order to be able to auto-correct the hysteresis/band filter, so it will reject only the start values of a peak (the end values are the start values of the next one), but once values are stabilized to a new level, values will be kept again.
The size of the sliding window will set how much consecutive "stable" values are needed to be kept, or in other words how much UNstable values are rejected when you reach a new level.

For that, you can check the mode of the values (rounded) and then take all the numbers in a certain range around the mode. That range can be taken from the data itself, for example by taking the 10% of the max - min value. That helps you to filter your data. You can select the percent that fits your needs. Something like this:
let expectedData = [35.267,9.267,9.332,9.186,9.220,9.141,9.107,9.114,9.098,9.181,9.220,4.012,0.132];
expectedData.sort((a, b) => a - b);
/// Get the range of the data
const RANGE = expectedData[ expectedData.length - 1 ] - expectedData[0];
const WINDOW = 0.1; /// Window of selection 10% from left and right
/// Frequency of each number
let dist = expectedData.reduce((acc, e) => (acc[ Math.floor(e) ] = (acc[ Math.floor(e) ] || 0) + 1, acc), {});
let mode = +Object.entries(dist).sort((a, b) => b[1] - a[1])[0][0];
let newData = expectedData.filter(e => mode - RANGE * WINDOW <= e && e <= mode + RANGE * WINDOW);
console.log(newData);

Modifying hoverformat in plotly

How would I make it so that when hovering over a plotly graph, the displayed value in the hoverbox can be divided by 1000 and have a "K" appended to it?
For instance, when hovering over "$10,000,000" in a plotly graph, what could be done so that "$10,000K" is displayed in the box?
Specifically, can anything be done with the "hoverformat" property? It currently looks like this:
hoverformat: ",.0f"
I'm aware that this prints a float with 0 decimal places...but how would I specify the displayed value to be divided by 1000 and then have a "K" appended?
Thanks.

You might consider using numeral for that purpose. Numeral is a library used to format numbers and you can find it here:
http://numeraljs.com/
You then could do something like
const number = 10000;
const formattedNumber = numeral(10000).format('$ 0.00 a');

d3 js axis Percentage Values with decimals

I'm using d3 to create a graph - (still learning both of those). I'd like my Y-axis to display values in percentages. The min and max values on the Y-axis are allocated dynamically, so the axis scale could sometimes be 25% to 38% or sometimes even 13% to 16%.
When there is a relatively larger range such as 25% to 28%, I'm fine with the numbers appearing as they are, i.e. 25%, 28%, 31% ....
However, for a really small range, I notice it appears as 13%, 14%, 14%, 15%, 15%,... (numbers repeating, probably because behind the scenes they might be something like say, 14.2% and 14.8%).
In such cases, I'd like the numbers to appear with 1 decimal place specified, such as 13.5%.
I know I can specify
.tickFormat(d3.format(".1%"))
and this gives me what I need in that case. The problem is when I have a larger scale, it essentially gives me 25.0%, 28.0%, 31.0%,... and in this case, I'd like no decimal precision.
Is there a simple way I can handle each scenario? Any guidance would be most appreciated.

As mentioned in the comments, you could check if the range is big enough and add the decimal point format depending on that but that wouldn't be very clean. I suggest you pass your own function as the tickFormat that applies the format depending if the value has a decimal portion or not. Something like:
// defined your formatters somewhere above
var decimalFormatter = d3.format(".1");
myAxis.tickFormat(function(d) {
if (!Number.isInteger(d)) {
d = decimalFormatter(d); // add the decimal point for non-integers
}
return d + '%';
}
Sample code:
// defined your formatters somewhere above
var decimalFormatter = d3.format(".1");
var tickFormat = function(d) {
if (!Number.isInteger(d)) {
d = decimalFormatter(d); // add the decimal point for non-integers
}
return d + '%';
};
console.log(tickFormat(12.5)); // 12.5%
console.log(tickFormat(14)); // 14%
console.log(tickFormat(15)); // 15%
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.11/d3.min.js"></script>
Where you can adjust the function above in any specific way you might need. For example, if you are using strings instead of numbers, you will need to first convert to a number and then use Number.isInteger because it will always return false for a string.

You already have the solution for numbers of a small scale: d3.format(.1%). For numbers of a large scale, you can use d3.format(~%). This will automatically delete the trailing zero for, say, 25.0%, 28.0%, 31.0%,...
The documentation of d3-format mentions that:
The ~ option trims insignificant trailing zeros across all format types. This is most commonly used in conjunction with types r, e, s and %.

Presimplify topojson from command line

As far as I understand topojson.presimplify(JSON) in D3 adds Z coordinate to each point in the input topojson shape based on its significance, which then allows to use it for the dynamic simplification like in http://bl.ocks.org/mbostock/6245977
This method topojson.presimplify() takes quite a long time to execute on complicated maps, especially in Firefox which makes the browser unresponsive for few seconds.
Can it be baked directly into the topojson file via the command line as it is done with projections:
topojson --projection 'd3.geo.mercator().translate([0,0]).scale(1)' -o cartesian.topo.json spherical.topo.json

I found a workaround for this which is not completely as simple as I wanted but still achieves the same result.
After the topojson.presimplify(data) is called, data already holds the pre simplified geometry with added Z axis values.
Then I convert it to the JSON string and manually copy it to a new file with JSON.stringify(data)
Nevertheless these conversion to a JSON string has a problem with Infinity values which often occur for Z and with JSON.stringify method are converted to null. Also when there is a value for Z coordinate it is usually too precise and writing all decimal points takes too much space.
For that reason before converting data to a JSON string I trim the numbers:
// Simplifying the map
topojson.presimplify(data);
// Changing Infinity values to 0, limiting decimal points
var arcs = data.arcs;
for(var i1 = arcs.length; i1--;) {
var arc = arcs[i1];
for(var i2 = arc.length; i2--;) {
var v = arc[i2][2];
if(v === Infinity) arc[i2][2] = 0;
else {
arc[i2][2] = M.round(v * 1e9)/1e9;
}
}
}
This makes Infinity values to appear as exactly 0 and other values are trimmed to 9 decimal points which is enough for dynamic simplification to work properly.
Since such string is too long to easily print it for copying to the new json file it is much easier to store it in the localStorage of the browser:
localStorage.setItem(<object name>, JSON.stringify(data))
Then in Safari or Chrome open the developer console and in the tab Resources -> Local Storage -> <Website URL> the stored object can be found, copied and then pasted into a text editor.
Usually it is pasted as a <key> <value> pair, so one needs to remove from the beginning of the pasted string so that it starts from {.
Since Infinity values have been converted to 0, in the dynamic simplification function it should be taken into account so that points with Z = 0 are treated as Z = Infinity and are always plotted with any simplification area:
point: function(x, y, z) {
if (z===0 || z >= simplificationArea) {
this.stream.point(x, y);
}
}

Optimise my Javascript percentage calculator

I have a javascript that calculates the percentage from two fields (retail and network) and then dumps that percentage into another field (markup).
As I am relatively new to the world of JS I have ended up reusing the code for several rows of fields. This goes against DRY and KISS principles so I was wondering if you could give me some input on how to optimise my code so that it can handle any two fields and then dump a value to a third field.
Here is a screenshot of my form segment that is using it.
http://i.imgur.com/FHvDs.png
Here is my code I am using, I have had to reuse it four times and place the code in four functions e.g. (percentage1, percentage2, percentage3, percentage4) each one of these functions deals with a row of fields show in the screenshot.
function percentage1()
{
//the dividee
x = document.getElementById('tariff_data');
//the divider
y = document.getElementById('network_data');
//if the first value is lower than the second, append a "-" sign
if (x.value < y.value)
{
z = "-"+(x.value/y.value)*100;
document.getElementById('markup_data').value = z;
}
//not a negative percentage
else
{
z = (x.value/y.value)*100;
document.getElementById('markup_data').value = z;
}
}
function percentage2()
{
//the dividee
x = document.getElementById('tariff_rental');
//the divider
y = document.getElementById('network_rental');
//if the first value is lower than the second, append a "-" sign
if (x.value < y.value)
{
z = "-"+(x.value/y.value)*100;
document.getElementById('markup_rental').value = z;
}
//not a negative percentage
else
{
z = (x.value/y.value)*100;
document.getElementById('markup_data').value = z;
}
}
etc etc....
These functions are called using the onchange HTML attribute
Also when I divide by a decimal number it gives the wrong value, any Ideas how to make it calculate the correct percentage of a decimal number?
My code also gives out these strange outputs:
NaN , Infinity
Thanks

Rather than optimization, let's focus on correctness first =)
Note that the HTMLInputElement.value property has type "string", so your arithmetic operators are doing implicit type conversion which means you are likely often doing string concatenation instead of the numeric operations you expect.
I strongly recommend explicitly converting them to numbers first and checking for invalid input, also, don't forget to declare your variables first using var so they don't potentially clobber globals, e.g.:
var x = Number(document.getElementById('tariff_data'));
var y = Number(document.getElementById('network_data'));
if (!isFinite(x) || !isFinite(y)) {
// Handle non-numerical input...
}
You can also use the parseFloat function if you prefer, e.g.:
var x = parseFloat(document.getElementById('tariff_data'), 10);
I highly recommend doing some formal learning about the JavaScript language; it is full of pitfalls but if you stick to the "good parts" you can save yourself a lot of hassle and headache.

With regard to DRYing your code out; remember that you can:
Pass parameters to your functions and use those arguments within the function
Return values using the return keyword
In your case, you've got all your multiplication code repeated. While trying to fix the string vs. number problems maerics has already mentioned, you could do something like this:
// We're assuming 'dividee' and 'divider' are numbers.
function calculatePercentage(dividee, divider) {
var result;
// Regardless of the positive/negative result of the calculation,
// get the positive result using Math.abs().
result = Math.abs((dividee.value / divider.value) * 100);
// If the result was going to be negative...
if (dividee.value < divider.value) {
// Convert our result to negative.
result = result * -1;
}
// Return our result.
return result;
}
Then, in your percentage functions, you can just call this code like so:
function percentage1() {
var tariff, network, markup;
tariff = parseFloat(document.getElementById('tariff_data').value, 10);
network = parseFloat(document.getElementById('network_data').value, 10);
markup = document.getElementById('markup_data');
markup.value = calculatePercentage(tariff, network);
}
Obviously, you could take this further, and create a function which takes in the IDs, extracts the values from the elements etc., but you should try and build that yourself based on these tips.
Maerics also makes a very good point which you should take note of; learn more about the Good Parts of JavaScript. Douglas Crockford's book is excellent, and should be read and understood by all JS developers, IMHO.
Hope this helps you clean your code up!