Handling large datasets in dimple.js to render a chart - javascript

I wish to draw charts for large datasets using dimple.js. My code works absolutely fine. But the only problem is that the chart takes more than 45 seconds to come up. I am looking for some kind of an optimization in my code to reduce the time taken for the rendering of the chart. The following is the code for my area chart:
var dataset = [];
// The arrays xpoints and ypoints are populated dynamically
// with hundreds of thousands of points
var xpoints = chartData["xdata"];
var ypoints = chartData["ydata"];
var area1;
var svg = dimple.newSvg("#" + mychart, 700, 600);
var x, y;
for (var i = 0; i < xpoints.length; i++)
dataset.push({
x : xpoints[i],
y1 : parseFloat(ypoints[i])
});
var myChart = new dimple.chart(svg, dataset);
myChart.setBounds(75, 30, 480, 330);
y = myChart.addMeasureAxis("y", "y1");
x = myChart.addCategoryAxis("x", "x");
area1 = myChart.addSeries("First", dimple.plot.area, [ x, y ]);
var l = myChart.addLegend(65, 10, 510, 20, "right");
myChart.draw(1500);
Is there some way to optimize this code in either dimple.js itself or maybe using d3.js?

I'm afraid Dimple is not very performant for hundreds of thousands of points. It's drawing logic is built for flexibility and for cases like this you need to write specific d3 code (think of Dimple as a Swiss-Army Knife but here you need a scalpel). Even with raw d3 you might run into problems with a path containing that number of points. Certainly try raw d3 but you might need to write some more complex additional logic to average every n points together and then fill in detail on zoom. Also remember that even with perfect client code you will suffer a noticeable wait simply getting that volume of data from the server.

I found a solution!!. I was adamant on using dimple.js itself and not raw d3.
What I did was I aggregated the values first and then passed them to the chart.draw() function
The time taken to render the graph now is reduced from 40 seconds to 12 seconds, which is much better.
For now, my aggregation function just sums up the values for a particular category. Maybe the implementation in the draw() function is a little more complex and is therefore taking extra time. xpoints[] and ypoints[] are my arrays with lakhs of points.
Earlier, I just did this:
dataset.push({
x : xpoints[i],
y1 : parseFloat(ypoints[i])
});
Now, I first apply an aggregation as follows:
var isPresent = false;
for (var j = 0; j < unique_x.length; j++) {
if (xpoints[i] == unique_x[j]) {
y_val = parseFloat(ypoints[i]);
if (isNaN(y_val)) {
y_val = 0;
}
y_sum[j] = y_sum[j] + y_val;
isPresent = true;
break;
}
}
if (isPresent == false) {
unique_x.push(xpoints[i]);
y_sum.push(parseFloat(ypoints[i]));
}
Then, I do this:
for (var i = 0; i < unique_x.length; i++) {
dataset.push({
x : unique_x[i],
y1 : y_sum[i]
});

Related

Recale plotly js z axis on zoom

plotly.js 2D histograms and contour plots automatically generate a z-axis range that accommodates the entire range of z values in the dataset being plotted. This is fine to start, but when I click-and-drag on the plot to zoom in, I'd like the z axis range to also zoom in to accommodate only the range of z values currently on display; instead, the z axis never changes. Here's a codepen (forked from the plotly examples, thanks plotly) to play around with: http://codepen.io/anon/pen/MKGyJP
(codepen code inline:
var x = [];
var y = [];
for (var i = 0; i < 500; i ++) {
x[i] = Math.random();
y[i] = Math.random() + 1;
}
var data = [
{
x: x,
y: y,
type: 'histogram2d'
}
];
Plotly.newPlot('myDiv', data);
)
This seems like pretty conventional behavior - am I missing an option in the docs somewhere to do this?
If there's no built-in option to do this, an acceptable alternative solution would be to manually set new z limits in a zoom callback, which is easy enough to implement per this example: http://codepen.io/plotly/pen/dogexw - in which case my question becomes, is there a convenience method to get the min and max z currently on display?
Thanks in advance,
plotly.js doesn't have a zoom-specific callback at the moment(follow this issue for updates).
One alternative would be to add a mode bar button updating the colorscale range:
Plotly.newPlot('myDiv', data, {}, {
modeBarButtonsToAdd: [{
name: 'click here to update the colorscale range',
click: function(graphData) {
var xRange = graphData.layout.xaxis.range,
yRange = graphData.layout.yaxis.range;
var zMin, zMax;
// code that would compute the colorscale range
// given xRange and yRange
// for example given these values:
zMin = 10;
zMax = 20;
Plotly.restyle('myDiv', {zmin: zMin, zmax: zMax});
}
}]
});
Complete example: http://codepen.io/etpinard/pen/JGvNjV

Handling collision in THREE.js

Hey guys I have been working on a little project of mine utilizing the awesome library three.js
Now I have been working with the example of https://github.com/stemkoski/stemkoski.github.com/blob/master/Three.js/Collision-Detection.html to handle collision detection, more so on when an object over laps another object utilizing voxel's.
To reference my problem I am using the http://threejs.org/examples/#canvas_interactive_voxelpainter example.
Anyways to continue, when I render a voxel onto the screen, anything above the cube will allow me to render another voxel anything within a certain radius below the volex will not let me render:
Here is shown above cube:
Now here is my neat little function I put together using the example provided by stemkoski:
checkOverlapObject: function(voxel) // THIS IS USED TO SEE IF WE ARE OVER LAPPING ANY OBJECTS
{
var originPoint = voxel.position.clone();
var collidableObjs = this.rooms;
for (var vertexIndex = 0; vertexIndex < voxel.geometry.vertices.length; vertexIndex++)
{
var localVertex = voxel.geometry.vertices[vertexIndex].clone();
console.log(localVertex);
var globalVertex = localVertex.applyMatrix4( voxel.matrix );
console.log(globalVertex);
var directionVector = globalVertex.sub( voxel.position );
console.log(directionVector);
console.log(originPoint);
console.log(directionVector.clone().normalize());
if(collidableObjs.length > 0)
{
var ray = new THREE.Raycaster( originPoint, directionVector.clone().normalize() );
var collisionResults = ray.intersectObjects( collidableObjs );
if ( collisionResults.length > 0 && collisionResults[0].distance < directionVector.length() )
{
console.log(collisionResults);
console.log(collisionResults[0].distance);
console.log( directionVector.length() );
return false
}
}
}
return true;
},
Now what happens here is, before actually adding a rendered volex the user gets a preview of if they have permission to add the volex So we pass a volex made by:
var voxel = new THREE.Mesh( this.room.cubeGeometry, this.room.cubeTmpHoverMaterial );
voxel.geometry.computeBoundingBox();
voxel.position.copy( intersect.point ).add( intersect.face.normal );
voxel.position.divideScalar( 50 ).floor().multiplyScalar( 50 ).addScalar( 25 );
voxel.material.color.setHex(this.colorTmpHover);
into our checkOverlapObject function to see if the object is overlapping a object that has been rendered onto the screen/grid
Now following my little neat function I made, I have placed console.log to output parameters and here is some of that output:
T…E.Vector3 {x: 25, y: 25, z: 25} <!-- our localVertex
T…E.Vector3 {x: 25, y: 25, z: 25} <!-- our globalVertex
T…E.Vector3 {x: 0, y: 0, z: -350} <!-- our directionVector
T…E.Vector3 {x: 25, y: 25, z: 375} <!-- our originPoint
T…E.Vector3 {x: 0, y: 0, z: -1} <!-- our directionVector.clone().normalize()
[Object, Object] <!-- our collisionResults
225 <!-- our collisionResults[0].distance
350 <!-- our directionVector.length()
This data is based off of the first picture.
Please understand that I have other volex that take up 2 blocks on the grid or more. So the reason for this is, I have a center location of the position but I need to take into account the rest of the object if it takes up 2 blocks on the grid to check if that overlaps the already rendered volex I don't care if they touch each other.
Any suggestions or thoughts as to what might be the problem?
As asked above I finally found a solution. I created a custom function for object collision/overlap.
Firstly, here is whats wrong with the above collision code as shown above, it process's the raycaster starting with the direction of our object - shooting a line straight across from its orgin to see if anything comes into contact with that ray. Now that is great and all but not very sufficient in telling when we have a overlap/collision if any part of the object is touching another object
As shwon with our rays:
So here is my solution.
When ever we have an object on our plane, we know everything about it, our coordinates of x,y,z - I am not dealing with y in this example so ignore that.
What we care about is our objects width = x and our objects depth/length = z. So in essence when we are checking to see if an object is overlapping another object we simply need to process through each min x/z and max x/z of our to be added object and our already rendered obj
How we do that is, loop through our object to be added starting min z position with a for loop. This starting min z position contains our position z + min z of object. This gives us the exact z position on our plane.
We than move on to our width or our object to be added starting min x position with a for loop. Don't forget our x gives us the exact position on our plane
Now the reason for this is simple. We want to start at position 0,0 of our object to be added. Increment by one up our length/z - increment all the way down our width/x and while doing this process each object on our plane that is already rendered and check to see if any exact point matches ours.
Here is the actual method:
checkOverlapObject: function(voxel) // THIS IS USED TO SEE IF WE ARE OVER LAPPING ANY OBJECTS
{
var collidableObjs = this.rooms;
//lets get our voxel min and max vars
var voxelMX = voxel.geometry.boundingBox.max.x;
var voxelMZ = voxel.geometry.boundingBox.max.z;
var voxelmX = voxel.geometry.boundingBox.min.x;
var voxelmZ = voxel.geometry.boundingBox.min.z;
// we need to get our voxel position ad to do some math to see if voxel min and max do not go beyound our boundries for our plane
var voxelPX = voxel.position.x;
var voxelPZ = voxel.position.z;
var totalPositionVoxelminZ = (voxelPZ + voxelmZ);
var totalPositionVoxelminX = (voxelPX + voxelmX);
var totalPositionVoxelMAXZ = (voxelPZ + voxelMZ);
var totalPositionVoxelMAXX = (voxelPX + voxelMX);
// start loop for object to add Z cordinate
for(var length = totalPositionVoxelminZ; length < totalPositionVoxelMAXZ; length++)
{
// start loop for object to add X cordinate
for(var width = totalPositionVoxelminX; width < totalPositionVoxelMAXX; width++)
{
for(var i = 0; i < collidableObjs.length;i++)
{
//lets get our voxel min and max vars
var thisvoxelMX = this.rooms[i].geometry.boundingBox.max.x;
var thisvoxelMZ = this.rooms[i].geometry.boundingBox.max.z;
var thisvoxelmX = this.rooms[i].geometry.boundingBox.min.x;
var thisvoxelmZ = this.rooms[i].geometry.boundingBox.min.z;
// we need to get our voxel position ad to do some math to see if voxel min and max do not go beyound our boundries for our plane
var thisvoxelPX = this.rooms[i].position.x;
var thisvoxelPZ = this.rooms[i].position.z;
var thistotalPositionVoxelminZ = (thisvoxelPZ + thisvoxelmZ);
var thistotalPositionVoxelminX = (thisvoxelPX + thisvoxelmX);
var thistotalPositionVoxelMAXZ = (thisvoxelPZ + thisvoxelMZ);
var thistotalPositionVoxelMAXX = (thisvoxelPX + thisvoxelMX);
for(var insideZ = thistotalPositionVoxelminZ; insideZ < thistotalPositionVoxelMAXZ; insideZ++)
{
for(var insideX = thistotalPositionVoxelminX; insideX < thistotalPositionVoxelMAXX; insideX++)
{
if(insideZ == length && insideX == width)
{
return false;
}
}
}
}
}
}
return true;
}
Here is our result:
This really does provide an exact answer on if your object is touching something it should not be touching because of the point by point it processes.
I hope this helps! Please feel free to use this at anytime.
Also note this really could effect overhead/memory usage if you are not careful on how you use this. I am working on a way to better optimize this in the case of having hundred of objects to process. So if you have any suggestions on modifying or adding to my existing code to accomplish this better and offer better performance in case of hundreds of objects passed, feel free provide details!

How to print not time-series using dygraphs

I have the following dataset
X Y
------------
9.2294 40
9.65712 60
10.0633 80
10.1865 90
10.2844 100
10.4122 120
10.5217 140
10.5776 160
10.5995 180
10.6237 200
10.563 250
and I want to plot a Profile (Y is the elevation) connecting the point on the basis of the Y order and not to the X order (it is a XY graph and not a time-series)... with dygraphs it seems to be not possible.
dygraphs requires that your data be sorted by the first column (the x-axis). You might be able to pull of what you're asking for using a custom plotter, but it's not a natural fit. You'll have an easier time using either D3 or a library with built-in support for plots like this.
I was able to achieve this with the use of a custom plotter function based on the link #danvk posted.
It works, but it's not pretty. I would love to hear suggestions on how to
avoid using a custom plotter
finding the canvas x/y positions without linear regression
reusing existing drawing code for better features support and avoiding code duplication
below is the code and a screenshot of the graphs it renders
plotter: function(e) {
// dygraph series
var seriesPoints = e.allSeriesPoints[e.seriesIndex];
// real series
var realData = table.lists.get(e.seriesIndex);
// build a linear regression to map from real values to canvas x/y based on a sample of 2 points
var pa, pb;
seriesPoints.forEach(p => {
if (!pa && p.xval != null && p.yval != null) pa = p;
else if (pa && !pb && p.xval != null && p.yval != null) pb = p;
});
if (!pa || !pb) {
return;
}
baseDataX = pa.xval;
baseDataXMul = (pb.canvasx - pa.canvasx) / (pb.xval - pa.xval);
baseDataY = pa.yval;
baseDataYMul = (pb.canvasy - pa.canvasy) / (pb.yval - pa.yval);
var getPoint = function(i) { return { canvasx: pa.canvasx + baseDataXMul * (realData[i].x - baseDataX), canvasy: pa.canvasy + baseDataYMul * (realData[i].y - baseDataY) }; }
// draw the line, iterating on the data points of the REAL series, each of which os properly sorted
var ctx = e.drawingContext;
ctx.beginPath();
var p0 = getPoint(0);
if (p0) ctx.moveTo(p0.canvasx, p0.canvasy);
for (var i = 1; i < realData.length; i++) {
var p = getPoint(i);
if (p) ctx.lineTo(p.canvasx, p.canvasy);
}
ctx.stroke(); }

How to draw a exponential function y=ab^x using d3.js axis functionality

I am trying to draw an exponential function (y=ab^x) using the d3.js (javascript) library. I understand how to draw the axes themselves. I just need the magic that draws the actual line. I have seen description for the linear and quadratic equations but nothing more custom.
Any help would be appreciated.
I think that you need to construct the data yourself. For an exponential function, you can generate the data:
var data = [],
n = 100,
a = 1,
b = 2;
for (var k = 0; k < 100; k++) {
data.push({x: 0.01 * k, y: a * Math.pow(b, 0.01 * k)});
}
and then, use the standard code to generate a line graph, for instance, see http://bl.ocks.org/3883245.

Drawing zoomable audio waveform timeline in Javascript

I have raw 44,1 kHz audio data from a song as Javascript array and I'd like to create a zoomable timeline out of it.
Example timeline from Audacity:
Since there are millions of timepoints normal Javascript graphics libraries probably don't cut it: I think, not sure, that normal graph libraries will die on this many timepoints. But does there exist already libraries for this sort of visualization for JS? Canvas, webGL, SVG all are acceptable solutions.
A solution preferably with zoom and pan.
Note that this happens strictly on client side and server-side solutions are not accetable.
I've looked into this same problem pretty extensively. To the best of my knowledge, the only existing project that does close to what you want is wavesurfer.js. I haven't used it, but the screenshots and the description sound promising.
See also this question.
Best of luck.
You cannot simply take the the waveform data and render all data points, this is terribly inefficient.
Variable explanation:
width: Draw area width in pixels, max is screen width
height: Same as width but then height of draw area
spp: Samples per pixel, this is your zoom level
resolution: Number of samples to take per pixel sample range, tweak for performance vs accuracy.
scroll: You will need virtual scrolling for performance, this is the scroll position in px
data: The raw audio data array, probably several million samples long
drawData: The reduced audio data used to draw
You are going to have to only take the samples that are in the viewport from the audio data and reduce those. Commenly this results in a data set that is 2 * width, you use this data set to render the image.
To zoom out increase spp, to zoom in decrease it. Changing scroll value pans it.
The following code has O(RN) complexity where N is width and R is resolution. Maximum accuracy is at spp <= resolution.
The code will look something like this, this gets the peak values, you could do rms or average as well.
let reduceAudioPeak = function(data, spp, scroll, width, resolution) {
let drawData = new Array(width);
let startSample = scroll * spp;
let skip = Math.ceil(spp / resolution);
// For each pixel in draw area
for (let i = 0; i < width; i++) {
let min = 0; // minimum value in sample range
let max = 0; // maximum value in sample range
let pixelStartSample = startSample + (i * spp);
// Iterate over the sample range for this pixel (spp)
// and find the min and max values.
for(let j = 0; j < spp; j += skip) {
const index = pixelStartSample + j;
if(index < data.length) {
let val = data[index];
if (val > max) {
max = val;
} else if (val < min) {
min = val;
}
}
}
drawData[i] = [min, max];
}
return drawData;
}
With this data you can draw it like this, you could use lines, svg etc:
let drawWaveform = function(canvas, drawData, width, height) {
let ctx = canvas.getContext('2d');
let drawHeight = height / 2;
// clear canvas incase there is already something drawn
ctx.clearRect(0, 0, width, height);
for(let i = 0; i < width; i++) {
// transform data points to pixel height and move to centre
let minPixel = drawData[i][0] * drawHeigth + drawHeight;
let maxPixel = drawData[i][1] * drawHeight + drawHeight;
let pixelHeight = maxPixel - minPixel;
ctx.fillRect(i, minPixel, 1, pixelHeight);
}
}
I have used RaphaelJS for SVG rendering in the browser at it has performed very well. It is what I would go for. Hopefully SVG will be up to the task.

Categories