I'm using d3.js to plot the contents of an 80,000 row .tsv onto a chart.
The problem I'm having is that since there is so much data, the page becomes unresponsive for aprox 5 seconds while the entire dataset is churned through at once.
Is there an easy way to process the data progressively if it's spread over a longer period of time?
Ideally the page would remain responsive, and the data would be plotted as it became available, instead of in one big hit at the end
I think you'll have to chunk your data and display it in groups using setInterval or setTimeout. This will give the UI some breathing room to jump in the middle.
The basic approach is:
1) chunk the data set
2) render each chunk separately
3) keep track of each rendered group
Here's an example:
var dataPool = chunkArray(data,100);
function updateVisualization() {
group = canvas.append("g").selectAll("circle")
.data(dataPool[poolPosition])
.enter()
.append("circle")
/* ... presentation stuff .... */
}
iterator = setInterval(updateVisualization,100);
You can see a demo fiddle of this -- done before I had coffee -- here:
http://jsfiddle.net/thudfactor/R42uQ/
Note that I'm making a new group, with its own data join, for each array chunk. If you keep adding to the same data join over time ( data(oldData.concat(nextChunk) ), the entire data set still gets processed and compared even if you're only using the enter() selection, so it doesn't take long for things to start crawling.
Related
I have data which updates every 10 seconds and I would like to check that all the data is valid before progressing with updates. I am currently getting false data intermittently which occurs as a negative number in one of the values. If one of the objects has a negative value then I don't trust the whole set and don't want to update any elements.
Ideally I don't want to update some items and then bail once the incorrect value occurs, but rather, determine if the whole set is good before updating anything
I'm not sure how d3 can manage this but I've tried with this and it seems to work. But it doesn't seem particularly in keeping with the elegance of D3 so I think there's probably a correct and better way to do it. But maybe not?!
var dataValid = true;
abcItems.each(function (d, i) {
if (0 > dd.Number1 - dd.Number2) dataValid = false;
});
if (dataValid) {
abcItems.each(function (d, i) {
// updating elements here
});
} else {
console.log("negative value occurred");
}
Is there a better way to manage this through D3?
A little bit more context:
The data (JSON provided via a RESTful API) and visualisation (a bar chart) are updating every 10 seconds. The glitch in the API results in incorrect data once every hour or so at the most (sometimes it doesn't happen all day). The effect of the glitch is that the bars all change dramatically whereas the data should only change by ones or twos each iteration. In the next fetch of data 10 seconds later the data is fine and the visualisation comes right.
The data itself is always "well-formed" it's just that the values provided are incorrect. Therefore even during the glitch it is safe to bind the data to elements.
What I want to do, is skip the entire iteration and update phase if the data contains one of these negative values.
Perhaps also worth noting is that the items in the data are always the same, that is to say the only "enter" phase that occurs is on page load and there are no items that exit (though I do include these operations to capture any unexpected fluctuations in the data). The values for these items do change though.
Looking at your code it seams you already have bound the dataset to your DOM elements abcItems.each(...).
Why not bail out of the update function when the data is not valid.
d3.json("bar-tooltip.json", function(dataset) {
if (!dataset.every(d => d.Number2 <= d.Number1)) return;
// do the update of the graph
});
The example assumes you call d3.json() froma function that is called every update interval, but you can use a different update method.
I am working on a project dealing with sensor data. In my backend everything is stored in a database which is getting polled by a controller and converted into kml to display on the cesium globe. This poll happens every 5-10 seconds and contains around 4000-8000 objects (we store up to 1 minute worth of data so we are looking at somewhere like 20k - 50k points). Following this I have an update function which slowly fades the markers out which updates every 5 seconds.
To load the kml on the map I use the following function:
var dataSource = new Cesium.KmlDataSource();
dataSource.load('link').then(function(value);
viewer.dataSources.add(dataSource);
});
On the update color function I am iterating over all of the objects within the datasources entity collection and updating them like so (this is very inefficient):
var colorUpdate = Cesium.Color.fromAlpha(newColor, .4);
dataSource.entities.values[i].billboard.color = colorUpdate;
When I do and add or color update I see a large amount of lag and was curious if there was anything you would suggest to fix this? Generally I get a freeze up for a few seconds. After 60 seconds of the data being on the map it gets removed like so (just a different if case within the color update loop)
dataSource.entities.remove(dataSource.entities.values[i]);
Is there potentially a way to set a propertiy for an entire entity collection so when this collection becomes 30 seconds old it updates the color to a new one? It seems that I just need to find a way to set a property for the entire collection vs individual entities. Does anyone know how to do that or have a suggestion for something better?
I want two instances of d3.forceCollide(). In one, every node is pushed away from one another to prevent overlap. In the second, only a subset of nodes are pushed away from one another, with a much bigger radius.
To accomplish the second force, I tweak the initialize method to filter the incoming nodes, like so:
function selective(force,filter){
var init = force.initialize;
force.initialize = function(_){return init(_.filter(filter));};
return force;
}
var dpi = 90; // approximate pixels per inch in SVG
var size = dpi * (1/4); // quarter-inch unit size
var universally_applied =
d3.forceCollide()
.radius(size)
.strength(1);
var selectively_applied =
selective(
d3.forceCollide(),
function(d){return d.id === color;}
)
.radius(size*5)
.strength(1);
}
Now, this ALMOST works. I created a fiddle to see it in action: https://jsfiddle.net/jarrowwx/0dax43ue/38/ - every colored circle is supposed to repel every other circle of the same color, from a distance. Every other color, it just bumps into and pushes it out of the way.
If I do not change the order in which things are defined, then the selectively applied force is ONLY applied to the first color (red). If I shuffle the data array before applying forces, it is difficult to define exactly what happens, but the force is applied to some circles and not most of the others, even among the same color.
Any ideas what is going on here, or how to fix it?
The D3 team decided that this behavior was a bug (Isolating forces to a subset of nodes? #72), and fixed it. The fix was included in version 1.0.4 of d3-force, which is available as part of the full D3 build as of version 4.4.0.
The problem is resolved using the solution suggested by "Partial forces on nodes in D3.js", and the code works as intended now.
I have used jqplot line chart.
I have get data from php page using ajax.In some condition I will display specific series.So
How to pass series dynamically in jqplot line chart and also set legend of series ?
I have manually write code for above requirement.I have apply click event on legend series and draw graph as per click on legend.
I have also change y-axis value as per select/deselect series legend.
I originally tried the answer posted by #sdespont, but due to additional properties that need to be in place for the series, it wasn't working properly. I was able to get this working by doing the following:
plot1.data = data;
plot1.replot( data );
data is a 3D array of the same makeup as you would pass in when creating the plot. If I did either part without doing the other, it wouldn't refresh properly, but the combination of the two seems to do the trick. Refreshing the plot in this fashion would dynamically add or remove any series I added to the data array.
Hope that helps.
You could add or remove series by playing with plot1.series array.
Here is a good jsfiddle : jsfiddle.net/fracu/HrZcj
The idea is to create an array with data
myNewSerie = Array();
x = (new Date()).getTime();
y = Math.floor(Math.random() * 100);
myNewSerie.push([x, y]);
Then add it to the graph using the next available slot
plot1.series[plot1.series.length] = myNewSerie
And finally redraw using plot1.replot();
Check out the updateSeries function in the end of the fiddle
Not tested, but should work
I had the same problem recently. "replot" works but is veeerrryyy slow. I used "jQPlot.drawSeries" which is blazingly fast. Just give your new series data to jQPlot as usual and call jQPlot.drawSeries({}, <nr of your series from 0...xxx)
My realtime chart with 800 values runs with >> 60 FPS on my PC and also very fast on my mobiles.
I need to plot thousands of points, perhaps close to 50,000 with the dojo charting library. It works, but it's definitely very slow and lags the browser. Is there any way I can get better performance?
EDIT:
I solved by applying a render filter to the data. Essentially, I have a new item parameter called "render" which is set to false by my json source if the point is expected to overlap others. My DataSeries then queries for all points where render:true. This way all of the data is there still for non-visual sources that want all of the points, while my charts now run smoothly.
Psuedocode:
def is_overlapped(x, y, x_round, y_round)
rounded_x = round(x, x_round)
rounded_y = round(y, y_round)
hash = hash_xy(rounded_x, rounded_y)
if(#overlap_filter[hash].nil?)
#overlap_filter[hash] = true
return false
end
return true
end
x_round and y_round can be determined by the x and y ranges, say for example range / 100
I know this isn't probably exactly the answer you're looking for, but have you considered simply reducing the number of points you are plotting? I don't know the specific function of the graph(s), but I'd imagine most graphs with that many points are unnecessary; and no observer is going to be able to take that level of detail in.
Your solution could lie with graphing techniques rather than JavaScript. E.g. you could most likely vastly reduce the number of points and use a line graph instead of a scatter plot while still communicating similar levels of information to your intended target.