We Keep Coding

D3 geo: getting projection.clipAngle to work on all specified elements

I'm a newcomer to D3 and I'm trying to make a world globe with some points ("pins") on it. Demo here: http://bl.ocks.org/nltesown/66eee134d6fd3babb716
Quite commonly, the projection is defined as:
var proj = d3.geo.orthographic()
.center([0, 0])
.rotate([50, -20, 0])
.scale(250)
.clipAngle(90)
.translate([(width / 2), (height / 2)]);
the clipAngle works well for the svg paths, but not the pins (which are svg circles). As you can see on the demo, the pin that sits between Iceland and Greenland should be hidden (it's Taiwan).
So I suppose the problem comes from these lines, but I can't understand why:
.attr("transform", function(d) {
return "translate(" + proj([ d.lng, d.lat ]) + ")";
});

It is not sufficient to just set the clipping radius via clipAngle() to get the desired behavior. The projection alone will not do the clipping, but just calculate the projected coordinates without taking into account any clipping. That is the reason, why Taiwan gets rendered, although you expected it to be hidden.
But, thanks to D3, salvation is near. You just need to re-think the way you are inserting your circles representing places. D3 has the mighty concept of geo path generators which will take care of the majority of the work needed. When fed a projection having a clipping angle set, the path generator will take this into account when calculating which features to actually render. In fact, you have already set up a proper path generator as your variable path. You are even correctly applying it for the globe, the land and the arcs.
The path generator will operate on GeoJSON data, so all you need to do is convert your places to valid GeoJSON features of type Point. This could be done with a little helper function similar to that used for the arcs:
function geoPlaces(places) {
return places.map(function(d) {
return {
type: "Point",
coordinates: [d.lng, d.lat]
};
});
}
With only minor changes you are then able to bind these GeoJSON data objects to make them available for the path generator which in turn takes care of the clipping:
svg.selectAll(".pin") // Places
.data(geoPlaces(places))
.enter().append("path")
.attr("class", "pin")
.attr("d", path);
Have a look at my fork of your example for a working demo.

d3.js scatter matrix with brushing - scale error

I try to reuse the example of 'scatter matrix with brushing': http://bl.ocks.org/mbostock/4063663
It seems that the code is not directly reusable with another csv. Scales seems to be somehow hard coded or so: i change the csv by adding 10 to 75% of the first column values, and the xscale is not directly updated.
To visualize the problem, see the fork of the mbostock gist: http://bl.ocks.org/fdeheeger/7249196
I cannot figure out where/how the scale is computed or updated in the javascript code.
Any advice from a d3 expert?

The scales are computed dynamically -- the problem is that the numbers in the CSV are parsed and processed as strings and not numbers. This is also the case in the original block, but there it doesn't matter because the ordering of the strings is the same as the ordering of the numbers.
All you need to do to fix this is parse the strings to numbers:
domainByTrait[trait] = d3.extent(data, function(d) { return +d[trait]; });
The plus makes all the difference here. Complete example here.

D3.js patterns for managing multiple elements bound to same data

I'm learning D3 and my toy application is a visualizer for a 2-dimensional gravity simulation using Symplectic Velocity Verlet Integration.
I have had quite some success animating and drawing a path using bound structured data, but it seems to me that D3 isn't really designed for attaching data to more than one element. It sort of assumes that for any given data that there is a clear and simple SVG element that should own it, and this is evident in the direct storage of data within the __data__ property inside the DOM.
It's not clear, though, the proper way to represent a datum with more than one SVG element. For instance, I'd really prefer to draw a path and a circle for each planet, the path traces its past position (and can have a bunch of clever line-length and color interpolation applied), and the circle plots its current position.
I can even come up with a few more elements I might want to draw: A vector-arrow for velocity... A vector-arrow for acceleration...
In my case, my master data structure is constructed like this, and is dynamically maintained in this structure:
var data = [];
function makeParticle(x, y, vx, vy) {
// state vector plus goodies
return [
x, y,
vx, vy,
0, 0,
[] // path
];
}
data.push(makeParticle(400, 100, -0.5, 1));
data.push(makeParticle(300, -100, 0.5, 2)); // and so on
Each element of data is a planet, which contains its current state vector (position, velocity, and cached acceleration (needed for the integrator)) as well as its path history which is an array of positions which is kept truncated to some reasonably large length.
Currently I am updating the path history like this:
var paths = d3.select('svg').selectAll("path").data(data);
paths.enter().append('path'); // put cool transitions here
paths.exit().remove();
paths.attr("stroke", "red")
.attr("d", function(d){
return lineDrawer(d[6]);
})
This works fine, each path tracks its own copy of its own planet's path.
It's not immediately clear to me how to extend this elegantly to include my circle at the head of each path. I certainly do not want to duplicate the entire datum into the circle's DOM element (as the path data is simply not necessary for it).
Edit in light of my self-answer:
I am hoping that someone can help to elucidate a way to use "groups" to make data-driven the set of things to draw for each datum. e.g. [position, velocity, force, path] as data where they are visualized using, respectively, a circle, an arrow closed path, an arrow closed path, and an open path. It is also possible that this is completely overthinking it because these properties are sort of fixed.

I guess in the process of thinking the problem through, it's become clear to me that all I have to do is filter out the data, so that I only attach the position state data to selectAll('circle.planet') rather than the full datum value which includes the massive path history array. This way it gets exactly the data it is responsible for displaying.
It seems like I have to do some more reading about subselections (I'm mostly puzzled by why (or if) the subselections are limited to two dimensions of hierarchy), but this seems completely reasonable. It seems logical, if I want to draw 4 items for each datum, I just have to somehow "assign" the correct subsets of my datum's structure to each SVG visualizer element.

How do I get resulted pathString after transformation is applied?

Consider I have the following HTML5 path:
var myPath = paper.path([
'M', 0, 0
'L', 100, 100,
'L', 150, 50,
'Z']
]);
myPath.transform(['s', 0.5, 0.5, 0, 0]);
After tranformation (scaling) my path resizes accordingly in half, but inspecting the element is the same path string but with transformation matrix applied. Is there any way to retrieve the pathString resulted (lets say M,0,0,L,50,50,L,75,24,z).

I think you need transformPath method: http://raphaeljs.com/reference.html#Raphael.transformPath

The only solution would be using Raphael 1.x which used to modify paths instead of applying transformations. Otherwise you'd need to write your own routines to convert apply matrix transformations to paths (really difficult).

Please check my answer HERE and the testbed HERE.
There is a function flatten_transformations() which can bake (or apply) transforms to paths, so that transform attribute can be removed. It can handle all path segments (also arcs).
OLD ANSWER (not so complete implementation):
Of course there is a way (example in JSBIN) to get a resulted path data after transformations applied. And even very easy way.
Let's suppose we have a SVG path pathDOM and it's root element svgDOM. We can get a transformation matrix of path's coordinate space to root element's coordinate space using native getTransformToElement() -function. It is used this way:
var matrix = pathDOM.getTransformToElement(svgDOM);
When we apply this matrix to all points in path, we get a new path data, where all coordinates are relative to root element. It can be done this way:
var pt = svgDOM.createSVGPoint();
pt.x = some_x_coordinate_of_path_data;
pt.y = some_y_coordinate_of_path_data;
var new_point = pt.matrixTransform(matrix); // <- matrix object, which we created earlier
var new_x = new_point.x;
var new_y = new_point.y;
And that's it! After conversion the transform attribute can be emptied.
Of course all coordinates in path have to be converted to absolute and eg. Arc segments have to be converted to Line or Cubic Segments, which both can be achieved with Raphaël's path2curve() function.
I made a full functional example of using this "Flattening transformations" functionality in JSBIN. There is a ready made function flatten_transformations(), which is the only one needed (the rest is needed for UI purposes). The example has got a path that is nested inside two g elements. Path has own transformations applied, as well as both g elements. Purpose is to test that also nested transformations are flattened.
The code works in newest main browsers and even in IE9. My code that modifies transformations is rather interesting mix of jQuery, Raphael and native code that it may be cause of some problems in IE9 when clicking buttons, but fortunately those essential native functions getTransformToElement(), createSVGPoint() and matrixTransform() work as expected also in IE. I wanted to test simultaneously how those different coding bases plays together. Because it's the fact that Raphael itself is not perfect enough to handle all possible coding needs (lack of styles and groups and lack of possibility to append svg elements as textual xml-data are just ones to note).

Charting thousands of points with dojo

I need to plot thousands of points, perhaps close to 50,000 with the dojo charting library. It works, but it's definitely very slow and lags the browser. Is there any way I can get better performance?
EDIT:
I solved by applying a render filter to the data. Essentially, I have a new item parameter called "render" which is set to false by my json source if the point is expected to overlap others. My DataSeries then queries for all points where render:true. This way all of the data is there still for non-visual sources that want all of the points, while my charts now run smoothly.
Psuedocode:
def is_overlapped(x, y, x_round, y_round)
rounded_x = round(x, x_round)
rounded_y = round(y, y_round)
hash = hash_xy(rounded_x, rounded_y)
if(#overlap_filter[hash].nil?)
#overlap_filter[hash] = true
return false
end
return true
end
x_round and y_round can be determined by the x and y ranges, say for example range / 100

I know this isn't probably exactly the answer you're looking for, but have you considered simply reducing the number of points you are plotting? I don't know the specific function of the graph(s), but I'd imagine most graphs with that many points are unnecessary; and no observer is going to be able to take that level of detail in.
Your solution could lie with graphing techniques rather than JavaScript. E.g. you could most likely vastly reduce the number of points and use a line graph instead of a scatter plot while still communicating similar levels of information to your intended target.