We Keep Coding

How should I bind data to appended elements to a transformed D3.js group?

I am approaching the problem of appending a complex (two or more glyphs) symbol to some data. With D3.js, it seems that the right way to do so is appending the glyphs (in this example, circles) to groups (g) joined to data:
datum <=> g <=> (circle, circle)
Both the groups and the appended glyphs have properties depending on data, so that for example g is translated by .start and the position of the second circle is given by .end for each datum.
In order to achieve this, I wrote the following code (see the notebook), which however does not work as expected
function updatea (){
a[0].start += 10*Math.sin(t);
a[0].end += 10*Math.cos(t);
console.log(a[0].end - a[0].start);
t += 0.1;
var miao = svg.selectAll('g.aaa').data(a).join('g')
.classed('aaa',true)
.attr('transform',(d, i)=>('translate('+d.start+','+(i+1)*50+')'));
miao.append('circle').attr('r', 10).attr('fill','red');
miao.append('circle').attr('r', 10).attr('cx', d=>d.end).attr('fill','red');
}
The expected result would be as follows: two circles oscillate around their initial position, with a phase of period/4 between them. Instead, the second circle (to which I assigned an attribute cx, in order to give the position relative to the first one) is not refreshed, but instead all of its positions are drawn one after the other, oscillating with the translation in the attribute "transform".
I think that the problem is appending circles every time I update data; but how should I then append them? I tried something like this, following https://bost.ocks.org/mike/nest/:
var groups = svg.selectAll('g').data(a).enter().append('g').attr('transform',(d, i)=>('translate('+d.start+','+(i+1)*50+')'));
var circle_start = groups.selectAll('circle').data((d)=>{return d.start;}).enter().append('circle').attr('cx', d=>d).attr('cy', d=>100).attr('r', 10);
var circle_end = groups.selectAll('circle').data((d)=>{return d.end;}).enter().append('circle').attr('cx', d=>d).attr('cy', d=>100).attr('r', 10);
but it gives no output. Doing a bit of debug, for example assigning another dataset to one of the two circles, apparently the problem lies in .data(d)=>{return d.end;}).

Problem
On the pragmatic side, your update function doesn't work as expected because each update you append two new circles to each g entered or updated with selectAll().join():
function updatea (){
// join g's
var miao = svg.selectAll('g.aaa').data(a).join('g')
.classed('aaa',true)
.attr('transform',(d, i)=>('translate('+d.start+','+(i+1)*50+')'));
// append two circles to each g entered or updated:
miao.append('circle').attr('r', 10).attr('fill','red');
miao.append('circle').attr('r', 10).attr('cx', d=>d.end).attr('fill','red');
}
If you inspect the page you'll see two new circles appended each update. You're never updating the circles that are already on the page, just the translate on the g.
On the more theoretical side, you are unclear if your approach is most appropriate for binding data to complex symbols.
Solution
In proposing a solution let's consider the theoretical side first.
D3 was designed with data binding in mind. Normally in D3 you want to bind one datum per element (or one datum per element with a single child in each level of nesting). Where data is grouped and each datum is represented with multiple children, we would often see a second, nested, selectAll().data().join() statement.
However, if your visualization uses a symbol that is always comprised of the same set of child elements, then we don't need to do a nested join. In fact we do not need to in order to stay true to the data binding philosophy in D3: we'll bind one datum to one symbol (symbol in the data visualization sense).
This is the approach I'll propose here.
Rationale
This approach has advantages depending on situation, for example, there may be cases where the symbol's elements share parts of the datum (as in your case where d.start and d.end are both used to set the position of one of the sub-components) - splitting the datum into a new data array would be unnecessarily cumbersome. Changes in the symbol's representation/behavior/etc may require different parts of the datum as well, in which case it doesn't make sense to split the parent datum up.
Also, another reason why the proposed approach is attractive is that if you break the datum into smaller sub-components by using a nested selection:
svg.selectAll("g").data(data).enter().append("g")
.selectAll("circle").data(function(d) { return [d.start,d.end]; })
...
Or by flattening your array:
svg.selectAll("g").data([data[0].start,data[0].end,data[1].start,...])
...
It isn't as clear what child datum corresponds to what property when entering/updating your elements or what even what child datum corresponds to what parent datum. But also, say you dislike the symbol and now want a circle and rect, or two circles and a rect, then you need to substantially adjust the above approaches (perhaps by creating a fancy enter function that returns different types of shapes depending on index or on some identifier that tells you what symbol sub-component the datum corresponds to).
I believe attempting to create one unique datum per element is not ideal in this case, which is why I'm advocating for one datum per symbol.
Implementation
So, let's do one datum per symbol (where the symbols have child elements). This should be fairly easy to implement, I'll go over a simple method to do this here:
We can create the symbol in the join's enter function, and update it in the update function:
function updatea (){
a[0].start += 10*Math.sin(t);
a[0].end += 10*Math.cos(t);
t += 0.1;
var miao = svg.selectAll('g').data(a).join(
enter => {
// create the parent element to hold the symbol
let entered = enter.append('g')
.attr('transform', (d,i) =>'translate('+d.start+','+(i+1)*50+')')
.attr('class','symbol');
// append the sub-components of the symbol
entered.append('circle').attr('r', 10).attr('fill','red');
entered.append('circle').attr('class','end').attr('r', 15).attr('fill','yellow').attr('cx',d=>d.end);
},
update => {
// update overall positioning
update.attr('transform', (d,i) =>'translate('+d.start+','+(i+1)*50+')')
// update the sub-components
update.select('.end').attr('cx',d=>d.end);
return update
},
exit => exit.remove()
)
First, it's important to note, even though you've likely noticed, the parent datum is passed to child elements when using selection.append().
In the enter function passed to selection.join() We enter the g, style it as appropriate. Then we add the symbol sub-components and set their initial properties.
In the update function we update the overall position of the symbol and then the sub components.
Nothing occurs outside the join method in this case.
I cannot fork your observable without creating another account somewhere, so I'll just make a snippet of your example:
const svg = d3.select("body").append("svg")
.attr("width", 600)
.attr("height", 150);
var a = [{'start': 100, 'end': 200},{'start':100, 'end':200}];
var t = 0;
function updatea (){
a[0].start += 5*Math.sin(t);
a[0].end += 5*Math.cos(t);
a[1].start += 5*Math.cos(t);
a[1].end += 5*Math.sin(t);
t += 0.1;
var miao = svg.selectAll('g').data(a).join(
enter => {
// create the parent element to hold the symbol
let entered = enter.append('g')
.attr('transform', (d,i) =>'translate('+d.start+','+(i+1)*50+')')
.attr('class','symbol');
// append the sub-components of the symbol
entered.append('circle').attr('r', 10).attr('fill','red');
entered.append('circle').attr('class','end').attr('r', 15).attr('fill','yellow').attr('cx',d=>d.end);
},
update => {
// update overall positioning
update.attr('transform', (d,i) =>'translate('+d.start+','+(i+1)*50+')')
// update the sub-components
update.select('.end').attr('cx',d=>d.end);
return update
},
exit => exit.remove()
)
}
updatea();
setInterval(updatea, 100)
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/6.0.0/d3.min.js"></script>

The idea is that since we have two circles, have two data with them. (Ignore what's in them for now:)
var a = [{...},
{...}];
Let's create a group:
var group = svg.append("g")
Then, d3 will "join" the data to DOM nodes. For each data, d3 creates a DOM node. In this case, since we're join()ing to circles, d3 will create a circle for each data. See this page for more details.
group
.selectAll('circle')
.data(a)
.join('circle')
.attr('r', 10)
.attr('fill','red')
.attr('transform', (d, i) => ('translate('+d.start+','+(i+1)*50+')'));
As for the actual logic, there's a couple things I changed.
Each circle now stores its own t and nothing else:
var a = [{t: 0},
{t: Math.PI/2}];
Then, the start and end attributes are set in order to have a representation independent of the current object's state. This allows us to have circles which have different t phases:
a.forEach((d, i) => {
d.start = 200 + 100*Math.sin(d.t);
d.end = 200 + 100*Math.cos(d.t);
d.t += 0.1;
})
Breaking it down:
(initial (range)
position)
200 + 100*Math.cos(d.t);
So it starts at position 200 and can either go to +100 or -100: the effective range is [100, 300].
You notice we do a lot of number crunching here. Basically, we're converting one domain of numbers (-1, 1) to a range (100, 300). This is a common use case for a scale, which can convert any domain to any range.
Observable notebook

How to use scatter chart with complex data?

I trying to implement scatter chart by example.
In example we can see creating dimension:
runDimension = ndx.dimension(function(d) {return [+d.Expt, +d.Run]; });
Example's data:
Expt Run Speed
1 1 850
1 2 740
1 3 900
I want to use same chart, but I have my data in next format:
[
{
"Timestamp":"2016-12-15T17:29:53Z",
"idgame":"euro",
"users":{
"Jo": {
"energy":200,
"jump_height":0.5
},
"Bob": {
"energy":220,
"jump_height":0.35
}
}
},
{
"Timestamp":"2016-12-15T17:29:55Z",
"idgame":"euro",
"users":{
"Jo": {
"energy":120,
"jump_height":0.15
},
"Bob": {
"energy":240,
"jump_height":0.75
}
}
}
]
I need to build next chart, where x-axis is timestamp and y-axis is jump_height:
My data is allready in crossfilter, so I can't change it.
How can I create good dimension with current format?

I'm still not convinced this is worth the effort, versus biting the bullet and flattening your data and fixing up the other charts. If your data isn't flat you will be fighting crossfilter and dc.js every step of the way.
That said, as usual, it's possible!
We can't use the series chart, because that requires all the data to be present in one group. But since you want to produce multiple symbols from each row of data, an ordinary crossfilter group can't produce the data you need.
Maybe we could use a fake group, but that would be complicated. Instead, let's produce a dimension and group for each user, and then sew them together using a composite chart.
First, we need to parse those timestamps:
data.forEach(function(d) {
d.Timestamp = new Date(d.Timestamp);
});
Next, we'll retrieve a list of all users, by pulling the user keys from each event (timestamp), concatenating them, and then using a d3.set to unique them:
// retrieve all users from all events
var users = data.reduce(function(p, event) {
return p.concat(Object.keys(event.users));
}, []);
users = d3.set(users).values();
In the rest of the code, we'll assume there are the same users for each event. It's possible for them to be different, but it adds extra complexity, and this answer is complicated enough. Just ping me me if you need that feature!
We'll create the chart, crossfilter, and a scale which will assign symbols to users:
var chart = dc.compositeChart("#test");
var ndx = crossfilter(data);
var symbolScale = d3.scale.ordinal().range(d3.svg.symbolTypes);
Now we can create the composite chart. (We'll add the scatter subcharts in the next step.)
chart
.width(768)
.height(480)
.x(d3.time.scale())
.xUnits(d3.time.seconds)
.elasticX(true)
.elasticY(true)
.brushOn(false)
.clipPadding(10)
.shareTitle(false) // allow default scatter title to work
.shareColors(true) // but use different colors for subcharts
.legend(dc.legend().x(350).y(350).itemHeight(13)
.gap(5).horizontal(1).legendWidth(140).itemWidth(70));
We set up the X axis with a time scale, with a resolution of seconds. Both axes have elastic. We need to share colors so that each subchart will be assigned its own color. (The legend is perhaps overspecified - I copied this from another example.)
Finally we get to the meat of it. For each user, we'll create a subchart, and we'll tell the composite chart to compose all of them:
chart.compose(users.map(function(user) {
var userDimension = ndx.dimension(function(d) {
return [d.Timestamp, d.users[user].jump_height];
})
var jumpGroup = userDimension.group();
console.log(user, jumpGroup.all());
var scatter = dc.scatterPlot(chart)
.symbol(symbolScale(user))
.dimension(userDimension)
.group(jumpGroup)
.colorAccessor(function() { return user; })
.symbolSize(8)
.highlightedSize(10);
return scatter;
}))
We're creating a new dimension for each chart. This is because the dc.js scatter plot expects the keys to include both X and Y coordinates, and we can only access the Y coordinate (jump_height) once we know the user. Once we get past that, the group is simple.
The chart will assign the symbol and color based on the user key. These both work the same; an ordinal scale will assign a new value from the range for each new value it encounters in the domain. The only difference is that we're using the default color scale, whereas we had to specify our own symbol scale.
Here's a fiddle: https://jsfiddle.net/gordonwoodhull/3m4mv3xf/19/

Plot within a tooltip in d3.js

I would like to create a d3-based plot which graphs a plot within a tooltip. Unfortunately, I haven't found any examples on the web. Here is a sample JSON file.
[{"x":[0.4],
"y":[0.2],
"scatter.x":[0.54,0.9297,0.6024,-1.9224,2.2819],
"scatter.y":[0.4139,1.1298,-0.1119,2.3624,-1.1947]},
{"x":[0.1],
"y":[0.9],
"scatter.x":[-0.8566,-0.5806,-0.9326,0.8329,-0.5792],
"scatter.y":[-0.5462,-0.7054,1.0264,-3.4874,-1.0431]}]
The idea is to have a scatter plot for (x,y) coordinates first. However, when one mouses over a point, a different scatter plot within a tooltip appears based on [scatter.x, scatter.y] coordinates for that respective point.
I can do the scatter plots separately but have been struggling to put them together. Could anyone shed some light on this and/or provide a minimal example?

This was too long for a comment but I'm not certain if it's the answer you were looking for. One of the issues you might find is that your nested data is formatted differently-- one uses JSON objects with x and y, while the other uses two arrays of points.
My solution to this would be to create an extensible function:
function makeScatterPlot(elem, width, height, data, fill)
elem, width, height, and data are the core parameters: which element to attach the chart to, the size of the chart, and the data for the chart (in the JSON object format).
This function would generate all necessary items for the chart and add the chart to the provided element.
Then you want to bind to mouseover of your main chart, and in that function you'll have to do a bit of data modification to re-organize the two arrays into the JSON object structure.
function mainMouseover(d){
var newData = [];
for (var i = 0; i < d["scatter.x"].length; i++){
var t = {x: [0], y: [0]};
t.x[0] = d["scatter.x"][i];
t.y[0] = d["scatter.y"][i];
newData.push(t);
}
var newG = mainG.append("g").attr("transform", "translate(200,200)");
makeScatterPlot(newG, 100,100, newData, "red");
}
Of course, you would modify the translate to match wherever you want your tooltip to be.
Putting this all together you get the following (very crude) fiddle. Hover over either of the black dots to see the sub-chart. Obviously this needs quite a bit of work to be a solid example (i.e. remove the sub-chart on mouseout), but hopefully it will set you in the right direction.
If the tooltip chart is significantly different styling-wise compared to your main chart it may not be the best idea to use an extensible function, and you could just create another custom function instead.

Adding data points to existing graph with Dygraphs

I have an existing graph created with Dygraphs based on a JavaScript array. I also have a process that generates new values, say once per second, as JavaScript array.
Is it possible to add the new values to the existing graph? I'd like to avoid redrawing the whole graph each time a new value is created.
var data = [[1,99,42],[2,98,52]]
var graph = new Dygraph(element, data, {});
// now graph is created and visible
// how to add newPoint to existing graph?
var newPoint = [3,100,20];

You can use option file for all kinds of data, not just for CSV files.
data.push(newPoint);
graph.updateOptions({file: data});

OpenLayers get position of newly created point

I'm using
lineLayer.events.on({
"vertexmodified": update
});
to get the line info once it's been modified. If a new point is added in between 2 existing points, is there a way to figure out the position of the new point in relation to everything else?
I'm updating a table based on the values of the line and need to know where to insert the new row.

You can figure out the position of the vertex by looking at its parent collection:
var vertex = event.vertex;
var idx = vertex.parent.components.indexOf(vertex);

You can’t (with actual OpenLayers code). I’d suggest you to get the whole geometry (you can access event.feature.geometry from your listener argument), and update the whole geometry at once rather than updating just a point.