We Keep Coding

Generating a continuous interpolated surface from point data with Mapbox GL JS

I'm relatively new to Mapbox and its GL JS Library, but so far have been really impressed at its capabilities.
I'm currently working with a dataset of approximately 100,000 points and am trying to devise a way of quickly generating/visualising a continuous interpolated surface within the convex hull of the dataset I have (essentially trying to generate something that's as fast/responsive as the heatmap function, but looking to interpolate from the point data values rather than spatial density).
The documentation for the heatmap functionality discusses this exact scenario:
Among maps you'll find on the web, there are two common categories of
heatmaps: those that encourage the user to explore dense point data,
and those that interpolate discrete values over a continuous surface,
creating a smooth gradient between those points. The latter is less
common and most often used in scientific publications or when a
phenomenon is distributed over an area in a predictable way. For
example, your town may only have a few weather stations, but your
favorite weather app displays a smooth gradient of temperatures across
the entire area of your town. For your local weather service, it is
reasonable to assume that, if two adjacent stations report different
temperatures, the temperature between them will transition gradually
from one to the next.
But then proceeds to explain this is less common and there's no documentation/example provided for this type of application.
At this stage I've tried converting the points to voronoi cell polygons and colour coding by data value (a nearest neighbour approach to visualising), but the render seems to struggle with 100,000 points at lower zoom levels (0 through 8). Does anyone know if it's possible to create a fast-rendering surface interpolation from point values? Any examples would be fantastic.

The Mapbox Delaunator library is a very fast JavaScript library for Delaunay triangulation of 2D points. d3-delaunay and d3-geo-voronoi are both based off of this library. You could use these libraries and then display the results on your map via a custom style layer.

Looks like what you are trying to do is some sort of Spatial Interpolation.
Here is a summary of popular methods typically used for this purpose.
https://michaelminn.net/tutorials/r-spatial-interpolation/

Mapbox GL JS line animation

I am working with ~150,000 polylines that span North America. Across all the lines I'd like to draw/animate from the start to the end of each line. I started down the road of adding vertices to an array and updating a Mapbox source like this example. This was not feasible given the size of the entire dataset. In geojson form it's about 800mb and manipulating this in the browser seemed unreasonable.
As an alternate approach, I've broken each polyline into equally sized segments of 10km in length. I added an attribute on each segment which represents its percent distance from the start. I created an interval below that triggers a setFilter event, filtering segments based in the percent from start.
currentLinPos=0;
lineInterval=setInterval(function(){
currentLinPos+=.01;
if(currentLinPos>0.9999){
currentLinPos=0
}
map.setFilter('lines',['<','linePrc',currentLinPos])
}, 250);
Even when working with a small subset of the data, performance choppy and lags. I expected a better result from Mapbox GL but perhaps I need to rethink my data structure or approach entirely. Or would using something like DECK.GL be far superior here?

Graphical visualisation of a big sparse matrix with 3D.js or Cairo

I designed an ad hoc library to write SVG images. I use it for the graphical visualisation of large sparse matrices : homology matrices of the comparison of two genomes.
In my case, a usual homology matrix is 20 000 x 20 000 with around 30 000 scattered dots (representing homologies), for instance this homology matrix.
My ad hoc library is not mature enough for new developpements and I am interested by the 3D.js library. I also found the graphical library Cairo and its binding pyCairo interesting.
My needs are :
the possibility to output SVG, PNG or PDF images of the matrix.
when the mouse get over a dot of the homology matrix I would like to show specific informations.
a plus would be the ability to zoom in an area of the matrix, and show new details above a zoom threshold.
I know that 3D.js has dynamic features (interesting for navigating/zooming through the sparse matrix) but I wonder if it will display fast enough my data. It might be a better idea to use Cairo in an application and recompute the image for different zoom levels.
In brief I feel that I have 2 solutions :
write a HTML file, with a 3D.js script that adds SVG elements of the homology matrix and interactivity. 3D.js would be great for sharing matrices from a server.
write a dedicated app (e.g. with GTK) and draw the matrix inside the app with Cairo. Interactions would then be added by asking Cairo to re-draw each time the user interacts with the app.
Do I miss something ? Which solution would you use ?

How to get the same node positions in d3's force layout graph

I would like to create a force directed graph, but i need it to stay the same every time it is generated (with the same data).
Is there any way to do this using d3.js?
UPDATE:
I found working solution, which is based on using seeded random number generator
// set the random seed
Math.seedrandom('mySeed');

You could by modifying D3's force layout, or by creating your own layout based on it. There are at least 3 places where randomness (Math.Random) is used in the positioning of nodes (there may be more, given the force layout references other code). You would have to eliminate all randomness in order to get the graph to display in the same way each time:
https://github.com/mbostock/d3/blob/master/src/layout/force.js
However, this would hamper how the layout works – it's using randomness to sort itself out into a legible diagram quickly. If your number of nodes is small, then it probably wouldn't be an issue, but a large number of nodes could just end up a tangle.

D3.js scattergraph with large (>500,000) points? Clustering?

I'm looking at plotting a scatterplot with a large number of points (500,000 and upwards).
Currently, we're doing this in Python with Matplotlib. It plots the points, and it provides controls to pan and zoom. I don't believe it provides any clustering or points, it just plots them all - doesn't make much sense at the zoomed out view, I suppose, but you can zoom in and they're all there.
I was looking at doing the chart in JavaScript, to make it a bit easier to distribute. I was looking at D3.js, to see if something similar is feasible there. I did find this example of a basic scatterplot:
http://bl.ocks.org/mbostock/3887118
Firstly, would you be able to plot that number of points? (500,000 and upwards) I was under the impression you couldn't due to the overhead of all the DOM objects? Are there ways around this?
Secondly, is there any kind of clustering available, either a library or even just an example of this being done in D3.js?
Thirdly, if anybody knows any good examples of pan/zoom functionality and clustering, or even just a packaged JS library that handles it, that would be awesome.
Fourth, it would be also nice to have click handlers for each point - and to display some text either in a overlay, or even just in a separate window. Any thoughts on this?

Can you draw half a million points with D3? Sure, but not with SVG. You'll have to use canvas (here's a simple example with 10,000 points that includes brush-based selection: http://bl.ocks.org/emeeks/306e64e0d687a4374bcd) and that means that you no longer have individual elements to assign click handlers to. You will not be able to render half a million points with SVG, because all those DOM elements will choke your interface, as you mentioned.
D3 does include quadtree support that can be leveraged for clustering. It's in use in the above example to speed up search but you could use it to nest elements in proximity at certain scales.
Ultimately, your choices are:
1) Some other library/custom implementation that renders in canvas and polls the mouse position to give you the data element rendered at that point.
2) A sophisticated custom D3 approach that nests elements in proximity and only renders SVG elements appropriate at the zoom level and canvas position (pan) you're at.

Yes, D3.js can be made to work with million scale data with two things:
pre-rendering on the server side. For more see here: https://mango-is.com/blog/engineering/pre-render-d3-js-charts-at-server-side/
By aggregating (or clustering) part of the data so that user can interact and expand the graph if need be. For this use collapsible nodes if you can (http://bl.ocks.org/mbostock/1062288).
Also avoid using force layout. It takes time to settle and converge to a stable positioning.
For clustering libraries, I would pick one up off the shelf. I would choose the scikits library from python, there are many in JavaScript but they are not very robust as they mostly cover k-means or hierarchical clustering. I would precalculate the coordinates using scikits by clustering and then render it using D3.
D3 handles Pan and zoom. Again click handlers and text display are available in D3. (http://bl.ocks.org/robschmuecker/7880033)