Performant data-tables with JavaScript - javascript

Introduction
I am making a request to a backend and getting a list of objects in JSON. Then I change it into an HTML table (it is a Vuetify data-table) where every object is a row.
Each row contains an array of exactly 72 ones and zeros ([1, 1, 1, 1, 0, ...]). They indicate activity back in time (72 hours).
In each row I have a for loop (kind of. It is Vue.js v-for directive) that goes through that array and loads an image 1.svg or 0.svg accordingly, to make a chart.
With 40 rows only, the table becomes to lag a little. Now, the table is quite wide and so it goes off screen (overflow: scroll or whatever).
TL;DR
Is it possible in JavaScript to somehow fluently hide DOM elements (in this case table cells (and rows)) (hide means remove from DOM, so that the browser doesn't have to render all of them) when they are off the screen?
Is there a literature you could recommend? Any tutorials? What to look for?
I remember seeing a Google talk on a list of thousands of elements scrolling smoothly on mobile, but can't seem to find it.
Demo
https://codepen.io/DCzajkowski/pen/yPbPqy

Hi Czajkowski Dariusz,
Yes it is certainly possible to fluidly handle large amounts of DOM nodes. The trick is to go one step past "hiding" the DOM nodes and to instead not render them at all.
A simplified view of this process could be broken into these steps:
Measure the size of the area in which you are rendering these nodes. -> RenderingHeight
Measure the size of one visible DOM node -> NodeHeight
Render only as many DOM nodes as will fit in the total plus a couple of extra which are used as a buffer -> ( RenderingHeight / NodeHeight) + 2 = NumberOfNodes
Create a subset of your data that should be populating these DOM nodes with values
When an action on the page occurs (scroll, click, etc..) update the subset of your data that should be displayed according to the action. Re-render the visible nodes with this new set of data.
Example: If your render a list of height 1000px and each list item will have a height of 100px and you have 1000 data points.
RenderingHeight = 1000px
NodeHeight = 100px
NumberOfNodes = 12
Render ten list item nodes using DataPoints[0] - DataPoints[11]. When a scroll event has moved the container down at least 100px you should update your selected subset of data to be DataPoints1 - DataPoints[12]. Then rather than delete nodes or append new ones, just update the data in the existing 12 nodes to use this new subset of data points.
This explanation is definitely a simplification of what you will end up doing in practice in your own applications but I hope it conveys the basic idea.
I believe the talk that you are thinking of might be this one from Google I/O this year: https://www.youtube.com/watch?v=mmq-KVeO-uU. Start at timestamp 15:45 to catch the example.
In this talk he uses the react-virtualized library as an example. Since you are working in Vue.js this library won't directly solve your problem but reading through the code might provide insight into how you might achieve this. I have used this library a couple of times and it has worked well for me.
A quick google search for virtualized lists in js yields some other vanilla js implementations that might also be helpful.
This is not a trivial mechanism which you are aiming to implement but it is amazingly powerful for managing performance when rendering large amounts of data.
Best of luck!

Example of Vue application with Polymer element canvas-datagrid with 5000 rows dataset.
new Vue({
el: '#app',
data: {
data: []
},
async created () {
var response = await fetch('https://jsonplaceholder.typicode.com/photos')
this.data = await response.json()
}
})
html, body, #app {
margin: 0;
padding: 0;
height: 100%;
}
<div id="app">
<canvas-datagrid>
{{ data }}
</canvas-datagrid>
</div>
<script src="https://unpkg.com/vue#2.5.3/dist/vue.min.js"></script>
<script src="https://unpkg.com/canvas-datagrid#0.18.12/dist/canvas-datagrid.js"></script>

Related

Change Ag-Grid infinite scroll block size?

Is there a way to change the block size when the row model used is "infinite" and a datasource is set?
I.e. when the datasource's getRows() is called, is there a way to set startRow and/or endRow? Default behaviour is to fetch 100 rows at a time which results in blank rows for me, since I only have about ~12 new data rows coming in at a time with infinite scroll. So, when getRows() is triggered, it'll try to fetch rows 100 to 200 (as an example) when there are only 45 data points in the first place. To compensate it adds a bunch of blank rows.
I've tried the following two additional grid options without success:
cacheOverflowSize: 2
infiniteInitialRowCount: 32
Fetching 100 rows at a time instead of 12 solves the issue, but I'd reeeeeally rather not do this (due to some design constraints of the product I'm working on).
Shoutout to #kamil-kubicki for their comment. cacheBlockSize was indeed what I was looking for. It didn't solve my whole problem though, so I'll outline my complete solution below.
I had ag-grid's row height set to the default 25px max. That meant that on load, when it was populating the initial data, it was coming close to rendering my entire initial data set of 32 items. On scroll, getRows() would then look for rows outside of the total data collection's bounds, so it added blank rows.
I changed how many data results are loaded on each scroll to 50. This is relatively large but it works and is performant for now, so I think I'll keep it.
For those with a similar problem:
Use cacheBlockSize in gridOptions
Make sure that your ag-grid isn't rendering anything close to your full initial data collection (e.g. if your collection is 32 items, don't render anything above like 16 items - control this by changing the height of your rows and the size of your grid)
Change how many data items are loaded in to your collection on scroll to a larger number; something that can populate data faster than the user can scroll. For me it was 50. For you it might be more, or less.

JS grid performance comparison

I was using angular-ui-grid (http://ui-grid.info/) to display tabular data. On the whole, it was quite slow and so we decided to use ag-grid (https://www.ag-grid.com/). This was much more performant and better to deal with for regular-sized data-sets.
However, now we are working with some tabular data of the size of 100 cols x 10,000 rows (~1M cells) and the grid seems quite slow in performance.
I was wondering if anyone had used hypergrid (https://fin-hypergrid.github.io/core/2.0.2/) -- it seems to 'solve' the issue of large cols x large rows and in their demo it seems much faster (almost by an order of magnitude) on large data sets.
How does hypergrid compare to ag-grid or react-virtualized in performance on large data sizes?
I haven't tried any of those example libraries you mentioned, but perhaps I could explain why fin-hypergrid stands out the most. My opinion is primarily based on my JavaScript knowledge and how this kind of stuff works in the back.
I should probably start with react-virtualized and ag-grid:
Both use the way of populating the DOM and only displaying a portion of data to the view, dynamically removing things from the DOM that aren't visible anymore and adding the upcoming things in advance. Now the problem here lies in adding and removing things in the DOM since this tends to be executed really fast / multiple times a second. Because of this we experience some lag or jitter. You can actually check Web Console > Profiles > Record JavaScript CPU Profiles and see that this method takes time to complete. So the only thing that differs from react-virtualized and ag-grid are their algorithms of applying these changes in the smoothest possible manner.
ag-grid, from what I can see, is the one that suffers the most from this issue as you could actually see some elements that haven't finished rendering yet and experience severe lag when you scroll too fast.
react-virtualized on the other hand does a splendid job implementing its algorithm in the smoothest possible manner. This might be the best library available in the DOM manipulation category though it still suffers from the problem of manipulating the DOM too fast which creates lag, though this is only seen when large chunks of data are involved.
Here are the reasons why fin-hypergrid excels:
The best asset of fin-hypergrid is it doesn't perform DOM manipulation at all so you are already saved from the problem caused by adding and removing things too fast since it uses <canvas>
fin-hypergrid also displays only data that the user sees and dynamically removes things that aren't visible. It also adds in advance to achieve a smooth scroll feeling, so no still-rendering items are shown.
fin-hypergrid also does a really great job on their scrolling algorithm to attain the smoothest possible manner, so there is no jitter or lag.
Now that doesn't mean that hypergrid is all good, it has some disadvantages too:
Since fin-hypergrid is made with HTML5 Canvas, styling it will become a real pain as it doesn't accept CSS. You would need to style it manually.
A few things to keep in mind are form controls such as <select>, radio buttons, checkboxes, etc. would be a real pain to implement. If you are trying to implement something like this then proceed with caution.
It's primarily used for displaying data with simple column edits, not involving anything other than a textbox, and achieving the most smooth scroll feeling.
Now in conclusion, I would probably suggest using react-virtualized instead since it offers the smoothest scroll, above fin-hypergrid. If you are willing to disregard the downsides of fin-hypergrid, then fin-hypergrid is the best option.
UPDATED:
Since we discussed JS / CSS, canvas implementations of these tables. I should have mentioned the last possible contender though this one is not primarily a js table library but a framework in which Google Sheets might have been used it is called d3.js.
d3.js has the speed and power of canvas and at the same time retains HTML structure, this means that it is possible to style it with CSS!
It maximizes the usage of HTML 5 SVG
I can't say anything more better in d3.js
The only downsides of d3.js in this discussion is that:
There is no available good table libraries out there that uses d3.js. Google Sheets that is. But they do not share code.
d3.js is just very hard to learn, though there are lots of stuff out there that helps us learn this faster but not that fast.
If you wanted speed of Canvas with CSS styling capabalities then d3.js is the key the problem is learning it.
I have gone through different data grid options. Then I found this.
Hope this answer helps to anyone who is looking for performance comparison between data grids.
Few points to be noted here, even you have gone through with the article I provided.
1 - Once a grid is 'fast enough', which means the rendering lag is not
noticeable, then it doesn't matter which grid is faster than the next.
2 - A canvas based grid is not a HTML grid - you cannot customise it
with HTML. The grid will struggle to be styled / themed / customised
by the standard JavaScript / CSS developer.
Pick your poison because it is not just the performance when it comes to the consumer level.
Have you considered using something designed for large data sets?
Clusterize.js
I believe the way it works is it only loads the elements data for the ones you are looking at. Therefore the browser doesn't lag because it has the elements need to display the viewport.
The demo page loads 3 examples with 500,000 elements each (1,500,000 total elements).
Update - With Example Snippet
Because I do not have 100,000 x 200 elements of data to load I build 100 x 200 using JavaScript.
Then I copy that array and insert it into the data array 1000 times.
This way I can reach your total data set size without overloading the JavaScript engine.
Since it is hard to tell that it is really doing 100,000 rows I called the getRowsAmount() function which is displayed at the top of the output.
You may need to play with the block & cluster sizes based on your viewport but this should show you that this library is totally possible of supporting your needs.
$(function() {
var dataBlock = []
var data = [];
const rows = 100000
const cols = 200;
const blockSize = 100;
const blocks = rows / blockSize;
for (let i = 0; i < cols; i++) {
$("thead tr").append(`<th>C${i}</td>`);
}
for (let j = 0; j < blockSize ; j++) {
var tr = $("<tr />");
for (var i = 0; i < cols; i++) {
tr.append(`<td>R${j}-C${i}</td>`);
}
dataBlock.push($("<div />").append(tr).html());
}
for (let i = 0; i < blocks; i++) {
$.merge(data, dataBlock);
}
var clusterize = new Clusterize({
rows: data,
scrollId: 'scrollArea',
contentId: 'contentArea',
rows_in_block: 10,
blocks_in_cluster: 2,
});
$("#totalRows").text(clusterize.getRowsAmount());
});
table td {
white-space: nowrap;
padding: 0 5px;
}
<html>
<head>
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" />
<link href="https://clusterize.js.org/css/clusterize.css" rel="stylesheet" />
<script src="https://clusterize.js.org/js/clusterize.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
</head>
<body>
Total Rows: <span id="totalRows"></span>
<div class="clusterize">
<div id="scrollArea" class="clusterize-scroll">
<table class="table">
<thead>
<tr></tr>
</thead>
<tbody id="contentArea" class="clusterize-content">
<tr class="clusterize-no-data">
<td>Loading data…</td>
</tr>
</tbody>
</table>
</div>
</div>
</body>
</html>
The library supports appending data so with your large data sets your may want to load some of your data through AJAX.
I used free version of handsontable for big datasets.
See example with 10000*100 cells - http://jsfiddle.net/handsoncode/Lp4qn55v/
For example, for angular 1.5:
<hot-table class="hot handsontable htRowHeaders htColumnHeaders" settings="settings"
row-headers="rowHeaders" datarows="dba">
<hot-column ng-repeat="column in Value" data="{{column.COL1}}" >
</hot-column>
</hot-table>
See documentation here

Most performant way to transform/move large number of DOM elements

I built up a tree table structure for AngularJS (with Angular Material) some time ago.
My target was to make it work on large screens only (1280 and higher) but right now I want to update it and make it work on smaller devices (mostly tablets) without limiting data. Because of performance, I want to keep HTML as simple as possible (tree table can have 1000+ rows so creating more complicated HTML for the single row will elongate the time needed to append and render table row (rows are dynamic so it's not only about initial rendering)).
I came up with idea that I will keep the "fixed" part with the first cell which contains a name and scroll the second part which contains all metrics and will be scrolled synchronically.
Current HTML of single row:
<div class="tb-header layout-row">
<div class="tb-title"><span>Name</span></div>
<div class="tb-metrics">
<div class="layout-row">
<div class="tb-cell flex-10">812</div>
<div class="tb-cell flex-7">621</div>
<div class="tb-cell flex-4">76.5</div>
<div class="tb-cell flex-7">289</div>
<div class="tb-cell flex-4">46.5</div>
<div class="tb-cell flex-7">308</div>
<div class="tb-cell flex-4">49.6</div>
<div class="tb-cell flex-7">390</div>
<div class="tb-cell flex-4">48.0</div>
<div class="tb-cell flex-7">190</div>
<div class="tb-cell flex-4">23.4</div>
<div class="tb-cell flex-7">0</div>
<div class="tb-cell flex-4">0.0</div>
<div class="tb-cell flex-8">6.4</div>
<div class="tb-cell flex-8">0.0</div>
<div class="tb-cell flex-8"></div>
</div>
</div>
My idea was to use touchmove event on parent container (wrapping the whole tree and bind as a directive) and check when touchmove starts over the metrics section then calculate the value which I should move metrics. And that part works fine.
The problem starts when I want to apply the offset on the .tb-metrics > .
My first try was to use jQuery:
function moveMetrics( offset ) {
var ofx = offset < 0 ? (offset < -getMetricsScrollWidth() ? -getMetricsScrollWidth() : offset) : 0;
$('.tb-metrics').children().css('transform', 'translateX(' + ofx + 'px)');
/*...*/
}
Unfortunately, this solution is quite slow when the table contains more rows (I cannot cache rows because they are dynamic).
In my second attempt, a tried to avoid as much DOM manipulation as I can.
To achieve that I decided to add <script> tag to dom which contains css which applies to .metrics > .layout-row.
Solution:
function moveMetrics( offset ) {
var ofx = offset < 0 ? (offset < -getMetricsScrollWidth() ? -getMetricsScrollWidth() : offset) : 0
, style = $( '#tbMetricsVirtualScroll' )
;
if ( !style.length ) {
body.append( '<style id="tbMetricsVirtualScroll">.tb-metrics > * {transform: translateX(' + ofx + 'px)}</style>' );
style = $( '#tbMetricsVirtualScroll' );
} else {
style.text( '.tb-metrics > * {transform: translateX(' + ofx + 'px)}' );
}
/*...*/
}
However, it doesn't seem to be much faster when the table contains a large number of rows. So it's not DOM manipulation but rendering/painting view seems to be the bottleneck here.
I tried to create some kind of virtual scroll but because tree structure is different for different sets of data and can have an "infinite" number of levels (each row can contain children rows in new ng-repeat) it's a really hard task.
I will appreciate any ideas about how I can improve performance in that situation without using the virtual scroll.
EDIT:
Screenshot of the Chrome timeline shows that most time of scrolling is consumed by rendering (I guess that it is because of complicated DOM structure)
EDIT 2:
I won't say that I achieved absolutely smooth scrolling, but I found a couple of things for significant performance improvement (some of them weren't obvious and the result is better than I expected after such small changes).
Simplify class selectors :
.tb-header > .tb-metrics > .tb-cell is much slower than .tb-specific-cell (it seems that it take more time to parse more complicated selectors?)
remove opacity and box shadows from transformed elements
try to distribute transformed element to new layer (use css will-change and/or translateZ(0))
I have had similar problems in the past with really big tables. So, leaving the UX side of things outside of the equation - in order to achieve what you are aiming for could be the following. Instead of keeping the first tables fixed and moving all the rest, maybe you could try the other way around. Aka, keep the table inside an overflown container to allow horizontal scrolling and move using translateY the first column table cells when scrolling vertically.
In the way I am describing, you can retrieve the max width of the first table cell (or even better, set a fixed width), absolutely position each cell on the left - careful though, the relative container should be OUTSIDE the overflown table, set a padding-left on the second table cell of each row, equal to the first table-cell and then, when scrolling vertically, add a negative translateY value. The selector then should be easy to access '.layout-row .table-cell:first-child'.
Also for minor optimizing, instead of using .css() try using .attr('style', value) instead. Also, I read that you have dynamic content and can't cache it, BUT, maybe you should consider caching once every time you manipulate the DOM. Cached selected elements are still better than calculating the selector in every scroll/touchmove event.
Last but not least, you could also add somekind of debouncing technique so that the scrolling does not occur on every scroll frame, but maybe once every 10 frames - it would still be invisible to the eye, or in any case better than bottle-necking the browser with all these repaint events.
I would suggest you use lazy loading.
please note that you have 4 options in lazy loading:
There are four common ways of implementing the lazy load design pattern: lazy initialization; a virtual proxy; a ghost, and a value holder. Each has its own advantages and disadvantages (link).
Good luck
So I'm no expert myself but I do have some ideas that I think are worth mentioning.
If I'm understanding correctly, the problem is that this project needs an increasing number of HTML elements, which would continually decrease performance
Here are 3 js concepts that I think would help you
*Dynamically creating the html element:
let div = document.createElement('div');
*Dynamically creating the attributes you're going to use for the elements, and dynamically being able to assign such attributes
function attributeSetter(atr){
div.setAttribute('class', atr)
}
That way you can call the function wherever you need it, and give it whatever class
name you want to give it dynamically. Furthermore, you can use the div element as many times as you want
final concept. You can also dynamically add pre-made classes, assuming that they're already pre-defined in css, like this.
div.classList.add("my-class-name")
One more concept that might come in handy, is de-structuring
let myClasses = ['flex-10', 'flex-7', 'flex-4', 'flex-9'] //etc
let [flex10, flex7, flex4, flex9] = myClasses;
This will turn all those strings into variables that you can easily iterate through and dynamically add them to your attributeSetter function, like this:
attributeSetter(flex10)
Wherever you need it. I don't know if that helps, or if that answered your question, but at least some ideas. Good luck :)

Implementing weighted flexible layout with CSS

Please take a look at this jsFiddle and press the only button to fill the list up. As you can see DIV elements inside the list are resized to fill the parent contained. This code does exactly what I want to do, but I think the way I implement this is too complex for such a seemingly simple task. Here is the code for the algorithm to assign height to inner elements:
fill = function() {
//Loop through all elements once to get total weight
var totalWeight = 0;
var totalHeight = $("#container").height() - 15; //need a little extra empty space at the buttom
$(".list").each(function(i) {
totalWeight += parseInt($(this).attr('weight'));
totalHeight -= parseInt($(this).css('margin'));
});
//Loop though the element a second time to set the relative height
$(".list").each(function(i) {
var element = $(this);
element.css("height", (element.attr("weight") / totalWeight) * totalHeight);
});
}
My question is, is the best we can do? Are there any other - hopefully faster or more elegant -- ways to achieve this functionality with reasonable cross-browser compatibility? I need to support newer Firefox, Chrome, and Safari only. I was reading about flex-box here and looking at its promising specs, but it does not look like flex-box can do weighted flexible layout consistently across browsers. If I am wrong, please show me a simple example of how this could be achieved.
This type of weighted flexible linear layout is not uncommon, for example it is available in Android SDK as one of the layout options. What is the recommended way to resize elements to fill their parent container, relative to a weight value assigned to them? A pure CSS solution would be wonderful, if at all possible.
Just a quick look over it, after about 30 mins googling. I may do a demo but don't really hav much time.
Have you looked here
html5rocks
coding.smashmag...
tutsplus
umarr
w3c
benfrain
There are some good examples on the 6th one
edit:
I was looking thought the firefox developer section on their website and found
developer.mozilla...
I also found another example with a download!!
github..
this might give you some direction for firefox and the rest should be in the other links I have provided

CSS Page Layout w/ Breaks

I'm trying to make a webpage where it basically looks like a word document. There would be multiple boxes that would scroll down and the text would flow and page break from one page to the next.
Does anyone have any idea where I would even start? Thanks.
Edit: It should be right in the browser, looking similar to this:
(Ignore the columns)
CSS mostly applies styles to a full element due to its box model. Exceptions are pseudo elements. So to create an appropriate break after a fixed length you would have to separate your text into correctly sized different elements.
EDIT:
It would be possible using javascript. But even in the simplest case, where everything inside the pages delivered as just one text element with no sub elements (not even other text elements), the code will be a development nightmare and will run quite crappy. This is because there is no measure function in javascript. So you would be forced to do trail and error to find the correct position to break the element. Since the properties of the elements are live it means, that the viewer of the website will see a lot of flickering of your page just after loading. If you dare put other elements inside the html element to break into pages you get even more problems. More or less you get hundreds of special cases (break inside other elements, what if those elements are inside even other elements) to look out for.
Something like that sounds possible using javascript, but it depends a bit on the structure of your html and whether or not you want to break paragraphs or just move the next paragraph to the next page if it doesn´t fit
So the simplest example, not breaking paragraphs / html elements with a flat html structure (no nested divs, columns, etc) like:
<div class="document">
<h1>title</h1>
<p>texts</p>
<h2>subtitle</h2>
<p>texts</p>
...
<p>texts</p>
</div>
would be to do something like:
height = 0
loop through all direct child elements of .document
{
if ( (height + element_height) > page_height)
{
add page_break_element before current element
height = 0
}
height = height + element_height
}
I´d use jquery because it makes it easy to loop through the elements, measure heights, etc.
I guess breaking paragraphs would be possible as well, but a lot of extra work.
<p style="page-break-before: always">This would print on the next page</p>

Categories