We Keep Coding

How to display large content quickly in browser [duplicate]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I need to present a large number of rows of data (ie. millions of rows) to the user in a grid using JavaScript.
The user shouldn't see pages or view only finite amounts of data at a time.
Rather, it should appear that all of the data are available.
Instead of downloading the data all at once, small chunks are downloaded as the user comes to them (ie. by scrolling through the grid).
The rows will not be edited through this front end, so read-only grids are acceptable.
What data grids, written in JavaScript, exist for this kind of seamless paging?

(Disclaimer: I am the author of SlickGrid)
UPDATE
This has now been implemented in SlickGrid.
Please see http://github.com/mleibman/SlickGrid/issues#issue/22 for an ongoing discussion on making SlickGrid work with larger numbers of rows.
The problem is that SlickGrid does not virtualize the scrollbar itself - the scrollable area's height is set to the total height of all the rows. The rows are still being added and removed as the user is scrolling, but the scrolling itself is done by the browser. That allows it to be very fast yet smooth (onscroll events are notoriously slow). The caveat is that there are bugs/limits in the browsers' CSS engines that limit the potential height of an element. For IE, that happens to be 0x123456 or 1193046 pixels. For other browsers it is higher.
There is an experimental workaround in the "largenum-fix" branch that raises that limit significantly by populating the scrollable area with "pages" set to 1M pixels height and then using relative positioning within those pages. Since the height limit in the CSS engine seems to be different and significantly lower than in the actual layout engine, this gives us a much higher upper limit.
I am still looking for a way to get to unlimited number of rows without giving up the performance edge that SlickGrid currently holds over other implementations.
Rudiger, can you elaborate on how you solved this?

https://github.com/mleibman/SlickGrid/wiki
"SlickGrid utilizes virtual rendering to enable you to easily work with hundreds of thousands of items without any drop in performance. In fact, there is no difference in performance between working with a grid with 10 rows versus a 100’000 rows."
Some highlights:
Adaptive virtual scrolling (handle hundreds of thousands of rows)
Extremely fast rendering speed
Background post-rendering for richer cells
Configurable & customizable
Full keyboard navigation
Column resize/reorder/show/hide
Column autosizing & force-fit
Pluggable cell formatters & editors
Support for editing and creating new rows."
by mleibman
It's free (MIT license).
It uses jQuery.

The best Grids in my opinion are below:
Flexigrid: http://flexigrid.info/
jQuery Grid: http://www.trirand.com/blog/
jqGridView: http://plugins.jquery.com/project/jqGridView
jqxGrid: https://www.jqwidgets.com/
Ingrid: http://reconstrukt.com/ingrid/
SlickGrid http://github.com/mleibman/SlickGrid
DataTables http://www.datatables.net/index
ShieldUI http://demos.shieldui.com/web/grid-virtualization/performance-1mil-rows
Smart.Grid https://www.htmlelements.com/demos/grid/overview/
My best 3 options are jqGrid, jqxGrid and DataTables. They can work with thousands of rows and support virtualization.

I don't mean to start a flame war, but assuming your researchers are human, you don't know them as well as you think. Just because they have petabytes of data doesn't make them capable of viewing even millions of records in any meaningful way. They might say they want to see millions of records, but that's just silly. Have your smartest researchers do some basic math: Assume they spend 1 second viewing each record. At that rate, it will take 1000000 seconds, which works out to more than six weeks (of 40 hour work-weeks with no breaks for food or lavatory).
Do they (or you) seriously think one person (the one looking at the grid) can muster that kind of concentration? Are they really getting much done in that 1 second, or are they (more likely) filtering out the stuff the don't want? I suspect that after viewing a "reasonably-sized" subset, they could describe a filter to you that would automatically filter out those records.
As paxdiablo and Sleeper Smith and Lasse V Karlsen also implied, you (and they) have not thought through the requirements. On the up side, now that you've found SlickGrid, I'm sure the need for those filters became immediately obvious.

I can say with pretty good certainty that you seriously do not need to show millions of rows of data to the user.
There is no user in the world that will be able to comprehend or manage that data set so even if you technically manage to pull it off, you won't solve any known problem for that user.
Instead I would focus on why the user wants to see the data. The user does not want to see the data just to see the data, there is usually a question being asked. If you focus on answering those questions instead, then you would be much closer to something that solves an actual problem.

I recommend the Ext JS Grid with the Buffered View feature.
http://www.extjs.com/deploy/dev/examples/grid/buffer.html

(Disclaimer: I am the author of w2ui)
I have recently written an article on how to implement JavaScript grid with 1 million records (http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records). I discovered that ultimately there are 3 restrictions that prevent from taking it highter:
Height of the div has a limit (can be overcome by virtual scrolling)
Operations such as sort and search start being slow after 1 million records or so
RAM is limited because data is stored in JavaScript array
I have tested the grid with 1 million records (except IE) and it performs well. See article for demos and examples.

dojox.grid.DataGrid offers a JS abstraction for data so you can hook it up to various backends with provided dojo.data stores or write your own. You'll obviously need one that supports random access for this many records. DataGrid also provides full accessibility.
Edit so here's a link to Matthew Russell's article that should provide the example you need, viewing millions of records with dojox.grid. Note that it uses the old version of the grid, but the concepts are the same, there were just some incompatible API improvements.
Oh, and it's totally free open source.

I used jQuery Grid Plugin, it was nice.
Demos

Here are a couple of optimizations you can apply you speed up things. Just thinking out loud.
Since the number of rows can be in the millions, you will want a caching system just for the JSON data from the server. I can't imagine anybody wanting to download all X million items, but if they did, it would be a problem. This little test on Chrome for an array on 20M+ integers crashes on my machine constantly.
var data = [];
for(var i = 0; i < 20000000; i++) {
data.push(i);
}
console.log(data.length);​
You could use LRU or some other caching algorithm and have an upper bound on how much data you're willing to cache.
For the table cells themselves, I think constructing/destroying DOM nodes can be expensive. Instead, you could just pre-define X number of cells, and whenever the user scrolls to a new position, inject the JSON data into these cells. The scrollbar would virtually have no direct relationship to how much space (height) is required to represent the entire dataset. You could arbitrarily set the table container's height, say 5000px, and map that to the total number of rows. For example, if the containers height is 5000px and there are a total of 10M rows, then the starting row ≈ (scroll.top/5000) * 10M where scroll.top represents the scroll distance from the top of the container. Small demo here.
To detect when to request more data, ideally an object should act as a mediator that listens to scroll events. This object keeps track of how fast the user is scrolling, and when it looks like the user is slowing down or has completely stopped, makes a data request for the corresponding rows. Retrieving data in this fashion means your data is going to be fragmented, so the cache should be designed with that in mind.
Also the browser limits on maximum outgoing connections can play an important part. A user may scroll to a certain position which will fire an AJAX request, but before that finishes the user can scroll to some other portion. If the server is not responsive enough the requests would get queued up and the application will look unresponsive. You could use a request manager through which all requests are routed, and it can cancel pending requests to make space.

I know it's an old question but still.. There is also dhtmlxGrid that can handle millions of rows. There is a demo with 50,000 rows but the number of rows that can be loaded/processed in grid is unlimited.
Disclaimer: I'm from DHTMLX team.

I suggest you read this
http://www.sitepen.com/blog/2008/11/21/effective-use-of-jsonreststore-referencing-lazy-loading-and-more/

Disclaimer: i heavily use YUI DataTable without no headache for a long time. It is powerful and stable. For your needs, you can use a ScrollingDataTable wich suports
x-scrolling
y-scrolling
xy-scrolling
A powerful Event mechanism
For what you need, i think you want is a tableScrollEvent. Its API says
Fired when a fixed scrolling DataTable has a scroll.
As each DataTable uses a DataSource, you can monitoring its data through tableScrollEvent along with render loop size in order to populate your ScrollingDataTable according to your needs.
Render loop size says
In cases where your DataTable needs to display the entirety of a very large set of data, the renderLoopSize config can help manage browser DOM rendering so that the UI thread does not get locked up on very large tables. Any value greater than 0 will cause the DOM rendering to be executed in setTimeout() chains that render the specified number of rows in each loop. The ideal value should be determined per implementation since there are no hard and fast rules, only general guidelines:
By default renderLoopSize is 0, so all rows are rendered in a single loop. A renderLoopSize > 0 adds overhead so use thoughtfully.
If your set of data is large enough (number of rows X number of Columns X formatting complexity) that users experience latency in the visual rendering and/or it causes the script to hang, consider setting a renderLoopSize.
A renderLoopSize under 50 probably isn't worth it. A renderLoopSize > 100 is probably better.
A data set is probably not considered large enough unless it has hundreds and hundreds of rows.
Having a renderLoopSize > 0 and < total rows does cause the table to be rendered in one loop (same as renderLoopSize = 0) but it also triggers functionality such as post-render row striping to be handled from a separate setTimeout thread.
For instance
// Render 100 rows per loop
var dt = new YAHOO.widget.DataTable(<WHICH_DIV_WILL_STORE_YOUR_DATATABLE>, <HOW YOUR_TABLE_IS STRUCTURED>, <WHERE_DOES_THE_DATA_COME_FROM>, {
renderLoopSize:100
});
<WHERE_DOES_THE_DATA_COME_FROM> is just a single DataSource. It can be a JSON, JSFunction, XML and even a single HTML element
Here you can see a Simple tutorial, provided by me. Be aware no other DATA_TABLE pluglin supports single and dual click at the same time. YUI DataTable allows you. And more, you can use it even with JQuery without no headache
Some examples, you can see
List item
Feel free to question about anything else you want about YUI DataTable.
regards,

I kind of fail to see the point, for jqGrid you can use the virtual scrolling functionality:
http://www.trirand.net/aspnetmvc/grid/performancevirtualscrolling
but then again, millions of rows with filtering can be done:
http://www.trirand.net/aspnetmvc/grid/performancelinq
I really fail to see the point of "as if there are no pages" though, I mean... there is no way to display 1,000,000 rows at once in the browser - this is 10MB of HTML raw, I kind of fail to see why users would not want to see the pages.
Anyway...

best approach i could think of is by loading the chunk of data in json format for every scroll or some limit before the scrolling ends. json can be easily converted to objects and hence table rows can be constructed easily unobtrusively

I would highly recommend Open rico.
It is difficult to implement in the the beginning, but once you grab it you will never look back.

I know this question is a few years old, but jqgrid now supports virtual scrolling:
http://www.trirand.com/blog/phpjqgrid/examples/paging/scrollbar/default.php
but with pagination disabled

I suggest sigma grid, sigma grid has embed paging features which could support millions of rows. And also, you may need a remote paging to do it.
see the demo
http://www.sigmawidgets.com/products/sigma_grid2/demos/example_master_details.html

Take a look at dGrid:
https://dgrid.io/
I agree that users will NEVER, EVER need to view millions of rows of data all at once, but dGrid can display them quickly (a screenful at a time).
Don't boil the ocean to make a cup of tea.

AngularJS Infinite Scrolling with lots of data

So I'm trying to create an infinite scrolling table using AngularJS, similar to this: http://jsfiddle.net/vojtajina/U7Bz9/
The problem I'm having is that in the jsfiddle example, if I keep scrolling till I have a million results, the browser is going to slow to a crawl, wouldn't it? Because there would now be 1,000,000 results in $scope.items. It would be better if I only ever had, for example, 1000 results at a time inside $scope.items, and the results I was viewing happen to be within those 1000.
Example use case: page loads and I see the first 10 results (out of 1,000,000). Even though I only see 10, the first 1000 results are actually loaded. I then scroll to the very bottom of the list to see the last 10 items. If I scroll back up to the top again, I would expect that the top 10 results will have to be loaded again from the server.
I have a project I did with ExtJS that a similar situation: an infinite scrolling list with several thousand results in it. The ExtJS way to handle this was to load the current page of results, then pre-load a couple of extra pages of results as well. At any one time though, there was only ever 10 pages of results stored locally.
So I guess my question is how would I go about implementing this in AngularJS? I kow I haven't provided much code, so I'm not expecting people to just give the coded answer, but at least some advice in which direction to go.

A couple of things:
"Infinite scrolling" to "1,000,000" rows is likely to have issues regardless of the framework, just because you've created millions and millions of DOM nodes (presuming you have more than one element in each record)
The implementation you're looking at doing with <li ng-repeat="item in items">{{item.foo}}</li> or anything like that will have issues very quickly for one big reason: {{item.foo}}} or any ngBind like that will set up a $watch on that field, which creates a lot of overhead in the form of function references, etc. So while 10,000 small objects in an "array" isn't going to be that bad... 10,000-20,000 additional function references for each of those 10,000 items will be.
What you'd want to do in this case would be create a directive that handles the adding and removing of DOM elements that are "too far" out of view as well as keeping the data up to date. That should mitigate most performance issues you might have.
... good infinite scrolling isn't simple, I'm sorry to say.

I like the angular-ui implementation ui-scroll...
https://github.com/angular-ui/ui-scroll
... over ngInfiniteScroll. The main difference with ui-scroll from a standard infinite scroll is that previous elements are removed when leaving the viewport.
From their readme...
The common way to present to the user a list of data elements of undefined length is to start with a small portion at the top of the list - just enough to fill the space on the page. Additional rows are appended to the bottom of the list as the user scrolls down the list.
The problem with this approach is that even though rows at the top of the list become invisible as they scroll out of the view, they are still a part of the page and still consume resources. As the user scrolls down the list grows and the web app slows down.
This becomes a real problem if the html representing a row has event handlers and/or angular watchers attached. A web app of an average complexity can easily introduce 20 watchers per row. Which for a list of 100 rows gives you total of 2000 watchers and a sluggish app.
Additionally, ui-scroll is actively maintained.

It seems that http://kamilkp.github.io/angular-vs-repeat would be what you are looking for. It is a virtual scrolling directive.

So turns out that the ng-grid module for AngularJS has pretty much exactly what I needed. If you look at the examples page, the Server-Side Processing Example is also an infinite scrolling list that only loads the data that is needed.
Thanks to those who commented and answered anyway.
Latest URL : ng-grid

You may try using ng-infinite-scroll :
http://binarymuse.github.io/ngInfiniteScroll/

Check out virtualRepeat from Angular Material
It implements dynamic reuse of rows visible in the viewport area, just like ui-scroll

Website text and code editor solution [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have spent days trying to incorporate different editors within my site. None seem to be up to the job. I have delved in to their code a lot and one thing or another doesn't work.
Here's my requirements:
Basic formatting such as size, bold, italic, strikeout, subscript and superscript
Image upload via ajax, image resize and positioning
Link handling
Code tag or styled div with a drop down in the top right for selecting syntax highlighting. And on leaving the code tag/styled div the text updates via ajax to include line numbering.
Paste but encode users code
The main issues with editors I have tried are:
Carriage Returns/Line breaks don't get the cursor out of the current element and if they do they re-create the current element or re-create the current element within itself.
They use a lot of unnecessary code, such as multiple nested divs. Heck looking at the source of some you get <div><div><div><div><div><div>Hello World</div></div></div></div></div></div>
They don't work in quirks which I need them to.
Here's an image of my idea, it's just a knock up:
So my idea is to use a content Editable div and regex to track the code tags and implement my own functionality. But I need to get selection start and length of a content div in all browsers inc' quirks. How can I do that?
Also replacing text within the content div.
Any suggestions, ideas or help would be great thank you.

It's pretty easy actually. You just need a <div/> with an attribute called contenteditable. From then on, the only browser issues you may have is to reliably insert HTML into that div. But I've seen it work well before, so it shouldn't be a real problem.
Since you mentioned quirks mode, I'm afraid it's not possible without hue hacks, however, there's an easy fix: put your editor (the div) in a separate page and embed it within an iframe.
Regarding the image uploading part, your server needs to handle that. From the editor's aspect, it's just an <img/> tag.

Alternative to SIMILE Timeline for timeline visualization? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
So I have been having lots of fun playing with SIMILE Timeline, but unfortunately, it appears to be all but abandoned (most recently closed ticket May 2009) and riddled with memory leaks, especially when you are dynamically loading and unloading events (which is especially true in my case).
I am a novice in JavaScript, so debugging these leaks is going to be more than a little complicated. Before I dive head-first into fixing a problem in a language I don't understand or a codebase I have never looked at, I want to know if there is anything out there that compares to the feature-set that I can get with Timeline:
Multiple events on one band
instead of having 10, 100, 1000 rows of data, use the space available to fit multiple events on one line when possible
Scrollable
CSS styling down to the specific event
I can specify a specific icon, font, etc., for each event if I need to. This is great for highlighting error events or interesting events
filtering/searching/highlighting
highlighting a band of time or point in time
This would be a vertical band that shows a 'now' moment, or a band that covers a time period's start and end time
I know that alot of these features can probably be put into things such as Flot or HighCharts, but these things were next to free for me to use in Timeline, so I'm weighing the cost/benefit of whether it is easier to fix a memory leak or roll out 3 - 4 features on some library that doesn't have it. My gut says "fix the leak, it's easier", but I would like to know if anyone can show me something that might be 90% of the way there with a low barrier of entry to getting that other 10%.

Maybe you can use the Timeline from the (open source) CHAP Links Library, developed by Almende.
http://almende.github.com/chap-links-library/timeline.html
Have a look at the examples to see what you can do with the Timeline. It is quite flexible, you can choose your create your own layout and use your own CSS/HTML in each individual event.

Check out Timeglider (see http://timeglider.com/jquery/) which is an open source (MIT), JavaScript (uses jQuery) scrolling timeline under active development. It meets your first 4 requirements I believe but I am unclear about the specifics of your 5th requirement. I have only done some simple experiments with Timeglider and don't know it very well but maybe this is a potential solution for you.

We needed the Simile Timeline at Lab 21k so we took the source code, cleaned it up and published it using bower. So you don't need an alternative anymore.
https://github.com/Lab21k/standalone-timeline/

The Timeline project is very much alive, they just don't use Google Code anymore. The Simile widgets are now hosted on a dedicated homepage, here is the homepage and documentation of Timeline. Keep in mind that the links on the page are not consistent, many pages still link to the outdated wikis on Google Code and the mit.edu wiki pages. Keep that in mind when checking the wiki.
There is also an active mailing list on Google Groups.

This is probably the best one around right now:
http://timeline.verite.co/

I'm actually working with the Simile Timeline in a project of my called Event Viewpoint. I've fixed quite a lot of bugs and enhanced it with many contributions that I have found online including images for the event, a zoom bar, and some additional events.
It is a brilliant piece of work just needs someone to take the time to update it and make it more current. That is currently out of scope for my project, but if I have time in the next year or so I will start to enhance it.
It would really be great if people could revive the project, add touch events and so on.
If someone needs the source I am willing to provide it.

They’re still very active. David Karger makes frequent updates. Now they have the version 3.0.
You can find reference documentation here:
http://simile-widgets.org/wiki/Reference_Documentation_for_Exhibit.
The code is here: https://github.com/simile-widgets/exhibit

I don't know exactly what you want, but i'm working on this jquery plugin that it's being used in a website with dynamically managed events.

javascript - google map like image scrolling [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I want to build an image scroller that works like google map. When the website first loads, there will be a viewing box in the middle of the page that contains part of an image. A user can click and drag the image, that is scrolling, to inspect other areas of the image. If the user happens to scroll beyond the perimeter of the image, the box will load an adjacent image. I also want to overlay these images with markers that the user can click on to zoom in for more details.
Is there a library that does what I want already? Or something close to what I want so that I can start to customize it? Or if I have to build from scratch, any suggestions on the general approach to take?
I like using jquery library, so anything built with jquery would be awesome.

I don't think this does everything you're looking for, but might be a good place to start: SpryMap

Take a look at the Overscroll jQuery plugin.

I've used OpenLayers for projects similar to this and it works great!
Not based on jQuery though.
It provides free maps but you can have your own images as well.

Just for some design ideas to take into account when writing/finding your solution:
There are two ways of doing this, one with multiple chopped up images that preload when the user scrolls near them, and the other which is simpler and loads one big image.
If your image(s) are very big, you should opt for a solution that preloads segments as this will be a lot more user friendly if you can do it seamlessly.
The big image will slow down some computers/mobile devices significantly!

If you built something yourself, it probably couldn't be pure jQuery. You'd want to write a server script that either processes your big images into segments that you cache, or serves segments on the fly. That's the point at which you could start using jQuery to control what gets loaded ans when.

I think CraftMap seems to provide what you are asking for. But I'm unsure if it supports zooming.