Fast managing of tables with ~10k rows in Javascript [closed] - javascript

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I've been working with tables and a huge amount of data. There are a tables in my website with 10 thousand rows. This table has dynamic search, filters, etc. I've been using pure JavaScript considering performance, but it gets laggy with this amount of rows.
Do you guys know any alternative for pure JavaScript with better performance?
EDIT> I REALLY need to load the 10 thousand rows at once. I can load them all in the browser in 5 seconds. The main problems are the filters and search...
EDIT2> The search is dynamic. I can search by name and filter it by first character.
I've been working on it for months...
SEARCH:
search when the field has more than 3 characters and only when its length and characters has changed (onchange event on input may trigger multiple times when a character changes, so I make sure it only triggers once using some verifications)
each row that matches the searched string is coppied to another table. The original table is hidden and the new one is displayed.
when the user changes the search field or cancel the dynamic search, the new table is erased.
Conclusion: it's faster to create a new table with desired elements than hide the undesired ones.
FITLER
The rows are actually in 25 tables (A to Z + non-alphabetical characters)
When you select a character, only that table is shown
Conclusion: it's faster to hide a whole table than hide the
undesired rows
Thanks for the repplies. I've edited with some extra info so we can narrow the possible solutions...

I'm asuming you get the contents from database and load with with something like PHP (I'm going to asume PHP for now)
You could make the Javascript make an ajaxcall to a php-file which does the filtering (actually you should make the database do it, a lot faster!) and place the resulting table back on screen.
A faster method combined with the above might be this: Get all id's on the initial rows in an array and save those (in a session might work pleasant).
When you have to filter, don't make PHP get the whole table, just apply the filter to only the stored id's and send javascript the matching rows.
Then make javascript do something like this:
- set all visible
- set resulting id's to hidden (hidden in favor of remove, because I think a user might perform multiple filter actions?)
Another idea just popped in my head: If you don't need to display it on load, you can start the initial load with all tables hidden and a message "please search to display".

A common technique to handle this case is to load the data in memory or a subset of the data, and recycle your table rows such that you aren't actually ever creating thousands upon thousands of rows. You can get creative with this and create a web interface that seemingly scrolls endlessly but in reality you are just reusing dom elements and shuffling them around.
Most well-built data grid widgets whether they are on the web, mobile or even a desktop interface will employ this technique to handle your particular problem.
In most cases a user will never actually find themselves benefiting from seeing 10's of thousands of rows of data at once anyway.

fetch from the server only the things to be seen by the user, Like everyone has pointed out 10,000 rows needn't be there on that page.
you can use the concept of pagination and for every page few rows are fetched and shown . JQuery's Ajax is capable of calling the server side function to fetch rows to add them to your page.
don't know any backend details here, but in struts framework there is display:table tag and I believe in .NET framework there is GridView for pagination in the client side that you can look into

I'm just a student, newly joined to the community. Take what I say with a grain of salt.
I'm not sure why everyone is so much as blinking at the ten thousand rows business when we're measuring modern personal computers' memory in gigabytes.
Alright. I'm going to assume that what you're doing needs to be done in the browser, and so you can't switch to doing native code. In that case, looking for an alternative to Javascript won't get you much of anywhere. In the context of a browser, you're looking at an interpreted language. In terms of number of instructions the program ultimately has to run, the difference between one language or another is negligible in the face of how long it takes to be interpreted. Besides, Javascript has gotten nicely polished over the years.
So never mind that. There's a much more important thing to consider here, and it applies no matter what you're programming in or on: The cache(s). Igor Ostrovsky explains it beautifully; read it until you grok it.
So I'm guessing you have objects that would stringify to something like, "obj1 = {field-1:'a', field-2:'b', ..., field-n:'n'}". And you can select a field-i to sort by. The trouble with this is that when you sort by field-i, you're loading the entirety of obj1 into the cache, even though you don't need it. What you really want to do is load the field-i's forobj1, obj2, obj3, ..., objm all at once. So you look at an array, stringifying to something like: "field1 = [refToRow1, field1inRow1, refToRow2, field1inRow2, ..., refToRowM, field1inRowM]".
You might not be able to do fit all M rows in the cache, after all M==10000! But you can group them together into chunks that you could reasonably expect a cache to manage. Anyone got a good number for this? Say, 64kB? So you figure for each i in M you've got a reference, and a field that's probably just a reference to a short string (it'd be better if you could have the string itself right there, but I don't think Javascript works that way). So that's 8B? 8B*1024 = 64kB? Hell, if that's right, you could fit it all into the cache in two chunks, which means you'd want to do it in 4.
So now you've got a collection of smaller arrays, and you want to sort them. This is a classic application for B-trees. And while having a separate B-tree for each and every column in the table may seem like a lot, it's not.
Okay, so that handles sorting. You tell it to sort by a column, and the truth is it's already sorted! You're just repopulating the visible table using a different b-tree. You still need to handle filtering, but that's fine. You do some cache juggling as you find something to display and follow the reference to get the other fields, but I'd still expect this to go fast since you're skipping over so many rows.
Normally, I would say if you want to speed things up, look into multiprocessing. But I think browsers are still working to make that a thing with their Javascript implementations. Plus, while it would be well-suited for sorting, it would be a lot of effort to make it useful for the filtering part, and I expect you can do fine without.
I hope this isn't too scatter-brained, and that it gives you some ideas. Good luck!

Related

How to implement autosave for a web application

I have a web page that contains a table wherein you can edit every cell content. The typical size of this table is about 12 columns and 40 rows. This page is typically used by teachers to encode grades here in our area. But our area sometimes has a poor connection, and what happens is sometimes, teachers would lose the encoding they did (they typically only save after encoding everything).
So right now, I am thinking of implementing an autosave feature. I think the solution I'm thinking of is straight-forward: every time a cell gets edited and then loses focus, do an ajax request to save the data. At least that's what I'm thinking.
My concern is since there's a lot of people using this at the same time, wouldn't that be too heavy, every edit, access the DB, and insert or update? And also, I've tried it a bit and the UI became a bit choppy when moving from one cell to another. My main question basically is, is there a pattern or guidelines I can use for implementing this?
I'd need more details on what your api design to help come up with the most optimal solution but here are some high level approaches you can consider
On each cell update only update the diff (the exited cell) and not the entire dataset to keep the api call as light weight as possible
Look into using IndexedDB or Local Storage to save a copy of the data in the browser as they update cells. In case of a lost connection you can restore the user's session.
There are some points here for your problem that I mentioned and try to say its solution:
First, you have to think about concurrency. If you need to show table content for a long time to your users, It is possible to change contents by other users at the same time, and the current content will expire! So you could keep your table content update continuously! one solution is to use RowVersion in DotNet For this purpose (Optimistic approach for concurrency). Another solution is using SingnalR to keep tables update all the time (I mean Real Time).
In addition, there is no serious problem with update and edit according to a user request in regular systems but in an enterprise solution you can scale up or scale out your structure.

Hierarchy table - activation of rows through javascript?

I've created a database-driven hierarchy table with 3 hierarchy levels. It works almost exactly how I want it to, but the javascript code that drives its functionality is UGLY. I would like to re-write some of it, but I don't quite know how. I imagine I'll have to use recursive functions, but there are some strange business rules that may make it more difficult than I imagined. Can you make any suggestions to simplify my current code (in particular, the ActivateRow() function)? See fiddle.
Note that there is some funky use of hidden fields due to ASP.NET Web Forms postbacks and losing input values, but there are some useful data-attributes written into the HTML such as data-region-id, data-subregion-id, data-market-id, and data-hierarchy-id. the main requirements are below:
There are two tables - the first is linked to the second, so all actions on the first table should be carried out on the second table as well (though the second table is slightly different).
Three levels of hierarchy. The highest level (Region), a middle level (Sub Region), and a final level (Market)
Row activation enables all controls in a given row. If the activated row is a Sub Region row, it deactivates its parent Region row controls and its sales values sum up to the Region level. This does not apply to activating a Market row.
Similarly, row deactivation should disable and clear all controls within that row. If the row is a parent row, it should clear all children rows and disable their controls as well.
Deactivating all children of a parent row should deactivate the parent row and clear all parent row controls.
Surely, there's a better alternative. Am I reinventing the wheel? Is there anything out there that I could use to improve my spaghetti code? I realize this isn't the best question for Q&A format - is there somewhere else I should ask my question?
It is not entirely clear what you are doing but from what I gather you have some kind of table you are filtering.
Also, you are using ASP.NET webforms.... so sorry for you.
I would not use any kind of form binding controls offered by ASP WebForms, because it is now obsolete, incorrect and very painful.
Instead, create an AJAX enabled page [WebMethod] that returns all the data or filtered with a parameter, what ever, in JSON (Because JSON is easier to read and debug, and much faster than XML)
Go and find the jQuery plugin DataTables. Everykind of functionality is built in and extentable, it works great, super efficient client side, with varios filtering and searching capabilities.
You either create a new model on the Server side and remap the data server side, or dump your existing model and remap it on the client, either way, you just plug JSON data into datatables.
Saving back to webforms is slightly more tricky (compared to MVC/Razor) because you have to take the String and deserialise it to your model, its a few extra lines but I do this all the time and you can carry on with your business logic.

Efficiently rendering a large number of "select" elements

I'm writing a web application that displays a large number of rows of data (~2000 at present), each of which has a drop-down "select" element with ~100 options. Any of those options can be selected by default. I'm generating all the actual DOM elements client-side. My problem: rendering this beast takes ~4 seconds on my relatively recent machine, which is really suboptimal. I know the problem is specifically with all the select elements, because replacing them with a bit of static text or a single-option list causes render time to be nigh imperceptible.
The vanilla code, minus failed experiments (see below) is here.
Avoiding the suggestions of "paginate your data" and "don't have so many options in a select", what is the most efficient way I can write my append / render code, assuming I do have a legitimate reason to display that much data and have that many options? For my purposes, Firefox is the only platform I care about.
Things I have tried:
Using an async loop to append rows to the table (slower than a regular loop, and oddly didn't render the intermediate results)
Building up a string that represents the body of the table and inserting it into the DOM in a single call (almost identical performance)
Instead of inserting the entire options list, inserting a single-option "select" element, and then populating the entire list when the "select" element gains focus (presumably because someone is trying to change it). This was actually pretty high-performing for the initial render, but then populating the element with the full list caused some weird behavior, losing focus and never actually being able to "open" the select element.
Right now my default assumption is that the third option is the way to go, and I need to figure out how to do some bookkeeping about what has already been populated. But my suspicion is that there is a plainly better / faster / more idiomatic way to do this. Is there?
Yes, I would "lazily" generate and/or populate the dropdowns.
That is, only create and populate the dropdowns when the user clicks on them, as probably almost all of the dropdowns in the 2000 rows will never be used right?
Perhaps a select element might not be the best UI here too, but instead some kind of HTML menu like so: https://jqueryui.com/menu/ that is created, populated and displayed only when the user clicks on some kind of button to display it.

Do you reuse values from a select box?

Let's say that I have a select box in an html page. And the values in that select box come from a db query that returns a not so small list of options.
Now if I need to add another 4 select boxes showing the same list of options, would I have to duplicate the code and so send back to the browser 4*(select_box_result_size) or this is normally done differently?
Regardless of how you are managing the server-side code, the answer is that each select box needs its own group of options. That said, if all options are the same then you should only make one query to your database, and generate the all the select boxes you need with a single function. Simply get the options from the database, store them in a variable and then create your boxes.
Do note that in your HTML each box will have the values hardcoded into it, so your client-side code will have duplicate data, but that doesn't really matter. What is important is that the code that you'll have to maintain is clean, and that you put as little stress as possible on your database. One query + one function is all you need.
If performance was really an issue, you always have the possibility of dynamically-generating the combo boxes using JavaScript. You could essentially embed one and then copy it four more times. However there are two things you need to consider for this:
Do you really have a performance issue that is being caused by these combo boxes? (Probably not)
Is the combo box so big that it will actually take longer to load the HTML across a network connection than it will to copy the box four times with JavaScript? (Again, probably not... remember that the longer the box the longer it will take to copy.)
Unless you have 1000+ options, I would recommend to simply stick to generating the box four times. If you do have that many then do two things:
Benchmark. Create two pages, one with a single combo box and one with four and compare the size. Then compare the load times.
Consider improving your UI. Perhaps if you have 1000 options it would be better for the end user to implement some sort of filtering process to reduce the number of necessary elements... i.e. If you have a box containing all the cities of the world, have one box to select the country and then populate the city box with only the applicable cities by way of a JSON request.
And the values in that select box come from a db query that returns a
not so small list of options.
For user friendliness you could try autosuggest, there are libraries for that but you could try writing your own.
If same data is displayed multiple times (4 times) depending on your server side script it might be best to fetch the data in memory and generate the selects instead of querying 4 times.
would I have to duplicate the code
It's best not to duplicate, at the very least you can write a helper function to generate the select list. If you're serious about removing code from html markup you can look into using one of the many MVC frameworks out there that use templates and allow you to bind data to it.

Small, fast, client-side Javascript database with 'update-as-you-type' functionality?

I have a database no more than say 100k which I'd like to use as reference documentation for my software. It's just a simple table really - around 5 columns by a couple of hundred rows. I am looking for a decent Javascript database library; one which would feature:
Sorting by column
A tiny size. Has to be small as it is sent to the user (since I don't want anything server-side). Say no more than 50-100k.
"Update-as-you-type" functionality, and by that I mean, you can type in a filter box, and the rows filter out instantly on the HTML page, or near instantly as you're typing (client-side processing only). If no input in the filter is given, all the results would display on a single large HTML page.
Searches that allow for partial matches of any cell in the table, and preferably allow NOT, OR and obviously AND.
Furthermore, it should be free/cheap, easy to use and install, perhaps working on a CSV data file for its data.
Is there anything out there that fits the bill?
Check jQGrid OR DataTables it has most of that what you are looking for.

Categories