How can I make Qooxdoo virtual list scalable? - javascript

I need to show list of data , at least 1 million rows (Big data , machine learning).
I do not need to show at once , remotetablemodel of qooxdoo table works fine but instead of table i choose list as design choice.
Below is a test i've made.
//create the model data, 1mil items
var rawData = [];
for (var i = 0; i < 1000000; i++) {
rawData[i] = "Item No " + i;
}
var model = new qx.data.Array(rawData);
//create the list
var list = new qx.ui.list.List(model);
this.getRoot().add(list);
I understand the point that it will take long to generate rawdata and assign it to list.
But the problem is after assigning the list , the virtual list itself is almost non-responsive.
Scrolling is very slow , navigating by down arrow freezes a few secs too.
Qooxdoo virtual infrastructure is suppose to render only visible items if i understand correctly? But in above test case it is so slow.
I expect to work like remote table model .
Tested with qooxdoo latest 4.0.0 and 3.5.1 , on Chrome 35 stable.

I can reproduce you issue only with the source version and not with the build version. I found the reason why the performance is so slow. There is an runtime check in an internal method from the SingleValueBinding which has a huge performance impact on the rendering.
I opened a bug report for that:
http://bugzilla.qooxdoo.org/show_bug.cgi?id=8439
But as I sad this issue only occurs with your developer version. So your customers are not effected.
You can disable the check if you want. Just remove the check block:
https://github.com/qooxdoo/qooxdoo/blob/master/framework/source/class/qx/data/SingleValueBinding.js#L915
You can also load your model data in parts to improve the model creation. You can maybe load the next part when the user has scrolled to the end of the list. You can use the example you have already seen:
Infinite scroll in qooxdoo with virtual list

Related

Facing issue in sessionStorage Proxy implementation in Ext JS Grid view

I just created a Sencha fiddle wherein I have one panel with two grid views inside it. Both Grid views facilitates item selection from left to right and vice-versa. I'm trying to show the selected item from left grid view into right grid view. I'm using sessionStorage proxy to hold data while moving items from left to right grid view. I have to use sessionStorage because my requirement is to persist this selection across the pages.
Sencha fiddle link : https://fiddle.sencha.com/#view/editor&fiddle/2j7u
Although, I'm not getting any error but my implementation is not working. Even the console logs show the correct selection data. But still I'm unable to populate the right grid view.
One thing I noticed is that in browser debugger under Applcation -> SessionStorage, my sessionStorage object is always null. Don't know why but on console it shows the data.
Console logs from fiddle
sessionStorage value from debugger
Any pointers would be highly appreciated!
On a side note, the same implementation was working fine with Ext JS 4.2. But I'm facing issue while migrating to Ext JS 6.5
Thanks,
Dhiraj
Further troubleshooting:
Further, I have updated the same fiddle and was successful to make it work further, by commenting out 'suspendEvents' and 'resumeEvents' for my right grid 'store' --> store2 and 'sessionStore' --> sessionStore2 in tropos.selectDevices.addToSelect method. Along-with this I figured out that line 'store2.loadPage(1)' in tropos.selectDevices.updateGridViews method was clearing my grid on loading. I tried to avoid that using 'clearOnPageLoad' config setting for store to false. But it DIDN'T work so I commented this line to make it work. Now, although, it works but my main requirement of PRESERVING user selection is still not met because sessionStorage is still not working.
Updated fiddle: https://fiddle.sencha.com/#view/editor&fiddle/2j7u
Thanks,
Dhiraj
Try this code in your addToSelect :
example.selectDevices.addToSelect = function(storeObj){
var addRecs = [], newRecs = [],
store2 = Ext.getCmp('grid2').getStore();
//********************
//I just comment all *sessionStore2* references because is redundand,
//store2 is the same as sessionStore2
//********************
//sessionStore2 = Ext.data.StoreManager.lookup('currentlySelectedDevices');
//store2.suspendEvents();
//sessionStore2.suspendEvents();
Ext.each(storeObj,function(record){
var rec = record.copy();
rec.phantom = true;
addRecs.push(rec);
newRecs.push(record.data.hid);
});
example.selectDevices.selectedDeviceIds = example.selectDevices.selectedDeviceIds.concat(newRecs);
store2.add(addRecs);
//store2.commitChanges();
for(i=0; i < store2.data.items.length; i++) {
console.log("addToSelect: Selected Device Serial [" + i + "] from STORE_2: " + store2.data.items[i].data.hid);
}
//sessionStore2.add(addRecs);
//for(j=0; j < sessionStore2.data.items.length; j++) {
// console.log("addToSelect: Selected Device Serial [" + j + "] from SESSION_STORAGE_2: " + sessionStore2.data.items[j].data.hid);
//}
//sessionStore2.sync();
store2.sync();
//sessionStore2.resumeEvents();
example.selectDevices.updateGridViews();
};
EDIT: You have to mark the record as phantom = true in order the sync work properly

Kendo Treeview Expanding big tree issue

I have created a tree control using kendo TreeView.it has more than 10,000 nodes and i have used loadOnDemand false when creating Tree.
I am providing a feature to expand the tree by its level, for this i have created a method which takes the parameter "level" as number and expand it accordingly and user can enter 15 (max level) into the method, it works fine with 500 to 600 nodes for all the levels but when tree has more than 5000 nodes than if user is trying to expand above the 2nd level nodes then browser hangs and shows not responding error.
Method which i have created to expand the tree is :-
function ExapandByLevel(level, currentLevel) {
if (!currentLevel) {
currentLevel = 0;
}
if (level != currentLevel) {
var collapsedItems = $("#treeView").find(".k-plus:visible");
if (collapsedItems.length > 0) {
setTimeout(function () {
currentLevel++;
var $tree = $("#treeView");
var treeView = $tree.data("kendoTreeView");
var collapsedItemsLength = collapsedItems.length;
for (var i = 0; i < collapsedItemsLength; i++) {
treeView.expand($(collapsedItems[i]).closest(".k-item"));
}
ExapandByLevel(level, currentLevel);
}, 100);
}
else {
//console.timeEnd("ExapandByLevel");
hideLoading();
}
}
if (level == currentLevel) {
hideLoading();
}
}
call above given method like this:-
ExapandByLevel(15);
here 15 is level to expand in tree.
when tree has more than 5000 nodes than if user is trying to expand above the 2nd level nodes then browser hangs and shows not responding error.
please suggest any way to do this,what i want is expand the tree which can contains more than 5000 nodes.
I had a similar problem with kendo TreeView, when I wanted to load a tree with 30,000 nodes. The browser would freeze for a long time to load this number of nodes even when loadOnDemand was set to true.
So we decided to implement the server-side functionality for expanding nodes, and that's what you should do. You need to have 2 changes in your existing code.
Change your tree use server side Expand method.
When you call expand, you should make sure the node is expanded.
These two steps will be explained below. The thing you should know is, this way your browser doesn't hang at all, but it may take some time to complete the operation, because there will be so many webservice calls to the server.
Change your tree to use server side Expand method:
Please see Kendo UI's demos for Binding to Remote Data in this
link. Note that loadOnDemand should be set to true. In addition the server side Expand web service should be implemented too.
When you call expand, you should make sure the node is expanded:
In order to do this, there should be an event like Expanded defined in Kendo UI TreeView, but unfortunately there is none, except Expanding event. Using setTimeout in this case is not reliable, because the network is not reliable. So we ended up using a while statement to check that the node's children are created or not. There might be a better solution for this, however this satisfies our current requirement. Here's the change you should make when expanding nodes:
if (collapsedItems.length > 0) {
currentLevel++;
var $tree = $("#treeView");
var treeView = $tree.data("kendoTreeView");
var collapsedItemsLength = collapsedItems.length;
for (var i = 0; i < collapsedItemsLength; i++) {
var node = $(collapsedItems[i]).closest(".k-item")
if (!node.hasChildren)
continue; // do not expand if the node does not have children
treeView.expand(node);
// wait until the node is expanded
while (!node.Children || node.Children.length == 0);
}
ExapandByLevel(level, currentLevel);
}
You can also do the expand calls in a parallel way in order to decrease the loading time, but then you should change the way you check if all the nodes are expanded or not. I just wrote a sample code here that should work fine.
Hope this helps.
The solution to your problem is pretty simple: Update the version of Kendo UI that you are using since they have optimized (a loooooooooot) the code for HierarchicalDataSource and for TreeView.
Check this: http://jsfiddle.net/OnaBai/GHdwR/135/
This is your code where I've change the version of kendoui.all.min.js to v2014.1.318. I didn't even changed the CSS (despite you should). You will see that opening those 5000 nodes is pretty fast.
Nevertheless, if you go to 10000 elements you will very likely consider it slow but sorry for challenging you: do you really think that 10000 nodes tree is User Friendly? Is a Tree the correct way of presenting such a huge amount of data?

AngularJS app freezes when loading in new data to $scope

This is probably a difficult question to answer as I'm not sure what the root problem is here, would appreciate if someone would take a look though.
http://threadfinder.net/search%3FnameTags=jacket/0
If you continually scroll down, more items are loaded in using ngInfiniteScroll and this function:
$scope.moreProducts = function() {
if ($scope.busy || $scope.noMore){return;}
else
if (!($scope.busy)) {
$scope.busy = true;
$scope.itemsLoaded += 27;
var theQuery = $routeParams.query.replace(/\?|%3f/gi, '');
$scope.itemSearch.get({
query: theQuery,
page: $scope.itemsLoaded
}, function(data) {
if (data.posts.length <= 0) {
$scope.noMore = true;
} else {
$scope.noMore = false;
for (var i = 0; i < data.posts.length; i++) {
if ($scope.user) {
if (!($scope.user.likes.indexOf(data.posts[i]._id) === -1)) {
data.posts[i].liked = 'http://i.imgur.com/CcqoATE.png';
} else {
data.posts[i].liked = 'http://i.imgur.com/tEf77In.png';
}
$scope.posts.push(data.posts[i]);
}
}
$scope.busy = false;
}
});
}
}
(I'm using AngularJS Deckgrid for the tile layout, but I've tried disabling it and there is no big change in performance.)
If you keep scrolling, after you get to ~300 items loaded on the page performance starts to take a hit and the app freezes while new items are being loaded into scope.
I understand that perhaps it's just a fact that loading in all this data takes a big load on the page - with every 27 items loaded in (one infiniteScroll GET call) the total weight of the data being loaded in is about 30kb a pop, so at around 300 items there's ~900kb of data in scope. This data is as little as I can make it (~500 lines of JSON).
The question is:
Are there any recommendations, plugins, directives or best-use practices for AngularJS for when loading a lot of data in the page $scope?
For clarification, the app is built on Node/ExpressJS/MongoDB
EDIT: I've checked out this issue on two computers (both on OSX) and this issue is MUCH more prevalent in Chrome than in Safari. Chrome totally staggers when loading in the data, Safari is really smooth and only has any noticable lag when you get to 600+ items and even then it's nowhere near as bad as Chrome.
I took a look at the page with AngularJS Batarang, and it doesn't appear that your app is spending a lot of time in the digest cycle. If you take a look at Chrome's "Timeline" panel during the periods of UI lag, you can see the following:
(full size)
Most of the time is spent in lots of "Parse HTML." A quick Google searched turned up this Stack Overflow question, which has some answers that may be useful; in particular, the Groups post about manual string concatenation by be worth trying here. You may also consider trying to break up the big block of HTML parses into smaller chunks by adding new items to the scope in small batches (perhaps using $evalAsync or a few timers) to see if that helps.

ExtJs 4.2 ref-selector vs Ext.getcmp() vs Ext.ComponentQuery.query()

I was asked to develop a tab panel with 6 tabs, each having 30 to 40 elements. Each tab is acting as a form in accumulating the details of a person and the last tab is a Summary page which displays all the values entered in the first five tabs. I was asked to provide summary as a tab because, the user can navigate to summary tab at any instance and look at the details entered by him/ or glace the summary. i am following ExtJs MVC pattern. Payload is coming from / going to Spring MVC Application. (JSON)
Using tab change event in controller and if the newtab is summary I am rendering the page with show hide functionality.
Method 1 :In controller I have used Ext.getCmp('id of each element inside the tabs') and show hide the components in summary tab based on the value entered by the user. This killed my app in IE8 popping a message saying that the "script is slow and blah blah..." i had to click on NO for 5 to 6 times for the summary tab to render and display the page.
Method 2 :In controller I used ref and selectos to acccess all the items in tabs. I have used itemId for each and every field in summary tab. like this.getXyz().show(). I thought it would be fast. Yes it was in Google chrome. but my app in IE8 is slow compared to goolge chrome/firefox
Any suggestions regarding this and plan to decrease page render time. The summary page has more than 1000 fields. Please feel free to shed ur thoughts or throw some ideas on this.
thank you!!
I've got a few suggestions you can try. First, to answer your title, I think the fastest simple way to lookup components in javascript is to build a hash map. Something like this:
var map = {};
Ext.each(targetComponents, function(item) {
map[item.itemId] = item;
});
// Fastest way to retrieve a component
var myField = map[componentId];
For the rendering time, be sure that the layout/DOM is not updated each time you call hide or show on a child component. Use suspendLayouts to do that:
summaryTabCt.suspendLayouts();
// intensive hide-and-seek business
// just one layout calculation and subsequent DOM manipulation
summaryTabCt.resumeLayouts(true);
Finally, if despite your best efforts you can't cut on the processing time, do damage control. That is, avoid freezing the UI the whole time, and having the browser telling the user your app is dead.
You can use setTimeout to limit the time your script will be holding the execution thread at once. The interval will let the browser some time to process UI events, and prevent it from thinking your script is lost into an infinite loop.
Here's an example:
var itemsToProcess = [...],
// The smaller the chunks, the more the UI will be responsive,
// but the whole processing will take longer...
chunkSize = 50,
i = 0,
slice;
function next() {
slice = itemsToProcess.slice(i, i+chunkSize);
i += chunkSize;
if (slice.length) {
Ext.each(slice, function(item) {
// costly business with item
});
// defer processing to give time
setTimeout(next, 50);
} else {
// post-processing
}
}
// pre-processing (eg. disabling the form submit button)
next(); // start the loop
up().down().action....did the magic. I have replaced each and every usage of Ext.getCmp('id'). Booooha... it's fast and NO issues.
this.up('tabpanel').down('tabName #itemIdOfField').actions.
actions= hide(), show(), setValues().
Try to check deferredRender is true. This should only render the active tab.
You also can try a different hideMode. Especially hideMode:'offsets ' sounds promising
Quote from the sencha API:
hideMode: 'offsets' is often the solution to layout issues in IE specifically when hiding/showing things
As I wrote in the comment, go through this performance guide: http://docs.sencha.com/extjs/4.2.2/#!/guide/performance
In your case, this will be very interesting for you:
{
Ext.suspendLayouts();
// batch of updates
// show() / hide() elements
Ext.resumeLayouts(true);
}

backbone.js and local storage with multiple browser tabs / windows overwrites data

just a very short question on using Backbone.js with LocalStorage:
I'm storing a list of things (Backbone collection) in LocalStorage. When my website is open in multiple browser windows / tabs and the user in both windows adds something to the list, one window's changes will overwrite the changes made in the other window.
If you want to try for yourself, just use the example Backbone.js Todo app:
Open http://backbonejs.org/examples/todos/index.html in two browser tabs
Add an item 'item1' in the first tab and 'item2' in the second tab
Refresh both tabs: 'item1' will disappear and you'll be left with 'item2' only
Any suggestions how to prevent this from happening, any standard way to deal with this?
Thxx
The issue is well-known concurrency lost updates problem, see Lost update in Concurrency control?.
Just for your understanding I might propose the following quick and dirty fix, file backbone-localstorage.js, Store.prototype.save:
save: function() {
// reread data right before writing
var store = localStorage.getItem(this.name);
var data = (store && JSON.parse(store)) || {};
// we may choose what is overwritten with what here
_.extend(this.data, data);
localStorage.setItem(this.name, JSON.stringify(this.data));
}
For the latest Github version of Backbone localStorage, I think this should look like this:
save: function() {
var store = this.localStorage().getItem(this.name);
var records = (store && store.split(",")) || [];
var all = _.union(records, this.records);
this.localStorage().setItem(this.name, all.join(","));
}
You may want to use sessionStorage instead.
See http://en.wikipedia.org/wiki/Web_storage#Local_and_session_storage.
Yaroslav's comment about checking for changes before persisting new ones is one solution but my suggestion would be different. Remember that localStorage is capable of firing events when it performs actions that change the data it holds. Bind to those events and have each tab listen for those changes and view re-render after it happens.
Then, when I make deletions or additions in one tab and move over to the next, it will get an event and change to reflect what happened in the other tab. There won't be weird discrepancies in what I'm seeing tab to tab.
You will want to give some thought to making sure that I don't lose something I was in the middle of adding (say I start typing a new entry for my to-do list), switch to another tab and delete something, and then come back I want to see the entry disappear but my partially typed new item should still be available for me.

Categories