I am currently working on a web application with MongoDB, and it populates the browser with pages of content.
I am aware of the two methods of paging (seen here: https://scalegrid.io/blog/fast-paging-with-mongodb/):
1) .skip() + .limit()
2) .find({_id > last_id}).limit(n)
I understand the performance issues with the first solution, so that option was out of the picture.
The second solution works great, but only if the results are sorted by _id. If you sort by a non-unique field such as a date, there can be more than one document with that date so some documents may be excluded from the results.
Also, the user must be navigating through the pages linearly because there is a last_id variable that constantly needs to be updated and is determined by the _id of the last document from the previous page. If the user were to jump from page 1 to 7, page 7 would show the results meant for page 2 because the last_id would hold the _id from the last document of page 1.
So my question is, how would one implement paging if we were sorting by a non-unique field (such as a Date or a firstName), and how would we allow users to jump multiple pages at once?
Any tips would be greatly appreciate, thank you!
The comment is correct, you should be fine with skip unless you know you're going to have a lot of documents. But even if you don't expect much but then get enormously successful/popular you don't want your code to break just is it gets used by millions of users.
If you sort by date and then by id you can do the following:
.find(
{
date:{
$lte:lastDate
},
_id:{
$lt: lastId
}
}
).limit(n)
Related
I'm working on a project wherein I am using a news api to show the contents of a specified user typed topic,for eg "technology" , but the problem is sometimes i get thousands of results and all are displayed on a single page,but i want the number of pages of the search result to change dynamically with the typed topic,
for example tech may have 100 pages while sports may have 500 pages.
How do i do this for my website?
Any sources,links,source code might help me a lot.
Thanks.
Use Aggregate paginate module (https://www.npmjs.com/package/mongoose-aggregate-paginate) for pagination and it will take page number and limit as an input and will provide you with that limit of results.
It will also help you to store how many pages are there with the limit decided.
I think you should look into 'pagination'. Basically, you set a number of items per page to return, divide the total number based on the items set per page, and pass in the starting and last index for fetching the data (like the first 2 parameters in a Array.slice method).
I looked at "Generating HTML Page on the fly" on this website, but most of it was over my head.
I have a 2 part question that I would like assistance with please.
I want to fill a narrow vertical container, <div id=”counter”> with the numbers 1 .. <xx>.
<xx> is determined by the record count of a database, filtered “on-the-fly”, by the user choosing a category (no problem there – I have an SQL background)
Eg. Category1: 1 .. 200
Category2: 1 .. 6
These numbers could change over time, as I want to allow users to add content to the database (vetted of course).
I have viewed a number of website source code pages (of similar ideas eg. Surgicalexam.com), but they have all been hard-coded and are distinct pages per category.
I have created a small website of a similar nature to that, hard-coding all the images and links, but I am looking at 3000+ images (as a starting point here), and they differ per page.
I have created this scenario many times in stand-alone apps and from past experience, I thought perhaps, I could create a javascript routine which would use a loop to
• print the numbers to the <div> using the getelementbyID ( ).
• Fill an array with the record number, a title and an image link.
Question 1: Is this possible or am I beating a “dead horse”?
If it is possible, any suggestions would be gratefully accepted.
Part 2:
My current idea is that, as the user hovers the mouse over any number, a mouseover ( ) event will occur that will read the appropriate array record and display the <title> as a tool-tip-text.
If the user clicks the number, a function (I have yet to write) will read the appropriate array record and attach the image link to an <a> tag, and subsequently display the appropriate image to the screen.
Question 2: repeat of question 1.
I have viewed a number of website source code pages (of similar ideas eg. Surgicalexam.com), but they have all been hard-coded and are distinct pages per category.
Why are you so sure about that? You can't see php-code, because it is executed on the server. There is no way to know if it was hardcoded or by php
Answer:
It is possible.
If I understand this correctly, you want to read some data from a database and if the user clicks / hovers something, you want to load more data?
You have to splitt this into two things:
Load data with PHP from the db (Server side)
If you want a live, visual feedback you need JavaScript (and/or CSS3) to do changes. (Client side)
One possible solution is to create a API with php (maybe REST-like) and then call that api with JavaScript.
You could also do everything with PHP but this will require a reload of the website on every click. PHP cannot do changes On-The-Fly.
First of all you should learn the basics about web development.
And most important: If you decide to learn Web-Programming: learn about security, too. For example things like Cross Site Scripting and SQL-Injection. Never trust data coming from a client (e.g. JavaScript)!
I need some help with oracle apex. The thing which I want to do is the following:
I have table with some data about people. So each row describes exactly one human. And I want to show some more information about certain human. For example, list of shops he or she has visited. Such data provided by other tables.
I see it in this way: right click on table with people on certain row -> select option (what kind of info to show) -> execute stored procedure and show new page with data table (e.g. list of shops). But how can I implement it?
I've already found this plugin. Now I can execute some JavaScript function after right-click. But how can I execute stored procedure and show new page?
I'm new in apex, any help would be appreciated.
You're trying to reinvent the wheel. You're new to apex. Have you taken a good look at the documentation?
Start out where everyone else has to start: at the beginning. Report + form. Column links.
There is ample help available to someone new to apex.
The 2-day developer guide, running you through some of the
basics of apex and a good familiarization.
Get a workspace at apex.oracle.com
Each workspace starts with the sample database application, based on
products and customers. You can view and edit this application and
thus you can glean plenty of information from it.
Furthermore, there are the packaged applications, many of which offer
good basic solutions to common situations. Again, you can glean a lot
of information on them, and they are even editable after you unlocked
them.
After you are familiar with the basics, you can look further ahead. What you are asking is simply too much for someone new to the matter. You even want to implement a jQuery plugin straight away. You're talking ajax. It's great if you know those subjects and they'll be of plenty of value to you, but it just seems you don't even know how to present and fetch your data.
A good start would be to make a report and a form. In the form you can then add some classic (or an interactive) report(s) to represent associated data.
It's possible. First of all, assume that each row contains unique identifier ID. You have to add hidden item to the page which would contain additional info about certain row. Let's name it P1_ID. Then add the following JavaScript code to page which contains initial data (in example from question, page with table with information about people):
function TestFunction(action, el, pos) {
var id = $(el).children('td[headers="ID"]').text();
var href = 'f?p={APPLICATION ID}:{PAGE_NUMBER}:&SESSION.::::P2_ID:'+id;
window.location = href
}
Function name should match name in plugin settings. Example: link
Replace APPLICATION_ID and PAGE_NUMBER with actual values according to application. PAGE_NUMBER is the page number which contains additional info about row.
Then you can add some reports to the page with additional info and use ID parameter to select information about certain row.
The only problem is that plugin mentioned in question stop working after table refreshing. For example, if we filter data in table then no menu on right click will be shown. I don't know how to fix it for now. Any ideas?
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I've been working with tables and a huge amount of data. There are a tables in my website with 10 thousand rows. This table has dynamic search, filters, etc. I've been using pure JavaScript considering performance, but it gets laggy with this amount of rows.
Do you guys know any alternative for pure JavaScript with better performance?
EDIT> I REALLY need to load the 10 thousand rows at once. I can load them all in the browser in 5 seconds. The main problems are the filters and search...
EDIT2> The search is dynamic. I can search by name and filter it by first character.
I've been working on it for months...
SEARCH:
search when the field has more than 3 characters and only when its length and characters has changed (onchange event on input may trigger multiple times when a character changes, so I make sure it only triggers once using some verifications)
each row that matches the searched string is coppied to another table. The original table is hidden and the new one is displayed.
when the user changes the search field or cancel the dynamic search, the new table is erased.
Conclusion: it's faster to create a new table with desired elements than hide the undesired ones.
FITLER
The rows are actually in 25 tables (A to Z + non-alphabetical characters)
When you select a character, only that table is shown
Conclusion: it's faster to hide a whole table than hide the
undesired rows
Thanks for the repplies. I've edited with some extra info so we can narrow the possible solutions...
I'm asuming you get the contents from database and load with with something like PHP (I'm going to asume PHP for now)
You could make the Javascript make an ajaxcall to a php-file which does the filtering (actually you should make the database do it, a lot faster!) and place the resulting table back on screen.
A faster method combined with the above might be this: Get all id's on the initial rows in an array and save those (in a session might work pleasant).
When you have to filter, don't make PHP get the whole table, just apply the filter to only the stored id's and send javascript the matching rows.
Then make javascript do something like this:
- set all visible
- set resulting id's to hidden (hidden in favor of remove, because I think a user might perform multiple filter actions?)
Another idea just popped in my head: If you don't need to display it on load, you can start the initial load with all tables hidden and a message "please search to display".
A common technique to handle this case is to load the data in memory or a subset of the data, and recycle your table rows such that you aren't actually ever creating thousands upon thousands of rows. You can get creative with this and create a web interface that seemingly scrolls endlessly but in reality you are just reusing dom elements and shuffling them around.
Most well-built data grid widgets whether they are on the web, mobile or even a desktop interface will employ this technique to handle your particular problem.
In most cases a user will never actually find themselves benefiting from seeing 10's of thousands of rows of data at once anyway.
fetch from the server only the things to be seen by the user, Like everyone has pointed out 10,000 rows needn't be there on that page.
you can use the concept of pagination and for every page few rows are fetched and shown . JQuery's Ajax is capable of calling the server side function to fetch rows to add them to your page.
don't know any backend details here, but in struts framework there is display:table tag and I believe in .NET framework there is GridView for pagination in the client side that you can look into
I'm just a student, newly joined to the community. Take what I say with a grain of salt.
I'm not sure why everyone is so much as blinking at the ten thousand rows business when we're measuring modern personal computers' memory in gigabytes.
Alright. I'm going to assume that what you're doing needs to be done in the browser, and so you can't switch to doing native code. In that case, looking for an alternative to Javascript won't get you much of anywhere. In the context of a browser, you're looking at an interpreted language. In terms of number of instructions the program ultimately has to run, the difference between one language or another is negligible in the face of how long it takes to be interpreted. Besides, Javascript has gotten nicely polished over the years.
So never mind that. There's a much more important thing to consider here, and it applies no matter what you're programming in or on: The cache(s). Igor Ostrovsky explains it beautifully; read it until you grok it.
So I'm guessing you have objects that would stringify to something like, "obj1 = {field-1:'a', field-2:'b', ..., field-n:'n'}". And you can select a field-i to sort by. The trouble with this is that when you sort by field-i, you're loading the entirety of obj1 into the cache, even though you don't need it. What you really want to do is load the field-i's forobj1, obj2, obj3, ..., objm all at once. So you look at an array, stringifying to something like: "field1 = [refToRow1, field1inRow1, refToRow2, field1inRow2, ..., refToRowM, field1inRowM]".
You might not be able to do fit all M rows in the cache, after all M==10000! But you can group them together into chunks that you could reasonably expect a cache to manage. Anyone got a good number for this? Say, 64kB? So you figure for each i in M you've got a reference, and a field that's probably just a reference to a short string (it'd be better if you could have the string itself right there, but I don't think Javascript works that way). So that's 8B? 8B*1024 = 64kB? Hell, if that's right, you could fit it all into the cache in two chunks, which means you'd want to do it in 4.
So now you've got a collection of smaller arrays, and you want to sort them. This is a classic application for B-trees. And while having a separate B-tree for each and every column in the table may seem like a lot, it's not.
Okay, so that handles sorting. You tell it to sort by a column, and the truth is it's already sorted! You're just repopulating the visible table using a different b-tree. You still need to handle filtering, but that's fine. You do some cache juggling as you find something to display and follow the reference to get the other fields, but I'd still expect this to go fast since you're skipping over so many rows.
Normally, I would say if you want to speed things up, look into multiprocessing. But I think browsers are still working to make that a thing with their Javascript implementations. Plus, while it would be well-suited for sorting, it would be a lot of effort to make it useful for the filtering part, and I expect you can do fine without.
I hope this isn't too scatter-brained, and that it gives you some ideas. Good luck!
I have a chat app where it shows users who are online (username + profile pic). I have an ajax poll that basically checks to see which users are still online, and refreshes the list automatically. Also, the list is ordered based on last activity.
The way I've been doing it is:
Get list of current online users
Clear existing elements
Re-add them (will be ordered correctly since the returned list from step1 is ordered)
This works fine in Chrome, but I notice in Firefox that it is causing a "flickering" effect while the images get re-added.
What is the best way to do this? It seems overly difficult to create an algorithm check which elements exist, if they are in the right order and move them, etc. Thoughts?
How often do you poll to see if users are still online?
I think the best way may be to give the user records unique ids so you can then check the list of users that were online against the new list of users that are now online.
fade away the users that have left and fade in any that have logged on.
It will be a much more elegant solution and it solves the problem you are having.
Firstly, I would try to "cache" the images separately, using the "preload" technique. That's when you create an Image object and set it's src to the URL of the userpic. Then you store all those objects in a global array. This will prevent the browser from getting rid of the images when they are no longer on the screen, so that it will not have to load them again when you reload the list.
If that doesn't help, I would actually reuse the existing list elements. I would just go over the elements, one by one, and replace their content with the appropriate content from the list. If I run out of existing elements in the process, I add new ones. If any elements are left over when the list ends, I remove them. This is a bit more complex, but actually not as complex as it looks at first glance.