I'm trying to use selenium to scrape a webpage, and one of my elements dynamically loads a finite amount of content (IE not like twitter where you can just keep scrolling) when I scroll to the bottom of it.
Now I could simply do something like
var scrollbar = document.getElementsByClassName("dataTables_scrollBody")[0]
var last = -1;
var curr = scrollbar.scrollHeight;
while (curr != last) {
last = curr;
scrollbar.scrollTop = curr;
curr = scrollbar.scrollHeight;
}
And in fact, that's what I'm doing. However, the actual load operation on the data table takes so long that this script exits by the time the load finishes, meaning I only ever get halfway down the scrollbar.
What's a good way to make sure that I've scrolled all the way to the bottom of the scroll bar?
This question is different from Scroll to bottom of div? because I don't have access to the raw HTML, and it's different from Scroll Automatically to the Bottom of the Page because my page is dynamically loading content.
Perhaps I could force an ajax load of all the content in the panel, but I don't know how to do that, and I don't want to wade through miles of minified code to figure out how the page owner does it.
Here's an ugly solution:
scroll_elem = document.getElementsByClassName("dataTables_scrollBody")[0];
table_elem = document.getElementsByClassName("dataTable")[0];
function scrollToBottom() {
scroll_elem.scrollTop = scroll_elem.scrollHeight;
};
new ResizeObserver(scrollToBottom).observe(table_elem)
Basically how this works is we have an object watching the element we want to scroll to the bottom of. If and when it resizes, it instantly scrolls to the bottom of the new window. This works well enough, then you could use the sleep function KunduK recommended in comments to wait for the process to complete
Related
I'm writing an Chrome Extension for Google Calendar, which adds an div under the header section of Google Calendar.
For simplicity, this is what essentially happens in Chrome Extension content script, you can just paste this code in the console of calendar.google.com to check what happens:
let header = document.querySelector("header[role='banner']");
let appContainer = document.createElement('DIV');
appContainer.style.height = '200px';
appContainer.style.backgroundColor = 'red';
header.parentNode.insertBefore(appContainer, header.nextSibling);
The problem I experience is, after creating or canceling the creation of an event in calendar view, the window scrolls up and doesn't show the page header now.
Anyone have an idea how to keep page steady after creating or canceling of an event in calendar view, and keep the div I append via Chrome Extension content script too?
EDIT: Here are screenshots of how it looks like before the event creation/cancel and after:
Normally, without your extension, header.parentElement contains two elements that count towards header.parentElement.clientHeight: header, and the div that contains the calendar itself:
(It also contains a few floating buttons and hidden divs and stuff, but those aren't important here.)
These two elements, header and calendar, have heights that are calculated specifically so that header.clientHeight + calendar.clientHeight is equal to the height of the page. Under normal circumstances this is perfect since it means there's no need for scrolling.
However, in your case, that you add an extra div, which pushes calendar down:
Normally you would be able to scroll down yourself to see the bottom of calendar, but since the scroll bar is disabled, you can't. However, when you create an event, your browser sees that you are trying to access the bottom of calendar, so it scrolls down automatically to show you the bottom of calendar. Since the whole page is now scrolled down to make the bottom of the page visible, the top of the page is now invisible, resulting in the behavior that you describe.
The way to fix this is to make adjust the height of calendar so that header.clientHeight + appContainer.clientHeight + calendar.clientHeight is equal to the height of the page, rather than just header.clientHeight + calendar.clientHeight. This can be done by adding the following code:
//Getting calendar can be a bit tricky since it's just one dive among others.
//While it does have a CSS class, I don't know how that name is determined or if it will change in future versions of Google Calendar, so it's best not to use it.
//What I do here instead is that I select the first child of header.parentElement that has parts outside of the page frame
const getCalendar = () => [...header.parentElement.querySelectorAll(":scope > div")].find(d => {
let boundingRect = d.getBoundingClientRect();
return boundingRect.y + boundingRect.height > document.body.clientHeight;
});
let calendar = getCalendar();
//This function adjusts the height of calendar
const adjustHeight = () => {
calendar = calendar || getCalendar(); //calendar may be null in the beginning before all the elements were set, so if it's null try to select it again
if(calendar != null){
calendar.style.height = (calendar.parentElement.clientHeight - header.clientHeight - appContainer.clientHeight) + "px"; //Adjust the height of calendar
}
};
window.addEventListener("resize", adjustHeight); //Adjust the height whenever the window gets resized
window.addEventListener("load", adjustHeight); //Adjust the height on page load
I am using this way to load old content from messages when user scrolls top.
$("#Default3").scroll(function() {
if($("#Default3").scrollTop()<1) {
// load 10 more old data to div
});
});
However, if you scroll to top, it just loads for one time. You need to scroll a little bottom and then scroll top to load 10 more again. So I checked the facebook messaging, and noticed that they load more old content if the scroll is upper than 50% of the height. What is the correct way for doing that ?
You can scroll down one pixel, so the user would be able to scroll up again:
$("#Default3").scroll(function() {
if ($("#Default3").scrollTop() < 1) {
// load 10 more old data to div
$("#Default3").scrollTop(1);
}
});
Situation:
Suppose we are reading the content somewhere down the page that is built to be responsive. Suppose also that we resize the browser window to a smaller size and that some content above get extended down due to the thinner width, hence making the whole page longer. Then as we resize, whatever content we are looking at will get pushed down the page accordingly.
Example:
Suppose we were to look at the Helper classes section in this page. Then shrinking/expanding the window a sufficient amount moves the bit we were reading down/up the current view.
Prompt:
Is there any way we can fix this? I.e. maintain our current view of the page regardless of what happens to the contents above it when we resize the window.
Thoughts:
I am thinking that we could at least start with javascript and put an event on window resize. Then automatically scroll the page to the top-most element that was in our view on event fire. I don't know how this will affect the performance, however, especially in bigger pages.
There's also the problem of refering to the top-most element in current view. The top of our current view might be cutting off the top portion of some elements, not to mention that there's usually more than 1 element layered on top of one another at any point within the page. The notion of top-most element I've mentioned is not very well-defined :(
Also rather than a problem of responsive design in general, instead it seems to me like this is a problem with the default scrolling behaviour of web browsers? Or perhaps I am missing some circumstances where the current behaviour is desirable.
Edit 2 4
Updated fiddle (see fullscreen result) based on Rick Hitchcock's solution's solution.
With jQuery:
//onresize:
var scrollAmount;
if (topNode.getBoundingClientRect().top >= 0) {
scrollAmount = $(topNode).offset().top - topNode.getBoundingClientRect().top;
} else {
scrollAmount = $(topNode.offset().bottom - topNode.getBoundingClientRect().bottom;
}
$(window).scrollTop(scrollAmount);
The fiddle is acting a bit weird even in the same browsers, I've uploaded the same script using a free hosting here.
Still need to incorporate the IE, Opera and Safari fix for elementFromPoint.
Edit 3
Thanks for all the help, Rick Hitchcock. Welcome to stackoverflow, by the way :)
The discussion is turning into cross-browser compatibility issues so I've accepted your answer since we've pretty much got the answer to the original question. I'll still be fixing up my implementation though. The focus being cross-browser issues, topNode criteria, and topNode cut-off handling.
An edge case
While playing around with it, I noticed that when we were at the bottom of the page in a small viewport, then switch to a larger viewport (let us assume now that some more elements that were originally above the element we saw now came into view due to shorter container from wider viewport) the window cannot always lock the topNode to the top of the viewport in such a case since we've reached the scroll bottom. But then switching back to the small viewport now uses a new topNode that got into the viewport during the switch.
Although this should be expected from the behaviour being implemented, it is still a weird side-effect on scroll bottom.
I will also be looking into this in due course. Initially, I am thinking of simply adding a check for scroll bottom before we update topNode. I.e. to keep the old topNode when we've reached scroll bottom until we've scrolled up again. Not sure how this will turn out yet. I'll make sure to see how Opera handle this as well.
Here's what I've come up with:
(function(){
var topNode;
window.onscroll=function() {
var timer;
(function(){
clearTimeout(timer);
timer= setTimeout(
function() {
var testNode;
topNode= null;
for(var x = 0 ; x < document.body.offsetWidth ; x++) {
testNode= document.elementFromPoint(x,2);
if(!topNode || testNode.offsetTop>topNode.offsetTop) {
topNode = testNode;
}
}
},
100
)
}
)();
}
window.onresize=function() {
var timer;
(function(){
clearTimeout(timer);
if(topNode) {
timer= setTimeout(function(){topNode.scrollIntoView(true)},10);
}
}
)();
}
}
)();
If there were a window.onbeforeresize() function, this would be more straightforward.
Note that this doesn't take into account the scrolled position of the element's textNode. We could handle that if only the height of the window were resized. But resizing the width would generally cause reformatting.
This works in Chrome, Firefox, IE, and Safari.
Edit
How it works
The code's closures make variables private, and the timers prevent the code from running constantly during scrolling/resizing. But both tend to obfuscate the code, so here's another version, which may aid in understanding. Note that the onscroll timer is required in IE, because elementFromPoint returns null when it used in onscroll event.
var topNode;
window.onscroll=function() {
setTimeout(
function() {
var testNode;
topNode= null;
for(var x = 0 ; x < document.body.offsetWidth ; x++) {
testNode= document.elementFromPoint(x,2);
if(!topNode || testNode.offsetTop>topNode.offsetTop) {
topNode = testNode;
}
}
},
100
)
}
window.onresize=function() {
if(topNode) {
topNode.scrollIntoView(true)
}
}
topNode maintains the screen's top-most element as the window scrolls.
The function scans the screen left to right, along the 3rd row: document.elementFromPoint(x,2)*
It doesn't scan along the 1st row, because when IE does scrollIntoView, it pushes the element down a couple pixels, making the top-most screen element the previous element. (Figured this out through trial and error.)
When the window is resized, it simply positions topNode at the top of the screen.
[*Originally, onscroll scanned left to right along the 11th row (in pixels) until it found an element with just one child. The child would often be a textNode, but that wouldn't always be the case. Example:
<div><ul><li>...<li>...<li>...</ul></div>
The div has only one child – the ul. If the window were scrolled to the 50th li, scanning left to right would incorrectly return the div due to the inherent padding of lis.
The original code has been updated.
]
Ok - here is what I am trying to do. I was looking online for a cool timeline that I can purchase - allowing zoom in zoom out, posting of events on it, and so on. However, all the examples I found are either too expensive or just downright useless.
So, I have decided to create my own, but there are two elements that I am having trouble with.
1) Converting the wheel scroll to left-right scrolling (so not up-down). I can't seem to find an easy and quick way to do this.
But, more importantly..
2) I need the area I will be showing the timeline on to automatically expand as I go about my scrolling. So, if I scroll down, it will add an "equivalent" area on the right, and down, on the left. So I was thinking like making an iFrame (already use these) and when you scroll it just adds more "timeline" on the left or the right, loads what ever it needs to load from the DB/list of events, and so on, ad infinitum, thus creating an ever-expanding list of blocks that are time-sized.
If I can do the two things above, then I am set - the rest (loading/positioning) I can figure out - just these two things are eluding my imagination and ability to find an answer.
Basically you need a horizontal infinite scroll script.
Take this plugin I wrote:
$.fn.hScroll = function( options )
{
function scroll( obj, e )
{
var evt = e.originalEvent;
var direction = evt.detail ? evt.detail * (-120) : evt.wheelDelta;
if( direction > 0)
{
direction = $(obj).scrollLeft() - 120;
}
else
{
direction = $(obj).scrollLeft() + 120;
}
$(obj).scrollLeft( direction );
e.preventDefault();
}
$(this).width( $(this).find('div').width() );
$(this).bind('DOMMouseScroll mousewheel', function( e )
{
scroll( this, e );
});
}
Initialize it with:
$('body').hScroll();
Makes your website a horizontally scrollable website.
Your content div must be wider than your body (ex. 3000px).
As for the infinite scrolling effect you pretty much gotta do that your self because I can't know what kind of data you'll input. But I'll explain.
Your children elements in the content div must be floated to left. (every new appended div will not go to new line).
Set an interval to check if the user's scrollLeft position is near the end of the content (just like pinterest and similar site).
function loadNewData(){ /* Your search for data and update here. */ }
setInterval('loadNewData', 500);
search for new data according to your last one with AJAX. When you get new data, append it into your content div (in a div that's floated left, as I wrote previously), and mark it as your last item.
Maybe you could use your ID to mark the last item on it's div.
<div data-id="467" class="item"> // your data here </div>
You can fetch it with
$('.item:last').attr('data-id');
with jQuery.
I want a solution either using a hashtag pointing at the name of an anchor tag or javascript.
The javascript I am currently using looks like this window.scroll(0, 20000);. The problem is that this causes the window jerk down when a user arrives on the page.
I know there are jQuery animations that make this movement more gradual. However, what I want is something that makes the movement of the window imperceptible to the user. I want to be as if the user landed at the bottom of the page.
The problem you face is that you wish to go to the bottom of your page which has not loaded yet. I would consider loading the page in a hidden format then show it when it has all loaded and after scrolling the user at the location you want. Use the focus or scroll to methods.
Take a look at the filament group website.
http://filamentgroup.com/
they hide the page with a loading screen until it is ready.
This way there is no jerk.
Hope this helps.
In loop it works, if the page is fully loaded and shown:
for(var n=0;n<1000;n++) window.scrollBy(0,20);
(Notice that 20*1000=20000, which was the original place to scroll.)
Teemu's answer doesn't seem to work for me (it goes straight to the bottom, making the loop with scrollBy stepping invisible), because it doesn't implement a delay.
If you mean to animate from top to bottom of the page in a 1000ms, try something more like this:
for (var n = 0; n < 1000; n += 1) {
setTimeout(function () {
window.scrollBy(0, document.height / 1000);
}, n);
}
That will give a 1 second (1000ms) animation, scrolling to the bottom in roughly 1000 steps.