Automatically Load All Webpage Data/Automatically Scroll to End - javascript

Take https://www.instagram.com/instagram as an example. If you click on 'Followers', a list of all the followers is loaded, but only a few are loaded at a time, and you need to continually scroll until all data is loaded.
Is there a way in Javascript to automatically load all the data, or maybe even automate the scrolling for that particular div?
Thanks.

A few things to consider.
The method Instagram is using to show some followers until you scroll to get more is called Infinite scrolling.
Basically, all we have to know is that this method is implemented on the client, that is, in the browser, using JavaScript.
The first solution is: if you can reverse engineer the Instagram code (minified I suppose) and find the good methods to call, you can force fetch new items even if you didn't scroll.
On the other hand, another technique would be to constantly simulate a scroll to the end, I let you refer to this answer on how to.

Related

Trigger some JavaScript function with selenium on a webpage to load new elements

I'm trying to get some data from a webpage:
https://www.cabotcorp.com/solutions#product-index
On this webpage, whenever you scroll down to the end, it takes a couple of seconds and then loads a new set of items.
When that happens something gets added to the HTML which I want to look at with selenium in python.
So my question is pretty general:
Is there a way to force the function that loads the new elements?
My first idea is as follows:
driver.execute_script("arguments[0].scrollIntoView();", driver.find_element(By.CLASS_NAME, "some class at the bottom of the page"))
This is the code I would use to let selenium scroll down to some element on the webpage. After that I would just sleep for a set amount of time to assure that the new elements loaded in.
However, I noticed that this webpage uses "ddscrollspy"
https://github.com/dynamicdriverepo/ddscrollspy
Is there a way to use selenium to trigger a new set of items?
I'm basically looking for a function that triggers certain Javascripts that are available on a certain webpage.
Edit:
A friend suggested looking for triggers:
This is what I found in Chrome:
So I now have access to the code that gets executed whenever something new loads. Now I just need to know how to trigger that code on demand.
Thanks for the help.

Website Infinite scrolling - return scroll position after user 'go back' in browser

Quite often on pages with infinite scroll you might face scenario:
you're scrolling a lot,
then you click some link from inifinite list,
you dont like what you've clicked
so you go back
you'd like to continue scrolling from the point you've left.
but all items you were scrolling are not there and you need to scroll everything you've already seen again.
How do you handle those situations? Do you know any solution for that? Is there any way to save state of page or at least know what this situation was and load proper ammount of infinite list and scroll user to last seen item.
While scrolling when a certain condition is true the new content to be appended to the page has to be loaded somehow, for example you call some page with an ajax call. Let's say every time you load more data you load ten items. So your current state could be the amount of times you loaded 10 more items. You could save that information e.g. by using an anchor:
www.site.tld/index.php?id=999#load_counter=5
Another possibility would be to save it with cookies. You could try at which time the "saving" of that data is the best. E.g. after every AJAX call or when leaving the page using the onbeforeunload event.
When your page is loaded you can check if there is that kind of metadata available in the cookie or the url. If there is you can load exactly that content either by sending an AJAX request that loads the list elements that where loaded before. In my example this would be
5 * 10
because the load_counter is 5 and on every load there were loaded 10 more items. You also can reconstruct the position. Either by also saving that information of how much the user has scrolled or by just guessing assuming that the user was looking at the last bulk of loaded items. E.g. you could scroll to the element (5-1)*10. That's the first element of the last bulk.
This is a fairly unspecific, open-ended question, but let me share some thoughts:
One reason the page you're returning to is reloading (and thus resetting) is because it prohibits the browser from caching it. If the page containing the scroll doesn't absolutely have to have pragma:no-cache etc etc, then try without. This may already solve the problem, because the browser is allowed to return to the last known state.
If this doesn't work or you need to make sure you have fresh data on each page load, then the solution will depend on your content and your framework. Maybe you can store the state server-side in the session and rebuild the page accordingly, or you can set a cookie with a reference and let some client-side script poll up to that point when you reload.
Other than that, all I can say is that most infinite scrolling frameworks simply say "well, this is a problem", period. An alternative would then be to use explicit pagination.
This is a very generic approach and will need customization depending on what kind of front-end framework is being used.
scrollTop maintains the vertical scroll position, so if you find a way to memoize scrollTop, you can always come back to the same position where you last left.
// get reference to your scrollable div
const element = document.getElementById("scrollableDiv");
// store element.scrollTop in local storage.
windows.localStorage.setItem('scrollTop', element.scrollTop);
// get scrollTop value once back to the previous page
const scrollTop = windows.localStorage.getItem('scrollTop')
// apply scrollTop to your scrollable div, to take scroll thumb to the exact position where it was left before
document.getElementById("scrollableDiv").scrollTop = scrollTop;

How can I find the name of the event listener that Twitter uses to load more tweets both when you scroll to the bottom and when you load new tweets?

I'm playing around with a Chrome extension for Twitter for a fun project, and I'd like to go through all the tweets that are shown on the page--even if more of them load.
So if the user presses the new-tweets-bar, I want my function to handle those new tweets; and if the user scrolls all the way to the bottom (thus loading older tweets), I want my function to handle those.
I thought the most straightforward way would be for those events to re-call my function, either only for the new tweets (optimal) or for the whole page again.
But how do I figure out what the names of those events are? I've looked at Firebug and its profile command; and at Allan Jardine's Visual Event 2, but neither of them were really that helpful to me in this case.
1) Am I thinking about this the wrong way? Is there a better way to go through all new tweets on a page?
2) Is there some way to get the name of the triggering event(s)?
3) Do you have any advice on using Firebug or Visual Event?

Loading contents dynamically

I have product listing page, which displays all the products which satisfy the search criteria. And these could be any number of products.
What I want is something like FB, that I display only first 5-7 products and as the user scrolls down, the products should be loaded dynamically.
I'd consider switching to jQuery or Mootools as JS libraries if you want to do this - both have native support for the infinite scroller concept as it's commonly called. It's not that hard to implement yourself though, mainly a matter of keeping track what you loaded last, and installing an onScroll event to detect when the bottom of the page is reached.
Here's a good tutorial using native JS to implement it, both server and client side. You'll need to replace the XHR invocations by the proper Prototype alternatives yourself (or not, wouldn't really matter).

Is it possible to write an "Infinite Scroll" javascript that can handle the back button?

Go here: http://www.infinite-scroll.com/
Play with the infinite scroll. Notice that you can't click a link and then hit "back". It has glitches when you do that.
So, I plan on writing my own Infinite scroll.
When the user gets to the bottom, load an AJAX call. Then do a JQuery "append" the result to the div. Then, call my other functions on those elements. (I set the background images of all those elements in javascript).
Will this work? If I do that...will I be able to handle the back button?
I considered the very same thing on a project I worked on. One option I thought about was to allow the back button to return to the place where the link was clicked (just like normal browsing).
Firstly, you need to record what page of the infinite scrolling your on so that you can load up to that section again. You can do this with some clever playing around with the window.location.hash value. If you look at my answer to this question, I explain in more detail how to do it in plain JavaScript or jQuery using Asual's address plugin.
The basic part of it would like like this:
// Whatever you're using to load the next page
function scrollToNextPage(page){
// Your infinite scrolling stuff
$.address.parameter("currentPage", page);
}
$.address.externalChange(function(){
var page = $.address.parameter("currentPage"),
top = $.address.parameter("linkTop");
loadAjaxUpTo(page);
$('html, body').animate({ scrollTop: top }, 500);
});
// Set a parameter for the location of a clicked link
$("a").live('click', function(){
$.address.parameter("linkTop", $(this).scrollTop());
});
I didn't implement the last bit (getting the position of the link clicked) but I can't see why it wouldn't work. It would cause the window to scroll nicely to where you where. You can always set it with an anchor to the loaded page instead (but when you scroll to it, it will always go to the top of the page).
A couple of points though, I wouldn't recommend this. At least for the project I was doing, it wasn't really necessary. In my project, we expected the user to go back to change their options and figured that scrolling wouldn't be a problem (although we didn't have that many pages to scroll through). The AJAX (and your JavaScript to set the images) needs to be loaded and executed again which takes a lot of time depending on how many pages you need to reload. On top of the time to scroll to the link (you could just window.scrollTo but you don't get animation so everything is very jerky. You could always just load the page the person was on and forget about the previous pages, but you're still breaking the user experience that way. Or (what I tried) was to implement a two way infinite scroll. So it would load the page the user clicked from and prepend the previous pages if they scrolled up - this was too much work for what it was though.
Another point is that, if you were to still do it, you want to use a GET request to get your pages (and make sure the cache isn't set to expire immediately). I found that pages requested by Ajax with the GET request would take from the cache (at least on some of the browsers I tried). Sending your details over a POST will always ignore the cache.
The only way to be able to handle the back button is to alter the location hash (the bit after the # symbol). Handling pressing of the back and forward buttons can be done in two ways:
Have an <a name="bookmark1"></a> which is unique to each section added, and change the location.hash to match it.
Use a setInterval with a short time like 100ms to watch the location.hash for changes and go to the required part of the page. You have to do this because you can't actually detect when the back/forward buttons are clicked.
This is an implementation of the second method to show you how it works. In this, each ajax request contains <a name="ajax-section-1"></a> and ajax-section-2 etc.
function addToBottom(htmlToAdd) {
$('#main-div').append(htmlToAdd);
window.location.hash = $('#main-div a[name^=ajax-section-]').last().attr('name');
}

Categories