I'm trying to do web scraping using puppeteer. The element I need to handle loads lately. When I click on the search button the result loads in AJAX and I need to pick the element I am trying to pick is in the search results but not in the initial load of the page. The page screenshot it is producing contains search results too and if it output the HTML source I can see the element there too. but not sure why I cannot pick it.
You can use await page.waitForSelector(cssSelector); to ask Puppeteer to wait for any element to be displayed in the UI before continuing on to further steps in your script. By default, the timeout for the wait is 30 seconds but you can set it to any timeout you wish.
So in your case I would:
Enter your search text into the search bar.
Click on the search button (this will execute your AJAX call to load the results).
Use await page.waitForSelector(cssSelector); to ask Puppeteer to wait until some element you are sure will be displayed in the UI after executing the search is visible.
Now that Puppeteer has registered the element as visible, you know that any actions you wish to perform on it will also execute correctly.
What you might find happens, if you don't use that waitForSelector() call is that the element is displayed but Puppeteer will timeout, for example, if you wish to execute a click command on an element. This is because the timeouts for click events (and other Puppeteer events which interact with elements) is very short and sometimes the script (especially in headless mode) can move to the next instruction too quickly to allow for the UI to update fast enough to keep up.
So by adding the additional waitForSelector calls, you're also making your scripts much more robust. Especially when data is being generated dynamically as they are in your case.
at the moment I am working on replacing pop-up on a website I inherited. Those pop-ups used modal dialogs, which are on their way out and even were dropped by pop-up blockers on client side.
My approach was loading the HTML of the pop-up into an div on the main site, while hiding the original content, then emptying the div and switch the main content back to visible.
This works so fa, but the loaded content has scripts that run to check if someone is already using that function. The first time you use that function all is fine. The script runs, sees noone is using the function, I go through the stuff, empty the div and return to the main content. When trying to use the function a second time the script to still run (console shows the requests), even though I emptied the div, prompting the eternal "please wait till other user is finished" lines, since the first script is still checking for use, signalling the second script "I'm buisy".
So right now, I am looking for a way to stop the first script, since removing the HTML-content doesn't suffice it seems.
The code so far:
$("#dialog").load("stuffToLoad.htm",function(response, status)
{
if(status=="success"){
$(".fullTable").toggle();
$("#dialog").toggle();
};
})
to get the content. The in-use-check is done with a post-request, that is repeated by a window.setTimeout every second. That part seems to still run.
When everything is done, or the user runs into an error I tried:
function returnToProcVal()
{
$("#dialog").html("");
$("#dialog").toggle();
$(".fullTable").toggle();
}
to delete the function content and scripts, to stop them running. While the DOM says all is gone I can see the post requests being repeated in the console.
I'd be grateful for any pointer or perhaps even better methods to get the function running and returning the user, without the use of pop-ups.
I realize the title is a bit unclear, so I'll elaborate as well as possible. Essentially, there is a webpage that the user visits. After the page completes its initial load, the user may then click a button on the page which will load new content into the page. Something like <div class="expanded" style> -- basically a bunch of text and perhaps some links.
I essentially need to alter the contents of this newly loaded area. Right now, I am running something like:
$('div.expand-button').click(function(){ //When the button to load new content is clicked...
And then fire off my document.querySelectorAll function at this point. However, if I do this, the new content is not finished loading when the function runs, and, as a result, nothing happens. What's the best way of delaying my function so that it will only run after the new content area is loaded?
I had a similar problem recently and the simplest solution I found (albeit probably not the best) was to use setTimeOut. So your code would look something like :
$('div.expand-button').click(function(){
YourLoadNewAreaFunction();
setTimeout(function() {
//do stuff you needed the page completely loaded to do
}, 1000);
}
1000 is of course an arbitrary number of milliseconds I chose, you could choose whatever time you deem necessary to completely load your new area. Assuming your new content loads in under a second, the above should work.
From what I've seen online, you could also try looking into callbacks.
I found myself in a following situation. I need to somehow not include <div class="article-meta-social"></div> element and all its contents into my document.ready function. The reason is, it has links to apis from facebook, twitter, g+ etc... and Multiplied by several posts it results in a little delay before contents within document.ready function are fired off.
Therefore, what can I change in order for this
$(document).ready(function(){
});
To not wait until .article-meta-social and its contents are ready?
$(document).ready relies on the native DOMContentLoaded event, which does exactly what it says - fires when the entire DOM has been parsed. So to achieve what you want, insert the contents of .article-meta-social dynamically inside your ready handler. This way it won't hold up the main rendering of your page.
I have a mobile single-page web application that is built using jquery-mobile (jqm) and knockout. The application itself has multiple pages but they are all contained within a single HTML document.
Problem: after changing my "create view model for page" from sync to async behavior, I have the problem that jquery-mobile fires its events before the data is ready.
Background: up until recently I had been working with sample data, basically a huge JSON blob, and everything worked smoothly. With the new async composition of view models from various sources, data is not ready immediately and my "buildViewModel" method takes a continuation callback instead of just synchronously returning data.
I'm subscribing to the pagebeforecreate and pagebeforechange events, and fire off the code to populate the viewmodel here. The problem is that after returning from the event handler, jqm triggers the remaining chain of events before the data is available. This causes a page transition to an unprepared page, which is undesirable.
I have tried to call event.preventDefault in all of the before-events and manually calling $.mobile.changePage once the page is ready to be a) enhanced and b) the page transition to occur, but without any luck.
I've scanned the jquery-mobile source, but couldn't spot anything that looked like it would allow me to delay the pagebeforeshow event, which is essentially what I need in order to be able to render the page properly.
How can I ensure that 1) data is available and 2) knockout has been applied to perform initial DOM manipulations, before jquery-mobile attempts to enhance the page and before it executes the in-transition to the page?
I also considered using synchronous ajax to fetch resources, but this will (I think) not work for resources loaded from the device (using PhoneGap/Cordova), and has other negative consequences that I'd like to avoid.
FWIW, I'd like to avoid having to manually handle all navigation events by wiring up click-handlers everywhere, but I'm open to all solutions if need be.
Apologies if this is a duplicate; I've searched and read a ton of questions, but not found an answer or question that was quite the same. It just sounds incredible that I would be the first to hit this problem, as I imagine it is a common scenario..
Update: clarified problem scenario description.
I had this exact same problem.
The only solution I've been able to come up with is to write a custom transition handler that defers starting the transition until the Ajax request completes.
Here's a fiddle showing the technique. The fiddle doesn't use Knockout, but does show how to defer the transition.
Basically, since $.ajax() returns a promise, I can pipe that into the promise returned by the default transition handler and return it from my new handler.
In my pagebeforeshow handler, I attach the Ajax promise to the page so that the transition handler has access to it. Not sure if this is the best way, but I liked it better than using a global variable.
The only thing I didn't like about this is that it delays the start of the transition until the Ajax response arrives so it could feel like the page has "hung" to the user making them click again. Manually showing the loading message makes it feel a bit more responsive.
Hope this helps and please let me know if you find a better solution!
Delaying the transition to a new page until its content is ready is a very common issue when facing dynamic content in jQuery Mobile. The most convenient ways to address this are:
Instead of classic href type navigation, base the links on "click" actions that will first retrieve the content, build a new page in the DOM, and then initiate a transition to this new page through $.mobile.changePage. The advantage of this approach is that it is easy to put in place, the disadvantage is that you do not navigate with classic href links
Bind the pagebeforechange event at the document level to detect if upon navigation the target page is one of your page that should contain dynamic content. In such a case, you can prevent default navigation from happening, take your time to generate the page, and transition upon success. This is described in the JQM docs on dynamically injected content. The advantage is that you can still rely on standard href links navigation, but it requires a bit more code and design upstream to properly detect and act upon navigation to the pages.
$(document).on( "pagebeforechange", function( e, data ) {
if ( typeof data.toPage === "string" ) {
if ( data.toPage === "myDynamicPageName" ) {
e.preventDefault(); //used to stop transition to the page (for now)
/*
Here you can make your ajax call
In you callback, once you have generated the page you can call
$.mobile.changePage
(you can pass the Div of the new page instead of its name as
the changepage parameter to avoid interrupting again the page change)
*/
}
}
});
Set your link to call a "load" function instead of doing a page transition. In your load function, display the "loading message" and make the JSON call. Finally, in the JSON callback function, change page to page2
The load function:
function loadPage2() {
/* show wait page */
$.mobile.loading( 'show', {
text: 'Loading massively huge dataset',
textVisible: true
});
/* perform JSON call then call callback */
}
Callback function
function callback() {
$.mobile.changePage("#page2");
}
Here is a working JSFiddle: http://jsfiddle.net/8w7PM/
Note that if you don't want users to be able to update input fields in Page 1 while waiting, introduce a "wait page" between page 1 and page 2, with the init of "wait page" doing the same as "loadPage2".
I think you have to fire again for all widget which you want to bind the data from response to
For example, you will have to invoke trigger with create or refrestevent for the element
$("#element").trigger('create');
JQuery Mobile will bind all default events to the element as it is
--- EDIT ---
I just created a sample code, I think it's same your issue, please try the link http://jsfiddle.net/ndkhoiits/BneqW/embedded/result/
Before rending the data, we have to invoke to service to get them for displaying, that why all events binded by jqm will be removed then.
I have a workaround for this, don't make jqm fire anything on the element, we'll trigger it after all data is binded by knockoutjs
Let try the fixed version
http://jsfiddle.net/ndkhoiits/c5a2b/embedded/result/
This is the code http://jsfiddle.net/ndkhoiits/c5a2b/
I have a small jQuery Mobile / KnockoutJS application and have struggled with the very same issue. My app contains about 5 pages. All are contained in a single physical HTML document with the standard <div data-role="page"> markup separating individual pages.
I finally went with click based navigation and fire $.mobile.changePage() as the result of $.ajax success.
One of the downsides to this technique is that you will lose your button highlighting when relying on onclick vs href attributes. See my related post: href vs scripted page transitions and button highlighting
I later chose to supply both and rely on the href to perform the navigation while using onclick to invoke my JavaScript logic to load ViewModels etc. The only place that I have found this to be an issue is when there is possible validation required on the source page. If it fails, the transition has already started and the UI then flashes back to the source page. Ugly but this only happens in limited instance within my app.
I don't think any of this is specific to Knockout. My exact solution may present issues for you in that your navigation is likely to complete before your model is fully loaded but if you rely on $.mobile.changePage(), it should all work and hide your page until after it is loaded. The transitions should work fine.
<a href="#MyNewPage" data-bind="click:LoadNewPage" data-role="button">
Load Page
</a>
$.ajax({
url: url,
cache: false,
dataType: "json",
data: requestData,
type: "POST",
async: true,
timeout: 10000,
success: function (data, textStatus, jqXHR) {
// use either href or changePage but not both
$.mobile.changePage("#NewPage");
},
error: function (jqXHR, textStatus, errorThrown) {
alert("AJAX Error. Status: " + textStatus);
// jqXHR.status
// http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
}
});
You should place the code for the page transition in a success function on the AJAX call.
$.ajax({
url:"request-url",
data: data,
type: "POST",
success: function(response){
// Add Transition Code Here
}
});