Dynamic content of Web Page not loaded totally using Htmlunit WebClient - javascript

I am trying to load web page (https://genpact.taleo.net/careersection/sgy_external_career_section/jobsearch.ftl?lang=en) for scraping using HtmlUnit WebClient. But the content is not being loaded properly. For example, i am unable to find the Apply buttons.
My webclient code is as below
webClient.setCssErrorHandler(new DefaultCssErrorHandler());
webClient.setJavaScriptErrorListener(new DefaultJavaScriptErrorListener());
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getCookieManager().setCookiesEnabled(true);
webClient.waitForBackgroundJavaScript(60000);
Can someone please help me with this

This works for me
public static void main(String[] args) throws IOException{
final String url = "https://genpact.taleo.net/careersection/sgy_external_career_section/jobsearch.ftl?lang=en";
try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60)) {
HtmlPage page = webClient.getPage(url);
// waitForBackgroundJavaScript has to be called after every action
// this page is really slow wait for the last part of the dynamic content
while(!page.asText().contains("Previous\r\n1\r\n2\r\n3\r\n4\r\n")) {
webClient.waitForBackgroundJavaScript(1_000);
}
System.out.println("-------------------------------------------------------------------------------");
System.out.println(page.asText());
System.out.println("-------------------------------------------------------------------------------");
}
}

Related

Android Web-view Error I/chromium: [INFO:CONSOLE(1)] "Uncaught ReferenceError:

I have a web-service In that I have added bar-code reader for android
So with the Help of JavaScript I am calling my bar-code reader from web-view
So for that I followed this
and designed on server side...
I have Given this
at JavaScript
function mybarcd() {
MyApp.mybarcdt();
}
function actfromAnd(msg){
document.getElementById("brcd").value = msg;
}
at HTML/PHP
<div class="data">
<input id="brcd" type="text" value=""/>
<button type="button" onClick="mybarcd()">SCAN</button>
</div>
On Android side
In webview
webView.addJavascriptInterface(new WebAppInterface(this), "MyApp");
and new js interface
#JavascriptInterface
public void mybarcdt() {
IntentIntegrator intentIntegrator = new IntentIntegrator(Main_A.this);
intentIntegrator.setBeepEnabled(true);
intentIntegrator.setOrientationLocked(true);
intentIntegrator.setPrompt("Scan Barcode");
intentIntegrator.initiateScan();
}
#Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
IntentResult result = IntentIntegrator.parseActivityResult(requestCode, resultCode, data);
if (result != null) {
if (result.getContents() == null) {
Log.d("ScanActivity", "Cancelled scan");
Toast.makeText(this, "Cancelled", Toast.LENGTH_LONG).show();
} else {
Log.d("ScanActivity", "Scanned");
String bcval = result.getContents();
if (android.os.Build.VERSION.SDK_INT < 19) {
webView.loadUrl("javascript:actfromAnd(\""+bcval+"\")");
}else{
webView.evaluateJavascript("javascript:actfromAnd(\""+bcval+"\")", null);
}
System.out.println("javascript:actfromAnd(\""+bcval+"\")");
}
} else
super.onActivityResult(requestCode, resultCode, data);
}
My Problem is that Its working fine in a single Html/PHP file with Js on same page or separate page I have tested its scanning and Updating the value in input box...
But its not working since I have using multiple pages or frame in one webview...
its missing JS value... How ever form server its opening scanner at on-click... but after scanning the value is not passing to the input box with JS I am getting this error.....
I/chromium: [INFO:CONSOLE(1)] "Uncaught ReferenceError: actfromAnd is not defined", source: (1)
Update
1)I hava Tried this in Static Page with JS in side that PHP/HTML page
2)I also tried with same in a static Page with JS seperate page
On the above two conditions its worked fine
But In my web-service I have Given Same JS file which is running successfully in Static page I have a single JS file for My Webservice and Static page its working fine in static but not working in MY webservice live.. How ever JS is loading Because on click its wokring from that JS and its opening Camera
But responce after scanning its not going to web input
I understand that I am getting Error Because...
In my Live Page I have a MainMenu Inside that menu when I select a application its loading in iframe So my Android Activity responce after scanning Is pinging to that Mainmenu page But for menu there is no Js function named actfromAnd So I am getting Error...
Here I can't give URL of that particular page(iframe) Because of depending on the menus it will change I can Give Only Login or MainMenu link directly.but not for a particular page inside the menu
Can Any one suggest me on this kind...
Add this script in Web-service at your Parent Menu which has your iframe or iframes
<script>
function actfromAnd(msg){
window.frames['yourExactframe'].document.getElementById("brcd").value = msg;
}
</script>
If you are using same in more than one frame then declare your frame name globally
I tried your Code Working Fine... In my example Frame Hope It works fine for you
Nice question....
You should execute the javascript when the page is loaded
mWebView.setWebViewClient(new WebViewClient() {
#Override
public void onPageFinished(WebView view, String url) {
webview.loadUrl("javascript:myFunction()");
}
});
The code will be executed and will find your javascript function. The way you are doing now does not wait.

Java, JavaFx : Inserting a HTML&JS(static,no internet) project inside a JavaFX container

I am working on a task in which we need to put one of our HTML & JS based project inside a JavaFX project or any other suitable containers which are out there. The purpose is to create an app, which can directly be deployed and would prevent any users from checking out the source code of HTML & JS.
Some time back when I was checking out JavaFX, I read that it supports JS, and JS can be used with it. Is there any way to create a container inside which I can put my HTML&JS files by giving path, etc?
How can I go about this? Whatever I am trying to do, what is it called. Any help, pointers, suggestions, would be nice.
Initial test
public class Main extends Application {
private Scene scene;
MyBrowser myBrowser;
#Override
public void start(Stage primaryStage) throws Exception{
primaryStage.setTitle("Test web");
myBrowser = new MyBrowser();
scene = new Scene(myBrowser, 1920, 1200);
primaryStage.setScene(scene);
primaryStage.show();
}
public static void main(String[] args) {
launch(args);
}
}
class MyBrowser extends Region {
final String hellohtml = "hello.html";
WebView webView = new WebView();
WebEngine webEngine = webView.getEngine();
public MyBrowser(){
URL urlHello = getClass().getResource("hello.html");
webEngine.load(urlHello.toExternalForm());
getChildren().add(webView);
}
}
As #sillyfly suggested use a WebView:
File f = new File(..);
// ..
final WebView webview = new WebView();
webview.getEngine().load(f.toURI().toURL().toString());
The hard part for me is always to figure out the right location to be used to reference the file.
Another option is when you have the HTML in the form of a string to load that as content:
String html = ...
webview.getEngine().loadcontent(html)
Be sure to check out at least the JavaDoc on WebView and WebEngine`

Load web page contents in console application using c#

I want to load contents of below web page in console application using c#.
http://justicecourts.maricopa.gov/findacase/casehistory.aspx
Using below code I am getting empty on the screen but it works perfectly if I load google.com web page.
By using WebClient and WebRequest I was getting error "Please enable javascript" and content was not loading so I used below code and javascipt error is not displaying now but web page content is not loading. I am struggling with this issue quite from long time, have seen lot of post regarding this and couldn't get this work.
Could anyone please help?
Thanks in Advance..
class Program
{
private static bool completed = false;
private static WebBrowser wb;
[STAThread]
static void Main(string[] args)
{
wb = new WebBrowser();
wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(wb_DocumentCompleted);
wb.Navigate("http://justicecourts.maricopa.gov/findacase/casehistory.aspx");
while (!completed)
{
Application.DoEvents();
Thread.Sleep(100);
}
Console.Write("\n\nDone with it!\n\n");
Console.ReadLine();
}
static void wb_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
Console.WriteLine(wb.Document.Body.InnerHtml);
completed = true;
}
}
If you literally just want to dump the contents of that URL out to the console, try this:
using(WebClient client = new WebClient()) {
Console.WriteLine(client.DownloadString(url));
}
try adding more wait.
static void Main(string[] args)
{
wb = new WebBrowser();
wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(wb_DocumentCompleted);
wb.Navigate("http://justicecourts.maricopa.gov/findacase/casehistory.aspx");
while (!completed)
{
Application.DoEvents();
Thread.Sleep(100);
}
//wait even more
for (int i = 0; i < 6; i++)
{
Application.DoEvents();
Thread.Sleep(1000);
}
Console.Write("\n\nDone with it!\n\n");
Console.ReadLine();
}
otherwise you can use EO Browser it is paid. but in your case trail will work cause it is not GUI application.as it shows trail message in GUI.
in EO you can say..
EOContorol.WebView.LoadUrlAndWait(URL);
Try using PhantomJs
basicaly like running a webbrowser without a window. (headless)

Facebook share button finished with blank page in Android webview

When I open webview and click Facebook share, it works.
But when share is finished, it opens a blank page.
Sometimes, it doesn't open blank page and show origin page. but usually it opens blank page.
I want to show page that includes share button. always
Here is my code:
//in onCreate.
String Url = //my url//;
CookieManager cookieManager = CookieManager.getInstance();
cookieManager.setAcceptCookie(true);
view = (WebView) this.findViewById(R.id.webView);
view.getSettings().setDomStorageEnabled(true);
view.getSettings().setSupportZoom(true);
view.getSettings().setBuiltInZoomControls(true);
view.getSettings().setUseWideViewPort(true);
view.getSettings().setJavaScriptEnabled(true);
view.getSettings().setAppCacheEnabled(true);
view.getSettings().setJavaScriptCanOpenWindowsAutomatically(true);
view.getSettings().setAllowFileAccess(true);
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.LOLLIPOP) {
CookieManager.getInstance().setAcceptThirdPartyCookies(view, true);
}
view.loadUrl(Url);
view.setWebViewClient(new myWebViewClient());
view.setWebChromeClient(new ChromeClient());
//
private class ChromeClient extends WebChromeClient{
#Override
public void onProgressChanged(WebView view, int newProgress) {
MainActivity.this.setProgress(newProgress*1000);
}
public void onCloseWindow(WebView window){
super.onCloseWindow(window);
}
And here is javascript code:
function fn_facebook_share() {
var url = //my url//;
var title = "${VideoModelDetail.title}";
var user_nm= "${VideoModelDetail.user_nm}";
var univ="${VideoModelDetail.univ_cd_nm}";
FB.ui({
method: 'feed',
link: url,
name:title,
caption:univ+" | "+user_nm,
description:'text',
}, function(response){});
}
Try moving your webview to xml layout file. The blank page error was caused due to js script fail while redirecting oAuth login to authorization acceptance page. You can overcome this issue by moving your webview into xml layout.I had the same issue on my android application. The cause of the issue is FB login javascript opens a new page on a new window

Waiting for Javascript with HtmlUnit

I was experimenting with HtmlUnit the other day. I wrote a program that performs a login to a site and gathers some information. But when clicking a specific button, htmlUnit doesn't wait for the resulting action. I tried to wait till all jobs from the JavascriptJobManager are done but it gets stuck at around 15 jobs. After that I tried to wait till the resulting htmlpage changes, but that does not work either. What could I try next ? Thanks for your time and I will try to implement any suggestion as fast as possible.
Edit: I´m completely aware that facebook doesn't like webcrawling, but I´m only doing this for study purposes, so no harm done. Following the error messages, the program throws. http://www.pastebin.ca/3007578
When the infoButton gets clicked, a new window appears making the old window unaccessible. http://imgur.com/aiF7nJR
final static WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31);
public static void main(String [] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException, InterruptedException{
//init webclient
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(true);
webClient.getOptions().setCssEnabled(true);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.waitForBackgroundJavaScript(12000);
webClient.setAlertHandler(new AlertHandler() {
#Override
public void handleAlert(Page arg0, String arg1) {
System.out.println("ALERT ON "+arg0.getUrl()+" :"+ arg1);
}
});
// perform the login
final HtmlPage loginPage = webClient.getPage("https://facebook.com");
final HtmlForm form = loginPage.getForms().get(0);
final HtmlTextInput username = form.getInputByName("email");
final HtmlPasswordInput password = form.getInputByName("pass");
final HtmlElement button = (HtmlElement) loginPage.getElementById("u_0_l");
username.setText("Your email");
password.setText("Your password");
final HtmlPage frontPage = (HtmlPage) button.click();
// The actual problem
final HtmlPage testPage = webClient.getPage("https://www.facebook.com/pages/Stackoverflow/1462865420609264");
HtmlElement infoButton = testPage.getFirstByXPath("//*[#class='share_action_link']"); // First share button.
HtmlPage testPage2 = infoButton.click();
JavaScriptJobManager manager = testPage2.getEnclosingWindow().getJobManager();
while (manager.getJobCount() > 0) {
Thread.sleep(1000);
webClient.waitForBackgroundJavaScript(100);
System.out.println(manager.getEarliestJob());
}
while(testPage == testPage2){
System.out.println("failed");
webClient.waitForBackgroundJavaScript(100);
Thread.sleep(5 * 1000);
}
}
With latest snapshot (in maven also), there was an error that window.performance is not defined, which was fixed.
EDIT: there was another error detected, fixed and snapshot deployed.
Please retest

Categories