We Keep Coding

WebView Crawler navigate to url based on page result

I'm trying to build a web crawler based on the requirements that were described here, and I figured WebView would be the most suitable way to implement this.
The problem seems to emerge when the next URL I need to visit is based on the HTML contents of the current page.
I am using view.evaluateJavascript to get the current page HTML and parse the URL part inside onReceiveValue, but then there is no way for me to navigate to the URL because onReceiveValue cannot access the view.
Also, using loadUrl in onPageFinished does not work as well, because it is done even before I retrieve the HTML content, thus navigating to the page with a null value
WebView myWebView = new WebView(this);
setContentView(myWebView);
myWebView.getSettings().setJavaScriptEnabled(true);
MyJavaScriptInterface jInterface = new MyJavaScriptInterface(this);
myWebView.addJavascriptInterface(jInterface, "HTMLOUT");
myWebView.setWebViewClient(new WebViewClient() {
#Override
public void onPageFinished(WebView view, String url) {
super.onPageFinished(view, url);
if (url.equals("http://url.com")) {
final String[] versionString = {
null
};
view.evaluateJavascript("(function(){return window.document.body.outerHTML})();",
new ValueCallback < String > () {
#Override
public void onReceiveValue(String html) {
String result = removeUTFCharacters(html).toString();
Matcher m = r.matcher(result);
versionString[0] = m.group(1);
}
});
String getFullUrl = String.format("https://url.com/getData?v=%s", versionString[0]);
view.loadUrl(getFullUrl);
}
}
});
myWebView.loadUrl("http://url.com");

Call your url from onReceiveValue
myWebView.setWebViewClient(new WebViewClient() {
#Override
public void onPageFinished(WebView view, String url) {
super.onPageFinished(view, url);
if (url.contains("https://www.google.com")) {
final String[] versionString = {
null
};
view.evaluateJavascript("(function(){return window.document.body.outerHTML})();",
new ValueCallback< String >() {
#Override
public void onReceiveValue(String html) {
String getFullUrl = String.format("https://cchat.in", versionString[0]);
view.loadUrl(getFullUrl);
}
});
}
}
});
myWebView.loadUrl("https://www.google.com");
I used 2 website to demonstrate. I am able to call 2nd URL from onReceiveValue.
You can try this.

Controling WebView Elements Not Working

I've looked at a few SO posts regarding this issue, and below is what I have came up with:
public class MainActivity extends AppCompatActivity {
private WebView webView;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
webView = (WebView) findViewById(R.id.webView);
webView.setWebViewClient(new WebViewClient());
webView.getSettings().setDomStorageEnabled(true);
webView.loadUrl("http://google.com");
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
String password = "a123345";
//webView.loadUrl("javascript:document.getElementById('lst-ib').value = '" + "hi" + "';");
webView.loadUrl("javascript:var uselessvar = document.getElementByName('q').value='" + password + "';");
Log.d("d","Done");
}
}
But nothing I have tried has worked. Any idea what I am missing?

Few things about this
Try to avoid using Thread.sleep in main thread
Dont load script with constant time
You can use of callback like WebViewClient callbacks which have at least onPageFinished method which you can use to load the JS
Add setJavaScriptEnabled to run the JS
Example :
WebView webView = (WebView) findViewById(R.id.webView);
webView.setWebViewClient(new WebViewClient() {
public void onReceivedError(WebView view, int errorCode, String description,
String failingUrl) {
}
#Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
return false;
}
#Override
public void onPageFinished(WebView view, String url) {
String password = "a123345";
view.loadUrl("javascript:(function() { " +
"document.getElementsByName('q')[0].value = '" + password + "'; " +
"})()");
}
});
webView.getSettings().setJavaScriptEnabled(true);
webView.loadUrl("https://google.com");

Javascript not working in WebView - Android

I have a code here where I want to retrieve an element from the webview of android and I can't seems to figure out what's wrong. It might be some silly mistakes but I just can't see why and what's wrong. Thanks.
private void retrieveContentWebView(String address){
Log.d("Web Download"," Inside ");
Log.d("Web Download ","address : " + address);
WebView webView;
webView = new WebView(context);
webView.loadUrl(address);
webView.getSettings().setJavaScriptEnabled(true);
webView.addJavascriptInterface(this, "myjava");
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP)
webView.getSettings().setMixedContentMode(WebSettings.MIXED_CONTENT_ALWAYS_ALLOW);
webView.setScrollBarStyle(View.SCROLLBARS_INSIDE_OVERLAY);
webView.getSettings().setDomStorageEnabled(true);
webView.setWebViewClient(new WebViewClient() {
#Override
public void onPageStarted(WebView view, String url, Bitmap favicon) {
super.onPageStarted(view, url, favicon);
Log.d("Web Download"," Inside Page Started");
}
#Override
public void onPageFinished(WebView view, String url) {
super.onPageFinished(view, url);
Log.d("Web Download"," Inside Page Finished");
view.loadUrl("javascript: {" +
"var titleTexts = document.getElementsByTagName('title')[0]" +
"myjava.onData(titleTexts.text);}");
}
});
}
//This method will be registered as a JavaScript interface
#JavascriptInterface
#SuppressWarnings("unused")
public void onData(final String value){
Log.d("Web Download"," Inside Handler");
final String getValue = value;
final Handler handler = new Handler();
handler.postDelayed(new Runnable() {
#Override
public void run() {
// Do something after 5s = 5000ms
Log.d("Web Download HTML :",getValue);
}
}, 5000);
}

Try adding this to your WebView:
webView.setWebChromeClient(new MyCustomChromeClient(this));
Create the MyCustomChromeClient for yourself.

Android webview not loading css and javascript when trying to access webpage using custom headers

I am using authorization headers to access the web page, but when using WebViewClient with authorization headers the webview not rendering the css and also the js not loading.
public class TableViewTest extends AppCompatActivity {
WebView webView;
SharedPreferences pref;
#SuppressLint("SetJavaScriptEnabled")
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_table_view_test);
String url = "http://myurl";
pref = getSharedPreferences("app", Context.MODE_PRIVATE);
webView = (WebView) findViewById(R.id.webView1Id);
webView.setWebViewClient(wvc);
//webView.setVerticalScrollBarEnabled(true);
//webView.setHorizontalScrollBarEnabled(true);
webView.getSettings().setJavaScriptEnabled(true);
webView.getSettings().setBuiltInZoomControls(true);
webView.getSettings().setDomStorageEnabled(true);
webView.getSettings().setUseWideViewPort(true);
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
webView.getSettings().setMixedContentMode(WebSettings.MIXED_CONTENT_COMPATIBILITY_MODE);
webView.getSettings().setMixedContentMode(WebSettings.MIXED_CONTENT_ALWAYS_ALLOW);
}
//webView.setLayerType(View.LAYER_TYPE_HARDWARE, null);
webView.loadUrl("http://myurl");
}
#Override
protected void onSaveInstanceState(Bundle outState) {
super.onSaveInstanceState(outState);
webView.saveState(outState);
}
#Override
protected void onRestoreInstanceState(Bundle savedInstanceState) {
super.onRestoreInstanceState(savedInstanceState);
webView.restoreState(savedInstanceState);
}
WebViewClient wvc = new WebViewClient() {
#SuppressWarnings("deprecation")
public WebResourceResponse shouldInterceptRequest(WebView view, String url) {
try {
final String acToken = pref.getString("token", "DEFAULT");
OkHttpClient okHttpClient = new OkHttpClient();
Request request = new Request.Builder().url(url).addHeader("Authorization", "Bearer " + acToken)
.build();
Response response = okHttpClient.newCall(request).execute();
return new WebResourceResponse(response.header("text/html", response.body().contentType().type()), // You can set something other as default content-type
response.header("content-encoding", "utf-8"), // Again, you can set another encoding as default
response.body().byteStream());
} catch (ClientProtocolException e) {
//return null to tell WebView we failed to fetch it WebView should try again.
return null;
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
};
}
Please help me. Any help is appreciated.

Used AdvancedWebView library to overcome the issues. The default webview in android can not handle the css issues sometimes.
And also both Android webview and AdvancedWebView Can not handle the headers except in initial request in the webview. It can not pass the headers across all the links in particular webview request. Please refer this to rectify your doubts Github Question.
My SO Question for the headers issue.
If anybody finds the solution for the above header passing issue, feel free to add your solution. Thank you.

This works for me.
(btw is deprecated now)
WebViewClient wvc = new WebViewClient() {
#Override
public WebResourceResponse shouldInterceptRequest (WebView view, String url){
try {
OkHttpClient httpClient = new OkHttpClient();
com.squareup.okhttp.Request request = new com.squareup.okhttp.Request.Builder()
.url(url.trim())
.addHeader("PIPPO", "PIPPO")
.build();
com.squareup.okhttp.Response response = httpClient.newCall(request).execute();
Log.d("TAG",response.header("content-type", "text/html"));
Log.d("TAG",response.header("content-encoding", "UTF-8"));
return new WebResourceResponse(
null,
null,
response.body().byteStream()
);
} catch (Exception e) {
//return null to tell WebView we failed to fetch it WebView should try again.
return null;
}
}
My only fear is that on smartphones with "problematic" default browser, the result may be broken.

Android JavaScript content click opens URL in same WebView [duplicate]

Short question that i cannot get my finger on, i'd like to launch a new browser from my WebView after people click a hyperlink. But how can i set that target of that link to escape the WebViewClient??
Here's my code, any help is greatly appreciated:
WebView site = (WebView)findViewById(R.id.WebView);
site.setWebViewClient(new WebViewClient());
site.setScrollBarStyle(View.SCROLLBARS_INSIDE_OVERLAY);
site.getSettings().setJavaScriptEnabled(true);
button1.setOnClickListener(button1OnClickListener);
button2.setOnClickListener(button2OnClickListener);
button3.setOnClickListener(button3OnClickListener);
button4.setOnClickListener(button4OnClickListener);
final AlertDialog alertDialog = new AlertDialog.Builder(this).create();
progressBar = ProgressDialog.show(FlitsersActivity.this, "Thingy1", "Load...", false, true);
site.setWebViewClient(new WebViewClient() {
public boolean shouldOverrideUrlLoading(WebView view, String url) {
Log.i(TAG, "Loading...");
view.loadUrl(url);
return true;
}
public void onPageFinished(WebView view, String url) {
Log.i(TAG, "Done: " +url);
if (progressBar.isShowing()) {
progressBar.dismiss();
}
}
public void onReceivedError(WebView view, int errorCode, String description, String failingUrl) {
Log.e(TAG, "Error: " + description);
Toast.makeText(MyActivity.this, "Oh no! " + description, Toast.LENGTH_SHORT).show();
alertDialog.setTitle("Error");
alertDialog.setMessage(description);
alertDialog.setButton("OK", new DialogInterface.OnClickListener() {
public void onClick(DialogInterface dialog, int which) {
return;
}
});
alertDialog.show();
}
});
site.loadUrl("http://www.etcetera.com");
}

Exactly, thanks! I think i found out around the time you wrote the second response ;)
Right now i am using the code below: it works like a charm :) Thank you very much for responding!
site.setWebViewClient(new WebViewClient() {
public boolean shouldOverrideUrlLoading(WebView view, String url) {
String myAlternativeURL = "http://newURL";
if (!url.equals(myAlternativeURL)) {
{
Intent i = new Intent(Intent.ACTION_VIEW);
i.setData(Uri.parse("http://newURL"));
startActivity(i);
}
return true;
} else {
Log.i(TAG, "Loading...");
view.loadUrl(url);
return true;
}
}

You are overriding the loading of all URLs by forcing the WebView to load them with this code:
public boolean shouldOverrideUrlLoading(WebView view, String url) {
Log.i(TAG, "Loading...");
view.loadUrl(url);
return true;
}
What you need to do is create a browser intent with the URL if you want it to open in the default browser instead of your WebView.
http://developer.android.com/guide/appendix/g-app-intents.html
The code would be something like this:
Intent i = new Intent(Intent.ACTION_VIEW);
i.setData(Uri.parse(url));
startActivity(i);
More info:
if (url.equals("http://www.etcetera.com")
// load url normaly
else
// Load in new window via intent

Static Problem
So I found out, that in my case, I have to add a Context before startActivity()
context.startActivity(browserIntent);
From: https://stackoverflow.com/a/6651535/13413319
In order to do that:
I have this variable:
Context context;
and of cource assigning its value by Constructor
Context = context;
After that calling the methode, wenn you need to perform the task.
openInBrowser(url);
methode:
public static void openInBrowser(String url) {
Intent browserIntent = new Intent(Intent.ACTION_VIEW);
browserIntent.setData(Uri.parse(url));
context.startActivity(browserIntent);