Using the browser if I visit a certain direct download URL, it automatically downloads the file. However, when I use Java code to download the file, I get the HTML code instead of the file contents:
<html>
<body>
<script type="text/javascript" src="/aes.js"></script>
<script>function toNumbers(d) {
var e = [];
d.replace(/(..)/g, function (d) {
e.push(parseInt(d, 16))
});
return e
}
function toHex() {
for (var d = [], d = 1 == arguments.length && arguments[0].constructor == Array ? arguments[0] : arguments, e = "", f = 0; f < d.length; f++) e += (16 > d[f] ? "0" : "") + d[f].toString(16);
return e.toLowerCase()
}
var a = toNumbers("f655ba9d09a112d4968c63579db590b4"), b = toNumbers("98344c2eee86c3994890592585b49f80"),
c = toNumbers("b5eb8dc5c53e5107faa7ec1c1f3e3dc7");
document.cookie = "__test=" + toHex(slowAES.decrypt(c, 2, a, b)) + "; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/";
location.href = "http://example.com/Test.txt?i=1";</script>
<noscript>This site requires Javascript to work, please enable Javascript in your browser or use a browser with
Javascript support
</noscript>
</body>
</html>
My file downloading code is e.g. the following:
URL url = new URL("...");
try (InputStream inputStream = url.openStream())
{
Files.copy(inputStream, downloadedFilePath, REPLACE_EXISTING);
}
How would it be possible to download the file programmatically in Java? There are ways to execute JavaScript but how is it supposed to work exactly? It seems like the document.cookie has to be set (correctly) to download.
Downloading works by passing the correct cookie value alongside of the request. The cookie value can be retrieved using e.g. the Google Chrome DevTools -> Application -> Cookies. Since in my example the cookie value does not change, a code like the following would do the trick:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class DownloadWithCookieExample
{
public static void main(String[] arguments)
{
try
{
URL url = new URL("...");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("Cookie", "blabla"); // Hard-coded correct cookie value
String readStream = readStream(con.getInputStream());
System.out.println(readStream);
} catch (Exception exception)
{
exception.printStackTrace();
}
}
private static String readStream(InputStream inputStream) throws IOException
{
StringBuilder stringBuilder = new StringBuilder();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream)))
{
String line;
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line);
stringBuilder.append(System.lineSeparator());
}
}
return stringBuilder.toString().trim();
}
}
An even better solution would be to execute the JavaScript to receive the cookie value and to pass the result. This task is left as an exercise for another answer for whoever knows how to do this elegantly.
Related
I want to retrieve data from a website using Nashorn script engine
I have the java code where I can retrieve data from a sample website template.
Now I want to call that java file from java script file.
following is the code:
JAVA CODE(Nsample.java):
package sample;
import java.net.*;
import java.io.*;
public class Nsample
{
public static void main(String[] args)
{
String output = getUrlContents("https://freewebsitetemplates.com/");
System.out.println(output);
}
public static String getUrlContents(String theUrl)
{
StringBuilder content = new StringBuilder();
try
{
URL url = new URL(theUrl);
URLConnection urlConnection = url.openConnection();
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(urlConnection.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null)
{
content.append(line + "\n");
}
bufferedReader.close();
}
catch(Exception e)
{
e.printStackTrace();
}
return content.toString();
}
}
JAVASCRIPT code:(sample.js)
var n = Java.type('C.JavaFolder.sample.Nsample');
var result = n.getUrlContents("https://freewebsitetemplates.com/");
print(result);
I'm trying to compile javascript code using command prompt but it is showing CLASSNOTFOUNDEXCEPTION.
The command was jjs sample.js.Im assuming I did some mistake in Java.type() function.
Can anyone solve this?
This line is the problematic line:
var n = Java.type('C.JavaFolder.sample.Nsample');
Java.type accepts fully qualified java type name. Based on your Java code, your package seems to be "sample" and class name is "Nsample". So the fully qualified class name would be "sample.Nsample".
You should compile your Java classes and specify the directory in -classpath option (of jjs tool or your java application if you use javax.script API with nashorn).
Instead of calling Java from JavaScript , I tried to call JavaScript from java and worked well.
I created some functions in JavaScript and invoked those functions from Java code.
Following is the code.Hope this helps.
Test.java:
import javax.script.*;
import java.io.*;
import java.util.*;
public class Test{
public static void main(String[] args) throws Exception{
ScriptEngine engine = new ScriptEngineManager().getEngineByName("Nashorn");
engine.eval(new FileReader("test.js"));
Invocable invoke = (Invocable)engine;
Object res = invoke.invokeFunction("httpGet","https://www.javaworld.com");
System.out.println(res);
}
}
test.js:
var httpGet = function(theUrl){
var con = new java.net.URL(theUrl).openConnection();
con.requestMethod = "GET";
return asResponse(con);
}
function asResponse(con){
var d = read(con.inputStream);
return d;
}
function read(inputStream){
var inReader = new java.io.BufferedReader(new
java.io.InputStreamReader(inputStream));
var inputLine;
var response = new java.lang.StringBuffer();
while ((inputLine = inReader.readLine()) != null) {
response.append(inputLine);
}
inReader.close();
return response.toString();
}
I have a client with written c# and a server with written java. I capture audio and send with socket to the server and server send with web socket to the browser and want to play with browser. But when i try browser says Uncaught (in promise) DOMException: Failed to load because no supported source was found.
Could you help me?
private static void Recordwav()
{
waveInEvent = new WaveInEvent();
int devicenum = 0;
for (int i = 0; i < WaveIn.DeviceCount; i++)
{
if (WaveIn.GetCapabilities(i).ProductName.Contains("icrophone"))
devicenum = i;
}
waveInEvent.DeviceNumber = devicenum;
waveInEvent.WaveFormat = new WaveFormat(44100, WaveIn.GetCapabilities(devicenum).Channels);
waveInEvent.DataAvailable += new EventHandler<WaveInEventArgs>(VoiceDataAvailable);
waveInEvent.StartRecording();
}
private static void VoiceDataAvailable(object sender, WaveInEventArgs e)
{
JObject jObject = new JObject();
jObject["voice"] = Convert.ToBase64String(e.Buffer);
byte[] messageByte = Encoding.ASCII.GetBytes(jObject.ToString().Replace("\r\n", "") + "\n");
socket.Send(messageByte);
}
$scope.socket.onmessage = function (response)
{
var data = JSON.parse(response.data);
if(data.id == $scope.id) {
if(data.voice) {
var voice = data.voice;
var sound = new Audio("data:audio/wav;base64," + voice);
sound.play();
}
}
};
you're just sending raw samples, not a properly formatted WAV file. You'd need to use WaveFileWriter to write to a MemoryStream (wrapped in an IgnoreDisposeStream) dispose the WaveFileWriter and then access the MemoryStream underlying byte array. Also you're not taking into account BytesRecorded.
Even if you get this working, I suspect you'll get very choppy audio, as each WAV file will be a few hundred ms, and they won't necessarily play perfectly one after the other.
I am actually trying to click on a link to download a file from :
http://www.histdata.com/download-free-forex-historical-data/?/metatrader/1-minute-bar-quotes/eurusd/2013
The html code for the line I am trying to download is:
<a id="a_file" title="Download the zip data file" href="javascript:return true;" target="nullDisplay">HISTDATA_COM_MT_EURUSD_M1_2013.zip</a>
And the java code is:
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_38);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
HtmlPage htmlPage=webClient.getPage("http://www.histdata.com/download-free-forex-historical-data/?/metatrader/1-minute-bar-quotes/eurusd/2016/7");
List<HtmlAnchor> anchors=htmlPage.getAnchors();
HtmlAnchor anchor = null;
for (int i = 0; i < anchors.size(); ++i) {
anchor = anchors.get(i);
String sAnchor = anchor.asText();
if (sAnchor.equals("HISTDATA_COM_MT_EURUSD_M1_201607.zip"))
break;
}
Page p = anchor.click();
webClient.waitForBackgroundJavaScript(60000);
InputStream is = p.getWebResponse().getContentAsStream();
int b = 0;
while ((b = is.read()) != -1) {
System.out.print((char)b);
}
The error message i get is:
Jul 12, 2016 1:29:57 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter error
SEVERE: error: message=[invalid return] sourceName=[javascript url] line=[88] lineSource=[return true;] lineOffset=[7]
Exception in thread "main" ======= EXCEPTION START ========
Exception class=[net.sourceforge.htmlunit.corejs.javascript.EvaluatorException]
com.gargoylesoftware.htmlunit.ScriptException: invalid return (javascript url#88)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:904)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:729)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:694)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:746)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptIfPossible(HtmlPage.java:902)
at com.gargoylesoftware.htmlunit.html.HtmlAnchor.doClickStateUpdate(HtmlAnchor.java:114)
at com.gargoylesoftware.htmlunit.html.HtmlAnchor.doClickStateUpdate(HtmlAnchor.java:179)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:800)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:747)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:694)
at clickPage.main(clickPage.java:38)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EvaluatorException: invalid return (javascript url#88)
at com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter.error(StrictErrorReporter.java:65)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addError(Parser.java:188)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addError(Parser.java:167)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:255)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:244)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:237)
at net.sourceforge.htmlunit.corejs.javascript.Parser.returnOrYield(Parser.java:1632)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1022)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:928)
at net.sourceforge.htmlunit.corejs.javascript.Parser.parse(Parser.java:572)
at net.sourceforge.htmlunit.corejs.javascript.Parser.parse(Parser.java:492)
at net.sourceforge.htmlunit.corejs.javascript.Context.compileImpl(Context.java:2660)
at net.sourceforge.htmlunit.corejs.javascript.Context.compileString(Context.java:1623)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory$TimeoutContext.compileString(HtmlUnitContextFactory.java:172)
at net.sourceforge.htmlunit.corejs.javascript.Context.compileString(Context.java:1615)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$2.doRun(JavaScriptEngine.java:720)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
... 12 more
Enclosed exception:
net.sourceforge.htmlunit.corejs.javascript.EvaluatorException: invalid return (javascript url#88)
at com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter.error(StrictErrorReporter.java:65)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addError(Parser.java:188)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addError(Parser.java:167)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:255)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:244)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:237)
at net.sourceforge.htmlunit.corejs.javascript.Parser.returnOrYield(Parser.java:1632)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1022)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:928)
at net.sourceforge.htmlunit.corejs.javascript.Parser.parse(Parser.java:572)
at net.sourceforge.htmlunit.corejs.javascript.Parser.parse(Parser.java:492)
at net.sourceforge.htmlunit.corejs.javascript.Context.compileImpl(Context.java:2660)
at net.sourceforge.htmlunit.corejs.javascript.Context.compileString(Context.java:1623)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory$TimeoutContext.compileString(HtmlUnitContextFactory.java:172)
at net.sourceforge.htmlunit.corejs.javascript.Context.compileString(Context.java:1615)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$2.doRun(JavaScriptEngine.java:720)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:729)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:694)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:746)
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptIfPossible(HtmlPage.java:902)
at com.gargoylesoftware.htmlunit.html.HtmlAnchor.doClickStateUpdate(HtmlAnchor.java:114)
at com.gargoylesoftware.htmlunit.html.HtmlAnchor.doClickStateUpdate(HtmlAnchor.java:179)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:800)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:747)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:694)
at clickPage.main(clickPage.java:38)
== CALLING JAVASCRIPT ==
return true;
======= EXCEPTION END ========
Please let me know what is wrong in my code and how to download file from the given link.
//Complete solution
//1. open page
//2. list the urls of that page using xpath
//3. download all file of that url .
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Date;
import java.util.List;
import java.util.Map;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.DomAttr;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class Crawler {
public static void main(String[] args) throws Throwable {
String baseUrl= "Enter base http/https url here";
String url1 = baseUrl+ "add addational url of main page";
String xpathofdownlaodlinks = "xpath of file url or--> html/body/div/div[3]/a/#href";
String pathToSaveFile="d:\\local\\to\\save\\files";
String fileExt = ".txt";
WebClient webclient = new WebClient(BrowserVersion.CHROME);
webclient.getOptions().setJavaScriptEnabled(true);
HtmlPage page = webclient.getPage(url1);
List<DomAttr> links = (List<DomAttr>) page.getByXPath(xpathofdownlaodlinks);
List<HtmlAnchor> anchors=page.getAnchors();
for (DomAttr object : links) {
String link = baseUrl+object.getValue()+"";
Date d=new Date();
downlaodRawFile(link,pathToSaveFile +d.getTime() + fileExt);
}
webclient.close();
}
public static void downlaodRawFile(String link,String fileName) throws IOException, Throwable{
URL url = new URL( link );
HttpURLConnection http = (HttpURLConnection)url.openConnection();
Map< String, List< String >> header = http.getHeaderFields();
while( isRedirected( header )) {
link = header.get( "Location" ).get( 0 );
url = new URL( link );
http = (HttpURLConnection)url.openConnection();
header = http.getHeaderFields();
}
InputStream input = http.getInputStream();
byte[] buffer = new byte[4096];
int n = -1;
OutputStream output = new FileOutputStream( new File( fileName ));
while ((n = input.read(buffer)) != -1) {
output.write( buffer, 0, n );
}
output.close();
}
private static boolean isRedirected( Map<String, List<String>> header ) {
for( String hv : header.get( null )) {
if( hv.contains( " 301 " )
|| hv.contains( " 302 " )) return true;
}
return false;
}
}
Thanks for reporting, the error is now fixed in SVN.
Please use latest build or snapshot.
I have a JavaScript code that I need to get into console application. The script works fine trough cmd but I want to make a console application out of it so its more user friendly. Can someone explain to me how I can write this code in console application or attach it inside the console application with links. I am new to console application so I apologize if I ask anything stupid :-)
When I use it trough cmd then I do the following;
- Run cmd.
- Type "cd downloads" and press enter.
- Type "cscript /nologo process.js log.txt 100 200" and press enter.
- Then I will get a list in the cmd window and I need to have process.js and log.txt in the download folder to make this work.
if(WScript.Arguments.Count() < 3)
{
WScript.Echo("Usage: cscript process.js <filename> <lower_value> <upper_value>");
WScript.Quit();
}
var filename = WScript.Arguments.Item(0);
var lowerBound = parseInt(WScript.Arguments.Item(1));
var upperBound = parseInt(WScript.Arguments.Item(2));
WScript.Echo("Here is the data from the file associated with the text 'verdi', where the");
WScript.Echo("number following 'verdi' is above " + lowerBound + " and below " + upperBound);
var fso = new ActiveXObject("Scripting.FileSystemObject")
var file = fso.OpenTextFile("log.txt", 1, false);
var lines = file.ReadAll().split('\r');
var failed = 0;
for(var idx in lines)
{
try
{
if(lines[idx].indexOf('verdi') > 0)
{
var tmp = lines[idx];
var regex = /verdi\s*\=\s*(\d+)/;
var result = regex.exec(tmp);
var num = parseInt(result[1]);
if(num >= lowerBound && num <= upperBound)
{
WScript.Echo(num);
}
}
}
catch(ex)
{
failed++;
}
}
if(failed > 0)
{
WScript.Echo("WARNING: one or more lines could not be processed!");
}
I have made this code in console application but it doesent work properly. I can choose the values and get the cmd to run. But I don't get the results in the window and print the result to a document.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApplication3
{
class Program
{
private static object cmd;
private static int verdi;
private static int s;
private static int d;
public static object WScript { get; private set; }
static void Main(string[] args)
{
//Choose lower and upper value
Console.WriteLine("Choose a lower and upper value:");
string value = Console.ReadLine();
//Choose file
Console.WriteLine("Choose a file to scan:");
string file = Console.ReadLine();
//Run the javascript code
Console.WriteLine("cd downloads");
Console.WriteLine("cscript /nologo process.js {0} {1} > mydata.txt", file, value);
string command = Console.ReadLine();
Console.WriteLine("Press any key to start scan");
System.Diagnostics.Process.Start("cmd.exe", "/C" + command);
//Quit Console Application
Console.WriteLine("Press any key to quit.");
Console.ReadKey();
}
}
}
Console.WriteLine only prints strings. It doesn't allow you to execute commands.
You could try this instead:
string command = $"cscript /nologo c:/downloads/process.js c:/downloads/{file} {lowerValue} {upperValue} > mydata.txt");
System.Diagnostics.Process.Start($"cmd.exe /C {command}");
There is also an error in process.js. That script always reads from log.txt and ignores the filename.
But why are you using two programs here? You could just have all the code in one file. And why use JavaScript for one and C# for the other?
I have Silverlight app. For example I want add some JS script which can interact with SL app. For example I want add google map use JS api. Can I do this, But I must send some data from SL to JS to add pins on map, draw figures on map ect.
If you are using Windows Phone and you dont mind that part of the xaml to be a webview yo can.
First add a webview to the xaml
<phone:WebBrowser Name="webView" BorderThickness="0" BorderBrush="Transparent" IsScriptEnabled="True"
ScriptNotify="WebBrowser_ScriptNotify" />
Then you have to bind the webview with the load event and then saveg files to storage and load your html and js files
webView.Loaded += WebBrowser_OnLoaded;
private void WebBrowser_OnLoaded(object sender, RoutedEventArgs e)
{
SaveFilesToIsoStore();
chatView.Navigate(new Uri("Assets/HtmlContent/index.html", UriKind.Relative));
}
private void SaveFilesToIsoStore()
{
//These files must match what is included in the application package,
//or BinaryStream.Dispose below will throw an exception.
string[] files = {
"Assets/HtmlContent/index.html",
"Assets/HtmlContent/js/libs/jquery-1.11.0.min.js", "Assets/HtmlContent/js/pagejs.js", "Assets/HtmlContent/css/style.css"
};
IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication();
if (false == isoStore.FileExists(files[0]))
{
foreach (string f in files)
{
StreamResourceInfo sr = Application.GetResourceStream(new Uri(f, UriKind.Relative));
using (BinaryReader br = new BinaryReader(sr.Stream))
{
byte[] data = br.ReadBytes((int)sr.Stream.Length);
SaveToIsoStore(f, data);
}
}
}
}
private void SaveToIsoStore(string fileName, byte[] data)
{
string strBaseDir = string.Empty;
string delimStr = "/";
char[] delimiter = delimStr.ToCharArray();
string[] dirsPath = fileName.Split(delimiter);
//Get the IsoStore.
IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication();
//Re-create the directory structure.
for (int i = 0; i < dirsPath.Length - 1; i++)
{
strBaseDir = System.IO.Path.Combine(strBaseDir, dirsPath[i]);
isoStore.CreateDirectory(strBaseDir);
}
//Remove the existing file.
if (isoStore.FileExists(fileName))
{
isoStore.DeleteFile(fileName);
}
//Write the file.
using (BinaryWriter bw = new BinaryWriter(isoStore.CreateFile(fileName)))
{
bw.Write(data);
bw.Close();
So on the js you have to talk to the c# like this
function sendMessageToCodeBehind(someData) {
window.external.notify(JSON.stringify({ method: 'AddMessage', data: someData }));
}
On the code behind you would recibe the messages from the webview like this:
private void WebBrowser_ScriptNotify(object sender, NotifyEventArgs e)
{
var example = new { method = string.Empty, data = new object() };
var obj = JsonConvert.DeserializeAnonymousType(e.Value, example);
switch (obj.method) {
case "methodName":
}
}
And you would send back messages to the js like this
webView.InvokeScript("jsMethodName", JsonConvert.SerializeObject(new { Message = "some json message" }));