I am learning pyqt,use for parse webpage.
Now i want use pyqt evaluate javascript function just like this answer do:
spidermonkey evaluate js function which in remote js file
import urllib2
import spidermonkey
js = spidermonkey.Runtime()
js_ctx = js.new_context()
script = urllib2.urlopen('http://etherhack.co.uk/hashing/whirlpool/js/whirlpool.js').read()
js_ctx.eval_script(script)
js_ctx.eval_script('var s = "abc"')
js_ctx.eval_script('print(HexWhirlpool(s))')
I want know how to achieve the same effect by using pyqt instead of spidermonkey.
PyQt is a UI framework, so it might not be the best arrow in your quiver.
You can use WebView to load and display the web page. Since WebView uses WebKit, it will load and parse the HTML and all CSS and JavaScript in it. But that means you just get the final result - you don't have much control what is loaded and when.
Or you can use a tool like Beautiful Soup to parse the HTML. This gives you full control. You can then try to use spidermonkey to run the JavaScript but that will fail since many global variables like window or document will be missing. Also, there won't be a DOM.
For this, you'd need something like Envjs but there is no Python integration.
Maybe PhantomJS (a script driven Chrome/Chromium browser) is an option. See this question: Is there a way to use PhantomJS in Python?
Related
Put it simple I want to make small currency exchange app (pet project- I want free API( 1000 requests per month including more currency is a perfect option)). I dont like the free APIs I have found so far but I have found this website https://bg.coinmill.com/ and I wanna use it for my purpose. Reading an answer to similar question:
The only way to make use of JS in Flutter is using WebView.
Dart compiles to JS only for browser applications, for Flutter it compiles >to native machine code.
convert js code direcly to dart, using package js
package JS doesn't convert JS, it just creates proxies for JS functions to >be able to call them from Dart, but that is also only supported in Dart web >applications.
Put it simple it isn't possible without hitting some compilation errors and some workarounds. However https://github.com/pichillilorenzo/flutter_inappbrowser looks promissing. Embedding the webpage that will look ugly and I won't have any control over ui/settings. My options now are looking for another free currency API or trying to find a workaround. I incline for another API, but not sure which one. Any suggestions ?
So basically what you actually want to do is use that website to do the currency conversion in the background (enter value, press "Convert"), then display the result in your Flutter app? You don't need javascript for that.
After entering pressing the submit button, the site simply redirects you to a different page (GET request) with an URL like this:
https://bg.coinmill.com/CAD_USD.html?CAD=22
Use dart's http library to perform the same request with the right currency/value parameters. The result of the request contains the source code of the web page.
Instead of displaying the web page, you just need to read the value you need from the source code of the web page:
<div id="currencyBox1">
<input class="currencyField" ... value="16.46" ...>
САЩ долар (USD)
</div>
So, how I understand your question, you have some js library, and you want to use it from Dart?
If question so, yes, you can do it using Dart JS Intertop. The more information in the link.
Edit
Yes, you are right, you can call js from Flutter only using evalJavascript function from flutter_webview_plugin.
You can use Firebase Cloud Functions and wrap your function in a callable function. You'll have all node js environment and Dart code will only call a function.
I am starting to study some web technologies to integrate content, markup, layout, styling and behaviors of stuff for personal use (NOT web developing for now) and am amazed with the power of JQuery selectors and functions.
I have heard that there are some ways to use javascript "outside" a browser, to do some DOM selection, manipulation, etc. I wonder if JQuery could be used that way too.
So, what I would like to do is:
Using some programming/scripting language (I use Python), access a XML file and parse its DOM;
Programmatically manipulate and modify the DOM with javascript/jquery selectors and functions;
Save the results to (possibly another) XML file.
If you like jQuery syntax, check out pyQuery:
from pyquery import PyQuery
_ = PyQuery('<body><p></p></body>')
_("p").text("hello").css({'color': 'red'})
print _.html()
>>> <p style="color: red">hello</p>
yeah, you just need a Javascript run time.
Check out node.js
What you're looking for is called a "headless" browser.
This SO post may help:
Real headless browser
Basically you need a javascript interpreter (ex: V8) + wrapper for your language of choice (ex: pyv8). Then you can do this (from pyv8 page):
import PyV8
ctxt = PyV8.JSContext()
ctxt.enter()
ctxt.eval("1+2") # 1+2 is a javascript code
I have been researching for the standard practice to analyze the markup of a web page after javascript processing within a script or from the command line, i.e. without any browser?
This needs to happen on a Linux environment. Are the are "installables" that would allow you to pass HTML markup including javascript and it would return the markup after simulating a standard browser request and all Javascript calls have been done?
If there are any Perl Modules you can think of than that would be of even more help.
I have been looking at https://developer.mozilla.org/en/SpiderMonkey and http://search.cpan.org/~mschilli/JavaScript-SpiderMonkey-0.12/SpiderMonkey.pm but I am not sure this would allow me to pass in a full HTML document in and get the processed version with all javascript DOM manipulations back?
Please let me know.
Update, I figure it out
I figured it all out - this is what needs to be done:
#!/usr/bin/perl
use WWW::Scripter;
$w = new WWW::Scripter;
$w->use_plugin('JavaScript');
$w->get('http://www.google.com');
print $w->content(),"\n";
You have to use a browser, a new one like WWW::Scripter::Plugin::Javascript
or an old one like WWW::Mechanize::Firefox
Maybe the solution could be headless browser like PhantomJS. Not a perl module, but very practical for front-end testing and automation.
How does one parse html documents which make heavy use of javascript? I know there are a few libraries in python which can parse static xml/html files and I'm basically looking for a programme or library (or even firefox plugin) which reads html+javascript, executes the javascript bit and outputs html code without javascript so it would look identical if displayed in a browser.
As a simple example
link
should be replaced by the appropriate value the javascript function returns, e.g.
link
A more complex example would be a saved facebook html page which is littered with loads of javascript code.
Probably related to
How to "execute" HTML+Javascript page with Node.js
but do I really need Node.js and JSDOM? Also slightly related is
Python library for rendering HTML and javascript
but I'm not interested in rendering just the pure html output.
You can use Selenium with python as detailed here
Example:
import xmlrpclib
# Make an object to represent the XML-RPC server.
server_url = "http://localhost:8080/selenium-driver/RPC2"
app = xmlrpclib.ServerProxy(server_url)
# Bump timeout a little higher than the default 5 seconds
app.setTimeout(15)
import os
os.system('start run_firefox.bat')
print app.open('http://localhost:8080/AUT/000000A/http/www.amazon.com/')
print app.verifyTitle('Amazon.com: Welcome')
print app.verifySelected('url', 'All Products')
print app.select('url', 'Books')
print app.verifySelected('url', 'Books')
print app.verifyValue('field-keywords', '')
print app.type('field-keywords', 'Python Cookbook')
print app.clickAndWait('Go')
print app.verifyTitle('Amazon.com: Books Search Results: Python Cookbook')
print app.verifyTextPresent('Python Cookbook', '')
print app.verifyTextPresent('Alex Martellibot, David Ascher', '')
print app.testComplete()
From Mozilla Gecko FAQ:
Q. Can you invoke the Gecko engine from a Unix shell script? Could you send it HTML and get back a web page that might be sent to the printer?
A. Not really supported; you can probably get something close to what you want by writing your own application using Gecko's embedding APIs, though. Note that it's currently not possible to print without a widget on the screen to render to.
Embedding Gecko in a program that outputs what you want may be way too heavy, but at least your output will be as good as it gets.
PhantomJS can be loaded using Selenium
$ ipython
In [1]: from selenium import webdriver
In [2]: browser=webdriver.PhantomJS()
In [3]: browser.get('http://seleniumhq.org/')
In [4]: browser.title
Out[4]: u'Selenium - Web Browser Automation'
What I need is simple: we have a console app project. We want to have such a function that would open a simple window with nothing but html (default system based) html + js rendering window that would read some default html+js string (form string or const char*). we want to have our own api joined with default js api so that when JS calls some our.function(argument); we would have our C++ application performe some function and for example cout argument. How to create such thing on windows?
CoCreate the MSHTML com object for HTML Documents:
CComPtr spDoc;
HRESULT hr = CoCreateInstance(CLSID_HTMLDocument, NULL, CLSCTX_INPROC_SERVER, IID_IHTMLDocument2, (void**)&spDoc);
Do something like this to read your HTML string and make the document render it.
Depending on what you need in terms of callbacks, you can use the COM DOM Interfaces to traverse the tree, and then sink the appropriate DispInterfaces to get events on the elements you're interested in. I would recommend this.
If what I suggest in #3 isn't good enough for you (and I'd like to hear why) then you can implement your own ActiveX control and have script on your page call methods on it as suggested by raj.
Brushing aside any security / cross browser/platforms concerns you can use implement an ActiveX object in your C++ that you can invoked from javascript.
http://msdn.microsoft.com/en-us/library/7sw4ddf8(v=vs.94).aspx
The host for the WebBrowser control can provide an object that will be accessible to scripts via the external object.
See http://msdn.microsoft.com/en-us/library/aa770041.aspx#GetExternal