Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
Is there any python module for rendering a HTML page with javascript and get back a DOM object?
I want to parse a page which generates almost all of its content using javascript.
The big complication here is emulating the full browser environment outside of a browser. You can use stand alone javascript interpreters like Rhino and SpiderMonkey to run javascript code but they don't provide a complete browser like environment to full render a web page.
If I needed to solve a problem like this I would first look at how the javascript is rendering the page, it's quite possible it's fetching data via AJAX and using that to render the page. I could then use python libraries like simplejson and httplib2 to directly fetch the data and use that, negating the need to access the DOM object. However, that's only one possible situation, I don't know the exact problem you are solving.
Other options include the selenium one mentioned by Łukasz, some kind of webkit embedded craziness, some kind of IE win32 scripting craziness or, finally, a pyxpcom based solution (with added craziness). All these have the drawback of requiring pretty much a fully running web browser for python to play with, which might not be an option depending on your environment.
You can probably use python-webkit for it. Requires a running glib and GTK, but that's probably less problematic than wrapping the parts of webkit without glib.
I don't know if it does everything you need, but I guess you should give it a try.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 months ago.
Improve this question
I would like to use Blazor, preferably server side. I also would like to use, reuse my existing themes and visual design, however this many time includes using great javascript libraries, which modifies DOM programmatically to achieve their effect.
As far as I understand Blazor works with virtual DOM. What I do not understand how can the virtual DOM be in sync the real DOM, if any javascript code modifies the real DOM behind Blazors back.
The only way I could imagine, if there is a mechanism built in to Blazor to deal with this scenario, however even after years after Blazor in the scene, I can not see it documented, the issue I am talking about does not seem even a topic in Google searches.
Question
Is there any way to use Blazor with javascript libraries, which are modifying DOM? I mean have Blazor any built in mechanism to deal with this, or in case it have no, is there any common best practice?
From the documentation:
Interaction with the Document Object Model (DOM)
Only mutate the Document Object Model (DOM) with JavaScript (JS) when
the object doesn't interact with Blazor. Blazor maintains
representations of the DOM and interacts directly with DOM objects. If
an element rendered by Blazor is modified externally using JS directly
or via JS Interop, the DOM may no longer match Blazor's internal
representation, which can result in undefined behavior. Undefined
behavior may merely interfere with the presentation of elements or
their functions but may also introduce security risks to the app or
server.
This guidance not only applies to your own JS interop code but also to
any JS libraries that the app uses, including anything provided by a
third-party framework, such as Bootstrap JS and jQuery.
In a few documentation examples, JS interop is used to mutate an
element purely for demonstration purposes as part of an example. In
those cases, a warning appears in the text.
For more information, see Call JavaScript functions from .NET methods
in ASP.NET Core Blazor.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I started learning JavaScript a while back and it didn't take long for me to realize that JS relies heavily on HTML/CSS (there isn't even a way to get an input from the user using just JS unless you use prompt or HTML).
However, I love the language itself and I was wondering if there is any way to have a project run on JUST JS. (Or if there is another language that's language is as beautiful and simple as JavaScript but still has the practicality of Java or C++. I find Python/Ruby's syntax to be odd, sorry).
I know this isn't the usual question here at Stack Overflow, but I'm somewhat new to this and there's no other clear, definitive answer.
You could look at Node.js. Which is a backend server framework that uses Javascript as the language.
But if you want to display user interface code to browsers, you still can't get away from HTML and Javascript.
If you want to interact with an user in a browser, you must need html, or prompts as you said otherwise the user have nothing to do there :).
And If you are looking a way to execute javascript at server side
Look at, node.js that take advantage of V8 JavaScript Engine
Another option is io.js
With Node.js, you can write a project beyond web.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I would like to make a web scraping application that is able to log in to a website (I was able to do this with twill (python)), and also to be able to execute JavaScript which trigger access to other pages.
I would definitely prefer to use something in python, but I am ready to try something new. I have installed mechanize, watir, Hojocki, etc. but not sure if this really helps.
I'd recommend PhantomJS.
It's a full Webkit browser, but headless and scriptable.
It's ideal for this sort of thing.
I believe there are a few modules (such as Ghost), but I have used Selenium/WebDriver for things like this. It is ostensibly a testing framework, but it provides you with a lot of methods to allow you to interact with the page just as if you had loaded it as a normal user. You also have the benefit of running it so that a browser actually opens and you can watch the code execute (makes debugging easier), or in a 'headless' mode where the code just executes (there are other sites/SO answers with much better explanations than I can give :) ).
That being said, Ghost looks great as well, so try them both and hopefully one will get you what you need!
Also, see Javascript (and HTML rendering) engine without a GUI for automation? for a similar question that may have some additional answers.
I would recommend Octoparse, a free web scraper for Windows.
It's not programmble but it's very easy to use. But there's no Mac version.So...
JavaScript can be handled by Octoparse btw.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
Has anyone here used KSS?
KSS is an AJAX framework.
KSS has both a client-side Javascript library and server-side support.
The client-side Javascript library needs to be included in your page. It fetches Kinetic style sheets from the server, parses them and binds a set of action to browser events and/or page elements. It is clean Javascript code that can peacefully coexist with other clean Javascript librarys(sic) like JQuery or ExtJS. It is about 100k in production mode. You can integrate your own Javascript code by using its extension mechanism through plugins.
I'm currently working on a project that uses it. Are there any drawbacks and gotchas to be aware of?
What's its cross browser support like?
At first as was really put off by the fact that you don't write the JS by hand, and actually translates a CSS-like file to JS behavior, but seeing in action, I've got to say that it really works quite well. But I haven't done any cross browser tests yet.
Some things that I've found:
it sends HTML from the server, instead of XML and/or JSON and replacing them clientside, meaning higher messages (understandable)
it has problems with scripts that add iframes dynamically on a KSS widget that you reload
some things are hard to debug, while others are made easy thanks to KSS' integration with Firebug
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Is there a plug-in for Firefox that would allow user's Javascript code like Greasemonkey and be able to browse from page to page?
I'd like to write a script to:
Log in to a website.
Follow several links.
Make a GET request to that host periodically with given data and time intervals.
Make a POST request based on the results of the previous in-loop requests.
Now I use Python's mechanize for a browser so I'm looking for something with similar (maybe not that rich) functionality within Firefox. Do you have experience with that type of things? What should I check out?
Selenium - which has an interface for recording and running tests inside the browser but can also export tests in many languages including Python for running as a suite in the SeleniumRC tool.
Or
Chickenfoot (You'll probably need to use setTimeout for the repeating requests.)
(source: mit.edu)
You also have iMacros
The software's description on Mozilla Addons says :
Automate Firefox. Record and replay
repetitious work. If you love the
Firefox web browser, but are tired of
repetitive tasks like visiting the
same sites every days, filling out
forms, and remembering passwords, then
iMacros for Firefox is the solution
you’ve been dreaming of! Whatever you
do with Firefox, iMacros can automate
it.
(source: extjs.com)
I would recommend Selenium RC. It comes as a Java command line tool and allows you to remote-control both Firefox, IE and Safari. Although it is officially a browser based web-testing tool, it can be very useful for crawling and scraping AJAX-based web applications and for all sort of automated tasks otherwise difficult to accomplish with non graphical HTTP clients such as Curl, Hpricot and Mechanize.
Moreover, it's widely spread, has an API for most popular programming languages (including python), and allows you inject custom javascript code within web-pages.
PS:
Documentation is here