Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Is there a plug-in for Firefox that would allow user's Javascript code like Greasemonkey and be able to browse from page to page?
I'd like to write a script to:
Log in to a website.
Follow several links.
Make a GET request to that host periodically with given data and time intervals.
Make a POST request based on the results of the previous in-loop requests.
Now I use Python's mechanize for a browser so I'm looking for something with similar (maybe not that rich) functionality within Firefox. Do you have experience with that type of things? What should I check out?
Selenium - which has an interface for recording and running tests inside the browser but can also export tests in many languages including Python for running as a suite in the SeleniumRC tool.
Or
Chickenfoot (You'll probably need to use setTimeout for the repeating requests.)
(source: mit.edu)
You also have iMacros
The software's description on Mozilla Addons says :
Automate Firefox. Record and replay
repetitious work. If you love the
Firefox web browser, but are tired of
repetitive tasks like visiting the
same sites every days, filling out
forms, and remembering passwords, then
iMacros for Firefox is the solution
you’ve been dreaming of! Whatever you
do with Firefox, iMacros can automate
it.
(source: extjs.com)
I would recommend Selenium RC. It comes as a Java command line tool and allows you to remote-control both Firefox, IE and Safari. Although it is officially a browser based web-testing tool, it can be very useful for crawling and scraping AJAX-based web applications and for all sort of automated tasks otherwise difficult to accomplish with non graphical HTTP clients such as Curl, Hpricot and Mechanize.
Moreover, it's widely spread, has an API for most popular programming languages (including python), and allows you inject custom javascript code within web-pages.
PS:
Documentation is here
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
I love the idea of crypto.cat for sharing sensitive information. Recently, i had to send my wife my social security number and i didn't want to use email/sms/IM/etc... I wanted to use crypto.cat, but she didn't want to install an extension on her work computer, so just ended up calling her.
I found myself wondering why an extension is even necessary. Looking back through their blog, I found that they switched from a pure web-based application to a browser extension. They claimed this improves security but they didn't explain why.
Looking through their github, the code appears to be all javascript so why not just skip the extension? Im thining about forking Crypto.cat and re-implementing a pure web-based version, but I'd like to understand why this is a bad idea before i start.
My ideas so far
Using an extension would make phishing more difficult
It helps prevent code injection attacks by men-in-the-middle. If you go to the Crypto.cat website every time you want to use the service, your browser will download the application source code to execute. A MITM could use this opportunity to inject code, which undermines the whole security of the service. Even SSL wouldn't necessarily help much unless you pay very close attention to the certificate and the entire chain of trust, since a MITM could wedge in his own certificate.
Installing a browser extension under trusted conditions once mitigates these concerns, since then the whole code is already on your machine and nobody can inject anything.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I know how to do my jobs by C++. I like to move most of
them to Windows Script Host JScript. But Windows JScript
doesn't do many important things.
Many JavaScript kernels are open source, and we have node.js,
processing.js already.
So we need to be able to build our own JavaScript interpreter
so as to bypass JScript. This sounds true for desktop applications
, right?
I need some hints to start it. It's worthy even if it takes years.
JScript limitations are :
1. can't access hardware.
2. can't do things that need administrator privilege.
A JScript wrapper like xNeat http://www.xneat.com helps JScript to call APIs. But it does not work with WinIO (access I/O ports and physical memory) because WinIO needs administrator privilege that is not allowed by JScript.
I'll try to compile V8 or the likes and remove restrictions it intentionally made for security then add some low level interfaces to call Windows APIs, which JScript can't do. I think this is cool. Maybe I don't even need to do that again because somebody already done that?
You seriously need a lot more research... I'll try & save a few flops of Google:
JScript != JavaScript. Assuming you are using them interchangably
without knowing about the existence of proprietary JScript from
Microsoft..
Javascript was never meant to do things you mention as limitations. It's supposed to run in web browsers & obviously a random website whose JS it is, should not be able to access/execute anything in the user's machine.
Although contrary to belief that JS does not need to touch the file system, the filesystem API was brought in. I feel, that expands the boundaries wide enough for any web application to leverage on.
Besides, you can always develop desktop apps for windows 8 in JS, by the libraries exposed by Microsoft, but that's another story.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I would like to make a web scraping application that is able to log in to a website (I was able to do this with twill (python)), and also to be able to execute JavaScript which trigger access to other pages.
I would definitely prefer to use something in python, but I am ready to try something new. I have installed mechanize, watir, Hojocki, etc. but not sure if this really helps.
I'd recommend PhantomJS.
It's a full Webkit browser, but headless and scriptable.
It's ideal for this sort of thing.
I believe there are a few modules (such as Ghost), but I have used Selenium/WebDriver for things like this. It is ostensibly a testing framework, but it provides you with a lot of methods to allow you to interact with the page just as if you had loaded it as a normal user. You also have the benefit of running it so that a browser actually opens and you can watch the code execute (makes debugging easier), or in a 'headless' mode where the code just executes (there are other sites/SO answers with much better explanations than I can give :) ).
That being said, Ghost looks great as well, so try them both and hopefully one will get you what you need!
Also, see Javascript (and HTML rendering) engine without a GUI for automation? for a similar question that may have some additional answers.
I would recommend Octoparse, a free web scraper for Windows.
It's not programmble but it's very easy to use. But there's no Mac version.So...
JavaScript can be handled by Octoparse btw.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 months ago.
Improve this question
Say a webpage loads an external javascript at load, is there any such FireFox plugin that I could use to modify the javascript before the page actually processes it?
(not just specifically javascript)
Thanks in advance.
(also I'm pretty sure Tamper Data plugin only changes header data and not actual content being received)
For everyone that has never used tamperdata: Tamper data is for OUTGOING requests. Tamper Data can modify the ENTIRE request, except the URL which requires you to replay the request.
Using GreaseMonkey you can make stand-alone custom plugins that can modify any element of the page before it loads.
Here is THE GUIDE you want which explains GreaseMonkey.
Here are a massive number of GreaseMonkey "UserScripts". This site contains many examples of what you are looking for.
You have a couple options:
Tamper Data will modify POST parameters (and GET really since you can modify the URL).
You can also combine FoxyProxy (https://addons.mozilla.org/en-US/firefox/addon/2464) with any number of free interactive proxies (Fiddler, Paros, Burp, Charles)
Finally you can choose to not use a proxy and write up a greasemonkey script.
I think you'll likely have the most luck with the FoxyProxy + proxy approach. Unfortunately it's not a single addon.
The minimalistic browser-agnostic approach would be to write your own bookmarklet. For example I have found the Show Hiddens bookmarklet to be extremely useful for debugging form submissions. While extremely simple the bookmarklet does things which Tamper Data cannot.
I have found it here:
http://www.squarefree.com/bookmarklets/forms.html
Also the Forms tab in the Web Developer toolbar has some useful options.
If You want to change a downloadable resource, use Opera, set it's cache to never expire, and modify the files cached. That's how I did it a year or two ago, successfully.
I believe GreaseMonkey can modify the data in the page, though I'm not sure if it's executed before or after the page loads.
Check out TamperMonkey for Chrome: http://tampermonkey.net/
Or if you want to do it manually, in Chrome, it's really simple.
In Chrome, browse to: chrome://extensions. Then drag your .js file into that page.
Chrome will automatically create a manifest.json file in the Chrome AppData folder.
You can change the manifest.json file to filter the websites you want to use your script on.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
Is there any python module for rendering a HTML page with javascript and get back a DOM object?
I want to parse a page which generates almost all of its content using javascript.
The big complication here is emulating the full browser environment outside of a browser. You can use stand alone javascript interpreters like Rhino and SpiderMonkey to run javascript code but they don't provide a complete browser like environment to full render a web page.
If I needed to solve a problem like this I would first look at how the javascript is rendering the page, it's quite possible it's fetching data via AJAX and using that to render the page. I could then use python libraries like simplejson and httplib2 to directly fetch the data and use that, negating the need to access the DOM object. However, that's only one possible situation, I don't know the exact problem you are solving.
Other options include the selenium one mentioned by Łukasz, some kind of webkit embedded craziness, some kind of IE win32 scripting craziness or, finally, a pyxpcom based solution (with added craziness). All these have the drawback of requiring pretty much a fully running web browser for python to play with, which might not be an option depending on your environment.
You can probably use python-webkit for it. Requires a running glib and GTK, but that's probably less problematic than wrapping the parts of webkit without glib.
I don't know if it does everything you need, but I guess you should give it a try.