Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I would like to make a web scraping application that is able to log in to a website (I was able to do this with twill (python)), and also to be able to execute JavaScript which trigger access to other pages.
I would definitely prefer to use something in python, but I am ready to try something new. I have installed mechanize, watir, Hojocki, etc. but not sure if this really helps.
I'd recommend PhantomJS.
It's a full Webkit browser, but headless and scriptable.
It's ideal for this sort of thing.
I believe there are a few modules (such as Ghost), but I have used Selenium/WebDriver for things like this. It is ostensibly a testing framework, but it provides you with a lot of methods to allow you to interact with the page just as if you had loaded it as a normal user. You also have the benefit of running it so that a browser actually opens and you can watch the code execute (makes debugging easier), or in a 'headless' mode where the code just executes (there are other sites/SO answers with much better explanations than I can give :) ).
That being said, Ghost looks great as well, so try them both and hopefully one will get you what you need!
Also, see Javascript (and HTML rendering) engine without a GUI for automation? for a similar question that may have some additional answers.
I would recommend Octoparse, a free web scraper for Windows.
It's not programmble but it's very easy to use. But there's no Mac version.So...
JavaScript can be handled by Octoparse btw.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
We have a webshop and we sell lots of items.
Our checkout process consists of 4 different pages where the user has to input their address, select a delivery method and confirm their order on different pages/urls. Each of those pages relies on communication with the server and lots of javascript / jquery.
Some of our users have reported problems at some parts of those pages. We suspect it could be a combination of OS/browser that can't understand a part of our javascript code.
Is there any way to automate testing of a checkout process of 4 different consecutive pages, each requiring user input?
We would like the testing environment to test on different brosers/browser versions.
We also had a customer recently that had an antivirus program that would change the urls of our js source files, is there any way to capture cases like that by testing?
Sounds like you might use E2E testing, using Protractor/Selenium.
It's basically about writing user behaviour, and the browser driver does it instead of user. You can write what should happen on the page and if any of these conditions is not met, it will be included in the post-test report. You can configure it to use whatever browser driver you like.
I suggest you to take a look at Selenium. The main purpose of it is to automate browsers actions.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I started learning JavaScript a while back and it didn't take long for me to realize that JS relies heavily on HTML/CSS (there isn't even a way to get an input from the user using just JS unless you use prompt or HTML).
However, I love the language itself and I was wondering if there is any way to have a project run on JUST JS. (Or if there is another language that's language is as beautiful and simple as JavaScript but still has the practicality of Java or C++. I find Python/Ruby's syntax to be odd, sorry).
I know this isn't the usual question here at Stack Overflow, but I'm somewhat new to this and there's no other clear, definitive answer.
You could look at Node.js. Which is a backend server framework that uses Javascript as the language.
But if you want to display user interface code to browsers, you still can't get away from HTML and Javascript.
If you want to interact with an user in a browser, you must need html, or prompts as you said otherwise the user have nothing to do there :).
And If you are looking a way to execute javascript at server side
Look at, node.js that take advantage of V8 JavaScript Engine
Another option is io.js
With Node.js, you can write a project beyond web.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
we want to provide an online bookmark service to our customers. Therefor it would be nice to provide also a function to store local bookmarks into our online tool. This shouldn't be done automatically - the user should confirm this action (only in case someone thinks I want to spy users data).
I'm looking for a while now, but I found only that there is no solution, because it's a security risk. Is there a way or already a tool for such a functionality in Javascript?
Regards
This is completely impossible.
You need to write a browser addon.
Is there a way or already a tool for such a functionality in Javascript?
Nope, definitely not. Maybe for browser extensions - if it's possible even for them.
No there isn't any solution to do this via javascript. The only solution I can think of is to use a Java Applet to read the client's computer and look for the bookmark.
Another way is you can ask them to export their favorites to a file and upload to your website.
Prompt the user to upload the bookmarks file: http://productforums.google.com/forum/#!topic/chrome/i1r3CgqvgCI
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I am currently working on a site that includes javascript code that we get from several different sources and need to run on the site I maintain. Every once and a while some of this code breaks without our knowing until its too late. Is there a monitoring tool that will crawl our site and look for javascript errors and report them or could this be incorporated into a selenium test somehow?
On the sites I develop, I wrap everything in try ... catch blocks, and if the exceptions I catch cannot be handled, I always generate an AJAX request to a script which emails an error report to the development team with as much information as I can gather.
If the code is code you didn't write yourself and try...catch blocks would be difficult to add, you can use the window.onerror handler instead:
<script type="text/javascript">
window.onerror = function()
{
// Your code to generate an AJAX request to your error report script here
}
</script>
I know this post is old, but recently we've launched a tool that does this :)
It's called ConsoleWatch - https://www.consolewatch.io/
It lets you scan whole websites for JS errors and also schedule repeating scans with reports, so you might find t handy!
It would also be smart to utilize a tool that will catch any JavaScript errors that might happen after production. There're several tools out there but I recommend RootCause because it will allow you to automatically reproduce any user errors.
Disclaimer: I work for RootCause. Our software automatically reproduces JavaScript errors and lets you replay user sessions live in your browser with the click of a button.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
Is there any python module for rendering a HTML page with javascript and get back a DOM object?
I want to parse a page which generates almost all of its content using javascript.
The big complication here is emulating the full browser environment outside of a browser. You can use stand alone javascript interpreters like Rhino and SpiderMonkey to run javascript code but they don't provide a complete browser like environment to full render a web page.
If I needed to solve a problem like this I would first look at how the javascript is rendering the page, it's quite possible it's fetching data via AJAX and using that to render the page. I could then use python libraries like simplejson and httplib2 to directly fetch the data and use that, negating the need to access the DOM object. However, that's only one possible situation, I don't know the exact problem you are solving.
Other options include the selenium one mentioned by Łukasz, some kind of webkit embedded craziness, some kind of IE win32 scripting craziness or, finally, a pyxpcom based solution (with added craziness). All these have the drawback of requiring pretty much a fully running web browser for python to play with, which might not be an option depending on your environment.
You can probably use python-webkit for it. Requires a running glib and GTK, but that's probably less problematic than wrapping the parts of webkit without glib.
I don't know if it does everything you need, but I guess you should give it a try.