I want to emulate some human(not for spam use) for some my daily works. I don't want to use some curl/wget solution because it will include some works to analyze the HTTP package sent by browser. So, basically:
I will use the browser(eg. chrome) to login the system, so I have login state in that browser.
I open the search page for the system, and by any way I start the script.
The script will input some string into the input, and submit the form.
The browser redirects to the result page, and my script will analyze the result page and get the things I want.
Is there any existing solutions that I can use to write such script? The functionality should include:
fill in some and submit the form
in the result page the script can analyze the webpage and get the desired data.
I was trying to do it in but the domain for my program and the system that I want to run in is not in the same domain.
You may be interested in famous Selenium:
Selenium automates browsers. That's it. What you do with that power is
entirely up to you. Primarily it is for automating web applications
for testing purposes, but is certainly not limited to just that.
Boring web-based administration tasks can (and should!) also be
automated as well.
See its DEMO here.
Or you may be interested in iMacros addon of Firefox depending on your requirements.
Related
My problem is a bit complex. Hard to explain with words, so I broke it down into steps with pictures at each step.
Select a single date from these boxes. Hit submit
I will land on a page with a table. Copy the <tbody> element from the developer console.
Paste it into a text file. Save the text file with the date that was selected.
Repeat steps 1-3 for as many times as needed, selecting a new date each time (01-15-2018, 01-14-2018, 01-13-2018, and so on...)
Is it even possible to build a bot that does this? If yes, what tools would I use?
I know a fair amount of JavaScript and Python, so I'd prefer to use those 2 if possible.
Would need to know the URL you're looking at/look at the page source. If the date is supplied as any part of a request, and the response contains this data you're looking for, it should be simple to farm and analyze that data from a python script.
Walk through your clicks with the network tab of your browser's developer tools and you should see a request go out when you hit submit. Expedia just uses query parameters, and so the entire URL that you'll need pops up in the URL bar of your browser after hitting submit...
Tools:
If request-based:
Python
Requests module
If something cached/more complicated, there are tools for automating clicks and saving the results...I would guess that this won't be necessary though...
Update:
AJAX calls are HTTP requests and responses, and so you should be able to observe them in your networking tab of our web browser developer tools, and then mimic that request from a script, rather than from your browser.
The readability of the requests/responses and/or any measures the organization has implemented to make any application other than a browser unable to get the same response would be potential impediments, but even those should be imitable. If your browser is making the request, then there is no reason your python script can't make the same.
The method you seem to be interested in, although it sounds more complicated to me, is possible with automation tools like Selenium, as the other poster answered. Best of luck.
It is possible:
Take a look at selenium library (its commonly used for automated testing) for python. It should be able to select single dates, hit the submit button then go through the HTML code and grab data in the tag. After that you can use python by itself to store this data in a text file with a name of your choice in a location of your choice.
For example, you could go to the login page of Facebook with Chrome, open the console and type
document.getElementById("email").value = "myemailaddress#example.com";
document.getElementById("pass").value = "hunter42";
document.getElementById("login_form").submit();
and it would log you in (assuming correct login information).
How would you do this from the Windows cmd or with a batch file?
Automating logging into a website is my main obstacle for creating an automation where a script sends me a text message when I receive a new grade on my school's web portal.
I've experimented with doing a form submission with cURL in PHP, but the login form is very complex with a lot of hidden fields and whatnot.
Personal project aside, I really want to know if you're able to run a JS Script on a specific web page because there are a ton of circumstances where this could be useful.
Thanks!
You're not going to be able to do it from the command line (which includes the run dialog). Your best bet is to find a headless browser framework like Selenium or PhantomJS. I've used PhantomJS, CasperJS, and NightmareJS to much success in the past and would recommend them especially if you're familiar with JavaScript already.
I'm not very familiar with browser extensions and before I begin to deeply explore them I have a few questions.
Let's say the extension injects JavaScript in the current website the user is visiting (if that's even possible). That injected JavaScript code will get, let's say the current URL for example purposes, and send it and store it on a database. Next time the user visits the same website, the user will get an extension notification informing that is the second or third or X time he or she has visited the same website.
Now that I have gave you the scenario, is the following possible? Injecting JavaScript from a browser extension to the current visiting website. If so, can I make some AJAX communication with the JavaScript and a PHP server?
Yes, you can inject stuff. See e.g. Insert code into the page context using a content script and How to inject javascript into page, from a Firefox add-on, and run it? or one of the many dupes there likely are.
You can then use whatever communication would be available between the site and a server, e.g. XHR, WebSockets, JSONP.
Please also check the policies of the Chrome Web Store and Mozilla Add-ons site regarding content/code injection and privacy rules. E.g. the Mozilla Add-ons will reject your add-on if you injected remote scripts (meaning code that is not bundled, e.g. originating from e.g. http:) and may also reject your stuff if you track users without prior explicit user consent.
Here's the situation: I'm redesigning our company's public facing website using ASP.NET, VB.NET and some javascript/jquery. Some of the features I'm adding require page reloads (which register as popups) and cookies. Works great if everything is enabled. But I've noticed on some browsers (such as Firefox) I still get prompted to ok these actions, retain these cookies, etc. Now I can code some contingencies for simpler pages for users who will not or can not enable these features, but I'd like to find a way to make it as simple as possible to enable the full features. From what I've read, there's no way I can actually force it to happen the way you can force a browsers document mode by settings in the web.config file, but I am hoping there is some way to give them a button to click (or something similar) where it will enable what I need. Is there a way to do this programmatically? what I'm looking for is some code that will make the changes, instead of directing them to go into e.g. Internet Explorer security settings, which most end users find tedious if not incomprehensible.
Advice?
You can avoid using cookies. Use Session or a database backend for things you would normally use cookies for. For popups, use overlaid divs such as Ajax Control Toolkit Modal Popup Extender or jQuery UI Dialog instead of starting a new browser window.
But really, ASP.NET is designed to function with cookies. If you're users aren't using them, tell them they're penalizing themselves.
Certain browser features are ONLY user-configurable for security reasons. You cannot provide a button to change these settings because they would then not user-configurable.
All you can do is warn the user.
JavaScript cannot change a client browser's setting due to security reason. Otherwise, all hell will break loose.
Note: you can if you create an executable program, and a user runs on his/her computer.
However, you should never change a user's browser setting.
Instead you should give the warning and instruction to a user which is a proper way of doing it.
Disable Javascript
Disable Cookie
I built a CRM for a client of mine, and now they've requested an interesting feature:
For each customer record, they have a matching directory of files on their local computer. They want the ability to open that folder in Windows Explorer directly from within the web app (the app doesn't need access to the directory/files; it just has to launch Windows Explorer so that the user can interact with their files).
This is obviously not possible with regular JavaScript running in the browser (thankfully). I thought there might be some way to accomplish this by building a Chrome extension for this purpose, but it seems Chrome extensions/apps can only access a sandboxed filesystem, which doesn't serve my needs at all. Building an NPAPI plugin in out of the question since Chrome is discontinuing support for NPAPI.
File URIs don't solve this problem either. Their display is ugly, there's no drag-and-drop, no big icons/thumbnails, no sorting etc. They want the full capability of the Windows Explorer.
The only viable option I thought of is to create a local node.js server, make a localhost CORS request to that server, and then run an exec command from node.
Any better idea?
One possibility is to register a custom URI protocol handler with the user's operating system, and then your web page can contain links using your custom protocol, such as openfolder://c/path/to/folder This sort of customization is probably most commonly seen in practice with itunes:// links.
A quick Google search led me to this decent looking tutorial: https://support.shotgunsoftware.com/hc/en-us/articles/200213756-How-to-launch-external-applications-using-custom-protocols-rock-instead-of-http-
The downside is that the user will have to run a small installer of some sort in order to set the correct registry entries (or whatever the non-Windows equivalent is for other OSes) and to drop a small script on disk. That would be much lighter-weight than running a node.js server like you proposed, though.
The linked tutorial uses a Python script, but even that is probably overkill for your needs. A batch file would likely suffice.
EDIT: One additional note, please be aware of the security implications of implementing a custom handler like this. Any webpage in any browser can potentially take advantage of your custom protocol, and an attacker would be able to pass arbitrary data to your script. You should take steps to ensure that the script will not accidentally execute arbitrary commands that may be injected by a malicious web page, and that it will only open a folder and nothing else.
That would require each customer to run a node.js server, which seems unrealistic in your case.
You could use File URIs.
Browsers will refuse to open them by default. However, as suggested in this answer, you could ask your customers to install LocalLinks.