I would like to automatize web tasks which normally should be achieved by a web browser, for example video upload to a certain video sharing website. I would like to make a pure command line program, which apply simple cURL command calls, because those calls are accessible for example in FireBug and Chrome / Chromium dev console / Network pane. So I woudn't like to use libcurl, or similar libraries. I would prefer programming in Ruby.
Task is straightforward: I upload a video while watching the dev tool network pane, and tracking the communication between the browser and the server. Copying POST and GET requests by "copy as cURL" menu. Applying some modification on the copied cURL command, eg. removing some header lines, which sends cookies, and substitute them with the cookies in a "cookie jar" text file (-c option in cURL). And later sending the needed cookies by applying that text file again (-b option in cURL). In the past I managed to make such Ruby scripts and they are simply working I can use those websites services by pure command line, so I can upload files from my VPS which is very fast unlike uploading from home machine.
Unfortunately the website I want to automatize apply a lot of redirections even at the login stage (for example 4 consecutive redirects), which aren't tracked by Chrome dev tool, so I can't see what really happen and when the needed cookies are stored, and which request is responsible for getting those cookies. Sometimes tricky javascript calls are applied by the website to store a cookie which is needed for the video upload and even exporting the video.
So my question is that besides Chrome dev tools and FireBug is there any automated and handy tool which can help achiving similar tasks?
Maybe BrowserAutomationStudio will help:
https://bablosoft.com/shop/BrowserAutomationStudio
This program can record your browser actions and replay them as a standalone bot.
Related
The difficulty comes from the fact that chrome extensions can only make external http requests through the background.js file which is separate from the extension and all communication between the two are serialized, which removes files.
I've found some solutions from 1++ years ago, but all seem to be very hacky and not working anymore anyway (as tends to happen to hacky solutions).
So I am left wondering if it is possible at all to upload files from a chrome extension that opens on any website.
If memory serves, I am fairly sure I am making external web requests from a content script in a Chrome extension.
However, if we just consider how to get a file into the background thread, the following options jump to mind, depending on your needs.
chrome.storage can be used to access local storage
chrome.fileSystem can be used to access the host file system
File content could be suitably encoded (say base64) and sent across with chrome.runtime.sendMessage in the JSON message. (I am not sure if there is a message size constraint. I suppose it could be sent in pieces if there were.)
I'm making an extension that among other things edit a javascript file in an external editor (one on the user's computer). The extension has the javascript file saved in chrome.storage but it will ofcourse be a lot easier for the user to write code in their own editor.
This is why I decided to find something that creates a file on the user's filesystem which the user can find and edit it themselves, and if any changes are made, sync that back up to the extension (either by periodically checking or by using some listener).
I have looked around but nothing really seems to fit what I'm trying to do. Chrome's fileSystem API only works for chrome apps, not chrome extensions and the HTML5 fileSystem API does not allow for a simple filesystem URL to be requested and opened, instead it obfuscates the stored file and makes it practically impossible to edit that file easily.
Something else I looked at which might be more promising is letting the user edit one of the files in the directory where the extension is stored and somehow retrieving that content. This is however going to be a bit tough to implement with chrome's all the hash checking going on in chrome extensions not to mention the general modifying of those files' contents by the extension (possibly by hacking around by specifying your own update URL and "updating" a dummy javascript file that is going to be written to).
Is there any way to simply ask for a location to store a file and then allow the user to edit that file and sync it back up?
No, extensions are sandboxed from the real filesystem.
As you said, it's possible to read extension's own files; however, this is read-only for the extension and modifying those files on a deployed extension will result in Chrome detecting extension "tampering" and immediate disabling as a precaution.
The only way for a Chrome extension to escape the sandbox is, as wOxxOm suggested, a Native Host module. Note that this cannot be distributed in Chrome Web Store with the extension; it needs a separate installer.
Alternatively, you could use some sort of cloud storage with API to access it; e.g. a user could store something in a Dropbox subfolder, and your extension can authorize access to it via Dropbox API. Unfortunately, there is no "native" solution like syncFileSystem for Apps.
We are currently looking at porting a enterprise silverlight application over to html5. The major roadblock that we have hit is the ability to open files from the user's local disk. Currently they have a document library which just links to files on their computer that they can open from within the app and view or print out. All that I read is that you can only access the local sandbox of the web app with the html5 file api's. We want to load these files from code.
Does anyone know of any workarounds to this?
Thanks
There is no way for html5 to access local file without user selection. But FSO: FileSystemObject works for IE and MAYBE could be regarded as a work around. But still there are some requirements to meet.
It is possible to use chrome's filesystem API to access files on a users local filesytem. So you'd have to be willing to make this a chrome only application.
Using java you can create a "Signed" applet which has access to the local filesystem. (if the applet is signed you can request filesystm permissions)
then there is a tutorial for accessing methods of your java code directly from javascript here: http://docs.oracle.com/javase/tutorial/deployment/applet/invokingAppletMethodsFromJavaScript.html
you should be able to perform something similar from silverlight.
There is no workaround in pure HTML5/Javascript. You need the use of plugins, such as Java or Silverlight (maybe you shouldn't port it after all). As for workarounds, HTML5 gives you an easy way drag and drop multiple files that you could transfer on the server and then display back to your users. Another workaround would be to install a custom agent (a software with only a tray icon) that would send the information about the current user "document library" to server and then again, you could display it back to the user.
Note: I've heard somewhere that browsers will eventually stop supporting plugins. http://www.howtogeek.com/179213/why-browser-plug-ins-are-going-away-and-whats-replacing-them/
Ya, I agree with Markain. However, if you were to limit your audience solely to chrome users, I daresay, you would most likely use some of your users. If Huazhihao is right, then your number of leaving customers should decrease but users who regularly use firefox won't be happy. Overall, I think that this will not work. Otherwise, there would be too many websites that trashed your hard driver (or at least wherever you have the rights to edit/delete files). I think it would be best if your product was setup to synchronize the file whenever an internet connection was detected and a change was made to the file. That way the user would not need to visit the website whenever the file was uploaded. If this is some kind of an error file, then it would be most beneficial if you were to make a link in the application that when clicked, would upload the file to the website and the website were to do whatever was necessary. If this is a purely online thing, then I don't see what business you would have looking through other peoples' files =-). Hope I helped!
I built a CRM for a client of mine, and now they've requested an interesting feature:
For each customer record, they have a matching directory of files on their local computer. They want the ability to open that folder in Windows Explorer directly from within the web app (the app doesn't need access to the directory/files; it just has to launch Windows Explorer so that the user can interact with their files).
This is obviously not possible with regular JavaScript running in the browser (thankfully). I thought there might be some way to accomplish this by building a Chrome extension for this purpose, but it seems Chrome extensions/apps can only access a sandboxed filesystem, which doesn't serve my needs at all. Building an NPAPI plugin in out of the question since Chrome is discontinuing support for NPAPI.
File URIs don't solve this problem either. Their display is ugly, there's no drag-and-drop, no big icons/thumbnails, no sorting etc. They want the full capability of the Windows Explorer.
The only viable option I thought of is to create a local node.js server, make a localhost CORS request to that server, and then run an exec command from node.
Any better idea?
One possibility is to register a custom URI protocol handler with the user's operating system, and then your web page can contain links using your custom protocol, such as openfolder://c/path/to/folder This sort of customization is probably most commonly seen in practice with itunes:// links.
A quick Google search led me to this decent looking tutorial: https://support.shotgunsoftware.com/hc/en-us/articles/200213756-How-to-launch-external-applications-using-custom-protocols-rock-instead-of-http-
The downside is that the user will have to run a small installer of some sort in order to set the correct registry entries (or whatever the non-Windows equivalent is for other OSes) and to drop a small script on disk. That would be much lighter-weight than running a node.js server like you proposed, though.
The linked tutorial uses a Python script, but even that is probably overkill for your needs. A batch file would likely suffice.
EDIT: One additional note, please be aware of the security implications of implementing a custom handler like this. Any webpage in any browser can potentially take advantage of your custom protocol, and an attacker would be able to pass arbitrary data to your script. You should take steps to ensure that the script will not accidentally execute arbitrary commands that may be injected by a malicious web page, and that it will only open a folder and nothing else.
That would require each customer to run a node.js server, which seems unrealistic in your case.
You could use File URIs.
Browsers will refuse to open them by default. However, as suggested in this answer, you could ask your customers to install LocalLinks.
Ok so I'm lost here, frustrated and pulling my hair and out. Plus probably about to be fired or take a pay cut.
I moved Files from a development server to my local machine. The files are consistent (used diff tool), all the dependencies are there. It works for the most part. The problem is that the some of the javascript (not all) is just not working. We're using jquery and a lot of plugins for it. I've checked with the web developer plugin in firefox and all the js files are loading. I cleared the cache in both firefox and chrome multiple times to no avail. The development server is a windows server running wamp. My local machine is running ubuntu. Somebody tell me what I missed.
Download firebug as a Firefox extension and view the http request and responses.
Easiest may be from within the 'net' tab to determine if your script is making a request.
Very likely that it is a source domain issue. There are no work-around for this issue. The ajax request and the source data must be on the same domain.
It may have something to do with JavaScript's security limitations. (In certain circumstances) You can only operate on URLs or pages from the current domain, which most likely changed when you moved the files off the other server. More here.
Are you running the files via a webserver, or just opening the files directly? If it's the latter, you'll want to set up a server on your local machine for local testing, and serve the files using it. Otherwise, you'll very likely run into the domain restrictions others have mentioned above.
You may need to host the site using a local server. VS IDE has an add-on called live server. You need to set up a workspace in order for it to work. The port used on my machine was 5500.
You need to make sure any dependencies for javascript are running on your server or the javascript will not be executed. These dependencies are listed in the json file.
ex. If you require express, you need to be running node or the javascript won't execute in your web browser.
In the terminal:
node app.js
Any dependencies that are not installed and running on the server will not execute.
Are you accessing the html web pages through the webserver and not simply double clicking the file to open it?
Also if you have WebDeveloper toolbar installed the click "Disable", "Disable Javascript" and make sure "All Javascript" isn't ticked.