I would like to automate the process of visiting a website, clicking a button, and saving the file. The only way to download the file on this site is to click a button. You can't navigate to the file using a url.
I have been trying to use phantomjs and casperjs to automate this process, but haven't had any success.
I recently tried to use brandon's solution here
Grab the resource contents in CasperJS or PhantomJS
Here is my code for that
var fs = require('fs');
var cache = require('./cache');
var mimetype = require('./mimetype');
var casper = require('casper').create();
casper.start('http://www.example.com/page_with_download_button', function() {
});
casper.then(function() {
this.click('#download_button');
});
casper.on('resource.received', function (resource) {
"use strict";
for(i=0;i < resource.headers.length; i++){
if(resource.headers[i]["name"] == "Content-Type" && resource.headers[i]["value"] == "text/csv; charset-UTF-8;"){
cache.includeResource(resource);
}
}
});
casper.on('load.finished', function(status) {
for(i=0; i< cache.cachedResources.length; i++){
var file = cache.cachedResources[i].cacheFileNoPath;
var ext = mimetype.ext[cache.cachedResources[index].mimetype];
var finalFile = file.replace("."+cache.cacheExtension,"."+ext);
fs.write('downloads/'+finalFile,cache.cachedResources[i].getContents(),'b');
}
});
casper.run();
I think the problem could be caused by my cachePath being incorrect in cache.js
exports.cachePath = 'C:/Users/username/AppData/Local/Ofi Labs/PhantomJS';
Should I be using something in adition to the backslashes to define the path?
When I try
casperjs --disk-cache=true export_script.js
Nothing is downloaded. After a little debugging I have found that cache.cachedResources is always empty.
I would also be open to solutions outside of phantomjs/casperjs.
UPDATE
I am not longer trying to accomplish this with CasperJS/PhantomJS.
I am using the chrome extension Tampermonkey suggested by dandavis.
Tampermonkey was extremely easy to figure out.
I installed Tampermonkey, navigated to the page with the download link, and then clicked New Script under tampermonkey and added my javascript code.
document.getElementById("download_button").click();
Now every time I navigate to the page in my browser, the file is downloaded. I then created a batch script that looks like this
set date=%DATE:~10,4%_%DATE:~4,2%_%DATE:~7,2%
chrome "http://www.example.com/page-with-dl-button"
timeout 10
move "C:\Users\user\Downloads\export.csv" "C:\path\to\dir\export_%date%.csv"
I set that batch script to run nightly using the windows task scheduler.
Success!
Your button most likely issues a POST request to the server.
In order to track it:
Open Network tab in Chrome developer tools
Navigate to the page and hit the button.
Notice which request led to file download. Right click on it and copy as cURL
Run copied cURL
Once you have cURL working you can schedule downloads using cron or Task Scheduler depending on operation system you are using.
Related
This is for Windows.
I have a flash application I am converting to AIR. I built a captive installer using NSIS and it works fine. However I would like to have an icon on a website which checks if the application is already installed and ask the user if they wish to run it. If it is not installed, they get the option to download it.
I am fairly certain this is doable, because Zoom and GoToMeeting both do this.
My searching skills seem to be failing me when looking for this.
Edit:
It appears the best/only way to do this is to create a custom protocol for the application. Something like DoDaApp://.
Which brings up the next set of questions;
How to create an NSIS file which will create the appropriate registry entries on the client computer? As a user, not admin.
How to check if the protocol is currently installed on the computer?
This is a partial answer as it does not work in Edge. I'll explain the issue below.
As recommended in How to detect browser's protocol handlers you can use timeout & blur event handlers. Here is my interpretation of the code;
function checkCustomProtocol(inProtocol,inInstalLink,inTimeOut)
{
var timeout = inTimeOut;
window.addEventListener('blur',function(e)
{
window.clearTimeout(timeout);
}
)
timeout = window.setTimeout(function()
{
console.log('timeout');
window.location = inInstalLink;
}, inTimeOut
);
window.location = inProtocol;
}
Microsoft Edge is ever so helpful by popping up a dialog box telling you "You'll Need a new app to open this" which "blurs" the screen, not allowing download of the file.
So I will be posting another question on how to make it work in Edge. I have reviewed ismailhabib's code but the known issues section says it doesn't work with Edge either.
Here is a more complete answer. It has been lightly tested in IE 11, Microsoft Edge, Chrome and Firefox. I also added comments;
/*
checkCustomProtocol - check if custom protocol exists
inProtocol - URL of application to run eg: MyApp://
inInstallLink - URL to run when the protocol does not exist.
inTimeOut - time in miliseconds to wait for application to Launch.
*/
function checkCustomProtocol(inProtocol,inInstalLink,inTimeOut)
{
// Check if Microsoft Edge
if (navigator.msLaunchUri)
{
navigator.msLaunchUri(inProtocol, function ()
{
//It launched, nothing to do
},
function()
{
window.location = inInstalLink; //Launch alternative, typically app download.
}
);
}
else
{
// Not Edge
var timeout = inTimeOut;
//Set up a listener to see if it navigates away from the page.
// If so we assume the papplication launched
window.addEventListener('blur',function(e)
{
window.clearTimeout(timeout);
}
)
//Set a timeout so that if the application does not launch within the timeout we
// assume the protocol does not exist
timeout = window.setTimeout(function()
{
console.log('timeout');
window.location = inInstalLink; //Try to launch application
}, inTimeOut
);
window.location = inProtocol; //Launch alternative, typically app download.
}
}
I'm am on an embedded system which has an implemented webserver and a ftp-server.
Both servers are handled by a RTOS and I cannot change server-side code at the time.
I need to access and download text-files programmatically over the ftp-server from my website.
So far I am dynamically creating an ftp-link and open it inside a new window, which only shows me the text-content of the requested file.
Now I want to download this text content for serveral files I am about to open in a loop:
Open window - download textcontent - Close window
... Open Window ... and so on.
I cannot "inject" Javascript for the new window, to make it download it content if Im right?
I open the window with
window.open('ftp://username:passwort#myfilename.txt')
from a script running on my webpage.
Is there any possibility to access and download that text content as described?
At the time I do not have the possibility to access the textfiles other than ftp, since the webserver and ftpserver are not on the same filesystem and I can not change the code on the serverside.
Now I know this is kind of hacky... yet I need a workaround for now to access the textfiles.
Thank you in advance!
If you're using node you can try node-ftp
Here's a quick example of how to use:
var Client = require('ftp');
var c = new Client();
c.on('ready', function() {
c.get('foo.txt', function(err, stream) {
if (err) throw err;
console.log(stream);
stream.once('close', function() { c.end(); });
});
});
// connect to localhost:21 as anonymous
c.connect();
I'm trying to trigger some sort of Folder Selection Dialog, I have a working model with nodejs and the powershell but it only works when the server and client are on the same machine. I need the prompt to occur on the client side triggered from the browser. From what i understand I can not trigger Powershell from Chrome? So is there an alternative or am i just screwed?
My current Powershell script
{
param([string]$Description="Select Folder",[string]$RootFolder="Desktop")
[System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms") |
Out-Null
$objForm = New-Object System.Windows.Forms.FolderBrowserDialog
$objForm.Rootfolder = $RootFolder
$objForm.Description = $Description
$Show = $objForm.ShowDialog()
If ($Show -eq "OK")
{
Return $objForm.SelectedPath
}
Else
{
Write-Error "Operation cancelled by user."
}
}
$folder = Select-FolderDialog # the variable contains user folder selection
write-host $folder
My javascript function
async function asyncfindDir() {
//executes powershell script
let promise = new Promise((resolve, reject) => {
const Shell = require('node-powershell');
const ps = new Shell({
executionPolicy: 'Bypass',
noProfile: true
});
ps.addCommand('./selectfolder.ps1');
ps.invoke()
.then(output => {
//console.log(output);
var shelloutput = output;
console.log (shelloutput + '^^from external script');
res.send(shelloutput);
})
.catch(err => {
console.log('please select a directory path')
//console.log('err');
});
});
};
Is there anyway to get that working locally?
Is there a trigger i'm not aware of to access that kind of dialog from the browser? I know i'm not the only person with this issue but i have yet to see a real solution.
Short answer: No.
Longer answer, is best illustrated by rephrasing your question with a different script name:
Using my browser, can I click on a link to visit a website, and have it run a random
PowerShell script called Delete_All_Files.ps1?
Answers why you will never be able to run a PowerShell script from a browser, on a remote machine, and why browsers will deliberately block you from doing it, because people usually don't want to have all their files deleted when they click on a random link in their email.
If you want to run PowerShell scripts on remote machines, then you should look into PSRemoting and Enter-PSSession.
#kuzimoto is right. If you just want to display a folder dialog box, there are easier ways to do that and Fine Uploader is an easier way.
Replying to your comment: If you want to specify a directory name, the reason you can't do it is because you are essentially asking:
Using my browser, can I click on a link to visit a website, and have
it run a script that will enumerate through all the files and folders
in my C:\ so that it can choose the folder C:\users\Justin
Miller\Desktop\SECRET FILES\?
The reason both operations do not work is because both operations require local computer access. i.e. local script execution access, and local directory knowledge access. Security-wize, we, in general, don't want to visit a random website and have it execute random code, or know what files/folders I have on my machine, which is why you won't be able to do what you want to try to do.
This is my first time learning to build a firefox addon. I want store all the open tabs in a window and for that I require sdk/tabs.
Here is my js file:
/*
Given the name of a beast, get the URL to the corresponding image.
*/
debugger;
var tabs = require("sdk/tabs");
function beastNameToURL(beastName) {
switch (beastName) {
case "Save Session":
debugger;
for (let tab of tabs)
console.log(tab.url);
return;
case "Load Session":
debugger;
return chrome.extension.getURL("beasts/snake.jpg");
case "Turtle":
return chrome.extension.getURL("beasts/turtle.jpg");
}
}
/*
Listen for clicks in the popup.
If the click is not on one of the beasts, return early.
Otherwise, the text content of the node is the name of the beast we want.
Inject the "beastify.js" content script in the active tab.
Then get the active tab and send "beastify.js" a message
containing the URL to the chosen beast's image.
*/
document.addEventListener("click", function(e) {
if (!e.target.classList.contains("btn")) {
return;
}
var chosenBeast = e.target.textContent;
var chosenBeastURL = beastNameToURL(chosenBeast);
chrome.tabs.executeScript(null, {
file: "/content_scripts/beastify.js"
});
chrome.tabs.query({active: true, currentWindow: true}, function(tabs) {
chrome.tabs.sendMessage(tabs[0].id, {beastURL: chosenBeastURL});
});
});
When I reach the var tabs = require("sdk/tabs") line I get a Reference error.
Github : https://github.com/sagar-shah/Session-manifest
Kindly let me know how do I resolve this error. This being my first time with add-ons I am completely lost.
Thanks in advance.
Update:
Tried to declare it globally in the js file. Now I am getting undefined error for tabs.
Update2:
I was mixing up development using sdk and webextensions as pointed out by #matagus. I have decided to go with development using the webextensions. Link to the new repository has been updated.
The error is on package.json line 6: you're telling to the addon sdk that the main file of your addon is manage.json. According to [the docs] the value of main should be:
A string representing the name of a program module that is located in one of the top-level module directories specified by lib. Defaults to "index.js".
So you need to change its value to index.js.
Besides that, I think you're missing a difference between Firefox addon built using the addon-sdk (which do not have a ´manifest.json´ and that you build using jpm tool) and the new WebExtensions which do require you to write a ´manifest.json´ like the one already have.
UPDATE:
Again: you're missing the difference between WebExtensions and SDK-based addons. Now you made a WebExtension but you're trying to use the SDK. It isn't possible. Just use chrome.tabs directly instead of trying to import it from the sdk (var tabs = require("sdk/tabs");).
I got task to write an add-on for Firefox which will add an div element to existing page. I downloaded Add-on SDK and wrote a main.js file that looks like this:
var data = require("sdk/self").data;
require("sdk/tabs").on("ready", ExecuteAd);
function ExecuteAd(tab) {
if ( tab.url.indexOf("some url checking") > -1 ) {
var image = "http://www.lavasoft.com/img/product_icons/aaw11/free.png";
var link = "http://www.google.me";
tab.attach({
contentScriptFile: data.url("myscript.js"),
contentScript: "appendFunc('"+image+"', '"+link+"');"
//contentScript: "alert('Works');"
});
}
}
When I execute command cfx run it starts Firefox and if I go to specific web pages this script works. But when I create XPI file with cfx xpi and then click on Firefox and open that file it installs my add-on but now when I go to same web pages I gave been before add-on does not work. I have this external Javascript file which is stored in folder 'data'.
appendFunc is in myscript.js file.
How to make my extension work in production environment not just testing environment? I think that main problem is that it does not find this data/myscript.js (does it include in .xpi file?)
Don't mix contentScript and contentScriptFile. Also, you cannot know what of both is loaded first.
Instead load your script, and communicate using port.
main.js
var data = require("sdk/self").data;
require("sdk/tabs").on("ready", ExecuteAd);
function ExecuteAd(tab) {
var image = "http://www.lavasoft.com/img/product_icons/aaw11/free.png";
var link = "http://www.google.me";
var worker = tab.attach({
contentScriptFile: data.url("myscript.js")
});
worker.port.emit("showAd", {image: image, link: link});
}
myscript.js
self.port.on("showAd", function(data) {
console.log("showing ad", data.link, data.image);
});
Also, it sounds like PageMod would be a better choice for what you're doing.
PS: Also consult the Add-on Policies if you're planning to host on the addons.mozilla.org website. The policies e.g. prohibit injecting ads that a) aren't clearly marked as such and b) where the user did not opt-in prior to that.