We Keep Coding

How to download a csv file after login by using Casperjs

I want to donwload a csv file by using Caperjs.
This is what I wrote:
var login_id = "my_user_id";
var login_password = "my_password";
var casper = require('casper').create();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 ');
casper.start("http://eoddata.com/symbols.aspx",function(){
this.evaluate(function(id,password) {
document.getElementById('tl00_cph1_ls1_txtEmail').value = id;
document.getElementById('ctl00_cph1_ls1_txtPassword').value = password;
document.getElementById('ctl00_cph1_ls1_btnLogin').submit();
}, login_id, login_password);
});
casper.then(function(){
this.wait(3000, function() {
this.echo("Wating...");
});
});
casper.then(function(){
this.download("http://eoddata.com/Data/symbollist.aspx?e=NYSE","nyse.csv");
});
casper.run();
And I got nyse.csv, but the file was a HTML file for registration of the web site.
It seems login process fails. How can I login correctly and save the csv file?
2015/05/13
Following #Darren's help, I wrote like this:
casper.start("http://eoddata.com/symbols.aspx");
casper.waitForSelector("form input[name = ctl00$cph1$ls1$txtEmail ]", function() {
this.fillSelectors('form', {
'input[name = ctl00$cph1$ls1$txtEmail ]' : login_id,
'input[name = ctl00$cph1$ls1$txtPassword ]' : login_password,
}, true);
});
And this code ends up with error Wait timeout of 5000ms expired, exiting..
As far as I understand the error means that the CSS selector couldn't find the element. How can I find a way to fix this problem?
Update at 2015/05/18
I wrote like this:
casper.waitForSelector("form input[name = ctl00$cph1$ls1$txtEmail]", function() {
this.fillSelectors('form', {
'input[name = ctl00$cph1$ls1$txtEmail]' : login_id,
'input[name = ctl00$cph1$ls1$txtPassword]' : login_password,
}, true);
}, function() {
fs.write("timeout.html", this.getHTML(), "w");
casper.capture("timeout.png");
});
I checked timeout.html by Chrome Developer tools and Firebugs, and I confirmed several times that there is the input element.
<input name="ctl00$cph1$ls1$txtEmail" id="ctl00_cph1_ls1_txtEmail" style="width:140px;" type="text">
How can I fix this problem? I already spent several hours for this issue.
Update 2015/05/19
Thanks for Darren, Urarist and Artjom I could remove the time out error, but there is still another error.
Downloaded CSV file was still registration html file, so I rewrote the code like this to find out the cause of error:
casper.waitForSelector("form input[name ='ctl00$cph1$ls1$txtEmail']", function() {
this.fillSelectors('form', {
"input[name ='ctl00$cph1$ls1$txtEmail']" : login_id,
"input[name ='ctl00$cph1$ls1$txtPassword']" : login_password,
}, true);
});/*, function() {
fs.write("timeout.html", this.getHTML(), "w");
casper.capture("timeout.png");
});*/
casper.then(function(){
fs.write("logined.html", this.getHTML(), "w");
});
In the logined.html user email was filled correctly, but password is not filled. Is there anyone who have guess for the cause of this?

The trick is to successfully log in. There are multiple ways to login. I've tried some and the only one that works on this page is by triggering the form submission using the enter key. This is done by using the PhantomJS page.sendEvent() function. The fields can be filled using casper.sendKeys().
var casper = require('casper').create();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 ');
casper.start("http://eoddata.com/symbols.aspx",function(){
this.sendKeys("#ctl00_cph1_ls1_txtEmail", login_id);
this.sendKeys("#ctl00_cph1_ls1_txtPassword", login_password, {keepFocus: true});
this.page.sendEvent("keypress", this.page.event.key.Enter);
});
casper.waitForUrl(/myaccount/, function(){
this.download("http://eoddata.com/Data/symbollist.aspx?e=NYSE", "nyse.csv");
});
casper.run();
It seems that it is necessary to wait for that specific page. CasperJS doesn't notice that a new page was requested and the then() functionality is not used for some reason.
Other ways that I tried were:
Filling and submitting the form with casper.fillSelectors()
Filling through the DOM with casper.evaluate() and submitting by clicking on the login button with casper.click()
Mixing all of the above.

At first glance your script looks reasonable. But there are a couple of ways to make it simpler, which should also make it more robust.
First, instead of your evaluate() line,
this.fillSelectors('form', {
'input[name = id ]' : login_id,
'input[name = pw ]' : login_password,
}, true);
The true parameter means submit it. (I guessed the form names, but I'm fairly sure you could continue to use CSS IDs if you prefer.)
But, even better is to not fill the form until you are sure it is there:
casper.waitForSelector("form input[name = id ]", function() {
this.fillSelectors('form', {
'input[name = id ]' : login_id,
'input[name = pw ]' : login_password,
}, true);
});
This would be important if the login form is being dynamically placed there by JavaScript (possibly even from an Ajax call), so won't exist on the page as soon as the page is loaded.
The other change is instead of using casper.wait(), to use one of the casper.waitForXXX() to make sure the csv file link is there before you try to download it. Waiting 3 seconds will go wrong if the remote server takes more than 3.1 seconds to respond, and wastes time if the remote server only takes 1 second to respond.
UPDATE: When you get a time-out on the waitFor lines it tells you the root of your problem is you are using a selector that is not there. This, I find, is the biggest time-consumer when writing Casper scripts. (I recently envisaged a tool that could automate trying to find a near-miss, but couldn't get anyone else interested, and it is a bit too big a project for one person.) So your troubleshooting start points will be:
Add an error handler to the timing-out waitFor() command and take a screenshot (casper.capture()).
Dump the HTML. If you know the ID of a parent div, you could give that, to narrow down how much you have to look for.
Open the page with FireBug (or tool of your choice) and poke around to find what is there. (remember you can type a jQuery command, or document.querySelector() command, in the console, which is a good way to interactively find the correct selector.)
Try with SlimerJS, instead of PhantomJS (especially if still using PhantomJS 1.x). It might be that the site uses some feature that is only supported in newer browsers.

CasperJS- Register on a site and validate the mail sent on Gmail -for both slimer and phantom-

Edit : this is the windows behaviour, with linux it just fails.
First, if you succeeded navigate on gmail with casper (without random waiting time -from 20sec to 5min-), please tell me.
I want to register on our site, then validate my registration automatically with Gmail (an entire register step). Did someone do that before?
I have no problem to register, and I can login on my mailbox (Gmail) but after i have some troubles to navigate and validate my registration in Gmail, and i observe different behaviors between phantomJS and slimerJS.
In phantom it will work (without special commands), but it may take until 5 minutes before pass in the next step (waitForSelector). And with slimerjs it just stays stuck on the mailbox page.
EDIT : A strange thing : if i click manually (slimer) on a link which opens a popup, it stops being blocked and my navigation continues, it's like it can't detect the end of the step itself and can't perform the waitFor after the submit click without another interaction. Is it a refresh/reload problem?
Try that to see yourself :
casper.thenOpen('https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=en', function(){
this.sendKeys("input#Email","your mail");
this.sendKeys("input#Passwd","your password");
this.click("input#signIn.rc-button-submit");
console.log(this.getCurrentUrl());
this.waitForSelector(".aeF",function(){//fail with linux -> timeout
this.test.pass("ok"); //windows -> stuck in slimer, several times in phantom
this.test.assertExists(".T-I.J-J5-Ji.T-I-KE.L3","Gmail Home ok");
console.log("url "+this.getCurrentUrl());
});
And i don't get any timeOut error. In slimerjs it just keeps the page opened.
If i do a waitForPopup instead of a waitForUrl, i have the error (timeout -> did not pop up), so why does a waitForUrl/waitForSelector... stay stuck ? I tried --web-security=no,--ignore-ssl-errors=true commands too (not linked but i tried --output-encoding=ISO 8859-1 too which doesn't work).
Here the differences between phantom and slimer (doc) :
http://docs.slimerjs.org/0.8/differences-with-phantomjs.html
(useless in this issue i think)

Well, we finally found a way to do it : the problem is by default gmail loop on ajax requests, to check some new mails, etc... see Page polls remote url, causing casper to block in step.
Fortunately google proposes a way to avoid that, using the simplified HTML version (you can for example use a special gmail address for your tests using this version) :
That way the script works as it should.
Bonus :
/*
* Click on an element specified by its selector and a part of its text content.
* This method resolves some problem as random space in textNode / more flexible too.
* Need to fix one bug though : when there is a tag in textContent of our selector.
*/
casper.clickSelectorHasText = function (selector, containsText){
var tmp = this.getElementsInfo(selector)
,i
,l
,bool=false
;
for (i=0,l=tmp.length;i<l; i++){
if(tmp[i].text && tmp[i].text.indexOf(containsText)!==-1){
this.clickLabel(tmp[i].text);
bool=true;
break;
}
}
casper.test.assert(bool, "clickSelectorHasText done, text and selector found -> click selector " + selector +" which contains text " + containsText);
};
casper.thenOpen('https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=en', function scrapeCode(){
//log in
this.sendKeys("input#Email","your email");
this.sendKeys("input#Passwd","your password");
this.click("input#signIn.rc-button-submit");
//wait to redirect to our mailbox
this.waitForSelector("form[name='f']",function(){
//check main block
this.test.assertExists("form[name='f']","Gmail Home ok");
this.test.assertSelectorHasText("span", "Your gmail title message");
this.clickSelectorHasText("font", "one string which appears in font tag");
//wait inscription message appears
this.waitForSelector("div.msg",function(){
this.test.assertSelectorHasText("a","the message which activates your account--> in <a>");
});
})
//validate our account
.then(function(){
this.clickLabel("the message which activates your account--> in <a>");
this.waitForPopup(/mail/, function(){
this.test.pass("popup opened");
});
this.withPopup(/mail/, function(){
this.viewport(1400,800);
this.test.pass("With Popup");
//wait something on your website (for me selector .boxValid)
this.waitForSelector(".boxValid", function(){
/*
* Here your code after validation
*/
});
});
})
It might be possible to do it with normal gmail using event, see resource.received.

Communicating between Chrome DevTools and content script in extension

(I have already read this and it didn't work, and I've done a lot of searching and experimentation to no avail.)
I am writing a Chrome extension (BigConsole) with the goal of building a better Console tab for the Chrome developer tools. This means I would like to execute user-input code in the context of the page with access to the DOM and other variables on the page. To do this, the communication is structured as follows:
devtools creates a panel where the user writes code
When the user wants to execute code from the panel, the panel sends a message to a background script with the code
The background script receives the message/code from panel and passes it on to the content script which is injected into the page
The content script receives the message/code from the background script and injects a script element into the page which then runs the code
The result of the script on the page is then posted back to the content script with window.postMessage
The content script listens for the message/result from the page and passes it on to the background script
The background script receives the message/result from the content script and passes it on to the panel
The panel receives the message/result from the background script and inserts it into the log of results
Whew.
Right now, when the user tries to run the code, nothing happens. I put a bunch of console.log()s into the code but nothing appears in the console. My main question is, what have I done wrong here with the message passing that results in nothing happening? Alternatively, I would love to be told that I am making this way too complicated and there is a better way of doing things. Simplified code below...
panel.js:
window.onload = function() {
var port = chrome.runtime.connect({name: "Eval in context"});
// Add the eval'd response to the console when the background page sends it back
port.onMessage.addListener(function (msg) {
addToConsole(msg, false);
});
document.getElementById('run').addEventListener('click', function() {
var s = document.getElementById('console').value;
try {
// Ask the background page to ask the content script to inject a script
// into the DOM that can finally eval `s` in the right context.
port.postMessage(s);
// Outputting `s` to the log in the panel works here,
// but console.log() does nothing, and I can't observe any
// results of port.postMessage
}
catch(e) {}
});
};
background.js:
chrome.runtime.onConnect.addListener(function (port) {
// Listen for message from the panel and pass it on to the content
port.onMessage.addListener(function (message) {
// Request a tab for sending needed information
chrome.tabs.query({'active': true,'currentWindow': true}, function (tabs) {
// Send message to content script
if (tab) {
chrome.tabs.sendMessage(tabs[0].id, message);
}
});
});
// Post back to Devtools from content
chrome.runtime.onMessage.addListener(function (message, sender) {
port.postMessage(message);
});
});
content.js:
// Listen for the content to eval from the panel via the background page
chrome.runtime.onMessage.addListener(function (message, sender) {
executeScriptInPageContext(message);
});
function executeScriptInPageContext(m) { alert(m); }

As pointed out by Alex, here's a typo in your code which prevents it from working.
Drop your current code and use chrome.devtools.inspectedWindow.eval to directly run the code and parse the results. This simplifies your complicated logic to:
devtools creates a panel where the user writes code
devtools runs code
devtools handles result
PS. There is a way to manipulate the existing console, but I recommend against using it, unless it's for personal use. Two different ways to do this are shown in this answer.

casperjs testing an internal site

I am trying to run a casper test for an internal site. Its running on pre-production environment, the code so far is
var casper = require('casper').create({
verbose: true,
loglevel:"debug"
});
// listening to a custom event
casper.on('page.loaded', function() {
this.echo('The page title is ' + this.getTitle());
this.echo('value is: '+ this.getElementAttribute
('input[id="edit-capture-amount"]',
'value'));
});
casper.start('https://preprod.uk.systemtest.com', function() {
this.echo(this.getTitle());
this.capture('frontpage.png');
// emitting a custom event
this.emit('age.loaded.loaded');
});
casper.run();
as you can see its not much but my problem is the address is not reachable. The capture also shows a blank page. Not sure what i am doing wrong. I have checked the code with cnn and google urls, the title and screen capture works fine. Not sure how to make it work for an internal site.

I had the exact same problem. In my browser I could resolve the url, but capserjs could not. All I got was about::blank for a web page.
Adding the --ignore-ssl-errors=yes worked like a charm!
casperjs mytestjs //didn't work
capserjs --ignore-ssl-errors=yes mytestjs //worked perfect!

Just to be sure.
Can you reach preprod.uk.systemtest.com from the computer on which casper runs ? For example with a ping or wget.
Is there any proxy between your computer and the preprod server ? Or is your system configured to pass through a proxy that should not be used for the preprod server ?
The casper code seems to be ok.
I know this should be a comment but I don't have enough reputation to post a comment.

As far as CasperJs tests are run in localhost, for testing a custom domain/subdomain/host, some headers need to be defined.
I experienced some problems when passing only the HOST header, for instance, snapshots were not taken properly.
I added 2 more headers and now my tests run properly:
casper.on('started', function () {
var testHost = 'preprod.uk.systemtest.com';
this.page.customHeaders = {
'HOST': testHost,
'HTTP_HOST': testHost,
'SERVER_NAME': testHost
};
});
var testing_url: 'http://localhost:8000/app_test.php';
casper.start(_testing_url, function() {
this.echo('I am using symfony, so this should have to show the homepage for the domain: preprod.uk.systemtest.com');
this.echo('An the snapshot is also working');
this.capture('casper_capture.png');
}

ApplicationUpdaterUI fails for Mac

LOG:
Runtime Installer begin with version 3.3.0.3650 on Mac OS 10.7.4 x86
Commandline is: -updatecheck
Installed runtime (3.3.0.3650) located at /Library/Frameworks/Adobe AIR.framework
Performing pingback request
Failure during pingback request: [ErrorEvent type="error" bubbles=false cancelable=falsr eventPhase=2 text="Unhandled exception Error: Error #3001" errorID=3001]
Runtime Installer end with exit code 0
It works fine on Windows but fails for Mac.
Digging around I found out that the error code #3001 has something to do with file/directory permission issues.
Checked /Users/internetslave/Library/Application Support/Adobe permissions seems ok. source.
Checked /Library/Frameworks/Adobe AIR.framework seems ok too.
Both had drwxr-xr-x.
UPDATE: permission is not an issue, successfully updated other applications on the same system.
var appUpdater;
function checkForUpdates() {
appUpdater = new air.ApplicationUpdaterUI();
appUpdater.configurationFile = new air.File("app:/update/update-config.xml");
appUpdater.addEventListener(air.ErrorEvent.ERROR, onCheckForUpdatesError);
appUpdater.initialize();
setTimeout(function() {
appUpdater.checkNow();
}, 500);
}
function onCheckForUpdatesError(event) {
alert(event.toString());
}
Cant seem to post the update configuration and descriptor files here.

For those might stumble upon the same problem.
The problem was the ApplicationUpdaterUI has not finished initializing when the method checkNow is called.
So either, change the second parameter of the setTimeout to a higher value or place this part
appUpdater = new air.ApplicationUpdaterUI();
appUpdater.configurationFile = new air.File("app:/update/update-config.xml");
appUpdater.addEventListener(air.ErrorEvent.ERROR, onCheckForUpdatesError);
appUpdater.initialize();
in page onLoad event handler and attach the checkNow method to a button onClick event
function checkUpdates() {
appUpdater.checkNow();
}
<input type="button" onclick="checkUpdates()" />
Thanks!