I am trying to run a casper test for an internal site. Its running on pre-production environment, the code so far is
var casper = require('casper').create({
verbose: true,
loglevel:"debug"
});
// listening to a custom event
casper.on('page.loaded', function() {
this.echo('The page title is ' + this.getTitle());
this.echo('value is: '+ this.getElementAttribute
('input[id="edit-capture-amount"]',
'value'));
});
casper.start('https://preprod.uk.systemtest.com', function() {
this.echo(this.getTitle());
this.capture('frontpage.png');
// emitting a custom event
this.emit('age.loaded.loaded');
});
casper.run();
as you can see its not much but my problem is the address is not reachable. The capture also shows a blank page. Not sure what i am doing wrong. I have checked the code with cnn and google urls, the title and screen capture works fine. Not sure how to make it work for an internal site.
I had the exact same problem. In my browser I could resolve the url, but capserjs could not. All I got was about::blank for a web page.
Adding the --ignore-ssl-errors=yes worked like a charm!
casperjs mytestjs //didn't work
capserjs --ignore-ssl-errors=yes mytestjs //worked perfect!
Just to be sure.
Can you reach preprod.uk.systemtest.com from the computer on which casper runs ? For example with a ping or wget.
Is there any proxy between your computer and the preprod server ? Or is your system configured to pass through a proxy that should not be used for the preprod server ?
The casper code seems to be ok.
I know this should be a comment but I don't have enough reputation to post a comment.
As far as CasperJs tests are run in localhost, for testing a custom domain/subdomain/host, some headers need to be defined.
I experienced some problems when passing only the HOST header, for instance, snapshots were not taken properly.
I added 2 more headers and now my tests run properly:
casper.on('started', function () {
var testHost = 'preprod.uk.systemtest.com';
this.page.customHeaders = {
'HOST': testHost,
'HTTP_HOST': testHost,
'SERVER_NAME': testHost
};
});
var testing_url: 'http://localhost:8000/app_test.php';
casper.start(_testing_url, function() {
this.echo('I am using symfony, so this should have to show the homepage for the domain: preprod.uk.systemtest.com');
this.echo('An the snapshot is also working');
this.capture('casper_capture.png');
}
Related
I'm coding a script in nodejs to automatically retrieve data from an online directory.
Knowing that I had never done this, I chose javascript because it is a language I use every day.
I therefore from the few tips I could find on google use request with cheerios to easily access components of dom of the page.
I found and retrieved all the necessary information, the only missing step is to recover the link to the next page except that the one is generated 4 seconds after loading of page and link contains a hash so that this step Is unavoidable.
What I would like to do is to recover dom of page 4-5 seconds after its loading to be able to recover the link
I looked on the internet, and much advice to use PhantomJS for this manipulation, but I can not get it to work after many attempts with node.
This is my code :
#!/usr/bin/env node
require('babel-register');
import request from 'request'
import cheerio from 'cheerio'
import phantom from 'node-phantom'
phantom.create(function(err,ph) {
return ph.createPage(function(err,page) {
return page.open(url, function(err,status) {
console.log("opened site? ", status);
page.includeJs('http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js', function(err) {
//jQuery Loaded.
//Wait for a bit for AJAX content to load on the page. Here, we are waiting 5 seconds.
setTimeout(function() {
return page.evaluate(function() {
var tt = cheerio.load($this.html())
console.log(tt)
}, function(err,result) {
console.log(result);
ph.exit();
});
}, 5000);
});
});
});
});
but i get this error :
return ph.createPage(function (page) {
^
TypeError: ph.createPage is not a function
Is what I am about to do is the best way to do what I want to do? If not what is the simplest way? If so, where does my error come from?
If You dont have to use phantomjs You can use nightmare to do it.
It is pretty neat library to solve problems like yours, it uses electron as web browser and You can run it with or without showing window (You can also open developer tools like in Google Chrome)
It has only one flaw if You want to run it on server without graphical interface that You must install at least framebuffer.
Nightmare has method like wait(cssSelector) that will wait until some element appears on website.
Your code would be something like:
const Nightmare = require('nightmare');
const nightmare = Nightmare({
show: true, // will show browser window
openDevTools: true // will open dev tools in browser window
});
const url = 'http://hakier.pl';
const selector = '#someElementSelectorWitchWillAppearAfterSomeDelay';
nightmare
.goto(url)
.wait(selector)
.evaluate(selector => {
return {
nextPage: document.querySelector(selector).getAttribute('href')
};
}, selector)
.then(extracted => {
console.log(extracted.nextPage); //Your extracted data from evaluate
});
//this variable will be injected into evaluate callback
//it is required to inject required variables like this,
// because You have different - browser scope inside this
// callback and You will not has access to node.js variables not injected
Happy hacking!
I'm looking for a way to take a screenshot of a long web page every time it changes. I would like to use Node.js for this. My question is about how to render the full page with images and save it to disk ad an image file.
Most images on the webpage is lazy loaded. So I guess that I need to scroll down the entire page first, before taking a screen shot.
I tried different tools:
casperjs
node-webshot
phantomjs
All of them seems way too complicated, if not impossible, to even install. I didn't succeed with any of them.
casperjs seems like a really nice choice, but I can't get it to work within node.js. It keeps complaining, that casper.start() is not a valid method...
I got closest with node-webshot, but I did not manage to scroll down page.
This is my code so far:
var webshot = require('webshot');
var options = {
shotSize: {
height: 'all',
streamType: 'jpg'
}
};
webshot('www.xx.com', 'xx.com.jpg', options, function(err) {
// screen shot saved to 'xx.com.jpg'
});
BTW I'm developing on a mac. The finished Node app will be on a linux server.
Any comments or experiences are appreciated!
Can't really help with installing CasperJS since on Windows it works by simply using npm install casperjs -g.
I've put up a simple script to do screenshots:
var casper = require('casper').create();
casper.options.viewportSize = {width: 1600, height: 950};
var wait_duration = 5000;
var url = 'http://stackoverflow.com/questions/33803790/capture-screen-shot-of-lazy-loaded-page-with-node-js';
console.log("Starting");
casper.start(url, function() {
this.echo("Page loaded");
});
casper.then(function() {
this.scrollToBottom();
casper.wait(wait_duration, function() {
casper.capture('screen.jpg');
this.echo("Screen captured");
});
});
casper.then(function() {
this.echo("Exiting");
this.exit();
});
casper.run();
The code is fairly straightforward:
Load the url
Scroll to the bottom
Wait for a specific duration (wait_duration) for stuff to load
Do a screenshot
End
Hopefully, that works for you!
this code work for me with node in OSX, save it like test.js and run node test.js in CLI
var webshot = require('webshot');
var options = {
streamType: 'png',
windowSize: {
width: 1024,
height: 768
},
shotSize: {
width: 'all',
height: 'all'
}
};
webshot("blablabla.com","bla-image.png",options,(err) => {
if(err){
return console.log(err);
}
console.log('image succesfully');
});
you can automate it via Selenium, http://webdriver.io/. Yes, it's most like a testing engine, not a screen shot application, but you can fully control the browser automation and see the browser on your display while debugging
Start selenium server, with, for example, Google Chrome
Load your page
Do scrolling, clicking, everything with webdriver.io
Take a picture when you think it's a good time
close session
fast way to install selenium with nodejs -> https://github.com/vvo/selenium-standalone
I'm setting the prerenderReady flag in my html (inline)
<script> console.log(Date()); window.prerenderReady = false; </script>
and setting it to true in my angular controller upon $http get success like this:
Post.getAll($stateParams.order, $stateParams.type, 10)
.success(function(response) {
Post.setPosts(response.data);
vm.nextUrl = response.next_page_url;
vm.loading = false;
$window.prerenderReady = true;
console.log(Date());
console.log($window.prerenderReady);
});
the difference between the inline date() and the success of the request is 3secs
nevertheless the prerender server reaches over 11500ms gets timed out and renders only the page's header.
I removed the comment from prerender logger and found out prerender had a problem with socket.io
I didn't knew what could be the problem since I am running prerender server and Socket.io server from different nodes and on different ports.
after trying to upgrade phantomjs from 1.9 to 2.0 with no avail
miraculously I tried replacing the socket.io cdn link in my index.html file
with inline script.
Now the pages are rendered perfectly :)
I want to donwload a csv file by using Caperjs.
This is what I wrote:
var login_id = "my_user_id";
var login_password = "my_password";
var casper = require('casper').create();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 ');
casper.start("http://eoddata.com/symbols.aspx",function(){
this.evaluate(function(id,password) {
document.getElementById('tl00_cph1_ls1_txtEmail').value = id;
document.getElementById('ctl00_cph1_ls1_txtPassword').value = password;
document.getElementById('ctl00_cph1_ls1_btnLogin').submit();
}, login_id, login_password);
});
casper.then(function(){
this.wait(3000, function() {
this.echo("Wating...");
});
});
casper.then(function(){
this.download("http://eoddata.com/Data/symbollist.aspx?e=NYSE","nyse.csv");
});
casper.run();
And I got nyse.csv, but the file was a HTML file for registration of the web site.
It seems login process fails. How can I login correctly and save the csv file?
2015/05/13
Following #Darren's help, I wrote like this:
casper.start("http://eoddata.com/symbols.aspx");
casper.waitForSelector("form input[name = ctl00$cph1$ls1$txtEmail ]", function() {
this.fillSelectors('form', {
'input[name = ctl00$cph1$ls1$txtEmail ]' : login_id,
'input[name = ctl00$cph1$ls1$txtPassword ]' : login_password,
}, true);
});
And this code ends up with error Wait timeout of 5000ms expired, exiting..
As far as I understand the error means that the CSS selector couldn't find the element. How can I find a way to fix this problem?
Update at 2015/05/18
I wrote like this:
casper.waitForSelector("form input[name = ctl00$cph1$ls1$txtEmail]", function() {
this.fillSelectors('form', {
'input[name = ctl00$cph1$ls1$txtEmail]' : login_id,
'input[name = ctl00$cph1$ls1$txtPassword]' : login_password,
}, true);
}, function() {
fs.write("timeout.html", this.getHTML(), "w");
casper.capture("timeout.png");
});
I checked timeout.html by Chrome Developer tools and Firebugs, and I confirmed several times that there is the input element.
<input name="ctl00$cph1$ls1$txtEmail" id="ctl00_cph1_ls1_txtEmail" style="width:140px;" type="text">
How can I fix this problem? I already spent several hours for this issue.
Update 2015/05/19
Thanks for Darren, Urarist and Artjom I could remove the time out error, but there is still another error.
Downloaded CSV file was still registration html file, so I rewrote the code like this to find out the cause of error:
casper.waitForSelector("form input[name ='ctl00$cph1$ls1$txtEmail']", function() {
this.fillSelectors('form', {
"input[name ='ctl00$cph1$ls1$txtEmail']" : login_id,
"input[name ='ctl00$cph1$ls1$txtPassword']" : login_password,
}, true);
});/*, function() {
fs.write("timeout.html", this.getHTML(), "w");
casper.capture("timeout.png");
});*/
casper.then(function(){
fs.write("logined.html", this.getHTML(), "w");
});
In the logined.html user email was filled correctly, but password is not filled. Is there anyone who have guess for the cause of this?
The trick is to successfully log in. There are multiple ways to login. I've tried some and the only one that works on this page is by triggering the form submission using the enter key. This is done by using the PhantomJS page.sendEvent() function. The fields can be filled using casper.sendKeys().
var casper = require('casper').create();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 ');
casper.start("http://eoddata.com/symbols.aspx",function(){
this.sendKeys("#ctl00_cph1_ls1_txtEmail", login_id);
this.sendKeys("#ctl00_cph1_ls1_txtPassword", login_password, {keepFocus: true});
this.page.sendEvent("keypress", this.page.event.key.Enter);
});
casper.waitForUrl(/myaccount/, function(){
this.download("http://eoddata.com/Data/symbollist.aspx?e=NYSE", "nyse.csv");
});
casper.run();
It seems that it is necessary to wait for that specific page. CasperJS doesn't notice that a new page was requested and the then() functionality is not used for some reason.
Other ways that I tried were:
Filling and submitting the form with casper.fillSelectors()
Filling through the DOM with casper.evaluate() and submitting by clicking on the login button with casper.click()
Mixing all of the above.
At first glance your script looks reasonable. But there are a couple of ways to make it simpler, which should also make it more robust.
First, instead of your evaluate() line,
this.fillSelectors('form', {
'input[name = id ]' : login_id,
'input[name = pw ]' : login_password,
}, true);
The true parameter means submit it. (I guessed the form names, but I'm fairly sure you could continue to use CSS IDs if you prefer.)
But, even better is to not fill the form until you are sure it is there:
casper.waitForSelector("form input[name = id ]", function() {
this.fillSelectors('form', {
'input[name = id ]' : login_id,
'input[name = pw ]' : login_password,
}, true);
});
This would be important if the login form is being dynamically placed there by JavaScript (possibly even from an Ajax call), so won't exist on the page as soon as the page is loaded.
The other change is instead of using casper.wait(), to use one of the casper.waitForXXX() to make sure the csv file link is there before you try to download it. Waiting 3 seconds will go wrong if the remote server takes more than 3.1 seconds to respond, and wastes time if the remote server only takes 1 second to respond.
UPDATE: When you get a time-out on the waitFor lines it tells you the root of your problem is you are using a selector that is not there. This, I find, is the biggest time-consumer when writing Casper scripts. (I recently envisaged a tool that could automate trying to find a near-miss, but couldn't get anyone else interested, and it is a bit too big a project for one person.) So your troubleshooting start points will be:
Add an error handler to the timing-out waitFor() command and take a screenshot (casper.capture()).
Dump the HTML. If you know the ID of a parent div, you could give that, to narrow down how much you have to look for.
Open the page with FireBug (or tool of your choice) and poke around to find what is there. (remember you can type a jQuery command, or document.querySelector() command, in the console, which is a good way to interactively find the correct selector.)
Try with SlimerJS, instead of PhantomJS (especially if still using PhantomJS 1.x). It might be that the site uses some feature that is only supported in newer browsers.
My usecase:
Implement an automation test to capture onbound client request after opening a page in browser. For example, open http://foo.com that later makes beacon call to http://bar.com with few parameters.
Question:
How do I test whether foo.com has initiated a call to bar.com? Can mocha do this?
Answering my own question.
I found a way to capture all requests being made by a page using phantomjs, it has a node module called phantom (https://www.npmjs.com/package/phantom) that can open a page and a listener would give all resources being requested.
Sample javascript code,
var phantom = require('phantom');
phantom.create(function (ph) {
ph.createPage(function (page) {
page.open("http://ramcountry.yahoo.com", function (status) {
console.log("opened site? ", status);
});
page.set('onResourceRequested', function(requestData, networkRequest) {
console.log("requested: ", requestData.url);
});
page.set('onResourceReceived', function(requestData, networkRequest) {
console.log("received: ", requestData.url);
});
});
});
Mocha (by design) doesn't provide this kind of thing, but can integrate with other tools (as you noted in your other answer) to provide test benefits. However your answer didn't display any testing, so I'm going to drop a few frameworks in here specifically designed for testing:
http://nightwatchjs.org/
http://www.cypress.io/
http://webdriver.io/