I'm trying to run a headless browser and run JS scripts (a bot) within it using but want to run it using php. Searching on Google, I found a implementation / wrapper of PhantomJS as php-phantomjs. Please bear with me, I'm very new to this stuff.
Here what I'm trying to do is to take screenshot of alert window (this is not necessary, but just to test if JS is executed and then the screenshot is taken.)
Here is my code:
// file: app.php
$client = PhantomJsClient::getInstance();
$client->isLazy();
$location = APP_PATH . '/partials/';
$serviceContainer = ServiceContainer::getInstance();
$procedureLoader = $serviceContainer->get('procedure_loader_factory')->createProcedureLoader($location);
$client->getProcedureLoader()->addLoader($procedureLoader);
$request = $client->getMessageFactory()->createCaptureRequest();
$response = $client->getMessageFactory()->createResponse();
$request->setViewportSize(1366, 768);
$request->setHeaders([
'User-Agent' => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36'
]);
$request->setTimeout(5000);
$request->setUrl('https://www.google.co.in');
$request->setOutputFile(APP_PATH . '/screen.jpg');
$client->send($request, $response);
Tried two custom scripts as given in the list here: http://jonnnnyw.github.io/php-phantomjs/4.0/4-custom-scripts/
// file: page_on_resource_received.partial, page_open.partial
alert("Hello");
OUTPUT: It just shows the page, not the alert window.
I repeat, Its not about taking screenshot, that's just to be sure that JS is executing.
I just want to execute my JS scripts (better say bots) like:
var form = document.getElementById('form');
var event = new Event('submit');
form.dispatchEvent(event);
or maybe using jQuery and then return the output of that page to php as response or so. So, if there is any other way to run bots using php in a headless browser please mention that in your answers or comments.
PhantomJS is headless, that is it doesn't have GUI. Therefore no window dialogs can be seen.
Try writing custom text to an element instead of alert, like
document.getElementById("#title").innerHTML = "It works!";
Hello everybody I am a newbie on javascript and I am trying to learn Casper.js/Phantom.js, but today I encountered a problem. I am trying to login in https://angel.co/login?utm_source=top_nav_home site with auto filling. My code fills username and password correctly however my
casper.getElementsByName("commit").click();
doesnt work at all. So my program doesnt login to the account. When I inspect element I dont see any id tag, I see name so I use getElementByName.My code is like this below
phantom.casperPath = 'C:/casperjs';
phantom.injectJs(phantom.casperPath + '/bin/bootstrap.js');
var casper = require('casper').create({
pageSettings: {
loadImages: false,
loadPlugins: false,
userAgent: 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36'
}
});
//First step is to open angellist
casper.start().thenOpen("https://angel.co/login", function () {
console.log("angel website opened");
});
//lets fill username and pass login
casper.then(function () {
console.log("Login using username and password");
this.evaluate(function () {
document.getElementById("user_email").value = "myemail";
document.getElementById("user_password").value = "mypassword";
//casper.getElementsByName("commit").click();
//THIS ONE ABOVE DOESN'T WORK
});
});
after this I take a screenshot but I see that it doesnt click the login button.
I tried to click it using xpath too but it didn'work. Thanks for your help.
I solved the problem by using fairly easy code
document.getElementsByName("commit")[0].click();
since getElementsByName returns an array we have to give an adress to is( like [0] or other values). I wont edit or remove this question, for the other newbies on javascript like me. Thanks to everybody
I am currently running version 1.9.8.0 PhantomJS. My problem is it works fine and does what it should (which in my case is going through the site counting certain elements to start off with) but I got this error:
PhantomJS is a headless WebKit with JavaScript API has stopped working.
The full error is:
Problem signature:
Problem Event Name: APPCRASH
Application Name: PhantomJS.exe
Application Version: 1.9.8.0
Application Timestamp: 5449270a
Fault Module Name: PhantomJS.exe
Fault Module Version: 1.9.8.0
Fault Module Timestamp: 5449270a
Exception Code: c0000005
Exception Offset: 00057976
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 2057
Additional Information 1: 8236
Additional Information 2: 823646afcac85a21ce127aeb0b347bb5
Additional Information 3: 137e
Additional Information 4: 137ec742f6481348348abf863da72fd4
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
It only seems to happen on one retailer site (currys) as I have it running on other retailers and it works fine. Also it always breaks on currys and it is since they have updated their site. If anyone could help it would be much appreciated.
Here is a code snippet:
var options = new PhantomJSOptions();
options.AddAdditionalCapability("phantomjs.page.settings.userAgent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
options.AddAdditionalCapability("phantomjs.page.settings.loadImages", "false");
options.AddAdditionalCapability("phantomjs.page.settings.resourceTimeout", "12000");
PhantomJSDriverService service = PhantomJSDriverService.CreateDefaultService(PhantomJSPath);
service.HideCommandPromptWindow = false;//make it true on deployment
using (PhantomJSDriver driver = new PhantomJSDriver(service, options))
{
driver.Manage().Window.Size = new System.Drawing.Size(1920, 989);
ITakesScreenshot screenShot = driver as ITakesScreenshot;
IJavaScriptExecutor jse = driver as IJavaScriptExecutor;
try
{
CAInformationProviderConfiguration.CAInformationSources.ForEach(source =>
{
DefaultCAInformationSourceResult result = new DefaultCAInformationSourceResult();
result.CAInformationSource = source;
try
{
driver.ExecutePhantomJS(
#"
var page = this;
page.onResourceRequested = function(requestData, networkRequest) {
//console.log(requestData.url);
if (requestData.url.match(/(.*ajax\.html.*)|(.*facebook.*)|(.*twitter.*)|(.*instagram.*)|(.*youtube.*)|(.*hotukdeals.*)|(.*pinterest.*)|(.*flix360.*)/)) {
networkRequest.abort();
}
};
page.onResourceReceived = function(response) {
//console.log('loaded ' + response.url+'\n'+ response.stage) ;
};
");
//scrape
driver.Navigate().GoToUrl(source.Url);
it will then loop through checking elements on the page.
Edit
I upgraded to v2.0 PhantomJS and I am getting the same issue.
I should mention I m using PhantomJS with Selenium.
what is the best way to restart PhantomJS - I was thinking if I could restart it after it has processed a certain amount of rows it may prevent the application crashing?
RESOLUTION:
It seems that the memory usage was building up for this and when it got over 1gb it crashed. To stop this from happening I ended up looping through a batch and then restarting phantomjs. Not ideal but fixes my issue for now.
Thanks
I want to donwload a csv file by using Caperjs.
This is what I wrote:
var login_id = "my_user_id";
var login_password = "my_password";
var casper = require('casper').create();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 ');
casper.start("http://eoddata.com/symbols.aspx",function(){
this.evaluate(function(id,password) {
document.getElementById('tl00_cph1_ls1_txtEmail').value = id;
document.getElementById('ctl00_cph1_ls1_txtPassword').value = password;
document.getElementById('ctl00_cph1_ls1_btnLogin').submit();
}, login_id, login_password);
});
casper.then(function(){
this.wait(3000, function() {
this.echo("Wating...");
});
});
casper.then(function(){
this.download("http://eoddata.com/Data/symbollist.aspx?e=NYSE","nyse.csv");
});
casper.run();
And I got nyse.csv, but the file was a HTML file for registration of the web site.
It seems login process fails. How can I login correctly and save the csv file?
2015/05/13
Following #Darren's help, I wrote like this:
casper.start("http://eoddata.com/symbols.aspx");
casper.waitForSelector("form input[name = ctl00$cph1$ls1$txtEmail ]", function() {
this.fillSelectors('form', {
'input[name = ctl00$cph1$ls1$txtEmail ]' : login_id,
'input[name = ctl00$cph1$ls1$txtPassword ]' : login_password,
}, true);
});
And this code ends up with error Wait timeout of 5000ms expired, exiting..
As far as I understand the error means that the CSS selector couldn't find the element. How can I find a way to fix this problem?
Update at 2015/05/18
I wrote like this:
casper.waitForSelector("form input[name = ctl00$cph1$ls1$txtEmail]", function() {
this.fillSelectors('form', {
'input[name = ctl00$cph1$ls1$txtEmail]' : login_id,
'input[name = ctl00$cph1$ls1$txtPassword]' : login_password,
}, true);
}, function() {
fs.write("timeout.html", this.getHTML(), "w");
casper.capture("timeout.png");
});
I checked timeout.html by Chrome Developer tools and Firebugs, and I confirmed several times that there is the input element.
<input name="ctl00$cph1$ls1$txtEmail" id="ctl00_cph1_ls1_txtEmail" style="width:140px;" type="text">
How can I fix this problem? I already spent several hours for this issue.
Update 2015/05/19
Thanks for Darren, Urarist and Artjom I could remove the time out error, but there is still another error.
Downloaded CSV file was still registration html file, so I rewrote the code like this to find out the cause of error:
casper.waitForSelector("form input[name ='ctl00$cph1$ls1$txtEmail']", function() {
this.fillSelectors('form', {
"input[name ='ctl00$cph1$ls1$txtEmail']" : login_id,
"input[name ='ctl00$cph1$ls1$txtPassword']" : login_password,
}, true);
});/*, function() {
fs.write("timeout.html", this.getHTML(), "w");
casper.capture("timeout.png");
});*/
casper.then(function(){
fs.write("logined.html", this.getHTML(), "w");
});
In the logined.html user email was filled correctly, but password is not filled. Is there anyone who have guess for the cause of this?
The trick is to successfully log in. There are multiple ways to login. I've tried some and the only one that works on this page is by triggering the form submission using the enter key. This is done by using the PhantomJS page.sendEvent() function. The fields can be filled using casper.sendKeys().
var casper = require('casper').create();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 ');
casper.start("http://eoddata.com/symbols.aspx",function(){
this.sendKeys("#ctl00_cph1_ls1_txtEmail", login_id);
this.sendKeys("#ctl00_cph1_ls1_txtPassword", login_password, {keepFocus: true});
this.page.sendEvent("keypress", this.page.event.key.Enter);
});
casper.waitForUrl(/myaccount/, function(){
this.download("http://eoddata.com/Data/symbollist.aspx?e=NYSE", "nyse.csv");
});
casper.run();
It seems that it is necessary to wait for that specific page. CasperJS doesn't notice that a new page was requested and the then() functionality is not used for some reason.
Other ways that I tried were:
Filling and submitting the form with casper.fillSelectors()
Filling through the DOM with casper.evaluate() and submitting by clicking on the login button with casper.click()
Mixing all of the above.
At first glance your script looks reasonable. But there are a couple of ways to make it simpler, which should also make it more robust.
First, instead of your evaluate() line,
this.fillSelectors('form', {
'input[name = id ]' : login_id,
'input[name = pw ]' : login_password,
}, true);
The true parameter means submit it. (I guessed the form names, but I'm fairly sure you could continue to use CSS IDs if you prefer.)
But, even better is to not fill the form until you are sure it is there:
casper.waitForSelector("form input[name = id ]", function() {
this.fillSelectors('form', {
'input[name = id ]' : login_id,
'input[name = pw ]' : login_password,
}, true);
});
This would be important if the login form is being dynamically placed there by JavaScript (possibly even from an Ajax call), so won't exist on the page as soon as the page is loaded.
The other change is instead of using casper.wait(), to use one of the casper.waitForXXX() to make sure the csv file link is there before you try to download it. Waiting 3 seconds will go wrong if the remote server takes more than 3.1 seconds to respond, and wastes time if the remote server only takes 1 second to respond.
UPDATE: When you get a time-out on the waitFor lines it tells you the root of your problem is you are using a selector that is not there. This, I find, is the biggest time-consumer when writing Casper scripts. (I recently envisaged a tool that could automate trying to find a near-miss, but couldn't get anyone else interested, and it is a bit too big a project for one person.) So your troubleshooting start points will be:
Add an error handler to the timing-out waitFor() command and take a screenshot (casper.capture()).
Dump the HTML. If you know the ID of a parent div, you could give that, to narrow down how much you have to look for.
Open the page with FireBug (or tool of your choice) and poke around to find what is there. (remember you can type a jQuery command, or document.querySelector() command, in the console, which is a good way to interactively find the correct selector.)
Try with SlimerJS, instead of PhantomJS (especially if still using PhantomJS 1.x). It might be that the site uses some feature that is only supported in newer browsers.
I'm a newcomer to CasperJS and after a couple hours I can login and navigate a few webpages with it, but I'm stumped by the alert message on this website: https://www.macysliquidation.com/
I need to get rid of the alert so I can login.
My simple (non-working) code is:
var casper = require('casper').create();
casper.userAgent('Mozilla/12.0 (compatible; MSIE 6.0; Windows NT 5.1)');
casper.on('remote.alert', function(message) {
this.echo('alert message: ' + message);
// how do i get rid of the popup??
this.thenClick();
});
casper.start('https://www.macysliquidation.com/');
casper.then(function() {
// login here
this.sendKeys('#txtUsername','username');
this.sendKeys('#txtPassword','password');
this.thenClick('#btnLogin');
});
casper.run(function() {
// see what went on
this.capture('page.png');
this.echo('done').exit();
});
Till the time the alert is clicked away, the login controls aren't visible/available. So the above js returns
Cannot get informations from #txtUsername: element not found
As you already noticed the function capser.waitForAlert() is available since version 1.1-beta4. You can copy the function from the code if you don't have the time to upgrade:
casper.waitForAlert = function(then, onTimeout, timeout) {
...
};
Problem:
Alerts and confirm just happen and they don't stop the execution in PhantomJS and CasperJS. They are also not part of the page and cannot be clicked on.
If you would register to the error events (resource.error and page.error and remote.message is always a good idea) in CasperJS, you would have seen that a specific resource error was thrown:
{"errorCode":6,"errorString":"SSL handshake failed","id":1,"url":"https://www.macysliquidation.com/"}
If you would have checked the status of the page, you would have seen that it wasn't loaded.
Solution:
Run CasperJS with --ignore-ssl-errors=true and depending on your PhantomJS version with --ssl-protocol=tlsv1. More information here.