I'm using PhantomJS via Selenium Webdriver in Python and I'm trying to execute a piece of JavaScript on the page in hopes of returning a piece of data:
from selenium import webdriver
driver = webdriver.PhantomJS("phantomjs.cmd") # or add to your PATH
driver.set_window_size(1024, 768) # optional
driver.get('http://google.com') # EXAMPLE, not actual URL
driver.save_screenshot('screen.png') # save a screenshot to disk
jsres = driver.execute('$("#list").DataTable().data()')
print(jsres)
However when run, it reports KeyError. I was unable to find much documentation on the commands available, so I'm a bit stuck here.
The method created for executing javascript is called execute_script(), not execute():
driver.execute_script('return $("#list").DataTable().data();')
FYI, execute() is used internally for sending webdriver commands.
Note that if you want something returned by javascript code, you need to use return.
Also note that this can throw Can't find variable: $ error message. In this case, locate the element with selenium and pass it into the script:
# explicitly wait for the element to become present
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "list")))
# pass the found element into the script
jsres = driver.execute_script('return arguments[0].DataTable().data();', element)
print(jsres)
Related
browser.execute_script("window.open('about:blank', 'tab2');")
browser.switch_to.window("tab2")
browser.get('http://bing.com')
I was searching online ways to open a new tab using selenium in python, and the method of ctrl + t wasn't working on chrome so I stumbled on the above piece of code, however I am not able to understand what 'excute_script' does.
execute_script method allows to execute JavaScript passed as string argument
Note that you can pass data from Python code into JavaScript code using arguments, e.g.
hello = "Hello"
friends = " friends"
browser.execute_script('alert(arguments[0], arguments[1]);', (hello, friends))
execute_script()
execute_script() synchronously executes JavaScript in the current window/frame.
execute_script(script, *args)
where:
script: The JavaScript to execute
*args: Any applicable arguments for your JavaScript.
This method is defined as:
def execute_script(self, script, *args):
"""
Synchronously Executes JavaScript in the current window/frame.
:Args:
- script: The JavaScript to execute.
- \\*args: Any applicable arguments for your JavaScript.
:Usage:
::
driver.execute_script('return document.title;')
"""
if isinstance(script, ScriptKey):
try:
script = self.pinned_scripts[script.id]
except KeyError:
raise JavascriptException("Pinned script could not be found")
converted_args = list(args)
command = None
if self.w3c:
command = Command.W3C_EXECUTE_SCRIPT
else:
command = Command.EXECUTE_SCRIPT
return self.execute(command, {
'script': script,
'args': converted_args})['value']
Examples
A couple of examples:
To open a new blank tab:
driver.execute_script("window.open('','_blank');")
To open a new tab with an url:
driver.execute_script("window.open('https://www.google.com');")
To retrieve the page title:
driver.execute_script('return document.title;')
To scroll inti view an element:
driver.execute_script("arguments[0].scrollIntoView(true);",element)
References
You can find a couple of relevant detailed discussions in:
What does arguments[0] and arguments[1] mean when using executeScript method from JavascriptExecutor interface through Selenium WebDriver?
What is JavaScriptExecutor in Selenium?
I'm trying to scrape the data within the table of this F5 Article (https://support.f5.com/csp/article/K15386).
The problem I'm facing is that it sometimes crawls the page correctly (tables are within the DOM). Other times however it does not crawl the javascript generated tables.
I have tried using implicit and explicit waits, but with no success. Using explicit wait I am not able to successfully select the table. I am always getting the timeouts.
Any idea on how to always access the data within the tables?
I'm using Java 8, Selenium 3.141.59 & ChromeDriver 85.0.4183.87.
driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
WebDriverWait wait = new WebDriverWait(driver, 15);
WebElement element = wait
.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("table")));
Edit:
I wish I could use their API, however since it is not documented, I am not allowed to.
What about this:
List<WebElement> tables = driver.findElements(By.tagName("table"));
if (tables.size() == 0) {
try {
wait.until(ExpectedConditions.presenceOfElementLocated(By.tagName("table")));
tables.addAll(driver.findElements(By.tagName("table")));
}
catch (TimeOutException e) {
};
}
When I say JavaScript file, I mean the whole file. Not a function. I've seen ways to run a JavaScript function from c# but nothing about running a file.
According to this question: https://stackoverflow.com/a/1469790/13105088, one could run the command with any normal shell.
string strCmdText;
strCmdText= "/C node myscript.js"; // the command to run from the command prompt
System.Diagnostics.Process.Start("CMD.exe",strCmdText);
Note that this will show the Command Prompt on Windows.
This following script prevents that.
System.Diagnostics.Process process = new System.Diagnostics.Process();
System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
startInfo.FileName = "cmd.exe";
startInfo.Arguments = "/C node myscript.js";
process.StartInfo = startInfo;
process.Start();
(these example scripts were both provided by the answer I linked to above.)
In addition, you should probably also note:
Important is that the argument begins with /C otherwise it won't work. How Scott Ferguson said: it "Carries out the command specified by the string and then terminates."
Note: this is all assuming you are referring to NodeJS when you are saying a "JavaScript file", but any other interpreter (eg. /C python3 myfile.py) should also work.
I am trying to fetch the links to all accomodations in Cyprus from this website:
http://www.zoover.nl/cyprus
So far I can retrieve the first 15 which are already shown. So now I have to invoke the click on the "volgende"-link. However I don't know how to do that and in the source code I am not able to track down the function called to use e.g. sth like posted here:
Issues with invoking "on click event" on the html page using beautiful soup in Python
I only need the step where the "clicking" happens so I can fetch the next 15 links and so on.
Does anybody know how to help?
Thanks already!
EDIT:
My code looks like this now:
def getZooverLinks(country):
zooverWeb = "http://www.zoover.nl/"
url = zooverWeb + country
parsedZooverWeb = parseURL(url)
driver = webdriver.Firefox()
driver.get(url)
button = driver.find_element_by_class_name("next")
links = []
for page in xrange(1,3):
for item in parsedZooverWeb.find_all(attrs={'class': 'blue2'}):
for link in item.find_all('a'):
newLink = zooverWeb + link.get('href')
links.append(newLink)
button.click()'
and I get the following error:
selenium.common.exceptions.StaleElementReferenceException: Message: Element is no longer attached to the DOM
Stacktrace:
at fxdriver.cache.getElementAt (resource://fxdriver/modules/web-element-cache.js:8956)
at Utils.getElementAt (file:///var/folders/n4/fhvhqlmx23s8ppxbrxrpws3c0000gn/T/tmpKFL43_/extensions/fxdriver#googlecode.com/components/command-processor.js:8546)
at fxdriver.preconditions.visible (file:///var/folders/n4/fhvhqlmx23s8ppxbrxrpws3c0000gn/T/tmpKFL43_/extensions/fxdriver#googlecode.com/components/command-processor.js:9585)
at DelayedCommand.prototype.checkPreconditions_ (file:///var/folders/n4/fhvhqlmx23s8ppxbrxrpws3c0000gn/T/tmpKFL43_/extensions/fxdriver#googlecode.com/components/command-processor.js:12257)
at DelayedCommand.prototype.executeInternal_/h (file:///var/folders/n4/fhvhqlmx23s8ppxbrxrpws3c0000gn/T/tmpKFL43_/extensions/fxdriver#googlecode.com/components/command-processor.js:12274)
at DelayedCommand.prototype.executeInternal_ (file:///var/folders/n4/fhvhqlmx23s8ppxbrxrpws3c0000gn/T/tmpKFL43_/extensions/fxdriver#googlecode.com/components/command-processor.js:12279)
at DelayedCommand.prototype.execute/< (file:///var/folders/n4/fhvhqlmx23s8ppxbrxrpws3c0000gn/T/tmpKFL43_/extensions/fxdriver#googlecode.com/components/command-processor.js:12221)
I'm confused :/
While it might be tempting to try to do this using Beautifulsoup's evaluateJavaScript method, in the end Beautifulsoup is a parser rather than an interactive web browsing client.
You should seriously consider solving this with selenium, as briefly shown in this answer. There are pretty good Python bindings available for selenium.
You could just use selenium to find the element and click it, and then pass the page on to Beautifulsoup, and use your existing code to fetch the links.
Alternatively, you could use the Javascript that's listed in the onclick handler. I pulled this from the source: EntityQuery('Ns=pPopularityScore%7c1&No=30&props=15292&dims=530&As=&N=0+3+10500915');. The No parameter increments with 15 for each page, but the props has me guessing. I'd recommend not getting into this, though, and just interact with the website as a client would, using selenium. That's much more robust to changes on their side, as well.
I tried the following code and was able to load next page. Hope this will help you too.
Code:
from selenium import webdriver
import os
chromedriver = "C:\Users\pappuj\Downloads\chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
url='http://www.zoover.nl/cyprus'
driver.get(url)
driver.find_element_by_class_name('next').click()
Thanks
I've written a script to test a process involving data input & several pages, but after writing it I've found the forms & main content to be generated from javascript.
The following is a snippet of the script I wrote, and after that initial link the content is generated by JS (its my first python script so excuse any mistakes);
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import time
browser = webdriver.Firefox()
browser.get('http://127.0.0.1:46727/?ajax=1')
assert "Home" in browser.title
# Find and click the Employer Database link
empDatabaseLink = browser.find_element_by_link_text('Employer Database')
click = ActionChains(browser).click(on_element = empDatabaseLink)
click.perform()
# Content loaded by the above link is generated by the JS
# Find and click the Add Employer button
addEmployerButton = browser.find_element_by_id('Add Employer')
addEmployer = ActionChains(browser).click(on_element = addEmployerButton)
addEmployer.perform()
browser.save_screenshot(r'images\Add_Employer_Form.png')
# Input Employer name
employerName = browser.find_element_by_id('name')
employerName.send_keys("Selenium")
browser.save_screenshot(r'images\Entered_Employer_Name.png')
# Move to next
nextButton = broswer.find_element_by_name('button_next')
moveForward = ActionChains(browser).click(on_element = nextButton)
# Move through various steps
# Then
# Move to Finish
moveForward = ActionChains(browser).click(on_element = nextButton)
How do you access page elements that aren't in the source? I've been looking around & found GetEval but not found anything that I can use :/
Well, to the people of the future, our above conversation appears to have lead to the conclusion that xpath is what mark was looking for. So remember to try xpath, and to use the Selenium IDE and Firebug to locate particularly obstinate page elements.