We Keep Coding

Selenium Python NoSuchElementException: Message: no such element: Unable to locate element:

Background:
So I have very little coding knowledge, I have tried to learn coding several times through guides and youtube tutorials but never got very far in my learning. This time around I am trying a different approach, of simply having an idea and doing it.
I am trying to make a program interact with the website fitbit. Very simply it needs to login and change the email of my account.
I have successfully used Selenium python to login and access the settings page, but I cannot interact with any of the elements beyond this.
Once the program has made it to the settings page it needs to click the "Change Email Address"
Then fill in the form and click submit. That is all I need the program to do.
My code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chromedriver = webdriver.Chrome(executable_path=r'C:/Utility/BrowserDrivers/chromedriver.exe')
browser = chromedriver
browser.get('https://www.fitbit.com/settings/profile')
browser.implicitly_wait(20)
username = browser.find_element_by_xpath('//*[#id="loginForm"]/fieldset/dl/dd[1]/input')
username.click()
username.send_keys('Email')
password = browser.find_element_by_xpath('//*[#id="loginForm"]/fieldset/dl/dd[2]/input')
password.click()
password.send_keys('Password')
browser.find_element_by_xpath('//*[#id="loginForm"]/div[1]/button').click()
browser.implicitly_wait(30)
#browser.find_element_by_xpath('//*[#id="ember798"]/div[3]/div[2]/button').click()
new_email = browser.find_element_by_xpath('//*[#id="ember1077"]')
new_email.click()
new_email.send_keys('New Email')
confirm_new_email = browser.find_element_by_xpath('//*[#id="ember1107"]')
confirm_new_email.click()
confirm_new_email.send_keys('New Email')
password = browser.find_element_by_xpath('//*[#id="ember1119"]')
password.click()
password.send_keys('Password')
#browser.find_element_by_css_selector('#ember832 > div:nth-child(4) > div.column.medium-5.change-email-button-container > button').click()
#browser.find_element_by_xpath('//*[#id="ember748"]/div[3]/div[2]').click()
My Error:
DevTools listening on ws://127.0.0.1:51658/devtools/browser/12ae1773-a1f7-473f-8128-32f2090202fc
[12820:7500:1215/095935.812:ERROR:shader_disk_cache.cc(257)] Failed to create shader cache entry: -2
[12820:7500:1215/095959.060:ERROR:shader_disk_cache.cc(257)] Failed to create shader cache entry: -2
Traceback (most recent call last):
File "looptest.py", line 25, in <module>
new_email = browser.find_element_by_xpath('//*[#id="ember1077"]')
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_
by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_resp
onse
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"
method":"xpath","selector":"//*[#id="ember1077"]"}
(Session info: chrome=70.0.3538.110)
(Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 1
0.0.17134 x86_64)
I appreciate the code is messy, and I plan to tidy things up and add in if statements, trys and everything else once the program actually works.
Solutions:
I have tried to make the browser wait for the elements to load.
I have tried fiddling around with the iframe command, I do not believe the first button is on a different iframe, but the second part maybe.
I have tried using xpath, and tried using the CSS selector command as well.
The HTML:
IMAGE OF SETTINGS PAGE
Above is an image of the settings page, I want to click the button circled in red
Here is the xpath: //*[#id="ember815"]/div[3]/div[2]/button
sol the element 100% exists.
HTML OF BUTTON
There is a screenshot of the inspect of the element.
CHANGE EMAIL FIELDS
Once the button has been click this box will appear and I need to then fill in these fields.
THE HTML OF THE NEW EMAIL FIELD
Any help on this will be greatly appreciated. I know there is a quite a bit of Javascript on this page, and I reckon that is my problem. The Javascript is making it so I cannot interact with these elements but I am unsure why.
UPDATE 1.0
A user suggest I screenshot all the parts I am unsure about using the command
browser.save_screenshot('error2.png')
Interestingly the screenshot is provides is blank. SEE IMAGE
Update 2.0
Thank you for the responses so far, I have implemented the suggestion.
I implemented the this line of code
browser.find_element_by_xpath('//button[#class="button change-email-button"]').click()
and it has progressed the program so it takes me to this stage.
Here
Now I am facing the same problem with filling out the form.
Traceback (most recent call last):
File "looptest.py", line 28, in <module>
new_email = browser.find_element_by_xpath('(//input[#data-test-qa="new-email"])[2]')
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_
by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_resp
onse
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"
method":"xpath","selector":"(//input[#data-test-qa="new-email"])[2]"}
(Session info: chrome=70.0.3538.110)
(Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 1
0.0.17134 x86_64)
I have added waits and things, but still having no luck
browser.find_element_by_xpath('//button[#class="button change-email-button"]').click()
browser.implicitly_wait(30)
#new_email = browser.find_element_by_xpath('//*[#id="ember1077"]')
new_email = browser.find_element_by_xpath('(//input[#data-test-qa="new-email"])[2]')

it seem the ID is dynamic try use the folowing Xpath
# Change Email Address button
//button[#class="button change-email-button"]
# email
(//input[#data-test-qa="new-email"])[2]
# confirm email
(//input[#data-test-qa="confirm-new-email"])[2]
# password
(//input[#data-test-qa="password-for-email-change"])[2]
after clicking the Change Email button you need to wait until form popup appear

Thank you to the users that responded. We have a solution
Here the key fact as #ewwink correctly said, I failed to notice that the elements I was trying to access had dynamic ids.
I managed to fix this issue by typing out my own xpath and came up with the following:
//*[#data-test-qa="new-email"]
//*[#data-test-qa="confirm-new-email"]
//*[#data-test-qa="password-for-email-change"]
These work each time the program is run!

Pepper naoqi 2.5 onConsoleMessage null

I am trying to print in python the messages from the web console using a callback on a onConsoleMessage event. Pepper (Edit: version 1.6) is running naoqi 2.5.5.5. I've modified the executeJS example as a test. The problem is I keep getting null for the message in the callback. Is it a bug that has been fixed in a newer version of naoqi ? I've had a look at the release notes but I didn't find anything.
Here is the code I am using:
#! /usr/bin/env python
# -*- encoding: UTF-8 -*-
"""Example: Use executeJS Method"""
import qi
import argparse
import sys
import time
import signal
def signal_handler(signal, frame):
print('Bye!')
sys.exit(0)
def main(session):
"""
This example uses the executeJS method.
To Test ALTabletService, you need to run the script ON the robot.
"""
# Get the service ALTabletService.
try:
tabletService = session.service("ALTabletService")
# Display a local web page located in boot-config/html folder
# The ip of the robot from the tablet is 198.18.0.1
tabletService.showWebview("http://198.18.0.1/apps/boot-config/preloading_dialog.html")
time.sleep(3)
# Javascript script for displaying a prompt
# ALTabletBinding is a javascript binding inject in the web page displayed on the tablet
script = """
console.log('A test message');
"""
# Don't forget to disconnect the signal at the end
signalID = 0
# function called when the signal onJSEvent is triggered
# by the javascript function ALTabletBinding.raiseEvent(name)
def callback(message):
print "[callback] received : ", message
# attach the callback function to onJSEvent signal
signalID = tabletService.onConsoleMessage.connect(callback)
# inject and execute the javascript in the current web page displayed
tabletService.executeJS(script)
print("Waiting for Ctrl+C to disconnect")
signal.pause()
except Exception, e:
print "Error was: ", e
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--ip", type=str, default="127.0.0.1",
help="Robot IP address. On robot or Local Naoqi: use '127.0.0.1'.")
parser.add_argument("--port", type=int, default=9559,
help="Naoqi port number")
args = parser.parse_args()
session = qi.Session()
try:
session.connect("tcp://" + args.ip + ":" + str(args.port))
except RuntimeError:
print ("Can't connect to Naoqi at ip \"" + args.ip + "\" on port " + str(args.port) +".\n"
"Please check your script arguments. Run with -h option for help.")
sys.exit(1)
main(session)
Output:
python onConsoleMessage.py --ip=192.168.1.20
[W] 1515665783.618190 30615 qi.path.sdklayout: No Application was created, trying to deduce paths
Waiting for Ctrl+C to disconnect
[callback] received : null
Did someone face the same issue?
Thanks

I have the same issue. You can easily reproduce it by opening two ssh consoles on the robot, and on the first one executing
qicli watch ALTabletService.onConsoleMessage
and on the second
qicli call ALTabletService.showWebview
qicli call ALTabletService.executeJS "console.log('hello')"
... and instead of "hello", you will see "null" appear in your first console.
HOWEVER - if your goal is to effectively test your webpage, what I usually do is just open the page on my computer and use the chrome console (you can set chrome up to act as if the page was a tablet of the right size, 1280x800); you can do this while still connecting the page to Pepper, as if it was on her tablet, using the method described here. This is enough for 99% of the case; the remaining 1% is things where Pepper's tablet is actually different from Chrome.

How to manage errors with bokeh library (Python + JavaScript) and show them on the user interface?

I am working in a Bokeh Server Application using the Python library "Bokeh" and I usually get two types of errors.
JavaScript errors on the Chromium console
Python errors on the server side, they appear in the Tornado logger. Bokeh works with the Tornado Web Server
What I could achieve is to create manual exceptions inheriting the Exception class in python. In this way I can write custom messages and run some actions when the error is raised with this class.
class ManualException(Exception):
def __init__(self, value):
self.value = value
# some actions
def __str__(self):
return repr(
'MANUAL ERROR\n' + self.value
)
But if any other exception occurs then I do not know how to capture them to run some custom actions. I have tried this:
try:
n = 9 / 0
except Exception:
tb = sys.exc_info()[2]
raise ManualException('ZERO ERROR').with_traceback(tb)
Here I have to catch any error manually and I raise them with my manual created class, then both appear in the python logger. Is this the proper way to do this?
2017-09-04 17:08:58,872 Error running application handler <bokeh.application.handlers.directory.DirectoryHandler object at 0x0000023E2435C208>: 'MANUAL ERROR\nZERO ERROR'
File "main.py", line 11, in <module>:
n = 9 / 0 Traceback (most recent call last):
File "[...]\main.py", line 11, in <module>
n = 9 / 0
ZeroDivisionError: division by zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[...]\lib\site-packages\bokeh\application\handlers\code_runner.py", line 125, in run
exec(self._code, module.__dict__)
File "[...]\main.py", line 14, in <module>
raise ManualException('ZERO ERROR').with_traceback(tb)
File "[...]\main.py", line 11, in <module>
n = 9 / 0
exceptions.ManualException: 'MANUAL ERROR\n'ZERO ERROR'
Anyway I want avoid this because I want to show all kind of errors to the user, even if I do not use try-except to capture exceptions.
On the other hand if a python error happens, a JavaScript message is shown. So I think the best option is to override some JavaScript method to run some custom js code. How could I achieve this? Is this achievable?

I got an answer in other SO question. Some listeners can be added to the window in order to get the JavaScript error messages
You can handle it as an event listener on window object.
window.onunhandledrejection = function(event) {
console.warn(`UNHANDLED PROMISE REJECTION: ${event.reason}`);
};
window.onerror = function(event) {
console.warn(`UNHANDLED ERROR: ${event.message}`);
};
Or also like this:
window.addEventListener('error', function(event) { ... })
You can read more about the unhandledrejection event on the MDN web docs here and the onerror event on the docs here

Can't find forms inside a html code - Web Scraping with Python and Selenium

I'm trying to scrap this website, but there are some forms to fill.
The mainly objective is to fill these 5 forms (one appears after selecting another) and download the data through the button "Consultar".
This forms are coded in javascript and I can't find them in the page's html code. When I inspect the frames trough Google Chrome, I find the forms ID's, but the code doesn't find them.
I have just a prototype of my code. I can't advance without knowing what I can do to find these forms.
from selenium import webdriver
from bs4 import BeautifulSoup
import time
import os
#Variables
url = 'http://www.anbima.com.br/pt_br/informar/sistema-reune.htm'
path_phantom = 'C:\\Users\\TBMEPYG\\AppData\\Local\\Continuum\\Anaconda3\\Lib\\site-packages\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
#Processing
driver = webdriver.PhantomJS(executable_path= path_phantom)
driver.get(url)
data = driver.find_element_by_id('data_ref')
data.send_keys("21/08/2017")
driver.quit()
Edit:
I updated the code to this:
from selenium import webdriver
path_phantom = 'C:\\Users\\TBMEPYG\\AppData\\Local\\Continuum\\Anaconda3\\Lib\\site-packages\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path= path_phantom)
driver.get('http://www.anbima.com.br/reune/reune.asp')
driver.switch_to.frame(driver.find_element_by_xpath('//iframe[#class="full"]'))
data = driver.find_element_by_name('Dt_Ref')
data.clear()
data.send_keys('21/08/
And I got this error:
CD: C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3
Current directory: C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3
python "C:\Users\TBMEPYG\Desktop\vamo.py"
Process started >>>
Traceback (most recent call last):
File "C:\Users\TBMEPYG\Desktop\vamo.py", line 8, in <module>
data = driver.find_element_by_name('Dt_Ref')
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 426, in find_element_by_name
return self.find_element(by=By.NAME, value=name)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 832, in find_element
'value': value})['value']
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with name 'Dt_Ref'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"89","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:62040","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"name\", \"value\": \"Dt_Ref\", \"sessionId\": \"bdd3fc70-8dd0-11e7-aeb1-85b8cfbe0d1c\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/bdd3fc70-8dd0-11e7-aeb1-85b8cfbe0d1c/element"}}
Screenshot: available via screen
Edit2:
Another possibility is to use the link inside of the the mains page http://www.anbima.com.br/reune/reune.asp
When I changed the code to this, I've got another error
from selenium import webdriver
path_phantom = 'C:\\Users\\TBMEPYG\\AppData\\Local\\Continuum\\Anaconda3\\Lib\\site-packages\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path= path_phantom)
driver.get('http://www.anbima.com.br/reune/reune.asp')
data = driver.find_element_by_name('Dt_Ref')
data.clear()
data.send_keys('21/08/2017')
Error:
Traceback (most recent call last):
File "C:\Users\TBMEPYG\Desktop\vamo.py", line 9, in <module>
data = driver.find_element_by_name('Dt_Ref')
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 426, in find_element_by_name
return self.find_element(by=By.NAME, value=name)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 832, in find_element
'value': value})['value']
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: {"request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"89","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:61820","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"name\", \"value\": \"Dt_Ref\", \"sessionId\": \"e61dd170-8dcf-11e7-a019-41573671066b\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/e61dd170-8dcf-11e7-a019-41573671066b/element"}}
Screenshot: available via screen

To be able to handle elements inside form you need to switch to iframe first:
driver.switch_to.frame(driver.find_element_by_xpath('//iframe[#class="full"]'))
data = driver.find_element_by_name('Dt_Ref')
data.clear()
data.send_keys('21/08/2017')

Just to update:
Problem not solved, but I know what is.
It's a bug in phantomJS that is occurring. So, if you have the same problem, try to use Chrome or Firefox.
Thanks for the answers.

I think you messed up with single and double quotes. Correct code would be-
driver.get("http://www.anbima.com.br/reune/reune.asp")
data = driver.find_element_by_name("Dt_Ref")
data.clear()
data.send_keys("21/08/2017")

Handle alert in Selenium Python

I have the following pop-up alert that I want to handle after a file upload. I have used the code below and it throws the error below.
wait.until(EC.alert_is_present())
driver.switch_to.alert().accept()
Traceback (most recent call last):
File "update.py", line 45, in
driver.switch_to.alert().accept()
TypeError: 'Alert' object is not callable
Why is this happening? I have handled a similar alert (that one had a cancel button?) in this manner.

There are two ways to accept alert available in Python + selenium (there is also JavaScript code for execute_script(), but it's not related to current issue):
driver.switch_to_alert().accept() # deprecated, but still works
driver.switch_to.alert.accept()
Note that in second line you don't need to call alert() as you did in your code

Problem with alert boxes (especially sweet-alerts is that they have a
delay and Selenium is pretty much too fast)
An Option that worked for me is:
while True:
try:
driver.find_element_by_xpath('//div[#class="sweet-alert showSweetAlert visible"]')
break
except:
wait = WebDriverWait(driver, 1000)
confirm_button = driver.find_element_by_xpath('//button[#class="confirm"]')
confirm_button.click()

Another option to handle alerts in Selenium Python would be to eliminate the notifications all together, if they are not needed.
You can pass in options to your webdriver browser that disables the notifications.
Example Python code using Chrome as the browser with options:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-notifications")
driver = webdriver.Chrome(ChromeDriverManager().install(),options=chrome_options)
driver.get('https://google.com')
print("opened Google")
driver.quit()