We Keep Coding

Unable to download research article from scihub using browser emulation with selenium

I am trying to automate the download of research articles from scihub (https://sci-hub.scihubtw.tw/) based on their corresponding article titles. I am using a library called scholarly (https://pypi.org/project/scholarly/) to get the url, author information related to the given article title as shown in the code below.
I use the fetched url (as described above) to emulate the download process using scihub. But I am unable to download directly, since I can't press the open button on the search page (https://sci-hub.scihubtw.tw/). And pressing enter after populating the query forwards me to another page with an open button. I am unable to fetch and press the open button for some reason and it always returns me a null element using the selenium library.
However, I am able to execute the following in the browser console and successfully download the pape,
document.querySelector("#open-button").click()
But, trying to get similar response from selenium is failing.
Kindly help me resolve this issue.
## This part of code fetches url using scholarly library from google scholar
from scholarly import scholarly
search_query = scholarly.search_pubs('Hydrogen-hydrogen pair correlation function in liquid water')
search_query = [query for query in search_query][0]
## This part of code uses selenium to automate download process
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
import time
download_dir = '/Users/cacsag4/Downloads'
# setup the browser
options = webdriver.ChromeOptions()
options.add_experimental_option('prefs', {
"download.default_directory": download_dir, #Change default directory for downloads
"download.prompt_for_download": False, #To auto download the file
"download.directory_upgrade": True,
"plugins.always_open_pdf_externally": True #It will not show PDF directly in chrome
})
browser = webdriver.Chrome('./chromedriver', options=options)
browser.delete_all_cookies()
browser.get('https://sci-hub.scihubtw.tw/')
# Find the search element to send the url string to it
searchElem = browser.find_element(By.CSS_SELECTOR, 'input[type="textbox"]')
searchElem.send_keys(search_query.bib['url'])
# Emulate pressing enter two different ways, either by pressing return key or by executing JS
#searchElem.send_keys(Keys.ENTER) # This produces the same effect as the next line
browser.execute_script("javascript:document.forms[0].submit()")
# Wait for page to load
time.sleep(10)
# Try to press the open button using JS or by fetching the button by its ID
# This returns error since its unable to fetch open-button id
browser.execute_script('javascript:document.querySelector("#open-button").click()')
#openElem = browser.find_element(By.ID, "open-button") ## This also returns a null element

Ok, so I got the answer to this question. Sci-hub stores its pdf inside an iframe, so all you got to do is fetch the src attribute of the iframe after pressing enter on the first page. The following code does the job.
from scholarly import scholarly
search_query = scholarly.search_pubs('Hydrogen-hydrogen pair correlation function in liquid water')
search_query = [query for query in search_query][0]
print(search_query.bib['url'])
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
import time
download_dir = '/Users/cacsag4/Downloads'
# setup the browser
options = webdriver.ChromeOptions()
options.add_experimental_option('prefs', {
"download.default_directory": download_dir, #Change default directory for downloads
"download.prompt_for_download": False, #To auto download the file
"download.directory_upgrade": True,
"plugins.always_open_pdf_externally": True #It will not show PDF directly in chrome
})
browser = webdriver.Chrome('./chromedriver', options=options)
browser.delete_all_cookies()
browser.get('https://sci-hub.scihubtw.tw/')
# Find the search element to send the url string to it
searchElem = browser.find_element(By.CSS_SELECTOR, 'input[type="textbox"]')
searchElem.send_keys(search_query.bib['url'])
# Emulate pressing enter two different ways, either by pressing return key or by executing JS
#searchElem.send_keys(Keys.ENTER) # This produces the same effect as the next line
browser.execute_script("javascript:document.forms[0].submit()")
# Wait for page to load
time.sleep(2)
# Try to press the open button using JS or by fetching the button by its ID
# This returns error since its unable to fetch open-button id
#browser.execute_script('javascript:document.querySelector("#open-button").click()')
openElem = browser.find_element(By.CSS_SELECTOR, "iframe") ## This also returns a null element
browser.get(openElem.get_attribute('src'))

Can't find forms inside a html code - Web Scraping with Python and Selenium

I'm trying to scrap this website, but there are some forms to fill.
The mainly objective is to fill these 5 forms (one appears after selecting another) and download the data through the button "Consultar".
This forms are coded in javascript and I can't find them in the page's html code. When I inspect the frames trough Google Chrome, I find the forms ID's, but the code doesn't find them.
I have just a prototype of my code. I can't advance without knowing what I can do to find these forms.
from selenium import webdriver
from bs4 import BeautifulSoup
import time
import os
#Variables
url = 'http://www.anbima.com.br/pt_br/informar/sistema-reune.htm'
path_phantom = 'C:\\Users\\TBMEPYG\\AppData\\Local\\Continuum\\Anaconda3\\Lib\\site-packages\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
#Processing
driver = webdriver.PhantomJS(executable_path= path_phantom)
driver.get(url)
data = driver.find_element_by_id('data_ref')
data.send_keys("21/08/2017")
driver.quit()
Edit:
I updated the code to this:
from selenium import webdriver
path_phantom = 'C:\\Users\\TBMEPYG\\AppData\\Local\\Continuum\\Anaconda3\\Lib\\site-packages\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path= path_phantom)
driver.get('http://www.anbima.com.br/reune/reune.asp')
driver.switch_to.frame(driver.find_element_by_xpath('//iframe[#class="full"]'))
data = driver.find_element_by_name('Dt_Ref')
data.clear()
data.send_keys('21/08/
And I got this error:
CD: C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3
Current directory: C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3
python "C:\Users\TBMEPYG\Desktop\vamo.py"
Process started >>>
Traceback (most recent call last):
File "C:\Users\TBMEPYG\Desktop\vamo.py", line 8, in <module>
data = driver.find_element_by_name('Dt_Ref')
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 426, in find_element_by_name
return self.find_element(by=By.NAME, value=name)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 832, in find_element
'value': value})['value']
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with name 'Dt_Ref'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"89","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:62040","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"name\", \"value\": \"Dt_Ref\", \"sessionId\": \"bdd3fc70-8dd0-11e7-aeb1-85b8cfbe0d1c\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/bdd3fc70-8dd0-11e7-aeb1-85b8cfbe0d1c/element"}}
Screenshot: available via screen
Edit2:
Another possibility is to use the link inside of the the mains page http://www.anbima.com.br/reune/reune.asp
When I changed the code to this, I've got another error
from selenium import webdriver
path_phantom = 'C:\\Users\\TBMEPYG\\AppData\\Local\\Continuum\\Anaconda3\\Lib\\site-packages\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path= path_phantom)
driver.get('http://www.anbima.com.br/reune/reune.asp')
data = driver.find_element_by_name('Dt_Ref')
data.clear()
data.send_keys('21/08/2017')
Error:
Traceback (most recent call last):
File "C:\Users\TBMEPYG\Desktop\vamo.py", line 9, in <module>
data = driver.find_element_by_name('Dt_Ref')
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 426, in find_element_by_name
return self.find_element(by=By.NAME, value=name)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 832, in find_element
'value': value})['value']
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: {"request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"89","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:61820","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"name\", \"value\": \"Dt_Ref\", \"sessionId\": \"e61dd170-8dcf-11e7-a019-41573671066b\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/e61dd170-8dcf-11e7-a019-41573671066b/element"}}
Screenshot: available via screen

To be able to handle elements inside form you need to switch to iframe first:
driver.switch_to.frame(driver.find_element_by_xpath('//iframe[#class="full"]'))
data = driver.find_element_by_name('Dt_Ref')
data.clear()
data.send_keys('21/08/2017')

Just to update:
Problem not solved, but I know what is.
It's a bug in phantomJS that is occurring. So, if you have the same problem, try to use Chrome or Firefox.
Thanks for the answers.

I think you messed up with single and double quotes. Correct code would be-
driver.get("http://www.anbima.com.br/reune/reune.asp")
data = driver.find_element_by_name("Dt_Ref")
data.clear()
data.send_keys("21/08/2017")

How do I recover the "text" from the page originating the alert, esp **after** the human user has *clicked* [dismiss] on the page's **alert**?

Basically, when an alert is popped up in javascript, I can dismiss() it from python perfectly OK, by calling selenium.webdriver.common.alert.Alert(browser).dismiss().
However, if the "browser user" dismisses the alert by clicking [OK] (on screen) with their mouse, then the browser alert gets "Lost in Space" the body.text can no longer be accessed from python.
So... How do I recover the "text" from the page originating the alert, esp after the human user has clicked [dismiss] on the page's alert?
Here are the hints and a script to demonstrate the problem...
FYI: The objective of the originating code is it allow the browser user intervene on screen in testing and manually response to specific alerts.
#!/usr/bin/env python
import os,sys,time
import selenium.webdriver
import selenium.webdriver.support.expected_conditions
print dict(python=sys.version,selenium=selenium.__version__)
path=os.path.join(os.getcwd(),"hello_worlds.html")
url="file:///"+path
open(path,"w").write("""<HTML>
<HEAD><TITLE>Head Title</TITLE></HEAD>
<BODY><H1>Hello, worlds!</H1></BODY>
</HTML> """)
browser=selenium.webdriver.Firefox()
browser.get(url)
body=browser.find_element_by_tag_name("body")
print "BODY:",body.text
try:
for enum,world in enumerate("Mercury Venus Earth Mars Asteroids Jupiter Saturn Uranus Neptune".split()):
if "Earth" in world: world+=": So do MANUALLY dismiss! {Click [OK] now!!!}"
else: world+=": AUTO PILOT... please DONT dismiss! {done via selenium.dismiss()!}"
browser.execute_script('alert("Hello, %s!")'%world)
if selenium.webdriver.support.expected_conditions.alert_is_present():
print selenium.webdriver.common.alert.Alert(browser).text
time.sleep(enum+5)
if "Earth" not in world: selenium.webdriver.common.alert.Alert(browser).dismiss()
print "BODY:",body.text
finally:
browser.quit()
Output: (Crash at Earth)
{'python': '2.6.6 (r266:84292, Aug 18 2016, 15:13:37) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]', 'selenium': '2.53.2'}
BODY: Hello, worlds!
Hello, Mercury: AUTO PILOT... please DONT dismiss! {done via selenium.dismiss()!}!
BODY: Hello, worlds!
Hello, Venus: AUTO PILOT... please DONT dismiss! {done via selenium.dismiss()!}!
BODY: Hello, worlds!
Hello, Earth: So do MANUALLY dismiss! {Click [OK] now!!!}!
BODY:
Traceback (most recent call last):
File "./js_alert.py", line 37, in <module>
print "BODY:",body.text
...
selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: Hello, Earth: So do MANUALLY dismiss! {Click [OK] now!!!}!
Message: Unexpected modal dialog (text: Hello, Earth: So do MANUALLY dismiss! {Click [OK] now!!!}!) The alert disappeared before it could be closed.
The strange thing is that if the browser user triggers another alert (on another page even!), then a selenium.dismiss() will pull body.text back from limbo and selenium with from then will operate as per (my) expectations.
Any suggestions on how to get the browser back to the page.body? (And escape the alert)
Addendum: Here are similar questions (found with intense searching):
downloading - Issues downloading file using Selenium + Firefox
java alert - Selenium WebDriver - Unexpected modal dialog Alert
UnexpectedAlertPresentException - Webdriver error: "No alert is present" after UnexpectedAlertPresentException is thrown
DesiredCapabilities - How to handle an Alert with "UnexpectedAlertBehaviour" capability in Selenium?
java&javascript - org.openqa.selenium.UnhandledAlertException: unexpected alert open
disables exception or DesiredCapabilities - org.openqa.selenium.UnhandledAlertException: unexpected alert open
non-click suggestion - When I show a dialog, an exception "UnhandledAlertException: Modal dialog present" is thrown
IE and in the wild - https://github.com/SeleniumHQ/selenium-google-code-issue-archive/issues/4839
java bug report? https://groups.google.com/forum/#!topic/webdriver/aNyOfEjMENg
"still doesn't work." catching exception does not reset the problem - (However almost exactly the same problem) - how to handle javascript alerts in selenium using python
suggested doing a driver.findElement(By.id("myAlert")); but that throws same exception - https://sqa.stackexchange.com/questions/22482/why-my-alert-is-not-popping-up
browser.refresh(); gives UnexpectedAlertPresentException so doesn't work. https://github.com/angular/protractor/issues/1486
browser.switch_to.window(browser.window_handles[0]) then body.text gives: UnexpectedAlertPresentException
Similar - Python webdriver to handle pop up browser windows which is not an alert

I have been struggling with this issue off and on for a long time; your comment on your question solved the problem for me:
After both UnexpectedAlertPresentException and NoAlertPresentException are thrown...
browser.execute_script('alert("Clearing out past dialogs.")')
browser.switch_to.alert.accept()
As you said in your answer, webdriver is creating a 'dialog' when the alert is present. Closing the alert by hand causes its reference to get lost in limbo, but it's still blocking access to body.text. Creating a new alert seems to allow webdriver to clear out that old 'dialog' and (after accepting) grants access to the page again.

No solution yet... I suspect it is a bug in Firefox's webdriver in file modals.js
Firefox's web driver captures the alert popup and replaces it with an element named 'dialog' { e.g. getElementsByTagName('dialog')[0]}
The problem is that if the tester is human and clicks on either "dismiss" or "accept", then the "onclick" for 'dialog' does not call fxdriver.modals.clearFlag_... hence the problem.
https://github.com/SeleniumHQ/selenium/blob/master/javascript/firefox-driver/js/modals.js#LC39
fxdriver.modals.isModalPresent = function(callback, timeout) {
fxdriver.modaltimer = new fxdriver.Timer();
fxdriver.modaltimer.runWhenTrue(
function() {
var modal = fxdriver.modals.find_();
return modal && modal.document && modal.document.getElementsByTagName('dialog')[0];
},
function() { callback(true) },
timeout,
function() { callback(false) });
};
fxdriver.modals.acceptAlert = function(driver) {
var modal = fxdriver.modals.find_();
var button = fxdriver.modals.findButton_(modal, 'accept');
button.click();
fxdriver.modals.clearFlag_(driver);
};
https://github.com/SeleniumHQ/selenium/blob/master/javascript/firefox-driver/js/modals.js#LC127
fxdriver.modals.setFlag = function(driver, flagValue) {
driver.modalOpen = flagValue;
};
fxdriver.modals.clearFlag_ = function(driver) {
fxdriver.modals.setFlag(driver, false);
};
Maybe the file_detector selecting local files has a work around...
http://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webdriver.WebDriver.file_detector_context
Similar problem:
How to debug Firefox alert box closing automatically and being unable to detect the alert in Serenity BDD?
Bug report:
https://github.com/SeleniumHQ/selenium/issues/4190

Handle alert in Selenium Python

I have the following pop-up alert that I want to handle after a file upload. I have used the code below and it throws the error below.
wait.until(EC.alert_is_present())
driver.switch_to.alert().accept()
Traceback (most recent call last):
File "update.py", line 45, in
driver.switch_to.alert().accept()
TypeError: 'Alert' object is not callable
Why is this happening? I have handled a similar alert (that one had a cancel button?) in this manner.

There are two ways to accept alert available in Python + selenium (there is also JavaScript code for execute_script(), but it's not related to current issue):
driver.switch_to_alert().accept() # deprecated, but still works
driver.switch_to.alert.accept()
Note that in second line you don't need to call alert() as you did in your code

Problem with alert boxes (especially sweet-alerts is that they have a
delay and Selenium is pretty much too fast)
An Option that worked for me is:
while True:
try:
driver.find_element_by_xpath('//div[#class="sweet-alert showSweetAlert visible"]')
break
except:
wait = WebDriverWait(driver, 1000)
confirm_button = driver.find_element_by_xpath('//button[#class="confirm"]')
confirm_button.click()

Another option to handle alerts in Selenium Python would be to eliminate the notifications all together, if they are not needed.
You can pass in options to your webdriver browser that disables the notifications.
Example Python code using Chrome as the browser with options:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-notifications")
driver = webdriver.Chrome(ChromeDriverManager().install(),options=chrome_options)
driver.get('https://google.com')
print("opened Google")
driver.quit()

Selenium, python, clicking on a javascript link regularly?

what should be right way to click on a javascript generated link on a regular time interval using python and selenium bindings? should it be using a thread?
as i would need to continue to process the input data, i need to refresh/reset a timer to continue to receive data, clicking on this given link to do this refresh (and this link is html directly generated by javascript).
best regards

You don't need thread to do this.
Use javascript function setInterval to continuously click the link.
For example:
import time
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://jsfiddle.net/falsetru/4UxgK/show/')
# Click the link every 3000 ms.
driver.execute_script('''
// argument passed from Python can be accessed by `arguments` array.
var link = arguments[0];
var timer = setInterval(function() {
link.click();
}, 3000);
''', driver.find_element_by_id('activity'))
while True:
data = driver.find_element_by_id('counter').text
print(data)
time.sleep(1)
NOTE
If you get error like follow, upgrade selenium to recent version. I experienced following error with Firefox 23.0 + selenium 2.32.0. Error was gone with selenium 2.35.0.
Traceback (most recent call last):
File "t2.py", line 12, in <module>
print driver.execute_script('''return 1 + 2;''')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 397, in execute_script
{'script': script, 'args':converted_args})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 165, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 158, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: u'waiting for evaluate.js load failed' ; Stacktrace:
at r (file:///tmp/tmpm1sJhH/extensions/fxdriver#googlecode.com/components/driver_component.js:8360)
at fxdriver.Timer.prototype.runWhenTrue/g (file:///tmp/tmpm1sJhH/extensions/fxdriver#googlecode.com/components/driver_component.js:392)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmpm1sJhH/extensions/fxdriver#googlecode.com/components/driver_component.js:386)
Alternative: using thread
import threading
import time
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://jsfiddle.net/falsetru/4UxgK/show/')
def click_loop(link, interval):
while True:
link.click()
time.sleep(interval)
link = driver.find_element_by_id('activity')
threading.Thread(target=click_loop, args=(link, 3)).start()
while True:
data = driver.find_element_by_id('counter').text
print(data)
time.sleep(1)