Selenium does not run a javascript - javascript

I'm trying to execute a simple java script command at selenium using python for scroll a part of a web page.
Here's the code:
command_js = 'document.querySelector(' + "'" + 'css_div_example[css_example="this is just a example for css selector"]' + "').scroll(0, 10000)"
#string joined "document.querySelector('css_div_example[css_example="this is just a example for css selector"]').scroll(0, 10000)"
driver.execute_script(command_js)
Picture:
Picture of scroll bar
OBS.:
I tried to execute directly from Firefox console and it worked very well.
Selenium does not return any error, just not execute it.
I tried to WebDriverWait with EC and time.sleep().
I am using python 3+ and PyCharm IDE, Firefox webdrivers...
Anyone can help me?

Related

How to get past Javascript is disabled in your browser error when web scraping with Python

I am trying to create a script to download an ebook into a pdf. When I try to use beautifulsoup in it I to print the contents of a single page, I get a message in the console stating "Oh no! It looks like JavaScript is disabled in your browser. Please re-enable to access the reader."
I have already enabled Javascript in Chrome and this same piece of code works for a page like a stackO answer page. What could be blocking Javascript in this page and how can I bypass it?
My code for reference:
url = requests.get("https://platform.virdocs.com/r/s/0/doc/350551/sp/14552484/mi/47443495/?cfi=%2F4%2F2%5BP7001013978000000000000000003FF2%5D%2F2%2F2%5BP7001013978000000000000000010019%5D%2F2%2C%2F1%3A0%2C%2F1%3A0")
url.raise_for_status()
soup = bs4.BeautifulSoup(url.text, "html.parser")
elems = soup.select("p")
print(elems[0].getText())
The problem is that the page actually contains no content. To load the content it needs to run some JS code. The requests.get method does not run JS, it just loads the basic HTML.
What you need to do is to emulate a browser, i.e. 'open' the page, run JS, and then scrape content. One way to do it is to use a browser driver as described here - https://stackoverflow.com/a/57912823/9805867

PhantomJS not retrieving correct data

I am trying to scrape a web page which has javascript in it using phantomjs. I found an element for button and when i click it, it show render next link. But i am not getting the exact output what i want. Instead, i am getting different output which is not required.
The code is:
from bs4 import BeautifulSoup
import requests
from selenium import webdriver
s = requests.session()
fg =s.get('https://in.bookmyshow.com/booktickets/INCM/32076',headers=headers)
so = BeautifulSoup(fg.text,"html.parser")
texts = so.findAll("div",{"class":"__buytickets"})
print(texts[0].a['href'])
print(fg.url)
driver = webdriver.PhantomJS()
driver.get(movie_links[0])
element = driver.find_element_by_class_name('__buytickets')
element.click()
print(driver.current_url)
I am getting the output as :
javascript:;
https://in.bookmyshow.com/booktickets/INCM/32076
https://in.bookmyshow.com/booktickets/INVB/47680
what i have to get is:
javascript:;
https://in.bookmyshow.com/booktickets/INCM/32076
https://in.bookmyshow.com/booktickets/INCM/32076#Seatlayout
Actually, the link which i have to get is generated by javascript of the previous link. How to get this link? (seatlayout link) Please help! Thanks in Advance.
PhantomJS in my experience don't work well.
Сhrome and Mozilla better.
Vitaly Slobodin https://github.com/Vitallium said he will not develop more Phantomjs.
Use Headless Chrome or Firefox.

Does Google Apps block code from the Chrome console?

I noticed the Chrome console gave an error when the following was typed:
$("body").hide();
Is there a solution to modify HTML elements when working with Google Apps?
Jquery can't be accessible in browser console directly, because it's a external library file & doesn't come inbuilt within browser.
So to access the Jquery from console, you can run below line in console;
var jqlib = document.createElement('script');
jqlib.src = "https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js";
document.getElementsByTagName('head')[0].appendChild(jqlib);
Here we get the script tag of current page & injecting JQuery library into it.
After running above lines, you can execute your code;
$(body).hide();

Selenium Webdriver Black Screenshot PhantomJS

Testing a site using Cucumber and Selenium. In my hooks.js file I have the following:
driver.get("https://localhost:8000/");
sleep(2000);
TakeScreenshot('./test_artifacts/img/', 'Load Success', driver);
var btn = this.driver.wait(selenium.until.elementLocated(By.css('#app > div > div > div.col-xs-6.textColumn > button'), seleniumTimeOut));
TakeScreenshot('./test_artifacts/img/', 'Load Success', driver);
this.driver.sleep(3000);
The objective here is to successfully load the page and to take a screenshot of it. The website is running off of localhost. The problem occurs when a screenshot is taken. No matter how long I get driver to sleep I get a black screenshot, indicating to me that the website is not 'building' in time (to use what may be an incorrect term, given the circumstances). I then get this error:
Waiting for element to be located By(css selector, #app > div > div > div.col-xs-6.textColumn > button)
Wait timed out after 20112ms
If I change the URL to https://google.com/ I get a screenshot of the site, no problem. Any ideas what is happening here? Is my above hypothesis correct?
Thanks in advance!
Please try to wait until some other elements to be available. Use xpath locator instead CSS. Is your machine behind proxy? if means please add the phantom browser in behind the proxy.
var phantom = require('phantom');
phantom.create(function(browser){
browser.createPage(function(page){
browser.setProxy('98.239.198.83','21320','http', null, null, function(){
page.open(
'http://example.com/req.php', function() {
});});});});
First, change the driver to chrome driver and see what is script is doing after changing the URL. The different environment some time have different id's and XPath which affect the script. so before moving to phantomjs directly first check the behaviour of your script with common drivers like chrome or firefox.
I have gone through the same scenario and I have experienced the same situation. It's a silly mistake which was wasted my half of the day :p
Hope it will help you :)
The problem seems to have been a certification problem. The tests would run on http but when localhost was using https the tests would fail.
I worked around it by adding the following to this.BeforeFreature in my hooks.js file:
this.driver = new selenium.Builder()
.withCapabilities({'phantomjs.cli.args': ['--ignore-ssl-errors=true']})
.forBrowser('phantomjs')
.build();

Python Get on a website and running $(document).ready(function()

I am doing some testing on my site, and I have a python program which does gets on few different pages. Some of these pages have $(document).ready(function(). I noticed that when I do get through python, I get the code, but for example $(document).ready(function() doesn't run.
How can I run the $(document).ready(function() of the site I am doing a GET on?
Thank you for help.
You should go for Selenium, it lets you control a real browser from your python code . That means your javascript will be executed by the browser .
Example code :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

Categories