Python Get on a website and running $(document).ready(function() - javascript

I am doing some testing on my site, and I have a python program which does gets on few different pages. Some of these pages have $(document).ready(function(). I noticed that when I do get through python, I get the code, but for example $(document).ready(function() doesn't run.
How can I run the $(document).ready(function() of the site I am doing a GET on?
Thank you for help.

You should go for Selenium, it lets you control a real browser from your python code . That means your javascript will be executed by the browser .
Example code :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

Related

Selected text with Selenium and Python?

In the web console, getting the selected (highlighted) text is a simple manner
window.getSelection().toString()
How about doing this in a headless browser? In particular, I'm using selenium with its python API. I cannot find methods similar to getSelection() around driver:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get("http://www.python.org")
For example, suppose I have selected/highlighted (with the cursor) the string "suppose I have " on this page, the desired output should be "suppose I have ". In case no text is selected/highlighted, return the empty string "".
The answer I found is to execute Javascript directly within selenium. For example, to fulfill what I want, run the following script.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.python.org")
# Manually highlight some text with your cursor.
driver.execute_script("return window.getSelection().toString()")
Slightly unrelated but useful: This works within the currently selected window. To switch among different windows, see [1].
[1] Python Selenium get current window handle

PhantomJS not retrieving correct data

I am trying to scrape a web page which has javascript in it using phantomjs. I found an element for button and when i click it, it show render next link. But i am not getting the exact output what i want. Instead, i am getting different output which is not required.
The code is:
from bs4 import BeautifulSoup
import requests
from selenium import webdriver
s = requests.session()
fg =s.get('https://in.bookmyshow.com/booktickets/INCM/32076',headers=headers)
so = BeautifulSoup(fg.text,"html.parser")
texts = so.findAll("div",{"class":"__buytickets"})
print(texts[0].a['href'])
print(fg.url)
driver = webdriver.PhantomJS()
driver.get(movie_links[0])
element = driver.find_element_by_class_name('__buytickets')
element.click()
print(driver.current_url)
I am getting the output as :
javascript:;
https://in.bookmyshow.com/booktickets/INCM/32076
https://in.bookmyshow.com/booktickets/INVB/47680
what i have to get is:
javascript:;
https://in.bookmyshow.com/booktickets/INCM/32076
https://in.bookmyshow.com/booktickets/INCM/32076#Seatlayout
Actually, the link which i have to get is generated by javascript of the previous link. How to get this link? (seatlayout link) Please help! Thanks in Advance.
PhantomJS in my experience don't work well.
Сhrome and Mozilla better.
Vitaly Slobodin https://github.com/Vitallium said he will not develop more Phantomjs.
Use Headless Chrome or Firefox.

How can i get embed JSON Data from Website with Python?

I have a device to collect energy data with a webinterface on it and sadly no API.
There is a JSON stored in window.dataJSON.
I can get the value of it with: console.log(JSON.stringify(window.dataJSON)); via the Chrome Debugger.
But my question is: How can i get this data with python?
I know i can get the Sourcecode of the page with:
import urllib2
response = urllib2.urlopen("10.10.10.10")
page_source = response.read()
But how can i read the JSON stored in window.dataJSON?
Thank you in advance!
window object exists only in a browser. So to get property of window, you should use a browser to do it.
You can use Selenium :
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('http://www.example.com')
result = driver.execute_script('return JSON.stringify(window.dataJSON)')
And you can change webdriver to use Headless Chrome or PhatomJS if you don't want a browser to show up.
Maybe you need to tell driver to wait if dataJSON is assigned to window asynchronously.

How to let phantomjs show dynamically loaded webpage?

When I access https://www.ncbi.nlm.nih.gov/pubmed/?term=cell, there is "Results by year", a histogram under it and "Download CSV".
But when I access the same URL with the following script, I don't see them. Does anybody know why?
Is there a way to get the histogram along with "Download CSV" using a command line scraper ? Thanks.
$ cat phjsget.py
#!/usr/bin/env python
import sys
from selenium import webdriver
browser = webdriver.PhantomJS(service_log_path='/dev/null')
browser.get(sys.argv[1])
print browser.page_source.encode('utf-8')
browser.close()
$ ./phjsget.py https://www.ncbi.nlm.nih.gov/pubmed/?term=cell
The "Results by year" and "Download CSV" are loaded in after the page has loaded with Javascript. wget will not execute Javascript. You can use a tool like PhantomJS or Selenium to simulate real browser behavior that will execute Javascript.

How to set up Selenium to automate a stand alone desktop webapp (Node-WebKit)

We have a desktop application written using Node-WebKit, javascript, html and css and packaged into an exe file (here is exactly how it is built http://tutorialzine.com/2015/01/your-first-node-webkit-app/)
All the tutorials I found are for apps that use regular browsers(chrome, IE...). I want to use javascript, Selenium, mocha or any other javascript framework... Any ideas please? Steps or tutorials would be great!
And if that is possible, how to launch an exe file from the tests too.
Just to clarify:
- I want to write the tests in javascript.
- It's already working with C# like this:
public ExeApp()
{
var service = ChromeDriverService.CreateDefaultService(ExePath _ ExeName);
service.Start();
var options = new ChromeOptions();
options.BinaryLocation = exePath;
Driver = new ChromeDriver(service, options);
Driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));
//Wait for page to load
}
To login into the app:
Driver.SwitchTo().Window(handle);
Driver.FindElement(By.Id(loginWindowId));
//Wait for page to load
var loginButton = Driver.FindElement(By.Id(buttonId));
loginButton.Click();
So how to do the same thing in javasript as I'm very new to this language. Thank you.
Might not be able to if it's not something you can access from a standalone browser.
NW.js provides a customized ChromeDriver compatible with Selenium, so you can test your NW.js app with any clients implementing the Json Wire protocol.
This is an example from the official documentation with Python:
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("nwapp=/path/to/your/app")
driver = webdriver.Chrome(executable_path='/path/to/nwjs/chromedriver', chrome_options=chrome_options)
time.sleep(5) # Wait 5s to see the web page
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Wait 5s to see the search result
driver.quit()
For more information:
NW.js Test with ChromeDriver

Categories