How can I webscrape data from website without HTML elements? [closed] - javascript

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 days ago.
Improve this question
How do I go about web scraping data from a website below to find specific case details?
Here are the manual steps to find case details:
Navigate to https://www.claytoncountyga.gov/government/courts/court-case-inquiry/
There seems to be possibly a JavaScript-loaded form with buttons/inputs to take your further to case details - need to select Name Search to search for cases by Last Name - click it
Then a new screen appears within the same element of (2) allowing user to select from drop down court (eg, Magistrate Court) and free form text input to enter Last Name and First Name (Smith John).
Clicking submit takes you to all the cases
Clicking the case number on one of the rows in the table populated within the same Element as all prior steps allows you to view case details - I’d like to scrape data from this page.
Because the inner form seems encapsulated (I’m guessing with Javascript), I can’t see the HTML elements that render after each input is provided. How can i automate this with Python?

The form is contained inside an iframe with ID "Clayton County". For selenium to be able to interact with elements inside it, we first have to switch to it using the EC.frame_to_be_available_and_switch_to_it method.
Then using Select() we can select an option from the dropdown menu.
In the last page we grab all the case numbers urls and save them in case_numbers_urls, so that we can loop over them, load each one, grab info and pass to the next one.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(service=Service(chromedriver_path))
driver.get('https://www.claytoncountyga.gov/government/courts/court-case-inquiry/')
# page 1
wait = WebDriverWait(driver, 9)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "Clayton County")))
driver.find_element(By.XPATH, "//a[contains(.,'Name Search')]").click()
# page 2
dropdown = wait.until(EC.element_to_be_clickable((By.ID, "ctt")))
Select(dropdown).select_by_value('M')
lname = 'Smith'
fname = 'John'
driver.find_element(By.NAME, 'lname').send_keys(lname)
driver.find_element(By.NAME, 'fname').send_keys(fname)
driver.find_element(By.ID, 'btnSrch').click()
# page 3
case_numbers_urls = [c.get_attribute('href') for c in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#myTable a[href]:not([rel])')))]
for url in case_numbers_urls:
driver.get(url)
# do something

Related

How to navigate a website ,hover mouse over sub menu and click on a website using selenium python

I was trying to automate one of my web application "maximo" by using selenium python.
I am able to login with username and password and also to click on any single button using "find_element_by_"etc.But I don't know how to hover mouse over sub menu and click.
Also I was trying to find a means to call Javascript code corresponding to a menu,which are shown ata the bottom of this maximo(picture attached).
enter image description here
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
browserchr3=webdriver.Chrome('C:\chromedriver\chromedriver.exe')
browserchr3.get('http://da-maximo.dca.com/maximo/webclient/login/login.jsp?welcome=true')
browserchr3.find_element_by_id('username').send_keys('bhsek015'+Keys.TAB)
userpassword=browserchr3.find_element_by_id('password')
userpassword
userpassword.send_keys('Adil#2020')
userpassword.submit()
#browserchr3.execute_script(topLevelMenus['m7f8f3e49_ns_mc'].menuClick({"id":"WOTRACK_APP","text":"Work Order Tracking","eventvalue":"WOTRACK","apptype":"RUN","target":"startcntr","event":"changeapp","value":"WOTRACK"}))
browserchr3.implicitly_wait(10)
Select(browserchr3.find_element_by_css_selector('#m7f8f3e49_ns_menu_WO_MODULE_a')).select_by_visible_text("Work Order Tracking")
wo
wo.click()
browserchr3.implicitly_wait(10)
wot=browserchr3.find_element_by_css_selector('#menuholdertd')
wot
wot.click()
For your question 1, hovering over element, you need to use Action Chains
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(browserchr3)
# Assuming wot is the element to move your mouse to
actions.move_to_element(wot)
actions.perform() #Performs chained action
You can chain actions like this
ActionChains(browserchr3).move_to_element(wot).perform()
For full reference, have a look at API => https://seleniumhq.github.io/selenium/docs/api/py/webdriver/selenium.webdriver.common.action_chains.html
For Question 2, how to execute javascript,
browserchr3.execute_script('Script statement')

jp#gc - WebDriver Sampler in JMeter

I just need to select a value from “smart search pick-list” (drop down list) from a web page using webdriver sampler in Jmeter. I already tried using one of it’s pick-list value’s xpath & id and it doesn’t work.
xpath value for "smart search pick-list" field is
.//*[#id='j_id0:frm:searchDiagnosisId']
xpath value for "smart search pick-list" field's one of the value is
.//*[#id='ui-id-21']
If I use sendKeys method, input value just appearing in the text box and not selecting values below. And also remaining script are not executed to fill remaining fields in the webpage. Can you give a solution to select a value from “smart search pick-list”.
Thanks in advance
People mostly use Select class in order to interact with the "dropdown lists", the relevant WebDriver Sampler code will look somehow like:
var select = new org.openqa.selenium.support.ui.Select(WDS.browser.findElement(org.openqa.selenium.By.xpath(".//*[#id='j_id0:frm:searchDiagnosisId']")))
select.selectByValue("....")
See The WebDriver Sampler: Your Top 10 Questions Answered article for more WebDriver sampler tips and tricks

Create a unique URL for linking to data pulled from a database [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Is it possible to create a unique URL to link to information that is called and displayed from a database?
[I am struggling to articulate this question, so any suggestions for revising it are welcome.]
Specifics: I have a page that includes an embedded map and related information about each location stored in a separate database. Users can navigate through the locations by clicking the markers on the map or by using navigation buttons in the sidebar. Either case zooms in on the marker and displays information about the location in the sidebar. All this works fine, but because the information is simply placed in divs, the page's URL is always the same regardless of what is being displayed.
I would like to be able to create and share links to the individual locations, which would open with the point selected in the map and the information displayed in the sidebar.
I use a javascript function to call the information from the database. Here is a simplified version of that code:
<a href="#" onClick='callMarker(markerNum);'>Click Me</a>
function callMarker(markerNum) {
var sql = new data.SQL({ user: 'me' });
var sql_query = "SELECT * FROM map_name WHERE marker_id = " + markerNum;
sql.execute(sql_query)
.done(function(data) {
console.log(data.rows[0]);
var title = data.rows[0].title;
// Displays the ID of the object in the external information box
document.getElementById("title").innerHTML= data.rows[0].title;
}
Since each location is tagged with a unique ID number (marker_id) in the database, it seems like this could be appended to the end of the current URL to create a unique URL associated with each point. Or maybe the ID number (marker_id) could somehow be assigned to a div ID as a variable within the page. I am not sure how to do either or if either would work.
Any ideas or direction would be appreciated. I have been searching a lot of different approaches, but have not found anything that seems to address this problem.
If you don't want to do anything on the server to provide a unique url, you can use a client side hash with urls in the form /mymap#uniqueId. The hashed part of the url is accessible in javascript like so:
window.location.hash
You can even set that property on the location as users click around on the map. So in your callMarker function you can do something like:
if(history.pushState)
history.pushState(null, null, '#myhash');
else
location.hash = '#myhash';
In newer browsers the first method avoids a hard refresh.
You can grab this value on the page load event to render the appropriate point on the map and sidebar information.

What do I call this type of list on ebay? [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
My apologies in advance if I posted this in the wrong place...
OK. I'd like to add a similar hierarchical/selectable list of data to my website as can be found on ebay's "sell a new item page" here:
http://cgi5.ebay.com/ws/eBayISAPI.dll?NewListing
Note: you may need to be able to log into ebay to follow the link. Sorry
(I've tried to post a screenshot to illustrate the table for those who can't follow the above link, but as I'm a new user to StackOverflow I'm not allowed to post images until my Reputation > 10. Bummer)
If you can't follow the link above, goto ebay.com, login (if you have an account), click the SELL tab->SELL AN ITEM and then the BROWSE CATEGORIES LINK which will bring up the list in question.
Ebay uses their version to make sure a seller lists their auction item in the proper category and I could use something similar for completely unrelated purposes on my own website.
It works like this: upon page load, column 1 is populated with a list of possible category options and when the user selects one of those options, a new (the 2nd) column of sub options is then displayed to the right of the first. When an option from column 2 is selected, column 3's options are displayed and so forth until the user has drilled down into the categories and arrives on one which does not have any additional subcategories. Ultimate, a single category number is displayed in the lower right hand corner one the user reaches the end of the category line.
2 questions:
To help me research how it works, what type of list is it? (Assuming it is a list and not a menu ect.)
In order for me to make one for my site, will it require adding code in javascript or are there online generators to make such an animal?
I've inspected the page's source before asking the question, but being fairly new to html/php and not having dabbled in javascript yet, I drew a blank on how to learn more.
Knowing what to call this type of list will help me google to find out how to make something that looks and functions in a similar manner for my data which is currently in a mySQL table.
Thanks in advance for any replies!
This can be achieved in HTML and its called 'selection list' with this code:
<select name="items" size="3">
<option value="ball">Ball</option>
<option value="toy">Toys</option>
<option value="something">Something</option>
</select>
You can see it in action here:
http://jsfiddle.net/5EXmc/
Of course you are going to need some javascript to understand which item was selected.
If you are interested in the javascript for this to displace what you selected:
http://jsfiddle.net/5EXmc/1/
$("select").change(function () {
var str = "";
$("select option:selected").each(function () {
str += $(this).text() + " ";
});
$("div").text(str);
}).change();
If you took 10 seconds to look at the HTML you'd learn that it's just a select element with a size property. This is just another mode of the input element.

Survey Question Reordering?

Im doing a survey application . It is having option to add questions.If on clicks on add button it will show the text box for adding question and a select box for selecting question type(radio button , check box) once it fill the field i will store than in database ans display in UI. then the user have option for adding question above and below. if he add question above i need to reorder the question number below. if he add question in middle i need to update the question number below that. how can i solve this situation ?
In a similar situation, I have shown the list of items to be sorted in a table/grid and given the option to the user to move up or down the items in the grid as per their choice. I kept a column for order in the grid (hidden in my case) and a corresponding column in the database table. Whenever there is a reordering in the grid, I update the order by a swap operation for the two rows that are swapping position. So, once the user has done the reordering and save the order using a button, I use to update the order column according to the order in the front page.
https://pixbyfunc.appspot.com/pub/asq/asq.html is a running example of a contingent questionnaire. A different set of questions are presented based on if the user likes wine or not.
It's designed using JavaScript that looks like:
Q().ask("Have you had your drink today?");
Q('like-wine').using('radio', 'Yes', 'No').ask("Do you like wine?");
Q('like-wine').matches('no').naming('reason').using('textarea').ask("Why not?").abort();//stop the survey
Q().ask("How much do you drink a day?");
Q('which-wine').using('checkbox', 'Reds', 'Whites', 'Other').ask("What do you like?");
Q('which-wine').matches('reds').ask("If you would like to receive a free bottle of Chateau d'Yquem Sauternes 2004 from your Santa, please provide your email.");
Q().start();
You can view the source to see how Q is implemented and to modify to suit your needs.

Categories