Javascript to input searches on multiple websites and take a screenshot - javascript

I am trying to automate the querying of names on 2 different websites. For each website I have to enter in the same name and take a screenshot of the results.I don't know how to approach this project. Is it possible to automate this with a javascript? Thank you for any input.
Website links are below:
[1] http://exclusions.oig.hhs.gov/search.aspx
[2] http://www.health.ny.gov/professionals/doctors/conduct/license_lookup.htm

You should be able to accomplish this with PhantomJS relatively easily. Check out the examples page - you'll probably want to use code evaluation to fill out and submit your forms, after which you can render the page to an image file.

You actually do this is one step: send a POST requests to the information you need. For the first example, you would send these POST data:
__VIEWSTATE=dDwtNzQ5MzEyMDI3Ozs%2BHWC8LXZfQQTCTJWtmudNLStjn%2Fk%3D&txtLastName1=test&txtFirstName1=&txtBusinessName1=&txtLastName2=&txtFirstName2=&txtBusinessName2=&txtLastName3=&txtFirstName3=&txtBusinessName3=&txtLastName4=&txtFirstName4=&txtBusinessName4=&txtLastName5=&txtFirstName5=&txtBusinessName5=&cmdSubmit=Search
So you can use PhantomJS, as suggested, with a POST request and take a screenshot. Or you can use a web service to create the screenshots. Unfortunately, very few allow POST requests with custom POST data. Browshot (see API) lets you send POST data and use a custom Referer. So your screenshot requests would include (after you encode the POST data):
url=http://exclusions.oig.hhs.gov/search.aspx&referer=http://exclusions.oig.hhs.gov/search.aspx&post_data=__VIEWSTATE%3DdDwtNzQ5MzEyMDI3Ozs%252BHWC8LXZfQQTCTJWtmudNLStjn%252Fk%253D%26txtLastName1%3Dtest%26txtFirstName1%3D%26txtBusinessName1%3D%26txtLastName2%3D%26txtFirstName2%3D%26txtBusinessName2%3D%26txtLastName3%3D%26txtFirstName3%3D%26txtBusinessName3%3D%26txtLastName4%3D%26txtFirstName4%3D%26txtBusinessName4%3D%26txtLastName5%3D%26txtFirstName5%3D%26txtBusinessName5%3D%26cmdSubmit%3DSearch
I have tested it, and do get a screenshot of the results, like if I filled out the form and submitted it.

Related

dynamically generate content for a page when clicking on product

everyone. I am making a website with t-shirts. I dynamically generate preview cards for products using a JSON file but I also need to generate content for an HTML file when clicking on the card. So, when I click on it, a new HTML page opens like product.html?product_id=id. I do not understand how to check for id or this part ?prodcut_id=id, and based on id it generates content for the page. Can anyone please link some guides or good solutions, I don't understand anything :(.
It sounds like you want the user's browser to ask the server to load a particular page based on the value of a variable called product_id.
The way a browser talks to a server is an HTTP Request, about which you can learn all the basics on javascipt.info and/or MDN.
The ?product_id=id is called the 'query' part of the URL, about which you can learn more on MDN and Wikipedia.
A request that gets a page with this kind of URL from the server is usually a GET request, which is simpler and requires less security than the more common and versatile POST request type.
You may notice some of the resources talking about AJAX requests (which are used to update part of the current page without reloading the whole thing), but you won't need to worry about this since you're just trying to have the browser navigate to a new page.
Your server needs to have some code to handle any such requests, basically saying:
"If anybody sends an HTTP GET request here, look at the value of the product_id variable and compare it to my available HTML files. If there's a match, send a response with the matching file, and if there's no match, send a page that says 'Error 404'."
That's the quick overview anyway. The resources will tell you much more about the details.
There are some solutions, how you can get the parameters from the url:
Get ID from URL with jQuery
It would also makes sense to understand what is a REST Api and how to build a own one, because i think you dont have a backend at the moment.
Here some refs:
https://www.conceptatech.com/blog/difference-front-end-back-end-development
https://www.tutorialspoint.com/nodejs/nodejs_restful_api.htm

How can a web page find what value to POST?

I'm currently making an application for a client to automatically fill some web forms on the website he uses to store his item pricing. The website doesn't have a documented public API, and there doesn't seem to be a way to add bulk pricing on the website itself. In order to accomplish this, I'm making a simple python application that reads his data, then sends POSTs to the website.
Their website is giving me a hard time, however, because it's sending payloads containing dozens of fields, while the form used to enter the pricing information only has 4 input fields. On top of that, their website uses angularjs to generate most of the web page, so I can't just find the <form>[...]</form> block and look at what's being sent, because that's not what they use.
Here is what the payload json looks like:
{
"entities":
[
{
"Price_Line_ID":"{}",
"Price_List_ID":"{}",
"Item_ID":"{}",
"Uofm_ID":"{}",
"Amount":"{}",
"Dtstamp":"{}",
"Tenant_ID":"{}",
"Created_On":"null",
"Created_By":"null",
"Changed_On":"null",
"Changed_By":"null",
"Seq":"0",
"Begin_Qty":"0",
"End_Qty":"0",
"Customer_ID":"null",
"Tax_Before_Discount":"false",
"Discount_Target":"All",
"Max_Discount_Amount":"null",
"Min_Discount_Amount":"null",
"Customer_Name":"null",
"Uofm":"null",
"Item_Number":"null",
"Uofm_Schedule_ID":"null",
"Uofm_Schedule":"null",
"Inactive":"false",
"entityAspect":
{
"entityTypeName":"PriceLine:#SalesPad.Spo.Api.Model",
"defaultResourceName":"PriceLines",
"entityState":"Added",
"originalValuesMap":
{},
"autoGeneratedKey":
{
"propertyName":"Price_Line_ID",
"autoGeneratedKeyType":"Identity"
}
}
}
],
"saveOptions":
{}
}
The 7 values at the top (with values of "{}") are found when I do a GET or POST action on the website's other pages. I've managed to find where all the values originate from, except the "Price_Line_ID" one, because it appears to change from page to page (and it changes after a price is added).
I know a web page can get the data needed for a POST event either in its own html (when using tags like <form>), and it can get them from other GET and POST events. Is there any other way for a web page to determine a value that will be sent in a POST event?
I'm not very familiar with angularjs, although from what I understand it only creates a bunch of Javascript for the page. Does it offer different ways of determining what values are sent in a POST or GET event?
Edit: I've already tracked all responses from GET and POST events from logging-in to adding a price, the Price_Line_ID field changes from page to page, and adding the price appears to use an ID different to the one received in the GET event. I just want to know the different ways that a web page (specifically one using angularjs might use to determine the value of the data sent in POST events.

ContactForm 7 and javascript to capture form data

I have a WordPress site and i use a cf7 form to make a booking enquiry. When the form is submitted an email is send to my info#.. and also a custom php file i made is passing data from the form to a CRM platform.
I use JS on_mail_sent action and i capture all the values of the form and im passing it to the php file via query (.php?id=id&......).
I want to ask if this method is acceptable or i must use another method. For example is there any way certain browsers for some reasons dont support javascript?
Thank you.
The better solution to use POST method to submit form.
About difference between Get and Post:
GET requests a representation of the specified resource. Note that GET should not be used for operations that cause side-effects, such as using it for taking actions in web applications. One reason for this is that GET may be used arbitrarily by robots or crawlers, which should not need to consider the side effects that a request should cause.
POST submits data to be processed (e.g., from an HTML form) to the identified resource. The data is included in the body of the request. This may result in the creation of a new resource or the updates of existing resources or both.
HTTP/1.1 specification (RFC 2616) section 9 Method Definitions contains more information on GET and POST as well as the other HTTP methods.
One more important moment:
Authors of services which use the HTTP protocol SHOULD NOT use GET based forms for the submission of sensitive data, because this will cause this data to be encoded in the Request-URI. Many existing servers, proxies, and user agents will log the request URI in some place where it might be visible to third parties. Servers can use POST-based form submission instead.
All popular browsers supports Javascript.
Regards.

Javascript: get data displayed from another url

I was wondering if I could get some data from another website to get it displayed on mine. The good example can be alexa.com. I need to display Alexa traffic rank and reputation in a div for example on my page, so it will be changed dynamically each time Alexa change its data.
Thank you for your help.
One way is to make an ajax request for the Alexa.com site, once you receive all the html, then you can use jquery or something to scrape it for the div you want.
It feels kinda dirty, but its an easy way to get what you want. Though this is assuming their page content isn't loaded dynamically.
Edit: See this for more info: Request external website data using jQuery ajax
yahoo yql... (instead of a php? proxy serverside script)..
I have a sneaky suspicion you do not own/control the external link site, so getting content from a different site, would fall under cross-domain security restrictions (to a modern browser).
So in order to regain 'power to the user', just use http://query.yahooapis.com/.
jQuery would not be strictly needed.
EXAMPLE 1:
Using the SQL-like command:
select * from html
where url="http://stackoverflow.com"
and xpath='//div/h3/a'
The following link will scrape SO for the newest questions (bypassing cross-domain security bull$#!7):
http://query.yahooapis.com/v1/public/yql?q=select%20title%20from%20html%20where%20url%3D%22http%3A%2F%2Fstackoverflow.com%22%20and%0A%20%20%20%20%20%20xpath%3D%27%2F%2Fdiv%2Fh3%2Fa%27%0A%20%20%20%20&format=json&callback=cbfunc
As you can see this will return a JSON array (one can also choose xml) and calling the callback-function: cbfunc.
Indeed, as a 'bonus' you also save a kitten every time you did not need to regex data out of 'tag-soup'.
Do you hear your little mad scientist inside yourself starting to giggle?
Then see this answer for more info (and don't forget it's comments for more examples).
Good Luck!

jQuery send values through POST

I'm trying to implement a third party tool. This tool uses a form with the post method to send data to their site. Is there any way that I can mimic this action without using the form tag? I don't know much about jquery post and same domain (this is sending it off to a different domain) so I don't know if there would be an issue with this.
Everything that I've found in my search talks about ajax and returning content after you post but all I want to do is to take the customer to the third party's site after they have submitted the form.
thanks!
You cannot send data to a different domain with AJAX. Its not permitted by the browser. As for can you do it without a form element, yes. Just encode the data as it would look in a browser get URL like http://site.com/search?query=I+love+js&perpage=10&page=2
datatosend="field1="+value+"&field2="+value2
$.post(url,datatosend,function(data){//do something with data. location.href="new location"}
Yes you can.
Check out this post.
https://stackoverflow.com/a/1078991
also found in the docs
http://api.jquery.com/jQuery.post/
http://api.jquery.com/jQuery.post/
and then you can do a location.replace() when it is completed
function replaceDoc()
{
window.location.replace("http://www.thatsite.com")
}

Categories