I'm currently making an application for a client to automatically fill some web forms on the website he uses to store his item pricing. The website doesn't have a documented public API, and there doesn't seem to be a way to add bulk pricing on the website itself. In order to accomplish this, I'm making a simple python application that reads his data, then sends POSTs to the website.
Their website is giving me a hard time, however, because it's sending payloads containing dozens of fields, while the form used to enter the pricing information only has 4 input fields. On top of that, their website uses angularjs to generate most of the web page, so I can't just find the <form>[...]</form> block and look at what's being sent, because that's not what they use.
Here is what the payload json looks like:
{
"entities":
[
{
"Price_Line_ID":"{}",
"Price_List_ID":"{}",
"Item_ID":"{}",
"Uofm_ID":"{}",
"Amount":"{}",
"Dtstamp":"{}",
"Tenant_ID":"{}",
"Created_On":"null",
"Created_By":"null",
"Changed_On":"null",
"Changed_By":"null",
"Seq":"0",
"Begin_Qty":"0",
"End_Qty":"0",
"Customer_ID":"null",
"Tax_Before_Discount":"false",
"Discount_Target":"All",
"Max_Discount_Amount":"null",
"Min_Discount_Amount":"null",
"Customer_Name":"null",
"Uofm":"null",
"Item_Number":"null",
"Uofm_Schedule_ID":"null",
"Uofm_Schedule":"null",
"Inactive":"false",
"entityAspect":
{
"entityTypeName":"PriceLine:#SalesPad.Spo.Api.Model",
"defaultResourceName":"PriceLines",
"entityState":"Added",
"originalValuesMap":
{},
"autoGeneratedKey":
{
"propertyName":"Price_Line_ID",
"autoGeneratedKeyType":"Identity"
}
}
}
],
"saveOptions":
{}
}
The 7 values at the top (with values of "{}") are found when I do a GET or POST action on the website's other pages. I've managed to find where all the values originate from, except the "Price_Line_ID" one, because it appears to change from page to page (and it changes after a price is added).
I know a web page can get the data needed for a POST event either in its own html (when using tags like <form>), and it can get them from other GET and POST events. Is there any other way for a web page to determine a value that will be sent in a POST event?
I'm not very familiar with angularjs, although from what I understand it only creates a bunch of Javascript for the page. Does it offer different ways of determining what values are sent in a POST or GET event?
Edit: I've already tracked all responses from GET and POST events from logging-in to adding a price, the Price_Line_ID field changes from page to page, and adding the price appears to use an ID different to the one received in the GET event. I just want to know the different ways that a web page (specifically one using angularjs might use to determine the value of the data sent in POST events.
Related
I have some problem while getting data from another site. In this case I want to get the reservation data from the booking engine site and they want to pass the data to me with Google Tag Manager. I don't really understand what I should do when they just need GTM code. What I should create in my server to get the data from the booking engine with Google Tag Manager ?
This is the illustrations:
I have two sites called sites1.com and sites2.com. In sites1.com I put the Google Tag Manager scripts to push form submit data like full name, last name, email, etc. After somebody submit the form I want to get the submited data in sites1.com to sites2.com with Google Tag Manager. My problem is how to get the data after somebody submited the form in sites1.com in my sites2.com ?
Please anybody knows how to resolve my problem . Thanks in advance .
Well if they implement your GTM from site2.com into site1.com all you need to do is:
Create a trigger for the submit button on the form (use the ID o class of the element and check for the that the Page URL contains site1.com/)
Create a tag where you want the information to be send
Scrap the fields with javascript or ask them to push to the dataLayer the information you need (in this case you can build the trigger based on this event)
And SUPER important: check all your triggers so no other tag fires on site1.com
2.1:
Im not sure if i get where you want this information to be stored but keep in mind GA does not accept PII. On the other hand if you want this in some DB you can just create and endpoint and send the information as parameters. Example:
site2.com/booking_info?field1={{DL variable}}
And just use a Custom IMG tag.
Ive made something like this using API Gateway, Lambda and DynamoDB and it took me 15 mins to set up. (just to give you perspective)
-- EDIT:
Ones you have the information avaliable you can send it to your database using two methods:
Using a HTML tag and making a request with javascript
Making a request with a custom image tag
On the past i ve just added the URL with parameters where you want the request to be made on an image tag and worked perfectly for me.
-- More info:
The custom image tag requests an image from a particular URL. The interesting part is that by making the request you’re actually transmitting information to a server. For example, if you request an image via URL https://www.example.com/something?parameter1=good¶meter2=123, the receiving server processes parameter1 and parameter2 and acts accordingly (e.g. registers those parameters as an event).
Source.
Currently I'm working on a project where a user enters a lot of data constantly for a hour long window. I'm looking to have one user control all the data via some control panel and then have a link they can distribute to other users that will allow them to view that data without the ability to edit it.
Right now I'm doing some extremely weird methods. I have an XHR request on the control page that fires whenever a field is finished being edited. From there the data is sent to a php file that converts the data into a simple text file. Then the distributed link file will load that file one time and translate it into the necessary format.
Some potential problems I've run into are it seems odd that I'm sending starting as javascript data then going to a php file then to a text file then translating the data all the way back into javascript data again. Another problem I've come into is I'm not sure of a way to force users to reload the page when a field is edited in the control panel after the user has opened the view page.
Have I totally gone overboard here? What are some better concepts I could employ to accomplish this task?
If i understand what you want to do this is how i will do this:
First the data entry
if you have lot of fields you better use a form wizard, i don't have a particular one in mind right now but there is lot of them just search jQuery Form wizard
Here is an example:
http://i.stack.imgur.com/Luk2b.jpg
The concept of the form wizard is to guide user via multiple page and also validate the data. And click save when and the end.
Then save date in database.
Display content
All you need to do is to create a global separate page to display your content.
Let see something like: http://yourserver.com/view/{id}
where id is the identifier of the particular row in your database.
i'm not sure if i totally understand what u about to do. i'm trying to make your work description shorter here:
want to build a website that one person can edit a single page's content in 1 hour, and others can view the content change in that 1 hour.
if this is what u want to build, here's the module:
teacher: the one who can edit the page
student: the one who can only view the page
server: information center
teacher client edits page -> teacher client sends update data to server -> server saves data -> server sends update notice to student client -> student client receives update notice -> student fetches update data from server
to make this module work well, i suggest try socket instead of http reqeust, just like online games or IMs do.
well, try socket.io
I am trying to automate the querying of names on 2 different websites. For each website I have to enter in the same name and take a screenshot of the results.I don't know how to approach this project. Is it possible to automate this with a javascript? Thank you for any input.
Website links are below:
[1] http://exclusions.oig.hhs.gov/search.aspx
[2] http://www.health.ny.gov/professionals/doctors/conduct/license_lookup.htm
You should be able to accomplish this with PhantomJS relatively easily. Check out the examples page - you'll probably want to use code evaluation to fill out and submit your forms, after which you can render the page to an image file.
You actually do this is one step: send a POST requests to the information you need. For the first example, you would send these POST data:
__VIEWSTATE=dDwtNzQ5MzEyMDI3Ozs%2BHWC8LXZfQQTCTJWtmudNLStjn%2Fk%3D&txtLastName1=test&txtFirstName1=&txtBusinessName1=&txtLastName2=&txtFirstName2=&txtBusinessName2=&txtLastName3=&txtFirstName3=&txtBusinessName3=&txtLastName4=&txtFirstName4=&txtBusinessName4=&txtLastName5=&txtFirstName5=&txtBusinessName5=&cmdSubmit=Search
So you can use PhantomJS, as suggested, with a POST request and take a screenshot. Or you can use a web service to create the screenshots. Unfortunately, very few allow POST requests with custom POST data. Browshot (see API) lets you send POST data and use a custom Referer. So your screenshot requests would include (after you encode the POST data):
url=http://exclusions.oig.hhs.gov/search.aspx&referer=http://exclusions.oig.hhs.gov/search.aspx&post_data=__VIEWSTATE%3DdDwtNzQ5MzEyMDI3Ozs%252BHWC8LXZfQQTCTJWtmudNLStjn%252Fk%253D%26txtLastName1%3Dtest%26txtFirstName1%3D%26txtBusinessName1%3D%26txtLastName2%3D%26txtFirstName2%3D%26txtBusinessName2%3D%26txtLastName3%3D%26txtFirstName3%3D%26txtBusinessName3%3D%26txtLastName4%3D%26txtFirstName4%3D%26txtBusinessName4%3D%26txtLastName5%3D%26txtFirstName5%3D%26txtBusinessName5%3D%26cmdSubmit%3DSearch
I have tested it, and do get a screenshot of the results, like if I filled out the form and submitted it.
I am working on a big site, and in the site there is a search module. Searching is done by using a a lot of user submitted values, so in pagination I must pass all these data to the next page, appending the values to url make the url very big.
Sso how can I solve this issue? I am planning to use a javascript based page submission (POST) with all the values in hidden fields to the next page the read all the values from the next page.
Will it cause any problems? Or should I use database to keep the search criterias?
I would create a server side object, possibly with a database backend which is updated by the different pages.
It is at my opinion the most clear and easy solution. Giving parameters from page to page, either by post or javascript or cookie will work too but it's more of a quirk in my experience.
Also if a search query is so complex that it needs multiple pages to create it, it might be helpfull for the user to have all the data stored on the server so he can change it more easily by switching back and forth between the different pages.
I would store all the search criterias in some kind of session-store on the server when the initial search is being triggered.
For pagination I would retrieve the criterias from the session-store and then just show the appropriate results. Also I would append some kind of key to the pagination links (so this would be the only hidden post-field) under which the search criterieas can be found.
Even though the session is per user, you might have several search windows open within the same session, and you don't want to mess them up with the pagination.
In order to make a reliable search with pagination, we need to do a bit more than normal.
We need to handle the following cases.
Once search is done, user may choose to do browser back and forward. Here, if you are doing form submission on every page, it would be an overload. Also, if user presses browser refresh button, it will unnecessarily warn him that data is being submitted.
Searching on a large database with lots of criteria is costly. Hence, optimization is important.
So you should NOT do the following:
Submit data on every page change
Not store data in cookie. (This is not secure and not even reliable.)
For large database with complex query, cache the result in session.
In case, you need very up-to-date and real-time result, ignore point (3) and try doing partial search for every page.
Thus, for your case, you can do the following:
When user searches first time, make the form POST data to a search page.
This search page will store the search query in session and generate a unique id for it.
Now render the result page. The result page will be passed the search id (generated in point 2) and the page number. Example result.aspx?searchId=5372947645&page=2
The result page will puck up the query from session using the searchId and then provide result based on the page number sent.
Using hidden fields and POST method should be fine too unless you are able to get them on the next page right.
To supplement Sarfraz's answer...
It's not necessary to use Javascript to make a POST.
<form action="destination_url" method="POST">
...
</form>
I have been finding ways around this for a long time but think it's time I addressed it.
If I have a page that has a dropdown menu, is there anyway I can select a value which will subsequently load other values further down.
Can this be done without a page reload?
I will give you an example.
Say I was making some tools for an admin panel, but first of all they needed to select a member to work with.
They would select the member and then below, the fields about that member would be populated based on what was selected in the first menu.
As I have already asked, can this be done without a page reload?
Thanks for reading.
Yes it can be done without AJAX. When the page is rendered pass all the collections that will be used by the dropdown lists as JSON objects into the HTML:
var collection = [{ id: 1, name: 'John' }, { id: 2, name: 'Smith' }];
...
Then register for the change event of the first drop down and based on the selected value go and fetch the data from the other collections. Of course if you have lots of data this might not be practical as your pages will become very large and in this case AJAX woulld be more appropriate.
Answer YES it can be done.
Firstly you'll need an event, in this case you need to take action on the onChange event for the selectBox. So when an item changes you run a function.
Now you have 2 choices. You can do this using AJAX or NOT, it really depends on the complexity / security of your application.
In the following I refer to
Users : those using the application
Hidden Client Side Data : Data sent to the client during page load, but not visible to all users, however using view source, or downloading JS files, the Data is not secured.
Method 1 - NO AJAX
Basics: You send all the possible display options down initially when the page is first loaded, but display only the sections relevant to the user during selectbox onchange events.
Recommended when: No security condiderations if hidden client side data is detected (or won't be detected, or you simply trust your audience to use the app in the intended manner). Lastly when your total view permutations are low.
Method 2 - AJAX
Basics: You send down initially only the page skeleton, when the user changes the value of the select box, you then do an AJAX request to the server - grab the new view info thats relevant to that user, send it back down to a script which will inject that user data into the DOM.
Recommended when: You have a public site, or a site where security is a consideration. Where you have lots of view permutations or want more customizations per user than in scenario 1.
As you see both methods do not require a repost - method 1 ships everything upfront, method 2 uses AJAX to fill in data when required. Both methods are valid depending on your requirement.
Yes. Ajax is mainly used for that i.e. (without a page reload)
You have to use following step to achieve
Create a link and call a JavaScript function on it's onchange function
In the JavaScript function you have to call Ajax request.
Update the div in your ajax response.