We Keep Coding

Capture the state of a web page in a URL

I find myself having to interact with a web page that hides state in various places so that one cannot easily share it as a URL, for example this page which allows users to look up information from city zoning applications:
https://aca.cityofberkeley.info/community/Default.aspx
You can interact with the page all you want, but the URL in the location bar will remain the same as the above.
Currently, city staff provide users with instructions like "Load this URL, click on the 'Zoning' tab, enter DRCP2020-0010 under the 'Permit Number' field, click 'Search', then when the records come up, click 'Record Info' and then select 'Attachments' from the dropdown menu, then click on the PDF document that says '2020-10-21_DRCP_APP_PCKT_2801 Adeline.pdf'". I would like to be able to replace these instructions with a URL.
Another example is the website where video from city council meetings is archived:
http://berkeley.granicus.com/MediaPlayer.php?publish_id=cbebb4e6-5b83-11eb-920e-0050569183fa
It would be nice to be able to produce a link which brings up one of the meeting videos, and seeks to a certain timestamp like 53:40, so that I can refer to something specific that was said at a meeting.
Looking at the pages that are loaded when I follow the instructions in each case, I can see that there are some POST forms, cookies, hidden input fields, and so on.
Is there some kind of tool that I can use to create "deep links" to pages like these, that were generated using non-URL hidden state, which will allow me to quickly share what I'm looking at with another user?
What I'm seeking is similar to the frmget "bookmarklet", which changes the forms on a page to use GET instead of POST. Sometimes this succeeds in producing a URL which captures form submission query parameters. However, it doesn't work for these applications, for whatever reason.
This question is possibly related to the idea of capturing a web page's DOM state using "browser screenshots" and a script called html2canvas. A possible solution might involve getting and setting cookies in a bookmarklet. Ideally something that produces a normal "https://" URL would be ideal, but if it is impossible to solve the problem except by outputting a "javascript:" URL (bookmarklet) then that is acceptable to me (in spite of the security implications). Thanks.

That seems like not a programming matter. It seems like the site has some security issues as well.
QUESTION A: About Zoning
Here are some links you can use
Direct link to Zoning (I've found it via Advanced search from the site):
https://aca.cityofberkeley.info/CitizenAccess/Cap/CapHome.aspx?module=Planning&TabName=Planning&TabList=Home%7C0%7CBuilding%7C1%7CHousing%7C2%7CPlanning%7C3%7CFire%7C4%7CLicenses%7C5%7CPublicWorks%7C6%7CCurrentTabIndex%7C3
A strange link to the list of files (I've found it via downloading a file, then going to chrome://downloads, then right-clicking the file I've download. The link has been the following):
https://aca.cityofberkeley.info/CitizenAccess/FileUpload/AttachmentsList.aspx?iframeid=ctl00_PlaceHolderMain_attachmentEdit&module=Planning&isInConfirm=False&isdetail=True&isaccountmanager=False&isAdmin=True&isPeopleDocument=&agencyCode=BERKELEY&isForConditionDocument=N
It still doesn't give the direct link to the file, but it it gives the list of attachement of the previously opened Zoning record.
Currently I have no idea what file is triggered by javascipt:__doPostBack('attachmentList$gdvAttachmentList$ctl02$lnkFileName','').
In any case, based on what we have, step one, and then step two seems like minimize the path to download the file. I guess there could be a way to download the file directly, but I currently don't see any easy way. Maybe someone else could figure it out.
QUESTION B: About video
I've used an embed link that shows all the attributes that can be used.
There is a pretty strange but working way to give the exact timestamp. Change starttime from the link below:
https://berkeley.granicus.com/MediaPlayer.php?publish_id=cbebb4e6-5b83-11eb-920e-0050569183fa&starttime=0&stoptime=undefined&autostart=1
So replacing 0 for 3600 will rewind the video forward by one hour (3600 seconds):
https://berkeley.granicus.com/MediaPlayer.php?publish_id=cbebb4e6-5b83-11eb-920e-0050569183fa&starttime=3600&stoptime=undefined&autostart=1
The problem here is that ... you cannot rewind back manually that particular hour (it just gets kind cropped out). But it works to show the exact episode.
That's a pretty strange site.

Appending only numbers to a friendly URL after selecting a number

My website basically consists of photo galleries that people can browse. On the main page, people can choose to access the gallery of images then in the gallery, they can select which photo they want to see.
This is an image of the possible different ways to give users picture choices. On my site now, I offer the left-most options because of compatibility for all browsers.
The option format I want on my site is defined under "Desired Options". When a user selects a boxed arrow, the number (shown as ## here) will automatically increment or decrement. then when GO is clicked, the URL is then http://example.com/picturenumber/##. So if users use the up and down arrow buttons to select number 10 and GO is clicked, then http://example.com/picturenumber/10 is the resulting URL.
Here's the issue
To achieve the above, I feel I need to use javascript. I believe at least one person in this whole world that wants to use my site uses a device or browser with no javascript support. This forces me to use the option format on the far right, a basic text box for the picture number to be typed in and a GO button.
My partial answer to my question is this:
Use this HTML:
Enter #:
and this PHP named imagepicker.php:
While this does work, the problem is that an extra request is required to fully process the user's request.
I feel my only other options to solve my problem are either:
Not use friendly URLs and make them compatible so that when the form is submitted, the correct URL is loaded without the need for an extra request. For example, make this URL the official image URL displayed in the address bar: http://example.com/imagepicker.php?number=##
OR
List every single image number on the main page which is rather redundant because the list is in the gallery page.
I even thought of using a combo box in place of a text box and that wouldn't help either.
Is there some way I can do this so that users with no javascript support can enter an image number and then with a click of a button, be taken to that image without requiring the resulting image page to be accessed at a not-so-friendly URL and without listing all image numbers at once? If so, what would you suggest?

So, basically it boils down to a HTML problem in which you need to increment some numbers pressing buttons that alter the format of the URL in order to be friendly, and after that, you have to produce a postback. I think you need javascript for it.
Without javascript, I think the best solution is to sacrifice friendly URLs and create a form with method GET, so the choice that the user writes will be sent to the server as part of the query string. But only for those users that don't have javascript on their browsers. You can use javascript for the rest, but if you do that, it would be better to keep the "unfriendly" URL format for consistency and maintainability.

Uncovering a hidden link [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm a complete layman and despite my utmost attempts (involving visiting about 100 different pages), I wasn't able to solve my problem. I'd be really grateful if anyone would be willing to help me out.
Say there is an elaborate website using Javascript, where you register, have your own profile etc. It asks you to register for one of a few time slots. You do it by clicking a hyperlink, which has this annoying feature (discussed elsewhere on this website, see here: how can url be hidden in hyperlink when mouse hover ) that it hides the link. Is there a way to uncover it?
By uncover I mean a way in which I would be able to register for a given time slot without clicking the hyperlink, but rather via copying that hidden link (or typing it in) to the address bar of the browser. I have tried using Firebug and Tamper Data, but to no success. Both programmes provided me with viable candidates for links, but when I copied them to address bar and pressed enter, it didn't result in a registration for a given time slot.
Finally, Tamper Data tells me that 'Content-Type=application/x-www-form-urlencoded', whereas Firebug indicates that 'enctype="multipart/form-data" right above the table with hyperlinks, so I begin to think there is some sort of encryption in place which is meant to prevent me from getting what I want. And I'm sure owners of this website would want to prevent me, but are they able to do it?
So all in all, is it even doable? Apologies if this all sounds too broad, I'll be more than happy to make it (or at least try to make it) more clear.
Thanks a lot and apologies, but I really don't know what I'm doing here.

As Yang Li said: it depends.
If a website is using a lot of javascript to perform on screen changes and network requests, the likelihood of you being able to just use a URL (even if it is the url used by a javascript script) to perform an action is low.
Most websites these days that rely on javascript would have data elements hidden to the user which may affect the way an action is relayed to the server. Think hidden form elements.
And on the server side, who is to say that the order you are trying to use that URL in is correct? You would have to run through the code piece by piece, along with observing the network requests to be certain.
So in answer to your question, it MAY be doable. The more important questions is: is it advisable? I would suggest not tampering too much with the way the website works on your browser, as it was intended to be used in a certian way. You can't be sure of all of the side effects of using it in an unintended fashion.

Many share buttons for social networks on one page

I am currently doing some work on a research database where they have decided that they want to be able to share links to articles from the site on social networks (Facebook, Twitter, LinkedIn and Google+).
Preferably this should be done through the share buttons provided by the respective networks. I quickly got the buttons working and displayed correctly on the site by following the implementation instructions from each network.
My problem is a consequence of that the site offers the possiblity to show 1000 (1K) post on a single search result page. This means that when such a page is created it needs to create 1000 share buttons for each social network (effectively 4000).
Sadly this seems to overwhelm the browser as it offers to stop the javascript provided by the social networks and whether you choose to stop it or not - the page ends up in deadlock waiting for a response from the social networks and never finishes the page loading process.
I have an idea that the problem may be that the large number of asynchronous requests means that the browser somehow misses some of the responses and thus ends up waiting forever for a response that will never come.
As mentioned it is only a problem with such a large number of posts, if a page for example displays 100 posts (effectively 400 share buttons) it works perfectly.
While it could be argued that 1000 post on a single page is overkill, limiting the maximum number of displayed post is sadly not an option.
My question therefore is whether any of you know of a way to solve this kind of problem or if my only real option is to create custom share buttons that doesn't need to be created through the javascript provided by the social networks ?
The following references leads to the documentation for each of the share buttons.
Twitter
Facebook
LinkedIn
Google+

For all these buttons, there is a main js file which does the heavy work.
So, for LinkedIn, add the script tag:
<script src="//platform.linkedin.com/in.js" type="text/javascript"></script>
once in the page. And use the below script as a placeholder for your linkedin button whereever you need it. (don't forget to replace the data-url attribute in below script)
<script type="IN/Share" data-url="http://developer.linkedin.com/plugins/share-plugin-generator" data-counter="top"></script>
For Twitter similarly, the below script tag needs to be added once in the page as it's job is to get the main js file and add it to the page.
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>
The below script needs to be added multiple times whereever you want. Replace the data-url attribute with your url which needs to be tweeted when you click on it.
Tweet
When you get the code for FB or Google Plus like, you will get a script which needs to be added once and then the code to be added where ever you need.
EDIT:
Based on your comment below: The scripts will surely cause issue because they need to convert each and every placeholder into a good looking 'like' button. Below are few ways to improve the performance:
run these scripts only on page load (i.e., add the main scripts at load time)
using setTimeout or setInterval, work on every 100 placeholders at a time (requires change in main scripts)
Lazy load the init of like buttons. When the user scrolls the page and the like buttons will show up in the page, then initialize the buttons (requires change in main scripts)
Recommended Approach: Keep just one set of like buttons. When user hovers over a search result, then add this set of buttons to that div and change the attributes related to url in the buttons. With this way, only one set of buttons will be shown and won't take time at all to init them.

Unobtrisuvely ask user for details

I am trying to figure out the best way to acompish "unobtrusive" forms for a user (within a web app).
The purpose: keep user on the site by not asking to fill unnecessary form in. Ask for the details as only when such are needed.
The requrements are:
User should provide additional details only when it is required (email to receive notifications, login required for account page, save credit card details when checking out).
User should not leave the current page providing the additional details.
The implementation would be fairly easy if all requests would be AJAX ones. It would be easy to analyse the response (401 or so) and show the appropriate lightbox-form.
I do not see how it can be done "the right way" with plain anchors and form submits as in both cases the user actually leaves the page (by following the link or submitting a form) and there is no way to analyse the response on the client side.
Converting all links and forms to AJAX ones would be just silly.
The closest analog to what I want to achieve is the default Basic Authentication dialog in most of the browser. But obviously that just doesn't fit my requirements.
Any creative suggestions how to do that for non-AJAX requests?
Regards,
Dmytrii.

In a page sense, where "page" refers to what the user sees and not what the URL is, I only can think of following ways to update independent parts in a page with JavaScript (and thus Ajax) switched off:
Frames
Iframes
Using held-open connections there are two more ways to update a page, however these do not work reliably in all cases:
Animated GIF
CSS DIV tags with absolute positioning.
Note that this needs that your Server can keep open a session for each person looking at the page, which can be thousands. If this does not work the only possible workaround is with FRAMEs and automatic refresh, which is somewhat clumsy.
As I think that you do not want to use Frames and you do not want to render animated GIFs, I explain the CSS DIV way:
When you load the page you do not finish loading it. Instead the connection is kept open by the web server and the script handling the connection waits for additional information to arrive. When there is additional data, this is sent to the browser by encapsulating it into additional DIV tags which can overwrite other parts of the page.
Using "style" in the DIV tag and CSS position:absolute these can overwrite other information on the page like a new layer. However you need either position:absolute or must add this data to the end of the page.
How does this work with forms?
Forms usually have a known size so you can put them into IFRAMEs. These IFRAMEs get submitted to the webserver. The script there notifies the other script that new data must be output, so the waiting script renders the response and displays it in the page while the script which took the submit redisplays the form with fresh values only.
How does this work with 404 and anchors?
I don't really know because this must be tested, but here is a hint how I would try to implement this:
We have 2 issues here.
First the URL must not point to other pages but back to a server script again, so the href is under control. This script then notifies the waiting script to update the page accordingly, for example by retrieving the page and sending it to your browser. The script can check for 404 as well.
Second you must hinder the browser to switch the page when clicking on the anchor. This probably involves some clever tricks using CSS, target and server side status codes (like "gone" or redirect to the current page, whatever) to keep the browser from switching the page. I am not completely sure if that works, but if you remember download pages, these show URLs which do not switch the page but have an effect (downloading the file). That's where to start to try to hack browsers not leaving the current page without using JavaScript.
One idea not followed here is not keeping the connection of the page open but the CSS file and send new css information to the browser which then "fills in empty stubs" using the CSS way. But I doubt that this works very well, most browsers probably will parse the CSS only after loading finished, but perhaps I am wrong.
Also note that keeping a connection open never finishes the page loading, so you will see the busy-logo spinning all the time, which is unavoidable with this technique.
Having said this all I doubt you get around JavaScript.
What I wrote here is very difficult to do and therefor usually is not used because it scales badly. And it is a lot more difficult than using JavaScript alone (that's why I explained it).
With proper AJAX it is much more easy to reach your goal. Also note that you do not need to change your page source much, all you need is to add a script which augments the page content such, that for example forms suddenly use AJAX instead of a direct POST with re-rendering the page. Things which cannot be detected easily then need some hints in the tags such that the tag scanner knows how to handle the tag. The good thing then is, that with JavaScript switched off your page still works - however it then "leaves the page".
Normal HTML just was not designed to create application-like web pages like we want to see today. This all was added using JavaScript.
About popup forms
The Basic-Auth-Handler reloads the page after the user enters something into this dialog, only if cancel is hit the current page is displayed.
But there are two ways to present additional query-popups in a page using JavaScript:
The first one is the javascript "prompt", like in following example:
http://de.selfhtml.org/javascript/objekte/anzeige/window_prompt_vor.htm
(Click on the "Hier").
The second one is "JavaScript forms" which are like popups within an HTML-page.
However I consider popups to be far too intrusive and bad design.
Ajax and JavaScript is the easiest way
Unfortunately using JavaScript is never easy, but if you think JavaScript is improper or too difficult, there is no other technique which is easier, that's why JavaScript is used everywhere.
For example your page onload-Script can cycle through all Anchor-Tags and modify them such, that clicking on them invokes a function. This function then must do something clever.
Same is true for Forms. Fields which can be modified (like the user's eMail address) then have two views, on is visible, the other one hidden. The hidden one is a form. Clicking on the eMail address then switches the view (disables the first div and enables the second), such that suddenly instead of the eMail address a text form field is there containing the eMail address. If you click on the "OK" button the button changes the look into a spinner until the data is submitted, then the view switches back to the normal one.
That's the usual way to do it using JavaScript and Ajax. And this involves a lot of programming until it works well.
Sorry for not shortening this post and missing code snippets, I am currently lacking time ;)

Hidden iframe.
Set target attribute of the form to the name of the iframe. use the onload event of the iframe to determine what is the response.
Or, if you really dont like any javascript, don't hide the iframe and instead present it in a creative manner.
CSS to hide an element
#myiframe { position:absolute; left: -999em; display: none; visibility: hidden; }
But normally, display: none is enough. This is just an overkill.