We Keep Coding

Delphi TWebBrowser JavaScript Errors and Cookies

I have been stuck on this for a couple of weeks now and this is a follow on from SO question Delphi REST Debugger Returns Error 429 Too Many Requests but Browser Returns JSON as Expected
I was wanting to get the content of a url response using the TNetHTTPRequest and TNetHTTPClient components. I was continually getting 429 errors “too many requests”. When using Firefox Inspect Element to look at network and storage, I discovered that I needed to receive cookies and then send those cookies with my request. Unfortunately, one of the cookies essential to the website content seems to be dependent (I think) on the execution of javascript. I went back to first principles and dropped a TWebbrowser on a form (VCL) and sure enough browser shows a javascript error “Expected Identifier”.
When I use the TWebbrowser in FMX it does not throw an error it just does not return the website contents at all and remains blank. I need FMX as I will be in a cross platform mobile environment.
The URL is https://shop.coles.com.au/a/national/home
I use Delphi Community Edition 10.3.3 Rio.
The URL returns perfectly in commercial browsers Firefox, Safari, Chrome and even CEF4Delphi. Unfortunately, I can’t use CEF as I need cross platform.
I would like to know how to get the website content returned to the browser (or even better NetHTTPClient) without script errors and how to access the browsers current cookies.
Any help will be most appreciated.
Thanks,
John.

URL returns perfectly in commercial browsers ... without script errors and how to access the browsers current cookies
If you'd inspect the network traffic (F12 > Network, then requesting your URL) or use uMatrix (to block everything that doesn't belong to the domain by default) you'd see the JS does at least one XHR to amazonaws.com. Your HTTP transfer alone (as done by TNetHTTP*) works fine and you get the same resource that each internet browser gets.
However, you don't operate with what you got (in contrast to the internet browser, which also automatically parses the HTML, sees JS resources, and executes them). TWebbrowser does not what you take for granted most likely due to security settings (try to get an error console in there, preferably F12 again). You need to do the same: parse the HTML resource for JS URIs, requesting those and executing what you get, while still providing the same cookie environment.
For executing JS you could use Chakra or mORMot or BESEN. It's challenging at first, but the more you understand about HTTP (including cookies) and a JS engine, the more you'll see why "things work" in one situation and not in another. There's a reason why an internet browser is a very complex software and not just a downloader.
As per this forcing IE11 Quirks mode might cure your problem already when using TWebBrowser:
TBrowserEmulationAdjuster.SetBrowserEmulationDWORD(TBrowserEmulationAdjuster.IE11_Quirks);

Find HTTP Request Initiator Line In Internet Explorer Developer Tools

I'm getting some odd behavior when my page is loaded in Internet Explorer 11. When my page loads in IE 11 and I watch the Network Tab of the Developer Tools I see a HTTP Get request to the base URL of the page. By base URL I mean the URL without any parameters. This is causing a 500 Internal Server Error to get fired because the Action on the other side of the Request doesn't accept no Parameters. I use Elmah to send me a mail for any unhandled exceptions so whenever a user is using IE I get a ton of Error notifications. This is only happening in IE. Both Firefox and Chrome work as expected. Also, this doesn't seem to break the page in anyway. I only know about this error because of the Emails I'm getting. No User has reported an issue when using IE 11.
Short of searching through every line of Javascript code both 3rd party and my own is there a way in Internet Explorer Developer Tools or even another 3rd party tool that will tell me the exact line (or even js file) that initiated the Http Get Request similar to Chrome's Development Tools?
Here are some images showing the call I'm talking about. The blacked out portion is my API Key.
The original Url being called in the below examples is http://localhost:44301/Grower/Profile/28
The first image is of the Internet Explorer 11 Development Tools with the offending Http Get Request highlighted.
Next is a Fiddler Screenshot of the same request highlighted.
And finally the Chrome Developer Tools showing no such call happening.

I think there's no such a way in IE dev tools to find the initiator line of HTTP request.
But in my research, I find that we can get the file path of the HTTP request initialed using Fiddler. We can find the file path in Inspectors -> Raw -> Referer:
The file path in Referer in Fiddler is exactly the path in Initiator in Network tab in Chrome. So you could try to check the file path using Fiddler.

What could be stopping AJAX from returning?

I'm trying to troubleshoot a problem on a client's machine for our website. We're using an ajax call to get information on a page to select additional parameters. Our callback function has a block of code for reading when the ajax coming back is an error or is correct. For every other computer that we've tested this with, the ajax comes back. However, for a particular client, we're seeing the ajax come back with the error message, meaning the response never got there successfully or that it's corrupted or broken.
Does anyone know how this would happen? The client is using IE 8 and I've tested IE 8, IE 9, IE 10, and Chrome and all of those work on my computers.
EDIT: As of now, we don't have access to the system and the network that would be causing the error. They are trying to see if they can accept everything from our domain and see if that fixes it, but right now, I can't put Fiddler on their computer.

I've seen any amount of random behaviour caused by virus scanners and so-called network security products. Try adding an exception for your site to any security software your client is running.
The other thing to do is to use Wireshark, Fiddler, etc. to see what's actually happening at the network level.

Developer Tools: Follow network requests across popups

We are trying to figure out how something works on the web (for web scraping/automation) and one of the web pages we are working on issues a popup to do some of the work. One of our most commonly used debug tools is the Chrome network tab in Developer Tools, hit "record" do some work, and then examine what was done and then replicate the work done "offline".
However the Developer Tools (in Chrome, Safari and Firefox - all work the same) do not follow requests across a popup, even if you hit "record".
Is there some configuration value I'm missing, or some way to record all network events? We can't use tcpdump/wireshark for this because it's all done over SSL. One option we've considered is a man-in-the-middle https proxy, but I can't find anything pre-written so we'd have to create one ourselves.

I don't know of any way to follow the requests across pop-ups, as each window has its own Web Inspector, however you can use Fiddler to inspect HTTPS requests. It will MITM, and subsequently throw a certificate error, which should allow you to inspect all requests in the order that they happened.

You can use Charles Web Debugging Proxy, which is an app that lets you see all the traffic and even replace some responses with your own. Of course that may break HTTPS so you have to accept the certificate errors, but that's usually a minor problem. It works on Win, Mac and even Linux.

The object inspector cannot inspect what isn't in the current page. Therefore, you will need to open the inspector inside the popup url with same parameters in order to see what it does.
As a tool, you can use a web sniffer to see exactly which url were called during the process.

Cross-domain SSL handshake failure in Firefox using xhr, client-certificate

The setup is as follows:
Firefox (both 3.x and 4b) with properly set up and working certificates, including a client certificate.
Web page with an XMLHttpRequest() type of AJAX call to a different subdomain.
Custom web server in said subdomain accepting requests, reponding with a permissive Access-Control-Allow-Origin header and requiring client verification.
The problem is that Firefox aborts the request (well, that's what it says in firebug anyway) abruptly. Running the setup with openssl s_server instead hints that Firefox actually doesn't even send the client certificate:
140727260153512:error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer
did not return a certificate:s3_srvr.c:2965:ACCEPT
The same exact setup works perfectly with Chrome, suggesting perhaps a bug in Firefox. However, performing the ajax call with a <script> element injected into the DOM seems to work as intended...
So, has anyone else run into this? Is it a bug? Any workarounds? Is there something obvious missing?

Chiming in 5 years later probably isn't much help to the OP, but in case someone else has this issue in the future...
Firefox appears to not send the client certificate with a cross-origin XHR request by default. Setting withCredentials=true on the XHR instance resolved the issue for me. Note that I also did not see this problem with Chrome--only Firefox.
For more info see this Mozilla Dev Network blog post. In particular, the following statement:
By default, in cross-site XMLHttpRequest invocations, browsers will
not send credentials. A specific flag has to be set on the XMLHttpRequest object when it is invoked.

The reason injecting the script works as opposed to a simple XHR request is because of the Single Origin Policy. This would probably explain why Chrome allows the XHR but not FF; Chrome considers the subdomain part of the same origin, but FF does not.
Injecting scripts from other domains (which is what Google Analytics does) is allowed and one of the practices to handle this situation.
The way my team handles this situation is by making a request through a server-side proxy.
I would recommend using a server-side proxy if you can, but the script injection method works fine as long as the code is coming from a trusted source.
I also found this article which describes your situation.