We Keep Coding

Delphi TWebBrowser JavaScript Errors and Cookies

I have been stuck on this for a couple of weeks now and this is a follow on from SO question Delphi REST Debugger Returns Error 429 Too Many Requests but Browser Returns JSON as Expected
I was wanting to get the content of a url response using the TNetHTTPRequest and TNetHTTPClient components. I was continually getting 429 errors “too many requests”. When using Firefox Inspect Element to look at network and storage, I discovered that I needed to receive cookies and then send those cookies with my request. Unfortunately, one of the cookies essential to the website content seems to be dependent (I think) on the execution of javascript. I went back to first principles and dropped a TWebbrowser on a form (VCL) and sure enough browser shows a javascript error “Expected Identifier”.
When I use the TWebbrowser in FMX it does not throw an error it just does not return the website contents at all and remains blank. I need FMX as I will be in a cross platform mobile environment.
The URL is https://shop.coles.com.au/a/national/home
I use Delphi Community Edition 10.3.3 Rio.
The URL returns perfectly in commercial browsers Firefox, Safari, Chrome and even CEF4Delphi. Unfortunately, I can’t use CEF as I need cross platform.
I would like to know how to get the website content returned to the browser (or even better NetHTTPClient) without script errors and how to access the browsers current cookies.
Any help will be most appreciated.
Thanks,
John.

URL returns perfectly in commercial browsers ... without script errors and how to access the browsers current cookies
If you'd inspect the network traffic (F12 > Network, then requesting your URL) or use uMatrix (to block everything that doesn't belong to the domain by default) you'd see the JS does at least one XHR to amazonaws.com. Your HTTP transfer alone (as done by TNetHTTP*) works fine and you get the same resource that each internet browser gets.
However, you don't operate with what you got (in contrast to the internet browser, which also automatically parses the HTML, sees JS resources, and executes them). TWebbrowser does not what you take for granted most likely due to security settings (try to get an error console in there, preferably F12 again). You need to do the same: parse the HTML resource for JS URIs, requesting those and executing what you get, while still providing the same cookie environment.
For executing JS you could use Chakra or mORMot or BESEN. It's challenging at first, but the more you understand about HTTP (including cookies) and a JS engine, the more you'll see why "things work" in one situation and not in another. There's a reason why an internet browser is a very complex software and not just a downloader.
As per this forcing IE11 Quirks mode might cure your problem already when using TWebBrowser:
TBrowserEmulationAdjuster.SetBrowserEmulationDWORD(TBrowserEmulationAdjuster.IE11_Quirks);

Safari localStorage not persisted between sessions

We have a web app that runs in Facebook (i.e. a running in an iFrame at a different domain). If a Safari user has Cookies and Website Data set to the default, "Allow from websites I visit", the data we store via localStorage.setItem is acting like sessionStorage, i.e. it's not available beyond the user's current session (i.e. after the user closes the tab). If we change the setting to "Always allow", it works fine just like in Chrome, IE , etc.
As a test, we've tried navigating the browser to our app's domain (https://ourappname.appspot.com) directly and it works fine there. And also then it should truly be a visited website, but when going back to the game within Facebook, the problem still exists.
Note that the setItem call is succeeding, it's just that getItem doesn't return anything in a subsequent session. (So it's not like when the user is Private Browsing and the setItem call itself fails with a Quota Exceeded error.)
What do we need to do to support Safari so that our app, running within Facebook, can use localStorage as intended where the data will survive between sessions?

It's either a bug in Safari or a security feature.
You are visiting FaceBook and not your website. Your app is in iframe which would violate security model if it let you read any data from the browser. Think if a competitor site read data it did/didn't set. That'd constitute an information leak.
Safari is doing it's job well in that regard.
Ideally, in "Allow from websites I visit" mode, no browser should let iframes to set data to localStorage; even if every domain has their own storage-sandboxes.
What's troubling me is why are they even letting you write to localStorage from iframe at all (in your 'Allow for only sites I visit' mode)? That might actually be a bug - a information spoof attack enabling bug.
I think it's because security-exceptions were dropped from localStorage in case of not-same-party origin of request. So Safari might actually will not throw error but let it silently fail (in some cases). That's probably why your setItem call is succeeding.
At this point, with the given information, I suspect, sir you are out of luck due to Safari programmers following standard to the letter.

I'm still waiting on a reply from Apple, but it's safe to say we're stuck with this behavior. So Anubhav's answer is accurate, but we still needed a solution.
So as a work around, we created new endpoints on our server for persisting/restoring game state. We only utilize this for Safari, for all other browsers we're still persisting our game state in localStorage.
There is a slight performance penalty for the user. And a slight server cost. Not a sexy solution, but now our Facebook canvas app supports Safari.

Internet Explorer not writing to document.cookie?

Ran into something lovely, and intermittent, while trying to write cookies today in an Iframe via javascript.
So say I assign a new cookie:
document.cookie = "key=value;";
Calling said cookie returns as:
document.cookie = "";
I've tried this across both IE 9 and IE10, and it's working for some users, and not others. Same browser versions.
Furthermore, it seems to be machine specific, we're in an enterprise network, and users can log into one machine, and be okay, whilest another station and we'll see this behavior.
I've thought about some group policy, but that last point has me at something of a loss.
Edit: More extra details.
Cookies are allowed on these machines.
This isn't a Session/Persistent Cookie mix up; literally trying to write to the document.cookie object returns "", even if you call it right after assignment.
Here's a big one I missed, I'm trying to assign these as an Iframe, and there doesn't seem to be an issue with P3P headers, as there are machines that are viewing it just fine; same browser versions.

I was experiencing the same problem and found that the users with IE that could not take the cookie had protected mode enabled.
Internet Options/Security/Enable Protected Mode (uncheck)

WebSocket on IE10 giving a SecurityError

I am currently developing a website under IE10 (on Windows 8), using WebSockets in JavaScript. It runs fine under Firefox 18 and Chrome 25, but on IE10 I get a SecurityError when I establish the connection.
What I am doing seems pretty straghtforward :
websocket = new WebSocket('wss://hello.dev.mydomain.net');
But IE doesn't like it :
SCRIPT5022: SecurityError
The script is on "https://test.dev.mydomain.net" (not the real address obviously).
What bothers me is that if I just double-click the file on my local computer (e.g. file://...) it just works. Even worse: if I use fiddler to monitor HTTP traffic... it also works. Whereas there seems to be no connection at all without fiddler, as detailed in the API's specs. (See below.)
Judging by websocket spec, the exception should also appear on Chrome/Firefox... but it does not. So I doubt it has anything related to HTTP/HTTPS. In any case, I am using a wsS socket on a httpS page... Moreover: when I replace the wss address by another valid server found on an online example, it works.
I don't know if this is relevant, but the IP from test.dev.mydomain.net is 10.14.x.x where hello.dev.mydomain.net is 194.247.x.x. I don't know if it could trigger some kind of security on IE only...
One more thing: I have a certificate for *.dev.mydomain.net, IE does not seems to have problems with it. The script originally resides on a server called my.name.dev.mydomain.net, but since I am accessing it from another URL (I got a redirect since we first thought it could have been some kind of Same Origin Policy issue), I don't see how it could matter. At least I hope it does not...
Any idea is welcomed.
EDIT: adding the sites to the trusted zone does not work either.

It looks like IE throws a SecurityError if you're trying to open a websocket on a local (intranet) domain. To overcome this, you may disable IE's automatic algorithm for recognizing local sites. This can be done in Tools > Internet Options > Security > Local Intranet > Sites.
Uncheck all checkboxes (or only a particular one, if you know how exactly your domain did end up in intranet ones).
Note that IE uses (among other things) its proxy settings to determine local sites: if your domain is listed as excluded from proxying in proxy settings, then it will probably be treated as intranet one. This is why WebSockets work if you enable Fiddler: it modifies IE proxy settings and thus the list of intranet sites changes.

I had this problem in Windows7/IE11 after applying a security patch. For Windows10/Edge is the same story.
As this is a local websocket (ws://localhost) you have to add ws:\\localhost\ to Internet Explorer configurations (Tools > Internet Options > Security > Local Intranet > Sites > Advanced).
In Windows 10/Microsoft Edge you will find this configuration in Control Panel > Internet Options.
UPDATE
The address of your webapp (https://test.dev.mydomain.net) must be added to the local intranet zone too. Note that in the image the webapp address should be added.

Well, my question wasn't that successful, so I'll post the "workaround" I found.
I got another address for the website, in 194.247.. too. This, magically, solved it. Guess IE doesn't like mixing local and external stuff and watches the IP.
Anyways, I hope this may come in handy to anyone who's got the same issue.
If you have a solution to solve the "real" issue by configuring IE, let me know :)
Cheers,

Browsers has a websocket limitation. For example Internet Explorer has default limit of websocket connections set to 6 per host header name. the same limitation is set for WinForms WebBrowser component.
The solution is to add values under key Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_WEBSOCKET_MAXCONNECTIONSPERSERVER in registry. Just add DWORD value with executable name , for example iexplore.exe (or your application executable name if you use Web browser component) and set value from range 2..128
Second option how to solve SecurityException is to create multiple subdomains.

The client hostname/IP Address should be same as server IP/Hostname thats listening to otherwise you would get the above error.
1) Make sure whether server hostname configured to listen at IP/localhost etc andif not explicitly specify the hostname ast server
2) use the same hostname in the client. THis will solve the issue. It worked for me...

I encountered the error (although it did not say the SCRIPT5022 part, rather it just reports "ScriptError"). I got around the issue by clicking on "Trusted Sites" and then adding the machine hosting the remote websocket. Note, to add to trusted sites,
I had to supply the address without the "ws://" part (like just mymahcine.mydomain.com)
I had to uncheck the box that says "Require server verification https:// " option.
After I was done adding the domain, I re-checked the box "Require server verification (https://). I would recommend everyone to do the same. Unchecking the box is only a workaround to add sites that don't begin with https (rather ws:// in my case)

I had the same issue at one of my customer's environment.
It turned out that they had a proxy configuration that did not allow the connection to the WebSocket endpoint directly and did not support the WebSocket protocol.
The temporary solution was to disable using the proxy and everything started working. The long term solution is to edit the proxy's configuration (.pac file) to exclude the address of the WebSocket endpoint.
To disable the proxy, go to: Internet Explorer Options > Connections tab > LAN settings button > un-check Automatically detect settings.
Hope this helps someone.

In addition to making sure that the internet zone is not localhost (as in above answers), ensure that if https is used, then wss should be used.
This is not an issue in other browsers, but IE is abit more finicky.

Replace remote JavaScript file with a local debugging copy using Greasemonkey or userscript

While debugging a client app that uses a Google backend, I have added some debugging versions of the functions and inserted them using the Chrome Developer Tools script editor.
However there are a number of limitations with this approach, first is that the editor doesn't seem to always work with de-minified files, and when the JS file is 35K lines long, this is a problem.
Another issue is that all the initialization that is done during load time, uses the original "unpatched" functions, hence this is not ideal.
I would like to replace the remote javascript.js file with my own local copy, presumably using some regex on the file name, or whatever strategy was suitable, I am happy to use either Firefox or Chrome, if one was easier than the other.

So basically, as #BrockAdams identified, there are a couple of solutions to these types of problem depending on the requirements, and they follow either 1 of 2 methods.
the browser API switcharoo.
The proxy based interception befiddlement.
the browser API switcharoo.
Both firefox and chrome support browser extensions that can take advantage of platform specific APIs to register event handlers for "onbeforeload" or "onBeforeRequest" in the case of firefox and chrome respectively. The chrome APIs are currently experimental, hence these tools are likely to be better developed under firefox.
2 tools that definitely do something like what is required are AdBlock plus and Jsdeminifier both of which have the source code available.
The key point for these 2 firefox apps is that they intercept the web request before the browser gets its hands on it and operate on the other side of the http/https encrpytion stage, hence can see the decrypted response, however as identified in the other post that they don't do the whole thing, although the jsdeminifier was very useful, I didn't find a firefox plugin to do exactly what I wanted, but I can see from those previous plugins, that it is possible with both firefox and chrome. Though they don't actually do the trick as required.
The proxy based interception befiddlement This is definitely the better option in a plain HTTP environment, there are whole bunch of proxies such as pivoxy, fiddler2, Charles Web HTTP proxy, and presumably some that I didn't look at specifically such as snort that support filtering of some sort.
The simplest solution for myself was foxyproxy and privoxy on firefox, and configure a user.action and user.filter to detect the url of the page, and then to apply a filter which swapped out the original src tag, for my own one.
The https case. proxy vs plugin
When the request is https the proxy can't see the request url or the response body, so it can't do the cool swapping stuff. However there is one option available for those who like to mess with their browser. And that is the man-in-the-middle SSL proxy. The Charles Web HTTP proxy appears to be the main solution to this problem. Basically the way it works is that when your browser makes a request to the remote HTTPS server, the ssl proxy intercepts the request and from the ip address of the server generates a server certificate on the fly, which it signs with its own root CA, and sends back to the browser. The browser obviously complains about the self-signed cert, but here you can choose to install the ssl proxy root CA cert into the browser, befuddling the browser and allowing the ssl proxy to man in the middle and make replacements and filters on the raw response body.
Alternative roll your own chrome extension
I decided to go with rolling my own chrome extension, which I am planning to make available. Currently its in a very hardcoded to my own requirements state, but it works pretty good, even for https requests and another benefit is that a browser plugin solution can be more tightly integrated with the browser developer tools.