We Keep Coding

Delphi TWebBrowser JavaScript Errors and Cookies

I have been stuck on this for a couple of weeks now and this is a follow on from SO question Delphi REST Debugger Returns Error 429 Too Many Requests but Browser Returns JSON as Expected
I was wanting to get the content of a url response using the TNetHTTPRequest and TNetHTTPClient components. I was continually getting 429 errors “too many requests”. When using Firefox Inspect Element to look at network and storage, I discovered that I needed to receive cookies and then send those cookies with my request. Unfortunately, one of the cookies essential to the website content seems to be dependent (I think) on the execution of javascript. I went back to first principles and dropped a TWebbrowser on a form (VCL) and sure enough browser shows a javascript error “Expected Identifier”.
When I use the TWebbrowser in FMX it does not throw an error it just does not return the website contents at all and remains blank. I need FMX as I will be in a cross platform mobile environment.
The URL is https://shop.coles.com.au/a/national/home
I use Delphi Community Edition 10.3.3 Rio.
The URL returns perfectly in commercial browsers Firefox, Safari, Chrome and even CEF4Delphi. Unfortunately, I can’t use CEF as I need cross platform.
I would like to know how to get the website content returned to the browser (or even better NetHTTPClient) without script errors and how to access the browsers current cookies.
Any help will be most appreciated.
Thanks,
John.

URL returns perfectly in commercial browsers ... without script errors and how to access the browsers current cookies
If you'd inspect the network traffic (F12 > Network, then requesting your URL) or use uMatrix (to block everything that doesn't belong to the domain by default) you'd see the JS does at least one XHR to amazonaws.com. Your HTTP transfer alone (as done by TNetHTTP*) works fine and you get the same resource that each internet browser gets.
However, you don't operate with what you got (in contrast to the internet browser, which also automatically parses the HTML, sees JS resources, and executes them). TWebbrowser does not what you take for granted most likely due to security settings (try to get an error console in there, preferably F12 again). You need to do the same: parse the HTML resource for JS URIs, requesting those and executing what you get, while still providing the same cookie environment.
For executing JS you could use Chakra or mORMot or BESEN. It's challenging at first, but the more you understand about HTTP (including cookies) and a JS engine, the more you'll see why "things work" in one situation and not in another. There's a reason why an internet browser is a very complex software and not just a downloader.
As per this forcing IE11 Quirks mode might cure your problem already when using TWebBrowser:
TBrowserEmulationAdjuster.SetBrowserEmulationDWORD(TBrowserEmulationAdjuster.IE11_Quirks);

Google Oauth2 in Chrome Extension in background.js

I am building a Chrome Extension for my personal use (i.e. it will not be packaged and distributed) to dump data from a website into Google Sheets. I'd like to click on the Extension and have it process the data to my sheet. I believe this means I need to do the Oauth flow in background.js.
I did the initial authorization flow by customizing this Google Sheets demo, can't figure out how to make it work in my Extension.
I've tried a number of approaches, including using the chrome.identity API, and gapi.client.init(), and following the Chrome App sample. No dice.
Some of my questions...thanks in advance:
To what extent do Chrome Extensions mirror Chrome Apps? I understand that Chrome Apps are being deprecated, so wondering if the docs are inconsistent.
Is it possible to do this without packaging and uploading my app? The Oauth credentials page in Console asks for a Web Store URL
Is it acceptable to store a copy of Google's api.js in my extension, or must I load it from https://apis.google.com/js/client.js? If so,
For the Chrome App Sample, Where do I get the key included in manifest.json? I've seen instructions like "Copy key in the installed manifest.json to your source manifest" but I don't understand.
Is anyone aware of a complete, self-contained Chrome Extension sample?

To what extent do Chrome Extensions mirror Chrome Apps? I understand that Chrome Apps are being deprecated, so wondering if the docs are inconsistent.
Extensions and Apps are similar in many ways, however for your situation the main hurdle to overcome is the two handle Google Authentication differently. Extensions have permission limitations, where javascript can't run in certain places. Therefore, Chrome Extensions use chrome.identity in background.js to establish a secure connection and token. The general process to implement it is as follows:
Make a Chrome Extension, zip it, upload to your Google Dev account & get extensionID#
In Google API Console, register an OAuth ClientID# using the extensionID#
Update your Chrome Extension manifest to include an 'oauth2' section with the OAuth ClientID# as well as the scopes you allow, and include 'identity' under "permissions:"
Enable the API of your choosing in the Google API Console and generate a key. Include this key in your background.js file so you can use the API.
Is it possible to do this without packaging and uploading my app? The Oauth credentials page in Console asks for a Web Store URL
No, mainly because you need both the chrome extension and the API to be aware of each other and be 'linked' in a sense so they can be secure and work properly. You can still have a private app however, as you only need to package (.zip it) and upload it into your Developer Dashboard, and you can leave it out of the public Chrome Store by simply not publishing. It can forever linger in 'Draft' stage for your personal use.
Is it acceptable to store a copy of Google's api.js in my extension, or must I load it from https://apis.google.com/js/client.js? If so,
For the Chrome App Sample, Where do I get the key included in manifest.json? I've seen instructions like "Copy key in the installed manifest.json to your source manifest" but I don't understand.
You don't need to store a copy within your extension, you can add the following to your manifest.json:
"content_security_policy": "script-src 'self' https://apis.google.com/; object-src 'self'"
and then this at the bottom of your popup.html:
<script src="https://apis.google.com/js/client.js?onload=onGAPILoad"></script>
It's a rather confusing process without a guide; here is the one that finally made sense of it all for me. This official example from Google is a good one as well.
Is anyone aware of a complete, self-contained Chrome Extension sample?
'self-contained' is a bit tricky here, as the manifest needs to reference keys specific to the OAuth ClientID and API that YOU are utilizing, however this (download link) along with the two links above should be enough to get you to a working extension.

Chrome Application - Interract with a tab [duplicate]

I need to combine functionality available only in a Chrome packaged app (access to syncFileSystem) and functionality available only in a Chrome extension (injecting a script into a 3rd party website).
It seems that neither a packaged app nor an extension can achieve both these things, so I'm now considering trying to achieve what I'm after with a separate packaged app and extension communicating.
I see that Chrome's documentation explains how two extensions can communicate via chrome.runtime.onMessageExternal.addListener and chrome.runtime.sendMessage, but nothing about packaged apps and extensions communicating.
Does anyone know if this is allowed? Is there any documentation, or a working example out there?

Yes, that is possible. The code sample in the documentation you linked works for any combination of app and extension.
The extension documentation for chrome.runtime.sendMessage says:
Sends a single message to onMessage event listeners within the extension (or another extension/app).
Messaging works the same in both extensions and apps, and they seem to be fully compatible; simply use the ID for the destination extension or app. If you look at the docs for the app version of chrome.runtime.sendMessage, you'll see that it is identical to the extension version.

Inter frame SOP - Chrome Extension

In a Chrome Extension, I'm trying to get gmail compose body content.
An error jumps out sporadically, and does not prevents it from working.
This is being run as a content script. I believe permissions are not the issue here, because when there is a permission missing, the error is different and the operation is blocked by Chrome, definitely not the case.
Error comes out in this line:
encodeURIComponent($canvas.find('iframe').contents().find('body').text());
where
var $canvas = $('#canvas_frame').contents();
Any information on this error and a possible turnaround?

You can not read cross site content using javascript. XSS Auditor of Google Chorme will never allow this.
Please have a look at http://en.wikipedia.org/wiki/Same_origin_policy
UPDATE 1 -
There is support for cross site communication using window.postMessage() in Chrome 2+. Please have a look at this documentation https://developer.mozilla.org/en-US/docs/DOM/window.postMessage

To give a good answer, we need more information:
where exactly you are running this code, is it in the web page, the extension's content script or the etension's background script?
Also, what does your manifest.json file look like? Which sites have you asked for permission to run on.
I suspect this is a case of a badly-written error message and really it is saying: your extension does not have permission to run on apis.google.com. In that case, simply add permissions for apis.google.com. More details are in the Chrome extensions docs: http://developer.chrome.com/extensions/manifest.html#permissions

Replace remote JavaScript file with a local debugging copy using Greasemonkey or userscript

While debugging a client app that uses a Google backend, I have added some debugging versions of the functions and inserted them using the Chrome Developer Tools script editor.
However there are a number of limitations with this approach, first is that the editor doesn't seem to always work with de-minified files, and when the JS file is 35K lines long, this is a problem.
Another issue is that all the initialization that is done during load time, uses the original "unpatched" functions, hence this is not ideal.
I would like to replace the remote javascript.js file with my own local copy, presumably using some regex on the file name, or whatever strategy was suitable, I am happy to use either Firefox or Chrome, if one was easier than the other.

So basically, as #BrockAdams identified, there are a couple of solutions to these types of problem depending on the requirements, and they follow either 1 of 2 methods.
the browser API switcharoo.
The proxy based interception befiddlement.
the browser API switcharoo.
Both firefox and chrome support browser extensions that can take advantage of platform specific APIs to register event handlers for "onbeforeload" or "onBeforeRequest" in the case of firefox and chrome respectively. The chrome APIs are currently experimental, hence these tools are likely to be better developed under firefox.
2 tools that definitely do something like what is required are AdBlock plus and Jsdeminifier both of which have the source code available.
The key point for these 2 firefox apps is that they intercept the web request before the browser gets its hands on it and operate on the other side of the http/https encrpytion stage, hence can see the decrypted response, however as identified in the other post that they don't do the whole thing, although the jsdeminifier was very useful, I didn't find a firefox plugin to do exactly what I wanted, but I can see from those previous plugins, that it is possible with both firefox and chrome. Though they don't actually do the trick as required.
The proxy based interception befiddlement This is definitely the better option in a plain HTTP environment, there are whole bunch of proxies such as pivoxy, fiddler2, Charles Web HTTP proxy, and presumably some that I didn't look at specifically such as snort that support filtering of some sort.
The simplest solution for myself was foxyproxy and privoxy on firefox, and configure a user.action and user.filter to detect the url of the page, and then to apply a filter which swapped out the original src tag, for my own one.
The https case. proxy vs plugin
When the request is https the proxy can't see the request url or the response body, so it can't do the cool swapping stuff. However there is one option available for those who like to mess with their browser. And that is the man-in-the-middle SSL proxy. The Charles Web HTTP proxy appears to be the main solution to this problem. Basically the way it works is that when your browser makes a request to the remote HTTPS server, the ssl proxy intercepts the request and from the ip address of the server generates a server certificate on the fly, which it signs with its own root CA, and sends back to the browser. The browser obviously complains about the self-signed cert, but here you can choose to install the ssl proxy root CA cert into the browser, befuddling the browser and allowing the ssl proxy to man in the middle and make replacements and filters on the raw response body.
Alternative roll your own chrome extension
I decided to go with rolling my own chrome extension, which I am planning to make available. Currently its in a very hardcoded to my own requirements state, but it works pretty good, even for https requests and another benefit is that a browser plugin solution can be more tightly integrated with the browser developer tools.