I have been stuck on this for a couple of weeks now and this is a follow on from SO question Delphi REST Debugger Returns Error 429 Too Many Requests but Browser Returns JSON as Expected
I was wanting to get the content of a url response using the TNetHTTPRequest and TNetHTTPClient components. I was continually getting 429 errors “too many requests”. When using Firefox Inspect Element to look at network and storage, I discovered that I needed to receive cookies and then send those cookies with my request. Unfortunately, one of the cookies essential to the website content seems to be dependent (I think) on the execution of javascript. I went back to first principles and dropped a TWebbrowser on a form (VCL) and sure enough browser shows a javascript error “Expected Identifier”.
When I use the TWebbrowser in FMX it does not throw an error it just does not return the website contents at all and remains blank. I need FMX as I will be in a cross platform mobile environment.
The URL is https://shop.coles.com.au/a/national/home
I use Delphi Community Edition 10.3.3 Rio.
The URL returns perfectly in commercial browsers Firefox, Safari, Chrome and even CEF4Delphi. Unfortunately, I can’t use CEF as I need cross platform.
I would like to know how to get the website content returned to the browser (or even better NetHTTPClient) without script errors and how to access the browsers current cookies.
Any help will be most appreciated.
Thanks,
John.
URL returns perfectly in commercial browsers ... without script errors and how to access the browsers current cookies
If you'd inspect the network traffic (F12 > Network, then requesting your URL) or use uMatrix (to block everything that doesn't belong to the domain by default) you'd see the JS does at least one XHR to amazonaws.com. Your HTTP transfer alone (as done by TNetHTTP*) works fine and you get the same resource that each internet browser gets.
However, you don't operate with what you got (in contrast to the internet browser, which also automatically parses the HTML, sees JS resources, and executes them). TWebbrowser does not what you take for granted most likely due to security settings (try to get an error console in there, preferably F12 again). You need to do the same: parse the HTML resource for JS URIs, requesting those and executing what you get, while still providing the same cookie environment.
For executing JS you could use Chakra or mORMot or BESEN. It's challenging at first, but the more you understand about HTTP (including cookies) and a JS engine, the more you'll see why "things work" in one situation and not in another. There's a reason why an internet browser is a very complex software and not just a downloader.
As per this forcing IE11 Quirks mode might cure your problem already when using TWebBrowser:
TBrowserEmulationAdjuster.SetBrowserEmulationDWORD(TBrowserEmulationAdjuster.IE11_Quirks);
Related
I hava situation where some javascript a web page works fine in Safari and Chrome, but fails in IE11. Unfortunately due to issues with confidentiality I cannot put the javascript up here.
In IE11 the web page's java script fails to operate correctly. By that I mean some of the javascript works and some doesn't. With no errors displayed or any other indication of whats wrong.
If I try to debug the page using IE's developers tools, all the javascript works perfectly without any errors or issues.
Searching on the net I found many people with the same IE problem - fails normally, works when debugging. The main issues they talk about is the console.log(...) statement. I checked my javascript and don't have any console.log(...) statements.
I then saw a stackoverflow thread where adding a cache:false to the $.ajax({... calls solved the issue. I added the same flag but the problem still persists.
Are there any other bugs I've not found?
The web page is using jQuery to handle most of it's manipulation of the DOM with a single $.ajax... call and a series of $.get(... calls polling the server.
How to debug your web pages.....IE11 tips.
All modern web browsers suppress scripting error messages and warnings by default. (In the early days web browsers would halt page loading/rendering and display a script error message with an alert statement)... this gives the best user experience who isn't concerned with the internal workings of web site code.
So, scripting errors will only BREAK execution if:
1. The browser debug tool is opened. and
2. The developer tools' Debug tab setting for Break on Exceptions has been turned on.
So to debug your web pages.
1. navigate to about:blank to start a testing cycle.....press f11 to display the dev tool, select "Break on all exceptions" from the dropdown (looks like a stop sign). Pin the dev tool to the bottom of the browser.
2. Return to the browser address bar and navigate to your test site (typed address of paste and go)...
The dev tool will now break on ALL exceptions and you will list them in the console tab.
IE has built-in content blocking and has ActiveX filtering (ad blocking) which can affect outcomes. You need to configure Internet Options so that the IE dev tool console will record any blocked content or security (XSS) errors.
Tools>Internet Options>Advanced tab, check "Always record developer console messages".
Also on the Emulation tab of the IE dev tool you will find the Emulation Mode (aka documentMode) that IE is using, and how it was established eg. x-ua meta, Enterprise site mode list, user Compatibility View list, etc
If you are developing an internal company website, the emulation mode used by IE may be for an earlier version of IE.. (IE8 on XP).. you should include this information with your questions.
You should also include the IE security zone that your site has been mapped to.. File>Properties menu in IE.... eg. Intranet zone as this can have different security and blocked content outcomes.
finally, the first step in troubleshooting web browser issues is to test in noAddons mode (for IE, winkey+r>iexplore.exe -extoff ). IE has built-in form-fillers and popup blockers... third-party addons can affect the outcomes expected.
This one has me stumped.
I have a web app that has a file upload/download area. Files are downloaded in the background via a temporary iFrame element. This is a single-paged AJAX application and the UI is written in Javascript, jQuery and uses the jQuery.FileDownloader.js to manage the iFrame. The application runs over HTTPS and the site and download URL are on the same exact domain. The back-end is a RESTful application. This has worked great for months. Until today.
All of a sudden, when attempting to download a file in Chrome, the browser reports an error of "Blocked a frame with origin https://example.com from accessing a cross-origin frame."
The problem is that the origin of the main site and that of the iframe are the exact same domain. I have ensured that the domains are the same as well as the protocol. Chrome is the only browser that throws up the cross-origin error. IE, Firefox, Opera, Safari... all work as expected. It's only in Chrome and it's only as of today. To make things worse, no updates were made to the browser. It truly is spontaneous. I've also ruled out plugins as the cause by running in Incognito mode, where none are allowed to run by my settings, as well as disabling my anti-virus software. This problem is being exhibited on other computers, in other locations (not on our LAN or subnet), all running Chrome.
And, again, both domains of the parent frame and the embedded iframe are identical. This only happens against the production server which runs over HTTPS. Other non-HTTPS sites (e.g. our dev environment, localhost) don't have the problem. Our SSL is valid. Since this is a single-paged AJAX application, we're trying to avoid popping up another window for the download.
Hopefully, someone can offer some advice. Thanks in advance.
Update: After additional research, I have found the solution to this problem is to enclose the filename in the response header in double-quotes.
I have found the cause of the problem. It turns out that Google Chrome has problems with files that have commas in their filename. When downloading the file via a direct link, Chrome will report that duplicate headers were reported from the server. This has been a long-standing problem with Chrome that has gone un-addressed. Other browsers are not susceptible to this problem. Still, it's a fairly easy problem to troubleshoot and, indeed, when I searched on this error, the first search result had the solution: remove commas from filenames when handling a request from Google Chrome.
However, this wasn't a direct link, it was an AJAX-request, which results in a different exception. In this case, the error provided by Chrome is the cross-origin request exception and this is what made it so difficult to troubleshoot.
So, the tl;dr of it all is to strip out commas in the names of uploaded files.
Another instance I found where this issue occurred is after executing code similar to:
document.domain = '[the exact same domain that the iframe originally had]'
Removing this line of code got rid of this error for me.
We are trying to figure out how something works on the web (for web scraping/automation) and one of the web pages we are working on issues a popup to do some of the work. One of our most commonly used debug tools is the Chrome network tab in Developer Tools, hit "record" do some work, and then examine what was done and then replicate the work done "offline".
However the Developer Tools (in Chrome, Safari and Firefox - all work the same) do not follow requests across a popup, even if you hit "record".
Is there some configuration value I'm missing, or some way to record all network events? We can't use tcpdump/wireshark for this because it's all done over SSL. One option we've considered is a man-in-the-middle https proxy, but I can't find anything pre-written so we'd have to create one ourselves.
I don't know of any way to follow the requests across pop-ups, as each window has its own Web Inspector, however you can use Fiddler to inspect HTTPS requests. It will MITM, and subsequently throw a certificate error, which should allow you to inspect all requests in the order that they happened.
You can use Charles Web Debugging Proxy, which is an app that lets you see all the traffic and even replace some responses with your own. Of course that may break HTTPS so you have to accept the certificate errors, but that's usually a minor problem. It works on Win, Mac and even Linux.
The object inspector cannot inspect what isn't in the current page. Therefore, you will need to open the inspector inside the popup url with same parameters in order to see what it does.
As a tool, you can use a web sniffer to see exactly which url were called during the process.
While debugging a client app that uses a Google backend, I have added some debugging versions of the functions and inserted them using the Chrome Developer Tools script editor.
However there are a number of limitations with this approach, first is that the editor doesn't seem to always work with de-minified files, and when the JS file is 35K lines long, this is a problem.
Another issue is that all the initialization that is done during load time, uses the original "unpatched" functions, hence this is not ideal.
I would like to replace the remote javascript.js file with my own local copy, presumably using some regex on the file name, or whatever strategy was suitable, I am happy to use either Firefox or Chrome, if one was easier than the other.
So basically, as #BrockAdams identified, there are a couple of solutions to these types of problem depending on the requirements, and they follow either 1 of 2 methods.
the browser API switcharoo.
The proxy based interception befiddlement.
the browser API switcharoo.
Both firefox and chrome support browser extensions that can take advantage of platform specific APIs to register event handlers for "onbeforeload" or "onBeforeRequest" in the case of firefox and chrome respectively. The chrome APIs are currently experimental, hence these tools are likely to be better developed under firefox.
2 tools that definitely do something like what is required are AdBlock plus and Jsdeminifier both of which have the source code available.
The key point for these 2 firefox apps is that they intercept the web request before the browser gets its hands on it and operate on the other side of the http/https encrpytion stage, hence can see the decrypted response, however as identified in the other post that they don't do the whole thing, although the jsdeminifier was very useful, I didn't find a firefox plugin to do exactly what I wanted, but I can see from those previous plugins, that it is possible with both firefox and chrome. Though they don't actually do the trick as required.
The proxy based interception befiddlement This is definitely the better option in a plain HTTP environment, there are whole bunch of proxies such as pivoxy, fiddler2, Charles Web HTTP proxy, and presumably some that I didn't look at specifically such as snort that support filtering of some sort.
The simplest solution for myself was foxyproxy and privoxy on firefox, and configure a user.action and user.filter to detect the url of the page, and then to apply a filter which swapped out the original src tag, for my own one.
The https case. proxy vs plugin
When the request is https the proxy can't see the request url or the response body, so it can't do the cool swapping stuff. However there is one option available for those who like to mess with their browser. And that is the man-in-the-middle SSL proxy. The Charles Web HTTP proxy appears to be the main solution to this problem. Basically the way it works is that when your browser makes a request to the remote HTTPS server, the ssl proxy intercepts the request and from the ip address of the server generates a server certificate on the fly, which it signs with its own root CA, and sends back to the browser. The browser obviously complains about the self-signed cert, but here you can choose to install the ssl proxy root CA cert into the browser, befuddling the browser and allowing the ssl proxy to man in the middle and make replacements and filters on the raw response body.
Alternative roll your own chrome extension
I decided to go with rolling my own chrome extension, which I am planning to make available. Currently its in a very hardcoded to my own requirements state, but it works pretty good, even for https requests and another benefit is that a browser plugin solution can be more tightly integrated with the browser developer tools.
I am developing JavaScript chat. I have done the prototype, and it seems to work pretty well. But our client says that it doesn't work. We both use IE7 on PC, and try to run the same JavaScript code. I have no idea about the reason.
On my client's machine there is "object is expected" error. I have thought about security restrictions, and tried to make security level higher in my IE, but in this case the script doesn't run and there is no error messages. I am running the script on the remote server.
Clients machine may be running within a group policy which for security reasons can restrict execution of scripts. Hence the client may need to include your prototype as part of their trusted sites and relax restrictions on trusted sites
Are you able to provide any more details about the specific error? Are you sending the client a zipped up version of the solution or is it hosted somewhere?
From the IEBlog; you could try using the Microsoft Script Debugger (see post)
One thing you may find, especially with IE is the error that is reported is actually caused by a previous error in the code.
Try running your javascript through jslint to ensure that it is well formed.
In Firefox with [firebug], enable strict warnings in the console.
Related SO posts.
debugging-javascript-in-ie7.
debugging-javascript-for-ie6.
Edit::
One thing to look for is comma's after the last element in an array, IE seems to ignore the closing brace } and keep adding following code to the array. Firefox gracefully just assumes that you forgot it.
If you're running the code from localhost and he's running it from a remote server then it could be a timing issue - some script could be running before the page has finished loading that's causing a problem on a slow connection that is hidden on a fast connection.
You can get this error if you try to use an exernal script file that the client's browser can not access.
Example :
<script src="http://localhost/scripts/freelib.js" language=Javascript> </script>
You can access freelib.js on your machine, but running it from any other machine it wouldn't work.
Are you sure all the scripts are being loaded? I've seen that error message before when some or all aren't loaded (maybe they didn't get deployed). Try manually copying the addresses of all the js files from the source and pasting them into the address bar to make sure that the browser can actually access them.
Can you run the page on your client's machine in Firefox? The firefox console should give you a more detailed error message than IE will.