Summary: I would like to make a Range header request from GitHub pages. However, in some browsers this is failing- possibly due to Gzip compression issues. It works in Chrome (v74) but not in FF (v66), Mac OS.
Goal: I would like to reliably make this request in all browsers, such as by forcing the response type to be encoded as text whenever a range request is made.
It is not clear to me whether this behavior is dictated by the browser, the server, or some combination of the two. Knowing origins could help to define a fix- working with Github pages would be nice but not mandatory.
It is also not clear to me whether this represents a bug, or if so, where. (in browser, in spec, etc)
Sample test case:
Possibly because this involves server-side gzip encoding, the sample test case doesn't reproduce locally. You'll need to enter these commands in the JS console at https://abought.github.io/weetabix/ to reproduce.
fetch('https://abought.github.io/weetabix/example_data/McDonald_s.csv', {headers: {range: 'bytes=1-100'}} ).then(resp => resp.text());
In chrome, this fetches the response text. In firefox, it gives a "decoding error".
If I omit resp.text, Firefox can complete the request- the decoding error is in reading the body, rather than any other code. Copying as curl shows that FF adds a --compress flag and Chrome does not.
Investigations
If the byte range is 0-100, the request works in FF. If the range is 1-100, it fails. This section of the file is all ASCII characters.
If I inspect the response headers (Array.from(r.headers.entries())), FF has an extra "content-encoding: gz flag" that I think is causing the issue.
(eg, gzip makes no sense without the secret decoder instructions)
I tried adding 'accept-encoding': 'identity' to the fetch request, but it is a forbidden header and modifying it via code has no effect.
The specs have changed quite recently here. Here is the link to the PR.
TLDR; They now ask the UA that the Acccept-Encoding/Identity header be added to all the Range-requests.
[§5.15]
If httpRequest’s header list contains Range, then append Accept-Encoding/identity to httpRequest’s header list.
Firefox has not yet followed up here, but a bug report has been filled.
For the time being, the Range requests in Firefox indeed are made with the Gzipped data, and thus, you must not break the bytes integrity (for instance the range 0-100 is decode-able in Firefox).
Related
As I continue trying to develop my robotics project, I have run into something that - though I am sure it is by design - borders on being a bug and a potential security risk - sticky cache.
Issue:
It is virtually impossible, short of an Act of God, to absolutely and positively clear cache on either Firefox or Chrome.
Issue:
While developing the browser-side scripting for my project, I frequently make changes to the underlying JavaScript, save, and relaunch both the server and the browser - only to find the problem still exists. When I examine the running script in "Developer Mode" - I discover that the script being run has absolutely no resemblance to the script on the server.
Sometimes, (in Chrome), I can right-click on refresh and select "drop cache and hard reset" to refresh things. Other times dropping cache or even restarting the browser is not sufficient. I end up having to rename the file to force a hard-cache-miss to invalidate the cache and reload the correct file.
Yes, I know, there are a thousand "duplicates" of this question all over Stack Exchange and the rest of the Internet, and I've tried it all - from passing --nocache to pointing the cache to /dev/null, (which doesn't work on a Windows system!), and everything I've tried either doesn't work or results in an unusable browser.
I can't prove this, nor have I seen it documented, but I'm beginning to believe that browsers implement some kind of "hidden" cache that is used when all else fails - like the Windows "Virtual Store" - if you don't clear it after an update, the system never sees the updated files.
This is causing me untold grief and angst and is severely limiting my ability to develop code sanely.
Note that Chrome seems to be, (at least to some degree), the most amenable to clearing cache whereas Firefox holds on to cached data like a miser with his last copper coin.
Short of a nuclear explosion, how do I tell these browsers to absolutely, positively, totally and completely, drop their cache of scripts and such like? There used to be a settings feature for this, but it's gone in later browsers.
Frustration reigns supreme and any help would be appreciated.
====================
Update:
Following one of the suggested answers, I discovered something interesting:
Caching appears to be controlled by the server, not the browser and to absolutely prevent caching, the server must send special headers to the browser to make sure caching doesn't take place.
Ref: How do we control web page caching, across all browsers?
Since the person who posted the question sounded like IT for a bank or accounting firm, it appeared to involve sensitive data from banking or such like, (and I am assuming access to money or important resources), and he accepted the answer, (which had gathered a consensus of support), that sounds good enough for me.
His suggested headers for Python/Flask:
response = make_response(render_template(...))
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response.headers["Pragma"] = "no-cache" # HTTP 1.0.
response.headers["Expires"] = "0" # Proxies.
However, two questions remain:
Is the answer still valid as it is from 13 years ago?
Where do I put it? Along with the generic response to a POST request? Or in the specific response when a resource, (like the JavaScript), is requested?
With regard to question #2, this is the server response code in my project - which I inherited from the earlier version of this written by someone else. (I added the CORS stuff.)
# Allow CORS (Cros Origin Resource Sharing) by the robot
# in response to browser "pre-flight" ("OPTION") requests.
#app.route("/robot", methods = ["OPTIONS"])
def create_CORS_response():
resp = Response()
resp.headers.add("Access-Control-Allow-Origin", "*")
resp.headers.add('Access-Control-Allow-Headers', "*")
resp.headers.add('Access-Control-Allow-Methods', "*")
resp.mimetype = "application/json"
resp.status = "OK"
resp.status_code = 200
return(resp)
#app.route("/robot", methods = ["POST"])
def get_args():
# get the query
args = request.args
# print(args)
process_robot_commands(args)
# After doing all that work, send a response.
resp = Response()
# Allow CORS (Cross Origin Resource Sharing) during POST
resp.headers.add("Access-Control-Allow-Origin", "*")
# ==> place the header data here?
resp.mimetype = "application/json"
resp.status = "OK"
resp.status_code = 200
return(resp)
# ==> or somewhere in one or more of these responses?
# ==> If so, how do I create the response and add the document data to it?
#app.route("/")
def index():
return page("index.html")
#app.route("/<string:page_name>")
def page(page_name):
return render_template("{}".format(page_name))
#app.route("/static/<path:path>")
def send_static(path):
return send_from_directory(directory_path, path)
I feel like I'm making progress, but I would appreciate any help I can get.
Thanks!
I found that fetch API always throw TypeError, ERR_EMPTY_REPONSE or NetworkError when attempting to fetch resource for 102 or any code smaller than 200, regardless of the body of the response.
This took me tremendous time to figure out. Behavior is consistent amongst Fx, Chrome and Opera. I also confirmed that Postman correctly identifies the 102 response.
I wonder why is this happening? 1xx code should be fine and it is not up to fetch to decide whether it is an error. I can't find any document and can't seem to find anyone who is in the same situation.
Is this by design? Or are there anyway to get around this problem? I need 102.
The Fetch spec requires browsers to follow the behavior described in the question. See specifically https://fetch.spec.whatwg.org/#ref-for-concept-response-status%E2%91%A0%E2%91%A8:
Any responses whose status is in the range 100 to 199, inclusive, and is not 101, are to be ignored.
Note: These kind of responses are eventually followed by a "final" response.
Just to follow up #sideshowbarker and for Googlers:
1xx codes are considered informational and non-final (except 101 for WebSocket)
Basically browsers adhere to this definition and expects more content to follow 1xx codes
I returned 102 and closed connection as would with other response codes
Browsers see this as an unexpectedly ended connection and complains
1xx responses should be properly completed with other codes (> 199)
And...
The errors thrown by browsers are too simple to give insights about what real happens
Partial response (1xx codes and body) is ignored thus not recorded by browsers and will NOT show up in the console, further complicates the problem
Even 999 will work for browsers as these are not ignored like 1xx codes
So...
Do not use 1xx unless you want keep some long running connections or use WebSocket
When signify "Request received, I'm processing it, come back later", do not use 102 Processing, but 202 Accepted for acknowledgement
On Chrome, I'm having no troubles making a cross domain request, however on Firefox (Ubuntu 14.04), I get an error that consists only of a colon on the line that calls for the xmlhttprequest.
xmlhttp.open("GET", "http://x.x.x.x:xxxx/folder/file.xml", false);
The error message is just ":".
Try Disabling AdBlock
I was having a similar issue where all of my XMLHttpRequests were going through except for a few very specific ones where even minor URL changes fixed the problem. And the only thing I was getting was a colon : in the console. In the end I realized that AdBlockPlus was blocking at least one these requests from going through (the request had 'Banners' in the URL).
So I'm not sure if this would have solved your problem, but I've encountered it multiple times.
using Firebug, the issue turned out to be
Blocked loading mixed active content
I had the same error on a CORS POST request.
I'm using https://cors-anywhere.herokuapp.com/ to bypass the Same-origin policy.
The problem was NoScript blocking the external domain, so in case you're using an external API in your request, this might solve it.
Same as this answer, however the problem for me isn't present neither with AdBlock nor with uBlock.
It's present with uMatrix.
You can find that it blocks the request at this URL:
chrome://umatrix/content/logger-ui.html
You can enable it manually when clicking on that field:
and choosing to enable XHR in the uMatrix popup:
The blocking persists even if you have the code packaged in an Add-on that's correctly installed (the blue bar containts Internal UUID of a Firefox Add-on), therefore if you've ever wondered why some Add-ons don't work while uMatrix (or likes) is enabled, this might be one of the reasons.
If I create a cookie with the value 'ë' on the server, it becomes 'ë' when I read it on the client side.
This only occurs with IE (10). I don't get this issue with other browsers (Chrome and Firefox).
The way the cookie is created:
HttpContext.Current.Response.Cookies.Add(new HttpCookie("test", "ë"));
Per RFC 2109 (sec. 4.2.2) and RFC 2616 (sec. 2.2, 4.2), HTTP headers may only be transmitted in ISO-8859-1. (There is an exception, but this is primarily used for MIME and is pretty much never seen in HTTP.) Since ISO-8859-1 is really just a series of octets, the server can choose to use a different encoding (UTF8, in this case) if it wishes, since cookies are intended to be treated opaquely by the client anyway. Thus it should be round-tripped correctly in all cases, but if the client tries parsing the cookie it will get different results based on whether or not it is following the RFCs.
Please try to avoid sending non-ASCII characters in headers if at all possible. The HTTP spec never really accounted for this properly, and as a result it causes pain and heartache when used in practice. :(
I mocking my API using Apiary.io. But somehow I cannot read any headers from response object using angularJS. And I am sure i have at least Content-Type: application/json correctly set-up by checking in firebug. Code in Angular should read headers correctly too as i can print them when sending request to somewhere else than apiary.io...
$http.get('http://ies.apiary.io/some').then(function(response) {
console.log("ok",response.headers('Content-Type'));
},function(response){console.log("err",response);});
http://plnkr.co/edit/zMO0pXGsIdJkV0fZdBdw
It all boils down to a bug in firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=608735
For CORS request, firefox is not returning anything for req.getAllRequestHeaders(), although req.getRequestHeader('Content-Type') returns properly.
Either FF bug has to be fixed or Angular must work around it (as jQuery does) or you must go deeper and use XmlHttpRequest instance directly.
This was maybe also related, but probably not the core issue:
However, few days ago, Apiary.io was not setting Max-Age in CORS headers. Thus, if you had a minimal blueprint, you might have CORS pre-flight response cached and thus subsequent requests may be disallowed even if you added additional resources to your blueprint.
Max-Age is now set to 10 seconds, so it should work properly. However, depending on your browser, pre-flight cache might still affect you. Try purging it or test this app in another browser.