I have two questions regarding HTTP cache using the Cache-Control header:
How does the browser identify which request can be fulfilled using the existing cache? Is the browser checking if the endpoint matches? But even requests to the same endpoint can have different body or config, I don't quite understand how does the browser know when to use the cache and when to not use the cache given the time it sends out the request is still within the time frame specified by max-age in the response's cache-control header?
I recently learned that both request and response can set max-age in their own cache-control header. I understand that the request's max-age would tell the server (or any intermediate caches) how fresh of response the client is willing to accept from them. The response max-age (or lack thereof) tells the client how long it can consider that response to be fresh. (feel free to correct me if I am wrong). But what would happen in this scenario:
let's say the response has a max-age for one year and then we send another request for the same resources with max-age being 0. Does that make the browser ignore the cache? or Does the browser accept the cache and not send out the request?
You can get information from this specification. According to the document,
The cache entry contains the headers of the request:
As discussed above, caching servers will by default match future
requests only to requests with exactly the same headers and header
values.
This means that you get one entry in your cache every time you make exactly the same request to the server (the cache can be personal or shared, like in a proxy). In practice, for entities that only cache GET requests, the key can be the URI of the request. By the process of normalization, two very similar requests can share a cache entry. The decision to use the cached entry depends on several factors, as detailed below. The figures in the document explain this very well. Bottom line, max-age only determines freshness, not the behavior of the cache.
According to this specification, the cache is never ignored if the entry exists. Even a fresh entry can be discarded to save disk space, and a stale entry can be kept long after it has expired. The diference is that a stale entry is not directly retrieved. In that case, the caching entity (browser/proxy/load_balancer...) sends a freshness request to the server. The server then decides whether the cached page is fresh.
In summary, if a cached page is fresh according to max-age and whatever other modifiers are used, the caching entity decides that the cached resource will be used. If it is stale, the server decided whether the cached resource can be used.
EDIT after comment:
To understand the difference between max-age sent by the client and the server, we need to dig into the http protocol. In section 5.2.1., It says
5.2.1. Request Cache-Control Directives
5.2.1.1. max-age
Argument syntax:
delta-seconds (see Section 1.2.1)
The "max-age" request directive indicates that the client is
unwilling to accept a response whose age is greater than the
specified number of seconds. Unless the max-stale request directive
is also present, the client is not willing to accept a stale
response.
The language seems to indicate that the server is not forced by the directive, but it is expected to honor it. In your example, this would means that the client directive prevails, as it is more restrictive. The client is saying "I do not want any page cached for more than 0 seconds", and the cache server is suposed to contact the server to fulfill the condition.
Related
In theory, one should use the HTTP GET method only for idempotent requests.
But, for some intricate reasons, I cannot use any other method than GET and my requests are not idempotent (they mutate the database). So my idea is to use the Cache-Control: no-cache header to ensure that any GET request actually hits the database. Also, I cannot change the URLs which means I cannot append a random URL argument to bust caches.
Am I safe or shall I implement some kind of mechanism to ensure that the GET request was received exactly once? (The client is the browser and the server is Node.js.)
What about a GET request that gets duplicated by some middle-man resulting in the same GET request being received twice by the server? I believe the spec allows such situation but does this ever happen in "real life"?
I've never seen a middle man, such as Cloudflare or NGNIX, preventing or duplicating a GET request with Cache-Control: no-cache.
Let's start by saying what you've already pointed out -- GET requests should be idempotent. That is, they should not modify the resource and therefore should return the same thing every time (barring any other methods being used to modify it in the meantime.)
It's worth pointing out, as restcookbook.com notes, that this doesn't mean nothing can change as a result of the request. Rather, the resource's representation should not change. So for instance, your database might log the request, but shouldn't return a different value in the response.
The main concern you've listed is middleware caching.
The danger isn't that the middleware sends the request to your server more than once (you mentioned 'duplicating' a request), but rather that (a) it sends an old, cached, no-longer-accurate response to whatever is making the request, and (b) the request does not reach the server.
For instance, imagine a response returning a count property that starts at 0 and increments when the GET endpoint is hit. Request #1 will return "1" as the count. Request #2 should now return "2" as the count, but if its cached, it might still show as 1, and not hit the server to increase the count to 2. That's 2 separate problems you have (caching, and not updating).
So, will a middleware prevent a request from reaching the server and serve a cached copy instead? We don't know. It depends on the middleware. You can absolutely write one right now that does just that. You can also write one that doesn't.
If you don't know what will be consuming your API, then it's not a great option. But whether it's "safe" depends on the specifics.
As you know, it's always best to follow the set of expectations that comes with the grammar of HTTP requests. Deviating from them sets yourself up for failure in many ways. (For instance, there are different security expectations for requests based on method. A browser may treat a GET request as "simple" from a CORS perspective, while it would never treat a PATCH request as such.)
I would go to great lengths to not break this convention, but if I were forced by circumstances to break this expectation, I would definitely note it in my APIs documentation.
One workaround to ensure that your GET request is only called once is to allow caching of responses and use the Vary header. The spec for the Vary header can be found here.
In summary, a Vary header basically tells any HTTP cache, which parts of the request header to take into account when trying to find the cached object.
For example, you have an endpoint /api/v1/something that accepts GET requests and does the required database updates. Let's say that when successful, this endpoint returns the following response.
HTTP/1.1 200 OK
Content-Length: 3458
Cache-Control: max-age=86400
Vary: X-Unique-ID
Notice the Vary header has a value of X-Unique-ID. This means that if you include the X-Unique-ID header in your request, any HTTP caching layer (be it the browser, CDN, or other middleware) will use the value in this header to determine whether to use a previously cached response or not.
Say your make a first request that includes a X-Unique-ID header with the value id_1 then you make a subsequent request with X-Unique-ID value of id_2. The caching layer will not use a previously cached response for the second request because the value of the X-Unique-ID is different.
However, if you make another request that contains the X-Unique-ID value of id_1 again, the caching layer will not make a request to the backend but instead reuse the cached response for the first request assuming that the cache hasn't expired yet.
One thing you have to consider though is this will only work if the caching layer actually respects the specifications for the Vary header.
The Hypertext Transfer Protocol (HTTP) is designed to enable communications between clients and servers.
where Get method is used to request the data from specified resources.
When we used 'Cache-control: no-cache' it means the cache can't store anything about the client request
or server responses. That Request hits to the server and a full response is downloaded each and every time.
This depends a lot on what's sat in the middle and where the retry logic sits, if there is any. Almost all of your problems will be in failure handling and retry handling - not the basic requests.
Let's say, for example that Alice talks to Bob via a proxy. Let's assume for the sake of simplicity that the requests are small and the proxy logic is pure store-and-forward. i.e. most of the time a request either gets through or doesn't but is unlikely to get stalled half-way through. There's no guarantee this is the case and some proxies will stop requests part-way through by design.
Alice -> Proxy GET
Proxy -> Bob GET
Bob -> Proxy 200
Proxy -> Alice 200
So far so good. Now imagine Bob dies before responding to the proxy. Does the proxy retry? If so, we have this:
Alice -> Proxy GET
Proxy -> Bob GET
Bob manipulates database then dies
Proxy -> Bob GET (retry)
Now we have a dupe
Unlikely, but possible.
Now imagine (much more likely) that the proxy (or even more likely, some bit of the network between the proxy and the client) dies. Does the client retry? If so, we have this:
Alice -> Proxy GET
Proxy -> Bob GET
Bob -> Proxy 200
Proxy or network dies
Alice -> Proxy GET (retry)
Proxy -> Bob GET
Is this a dupe or not? Depends on your point of view
Plus, for completeness there's also the degenerate case where the server receives the request zero times.
I'm reading this great article on caching and there is the following there:
Validators are very important; if one isn’t present, and there isn’t
any freshness information (Expires or Cache-Control) available, caches
will not store a representation at all.
The most common validator is the time that the document last changed,
as communicated in Last-Modified header. When a cache has a
representation stored that includes a Last-Modified header, it can use
it to ask the server if the representation has changed since the last
time it was seen, with an If-Modified-Since request.
So, I'm wondering whether browser continues to send requests (for example HEAD) for a resource even if I specified Cache-Control: max-age=3600? If it doesn't, than what's the point in this header? Is it used after the max-age time passes?
The Cache-Control: max-age=3600 header means that the browser will cache the response for up to 3600 seconds. After that time has passed it may no longer serve the response without first confirming that it is still fresh.
In order to this, the browser can either:
Fetch the full resource with a normal GET request (transfers the whole response body again)
Or perform a revalidation based on an ETag (If-None-Match) or the Last-Modified header (If-Modified-Since), i.e. the client only fetches the response body if it has actually changed. This is of course only possible if the validator was present in the original response.
In short: the reason to use both max-age and a cache validator is to first cache the response for some time and then perform a bandwidth-saving revalidation to confirm the resource's freshness.
I can build a full stack app using Ruby on Rails, JavaScript, React, HTML and CSS. Yet, I feel I don't understand completely how cookies actually work and what they are precisely. Below I write what I think they are, and ask that someone confirm or correct what is written.
An HTTP request contains an HTTP method, a path, the HTTP protocol version, headers, and a body.
An HTTP response contains the HTTP protocol version, a status code, a status message, headers, and a body.
Both are simply text (which means that they are simply sequences of encoded characters), but when this text is parsed it contains useful structure. Is there one single structure that an HTTP request is usually parsed into (an array, a hash)? What about an HTTP response?
Cookies represent some content associated with a specific header in an HTTP request, specifically the "Cookie" header.
When building an HTTP response, the server sets the 'Set-Cookie' header. This header needs the following information: a name for the cookie, a path, and the actual content of the cookie. The path is a description of the range of URLs for which this cookie should be sent from client to server.
Does the browser keep a list of cookies (ie, a list of elements that are each text of some sort), and it only sends the right ones to the right sites (say a google cookie to google.com)?
Let's say I visit site A and then site B and authenticate on both. Session management just adds a specific element in the cookies (perhaps a hash named Session inside another hash that corresponds to the totality of the cookie stored in Cookie), correct? How do sites alter my cookies? Do they append new information, do they ask my browser to append information?
A cookie is a string (with a specific format) that your browser stores. It can be set by a server when it sends a http-response, by the 'Set-Cookie' header. Each http-request that your browser sends that matches the cookie's path will contain that cookie in the 'Cookie' header.
The server cannot tell the browser to append data to the cookie. It can only get the current cookie value, add to it the new information, and then reset it.
When I reload page, Angular directive templates are loaded in two ways.
First one - browser makes a request to the server and it responses with 304 - it's ok.
But the second one - browser doesn't make a request. And I can't guess why.
As a result, when I make changes in templates from the first group, the changes are shown with the next page reload. But the changes in templates from the second group are not shown. That's the trouble.
And the question is - how to make the browser send request to the server for each template?
It seems that in the response headers for templates there are no Cache-Control headers. In such case a browser will use a heuristic to decide for how long a response can be cached.
To solve your problem of having always fresh templates fetched in development. You can:
check "Disable cache" in developer tools
set a proper Cache-Control header for resources you care about i.e :
Cache-Control: no-cache
If you want to understand different behaviours triggered by various Cache-Control values I highly recommend this article by Ilya Grigorik.
When using the EventSource API in JavaScript, is there any way to send a request body along with the HTTP request initiating the polling?
I need to send a large blob of JSON to the server at the SSE request so that the server can calculate what events to send to the client. It seems daft to do web-sockets when I don't need it or do weird things with cookies or multiple requests.
I worry i'll run in to length limits on query strings if I bundle the data in to that, which may be likely.
Thanks in advance!
The initial SSE request is a fairly ordinary HTTP GET request, so:
Given that SSE is only supported by modern browsers, the maximum URL length should not be assumed to be the old 255 bytes "for old browsers". Most modern browsers allow longer URLs, with IE providing the lowest cap of ~2k. (granted EventSource is not supported on IE anyway, but there's an XHR polyfill...) However, if by large blob you mean several kilobytes, the URL is not reliable. Proxies could also potentially cause problems.
See:
What is the maximum length of a URL in different browsers?,
Is there any limitation on Url's length in Android's WebView.loadUrl method?,
http://www.benzado.com/blog/post/28/iphone-openurl-limit
You could also store information in one or more cookies which will be sent along with the GET request. This could include a cookie you set on the request for the page that uses SSE, or a cookie you set in javascript (prior to creating your EventSource object). The max size for a cookie is specified as being at least 4096 bytes (which is the whole cookie, so somewhat less for your actual data portion) with at least 20 cookies per hostname supported. Emperical testing appears to bear this out: http://browsercookielimits.x64.me/ Worst case you could possibly chunk the information in multiple cookies.
Larger than that, and I think you need an initial request that uploads the JSON and sends back an ID that is referenced by the SSE request.
It is technically possible, but (strongly) discouraged, to send a body with a GET request. See HTTP GET with request body. The EventSource constructor only takes a URL and so does not directly support this.
As dandavis pointed out, you can compress your JSON.