I want to implement the following logic: fetch the big file (up to 100MB) and store it in the service worker. Then I want to check periodically (say once per hour) and if the remote file changes - need to re-download it.
Otherwise - I don't want to disturb the server.
I was thinking about sending something like 'OPTIONS' request, just to get checksum of the file.
How could I tackle this?
Use HTTP Conditional Requests. Here's a way you could implement this:
Configure the server to include a Last-Modified response header with the date when the requested file was last modified and to respond with 304 Not Modified if the file was not modified since the date indicated by the If-Modified-Since request header.
Make the service worker save the value of the Last-Modified header and use it in the next requests to the file. If the server returns a 304 Not Modified, it means that the cached file is still up-to-date. If the server returns 200 OK, cache that file and store the value of the Last-Modified header.
You can also use ETags with If-None-Match.
This is the standard way to handle caching with HTTP.
Related
I have two questions regarding HTTP cache using the Cache-Control header:
How does the browser identify which request can be fulfilled using the existing cache? Is the browser checking if the endpoint matches? But even requests to the same endpoint can have different body or config, I don't quite understand how does the browser know when to use the cache and when to not use the cache given the time it sends out the request is still within the time frame specified by max-age in the response's cache-control header?
I recently learned that both request and response can set max-age in their own cache-control header. I understand that the request's max-age would tell the server (or any intermediate caches) how fresh of response the client is willing to accept from them. The response max-age (or lack thereof) tells the client how long it can consider that response to be fresh. (feel free to correct me if I am wrong). But what would happen in this scenario:
let's say the response has a max-age for one year and then we send another request for the same resources with max-age being 0. Does that make the browser ignore the cache? or Does the browser accept the cache and not send out the request?
You can get information from this specification. According to the document,
The cache entry contains the headers of the request:
As discussed above, caching servers will by default match future
requests only to requests with exactly the same headers and header
values.
This means that you get one entry in your cache every time you make exactly the same request to the server (the cache can be personal or shared, like in a proxy). In practice, for entities that only cache GET requests, the key can be the URI of the request. By the process of normalization, two very similar requests can share a cache entry. The decision to use the cached entry depends on several factors, as detailed below. The figures in the document explain this very well. Bottom line, max-age only determines freshness, not the behavior of the cache.
According to this specification, the cache is never ignored if the entry exists. Even a fresh entry can be discarded to save disk space, and a stale entry can be kept long after it has expired. The diference is that a stale entry is not directly retrieved. In that case, the caching entity (browser/proxy/load_balancer...) sends a freshness request to the server. The server then decides whether the cached page is fresh.
In summary, if a cached page is fresh according to max-age and whatever other modifiers are used, the caching entity decides that the cached resource will be used. If it is stale, the server decided whether the cached resource can be used.
EDIT after comment:
To understand the difference between max-age sent by the client and the server, we need to dig into the http protocol. In section 5.2.1., It says
5.2.1. Request Cache-Control Directives
5.2.1.1. max-age
Argument syntax:
delta-seconds (see Section 1.2.1)
The "max-age" request directive indicates that the client is
unwilling to accept a response whose age is greater than the
specified number of seconds. Unless the max-stale request directive
is also present, the client is not willing to accept a stale
response.
The language seems to indicate that the server is not forced by the directive, but it is expected to honor it. In your example, this would means that the client directive prevails, as it is more restrictive. The client is saying "I do not want any page cached for more than 0 seconds", and the cache server is suposed to contact the server to fulfill the condition.
Actually, I´ve create a Batch HTTP API that receives a JSON array with many different requests to our backend server. The Batch API just call all of these requests to a load balancer, wait for the return of all of them and return a new JSON to the client.
The client receives a huge JSON array response with its indices in the same position as the request, so it is easy to know what response is addressed for what request.
The motivation for this API was to solve the 5 browser simultaneous connections and improve performance as the Batch API has a much more direct access to the server (we do not have a reverse proxy or a SSL server between then).
The service is running pretty well, but now I have some new requirements as it is gaining more use. First, the service can use a lot of memory as it has a buffer for each request that will only be flushed when all responses are ready (I am using an ordered JSON Array). More, since it can take some time to all requests be delivered, the client will need to wait everything be processed before receiving a single byte.
I am planning change the service to return each response as soon it is available (and solve both issues above). And would like to share and validate my ideas with you:
I will change the response from a JSON response to a multipart response.
The server will include, for every part, the index of the response
The server will flush the response once its available
The client XHR will need to understand multipart content type response and be able to process each part as soon as it is available.
I will create a PoC to validate every step, but at this moment I would like to validate the idea and hear some thoughts about it. Here some doubts I have about the solution:
From what I read, I am in doubbt of that content-type is right for the response. multipart/mixed? multipart/digest?
Can I use an accept request header to identify if the client is able to handle the new service implementation? If so, what is the right accept header for this? My plan is to use the same endpoint but very accept header.
How can I develop a XHR client that is able to process many parts of a single response as soon as they are available? I found some ideias on the Web but I am not entirely confident with then.
I will change the response from a JSON response to a multipart
response.
The server will include, for every part, the index of the
response
The server will flush the response once its available
The
client XHR will need to understand multipart content type response and
be able to process each part as soon as it is available.
The XHR protocol will not support this work flow through a single request from the client. Since XHR relies heavily on the HTTP protocol for communications, XHR follows the HTTP connection rules. The first and most important rule: HTTP connections are always initiated by the client. Another rule: XHR returns the entire content-body or fails.
The implications for your workflow is that each part of the multipart response must be requested individually by the client.
From what I read, I am in doubbt of that content-type is right for the
response. multipart/mixed? multipart/digest?
You should be in doubt as there is no provision in the specfication to do this. The response-type attribute is limited to the empty string (default), "arraybuffer", "blob", "document", "json", and "text". it is possible to set the override MIME type header, but that does not change the response type. Event given that case, the XHR spec is very clear about what it will send back. It is one of the types listed above as documented here.
Can I use an accept
request header to identify if the client is able to handle the new
service implementation? If so, what is the right accept header for
this? My plan is to use the same endpoint but very accept header.
Custom HTTP headers are designed to assist us in telling the server what our capabilities are on the client. This is easily done. it doesn't necessarily have to be in the accept header (as that also is a defined list of MIME types).
How
can I develop a XHR client that is able to process many parts of a
single response as soon as they are available? I found some ideias on
the Web but I am not entirely confident with then.
XHR is processed natively by the client and cannot be overridden for all sorts of security reasons. So this is unlikely to be available as a solution for this reason.
Note: ordinarily one might suggest the use of a custom version of Chromium, but your constraints do not allow for that.
I'm using the Google "Page Speed" plug-in for Firefox to access my web site.
Some of the components on my page is indicated as HTTP status:
200
200 (cache)
304
By Google's "Page Speed".
What I'm confused about is the difference between 200 (cache) and 304.
I've refreshed the page multiple times (but have not cleared my cache) and it always seems that my favicon.ico and a few images are status=200 (cache) while some other images are http status 304.
I don't understand why the difference.
UPDATE:
Using Google "Page Speed", I receive a "200 (cache)" for http://example.com/favicon.ico as well as http://cdn.example.com/js/ga.js
But, I receive a http status "304" for http://cdn.example.com/js/combined.min.js
I don't understand why I have two JavaScript files located in the same directory /js/, one returning a http status 304 and the other returning a 200 (cache) status code.
The items with code "200 (cache)" were fulfilled directly from your browser cache, meaning that the original requests for the items were returned with headers indicating that the browser could cache them (e.g. future-dated Expires or Cache-Control: max-age headers), and that at the time you triggered the new request, those cached objects were still stored in local cache and had not yet expired.
304s, on the other hand, are the response of the server after the browser has checked if a file was modified since the last version it had cached (the answer being "no").
For most optimal web performance, you're best off setting a far-future Expires: or Cache-Control: max-age header for all assets, and then when an asset needs to be changed, changing the actual filename of the asset or appending a version string to requests for that asset. This eliminates the need for any request to be made unless the asset has definitely changed from the version in cache (no need for that 304 response). Google has more details on correct use of long-term caching.
200 (cache) means Firefox is simply using the locally cached version. This is the fastest because no request to the Web server is made.
304 means Firefox is sending a "If-Modified-Since" conditional request to the Web server. If the file has not been updated since the date sent by the browser, the Web server returns a 304 response which essentially tells Firefox to use its cached version. It is not as fast as 200 (cache) because the request is still sent to the Web server, but the server doesn't have to send the contents of the file.
To your last question, I don't know why the two JavaScript files in the same directory are returning different results.
This threw me for a long time too. The first thing I'd verify is that you're not reloading the page by clicking the refresh button, that will always issue a conditional request for resources and will return 304s for many of the page elements. Instead go up to the url bar select the page and hit enter as if you had just typed in the same URL again, that will give you a better indicator of what's being cached properly. This article does a great job explaining the difference between conditional and unconditional requests and how the refresh button affects them:
http://blogs.msdn.com/b/ieinternals/archive/2010/07/08/technical-information-about-conditional-http-requests-and-the-refresh-button.aspx
HTTP 304 is "not modified". Your web server is basically telling the browser "this file hasn't changed since the last time you requested it." Whereas an HTTP 200 is telling the browser "here is a successful response" - which should be returned when it's either the first time your browser is accessing the file or the first time a modified copy is being accessed.
For more info on status codes check out http://en.wikipedia.org/wiki/List_of_HTTP_status_codes.
For your last question, why ? I'll try to explain with what I know
A brief explanation of those three status codes in layman's terms.
200 - success (browser requests and get file from server)
If caching is enabled in the server
200 (from memory cache) - file found in browser, so browser is not going request from server
304 - browser request a file but it is rejected by server
For some files browser is deciding to request from server and for some it's deciding to read from stored (cached) files. Why is this ? Every files has an expiry date, so
If a file is not expired then the browser will use from cache (200 cache).
If file is expired, browser requests server for a file. Server check file in both places (browser and server). If same file found, server refuses the request. As per protocol browser uses existing file.
look at this nginx configuration
location / {
add_header Cache-Control must-revalidate;
expires 60;
etag on;
...
}
Here the expiry is set to 60 seconds, so all static files are cached for 60 seconds. So if u request a file again within 60 seconds browser will read from memory (200 memory). If u request after 60 seconds browser will request server (304).
I assumed that the file is not changed after 60 seconds, in that case you would get 200 (ie, updated file will be fetched from server).
So, if the servers are configured with different expiring and caching headers (policies), the status may differ.
In your case you are using cdn, the main purpose of cdn is high availability and fast delivery. Therefore they use multiple servers. Even though it seems like files are in same directory, cdn might use multiple servers to provide u content, if those servers have different configurations. Then these status can change. Hope it helps.
When fetching data with a polyfilled fetch, I'm getting a 304 response as described here. Fetch apparently automatically generates the appropriate If-None-Match header for my request, and my backend replies with a 304 (data is already in cache). The request isn't properly fullfilled though.
I couldn't find any documentation on where this cached data can be found though. So should the cached data be retrieved automatically? I could set a 'cache-control': 'no-cache' header, but I'd prefer to use caching of course.
If I understand what you're asking, if the server returns a 304 for the request that the browser sent to it, then the browser fulfills the request from it's cache (depends on the browser) just as it does for any normal HTTP Request or XMLHTTP Request.
If however you do not want to get content from the cache, and you want to force a brand new load of content, you will want to set the applicable 'cache-control': 'no-cache' header AND ensure that the URL you are trying to fetch is unique (either by path or parameters). Often the easiest way to do this is append a dummy param with the current timestamp. e.g. ...&_=1483970310923
I'm reading this great article on caching and there is the following there:
Validators are very important; if one isn’t present, and there isn’t
any freshness information (Expires or Cache-Control) available, caches
will not store a representation at all.
The most common validator is the time that the document last changed,
as communicated in Last-Modified header. When a cache has a
representation stored that includes a Last-Modified header, it can use
it to ask the server if the representation has changed since the last
time it was seen, with an If-Modified-Since request.
So, I'm wondering whether browser continues to send requests (for example HEAD) for a resource even if I specified Cache-Control: max-age=3600? If it doesn't, than what's the point in this header? Is it used after the max-age time passes?
The Cache-Control: max-age=3600 header means that the browser will cache the response for up to 3600 seconds. After that time has passed it may no longer serve the response without first confirming that it is still fresh.
In order to this, the browser can either:
Fetch the full resource with a normal GET request (transfers the whole response body again)
Or perform a revalidation based on an ETag (If-None-Match) or the Last-Modified header (If-Modified-Since), i.e. the client only fetches the response body if it has actually changed. This is of course only possible if the validator was present in the original response.
In short: the reason to use both max-age and a cache validator is to first cache the response for some time and then perform a bandwidth-saving revalidation to confirm the resource's freshness.