We Keep Coding

How to save cache storage date with JavaScript Cache API?

I'm using JavaScript's Cache Web API to store responses from my server to requests made by my client application. Moreover, I need a way to programmatically remove them based on when the request was originally made. Here is the code I use to store the responses:
/** Searches for the corresponding cache for the given request. If found, returns
* the cached response. Otherwise, performs the fetch request and adds the response
* to cache. Returns the HTTP response.
*/
export async function fetchCachedData(request: Request) {
const cache = await caches.open(CACHE_NAME);
// Check if response is already cached
const cachedResponse = await cache.match(request);
if (cachedResponse) {
console.debug("Using cached response for", request.url);
return cachedResponse.clone();
}
// Fetch new response
console.debug("Fetching", request.url, "...");
const response = await fetchFromAPI(request);
const responseDate = new Date().getTime().toString();
response.headers.set("Date", responseDate);
// Cache the new response
if (
response.ok /*&& response.clone().headers.get("Cache-Control") !== "no-store"*/
) {
await cache.put(request, response.clone());
console.info("Cached response as", response.url);
}
return response.clone();
}
This approach seems to work on browsers like Firefox, however on Chrome I get an error telling me that headers is read-only:
TypeError: Failed to execute 'set' on 'Headers': Headers are immutable
I have also tried setting the Date header on the server side, however it appears that not all headers set in the express-based app are honoured when cloning and retrieving them from the cache. This is why I wish to manually set the request date when the response is retrieved on the client side.
I don't necessarily need the date to be stored in the cached response's headers, that's just the way I have my cache filtering code set up. Ideally, the request date should be stored somewhere in the response object so that it is preserved when using clone() and is present in the cache.

I've solved this by using the pragma HTTP header -- it appears to be unused as of the HTTP/1.1 spec, however when set on the server side it is preserved in the headers of the response object obtained after making a request from the fetch() API.
Server code (Express.js):
export function headerMiddleware(
_req: Request,
res: Response,
next: NextFunction
) {
const now = new Date().getTime();
res.setHeader("pragma", now);
next();
}
This implementation is probably discouraged since pragma is a deprecated header and its original intention was to signal if no-cache is how the response is to be handled, however when I set its value to a numeric string there appear to be no errors and the solution runs smoothly. Ideally, I'd use the Date header, however if I set that on the server side then the header is removed upon inspection on the client.

Trouble with Cloudflares Worker Cache API

I’ve now spent countless hours trying to get the cache API to cache a simple request. I had it working once in between but forgot to add something to the cache key, and now its not working anymore. Needless to say, cache.put() not having a return value that specifies if the request was actually cached or not does not exactly help and I am left with trial and error. Can someone maybe give me a hint on what I’m doing wrong and what is actually required? I’ve read all the documentation more than 3 times now and I’m at a loss…
Noteworthy maybe is that this REST endpoint sets pragma: no-cache and everything else cache-related to no-cache, but i want to forcibly cache the response anyway which is why I tried to completely re-write the headers before caching, but it still isn’t working (not matching or not storing, no one knows…)
async function apiTest(token, url) {
let apiCache = await caches.open("apiResponses");
let request = new Request(
new URL("https://api.mysite.com/api/"+url),
{
headers: {
"Authorization": "Bearer "+token,
}
}
)
// Check if the response is already in the cloudflare cache
let response = await apiCache.match(request);
if (response) {
console.log("Serving from cache");
}
if (!response) {
// if not, ask the origin if the permission is granted
response = await fetch(request);
// cache response in cloudflare cache
response = new Response(response.body, {
status: response.status,
statusText: response.statusText,
headers: {
"Cache-Control": "max-age=900",
"Content-Type": response.headers.get("Content-Type"),
}
});
await apiCache.put(request, response.clone());
}
return response;
}
Thanks in advance for any help, I've asked the same question on the Cloudflare community first and not received an answer in 2 weeks

This might be related to your use of caches.default, instead of opening a private cache with caches.open("whatever"). When you use caches.default, you are sharing the same cache that fetch() itself uses. So when your worker runs, your worker checks the cache, then fetch() checks the cache, then fetch() later writes the cache, and then your worker also writes the same cache entry. Since the write operations in particular happen asynchronously (as the response streams through), it's quite possible that they are overlapping and the cache is getting confused and tossing them all out.
To avoid this, you should open a private cache namespace. So, replace this line:
let cache = caches.default;
with:
let cache = await caches.open("whatever");
(This await always completes immediately; it's only needed because the Cache API standard insists that this method is asynchronous.)
This way, you are reading and writing a completely separate cache entry from the one that fetch() itself reads/writes.
The use case for caches.default is when you intentionally want to operate on exactly the cache entry that fetch() would also use, but I don't think you need to do that here.
EDIT: Based on conversation below, I now suspect that the presence of the Authorization header was causing the cache to refuse to store the response. But, using a custom cache namespace (as described above) means that you can safely cache the value using a Request that doesn't have that header, because you know the cached response can only be accessed by the Worker via the cache API. It sounds like this approach worked in your case.

webRequest API: How to get the requestId of a new request?

The chrome.webRequest API has the concept of a request ID (source: Chrome webRequest documention):
Request IDs
Each request is identified by a request ID. This ID is unique within a browser session and the context of an extension. It remains constant during the the life cycle of a request and can be used to match events for the same request. Note that several HTTP requests are mapped to one web request in case of HTTP redirection or HTTP authentication.
You can use it to correlate the requests even across redirects. But how do you initially get hold off the id when start a new request with fetch or XMLHttpRequest?
So far, I have not found anything better than to use the URL of the request as a way to make the initial link between the new request and the requestId. However, if there are overlapping requests to the same resource, this is not reliable.
Questions:
If you make a new request (either with fetch or XMLHttpRequest), how do you reliably get access to the requestId?
Does the fetch API or XMLHttpRequest API allow access to the requestId?
What I want to do is to use the functionality provided by the webRequest API to modify a single request, but I want to make sure that I do not accidentally modify other pending requests.

To the best of my knowledge, there is no direct support in the fetch or XHMLHttpRequest API. Also I'm not aware of completely reliable way to get hold of the requestId.
What I ended up doing was installing a onBeforeRequest listener, storing the requestId, and then immediately removing the listener again. For instance, it could look like this:
function makeSomeRequest(url) {
let listener;
const removeListener = () => {
if (listener) {
chrome.webRequest.onBeforeRequest.removeListener(listener);
listener = null;
}
};
let requestId;
listener = (details) => {
if (!requestId && urlMatches(details.url, url)) {
requestId = details.requestId;
removeListener();
}
};
chrome.webRequest.onBeforeRequest.addListener(listener, { urls: ['<all_urls>'] });
// install other listeners, which can then use the stored "requestId"
// ...
// finally, start the actual request, for instance
const promise = fetch(url).then(doSomething);
// and make sure to always clean up the listener
promise.then(removeListener, removeLister);
}
It is not perfect, and matching the URL is a detail that I left open. You could simply compare whether the details.url is identical to url:
function urlMatches(url1, url2) {
return url1 === url2;
}
Note that it is not guaranteed that you see the identical URL, for instance, if make a request against http://some.domain.test, you will see http://some.domain.test/ in your listener (see my other question about the details). Or http:// could have been replaced by https:// (here I'm not sure, but it could be because of other extensions like HTTPS Everywhere).
That is why the code above should only be seen as a sketch of the idea. It seems to work good enough in practice, as long as you do not start multiple requests to the identical URL. Still, I would be interested in learning about a better way to approach the problem.

Cannot construct a Request with a Request whose mode is 'navigate' and a non-empty RequestInit

Consider this sample index.html file.
<!DOCTYPE html>
<html><head><title>test page</title>
<script>navigator.serviceWorker.register('sw.js');</script>
</head>
<body>
<p>test page</p>
</body>
</html>
Using this Service Worker, designed to load from the cache, then fallback to the network if necessary.
cacheFirst = (request) => {
var mycache;
return caches.open('mycache')
.then(cache => {
mycache = cache;
cache.match(request);
})
.then(match => match || fetch(request, {credentials: 'include'}))
.then(response => {
mycache.put(request, response.clone());
return response;
})
}
addEventListener('fetch', event => event.respondWith(cacheFirst(event.request)));
This fails badly on Chrome 62. Refreshing the HTML fails to load in the browser at all, with a "This site can't be reached" error; I have to shift refresh to get out of this broken state. In the console, it says:
Uncaught (in promise) TypeError: Failed to execute 'fetch' on 'ServiceWorkerGlobalScope': Cannot construct a Request with a Request whose mode is 'navigate' and a non-empty RequestInit.
"construct a Request"?! I'm not constructing a request. I'm using the event's request, unmodified. What am I doing wrong here?

Based on further research, it turns out that I am constructing a Request when I fetch(request, {credentials: 'include'})!
Whenever you pass an options object to fetch, that object is the RequestInit, and it creates a new Request object when you do that. And, uh, apparently you can't ask fetch() to create a new Request in navigate mode and a non-empty RequestInit for some reason.
In my case, the event's navigation Request already allowed credentials, so the fix is to convert fetch(request, {credentials: 'include'}) into fetch(request).
I was fooled into thinking I needed {credentials: 'include'} due to this Google documentation article.
When you use fetch, by default, requests won't contain credentials such as cookies. If you want credentials, instead call:
fetch(url, {
credentials: 'include'
})
That's only true if you pass fetch a URL, as they do in the code sample. If you have a Request object on hand, as we normally do in a Service Worker, the Request knows whether it wants to use credentials or not, so fetch(request) will use credentials normally.

https://developers.google.com/web/ilt/pwa/caching-files-with-service-worker
var networkDataReceived = false;
// fetch fresh data
var networkUpdate = fetch('/data.json').then(function(response) {
return response.json();
}).then(function(data) {
networkDataReceived = true;
updatePage(data);
});
// fetch cached data
caches.match('mycache').then(function(response) {
if (!response) throw Error("No data");
return response.json();
}).then(function(data) {
// don't overwrite newer network data
if (!networkDataReceived) {
updatePage(data);
}
}).catch(function() {
// we didn't get cached data, the network is our last hope:
return networkUpdate;
}).catch(showErrorMessage).then(console.log('error');
Best example of what you are trying to do, though you have to update your code accordingly. The web example is taken from under Cache then network.
for the service worker:
self.addEventListener('fetch', function(event) {
event.respondWith(
caches.open('mycache').then(function(cache) {
return fetch(event.request).then(function(response) {
cache.put(event.request, response.clone());
return response;
});
})
);
});

Problem
I came across this problem when trying to override fetch for all kinds of different assets. navigate mode was set for the initial Request that gets the index.html (or other html) file; and I wanted the same caching rules applied to it as I wanted to several other static assets.
Here are the two things I wanted to be able to accomplish:
When fetching static assets, I want to sometimes be able to override the url, meaning I want something like: fetch(new Request(newUrl))
At the same time, I want them to be fetched just as the sender intended; meaning I want to set second argument of fetch (i.e. the RequestInit object mentioned in the error message) to the originalRequest itself, like so: fetch(new Request(newUrl), originalRequest)
However the second part is not possible for requests in navigate mode (i.e. the initial html file); at the same time it is not needed, as explained by others, since it will already keep it's cookies, credentials etc.
Solution
Here is my work-around: a versatile fetch that...
can override the URL
can override RequestInit config object
works with both, navigate as well as any other requests
function fetchOverride(originalRequest, newUrl) {
const fetchArgs = [new Request(newUrl)];
if (request.mode !== 'navigate') {
// customize the request only if NOT in navigate mode
// (since in "navigate" that is not allowed)
fetchArgs.push(request);
}
return fetch(...fetchArgs);
}

In my case I was contructing a request from a serialized form in a service worker (to handle failed POSTs). In the original request it had the mode attribute set, which is readonly, so before one reconstructs the request, delete the mode attribute:
delete serializedRequest["mode"];
request = new Request(serializedRequest.url, serializedRequest);

Can service workers cache POST requests?

I tried to cache a POST request in a service worker on fetch event.
I used cache.put(event.request, response), but the returned promise was rejected with TypeError: Invalid request method POST..
When I tried to hit the same POST API, caches.match(event.request) was giving me undefined.
But when I did the same for GET methods, it worked: caches.match(event.request) for a GET request was giving me a response.
Can service workers cache POST requests?
In case they can't, what approach can we use to make apps truly offline?

You can't cache POST requests using the Cache API. See https://w3c.github.io/ServiceWorker/#cache-put (point 4).
There's a related discussion in the spec repository: https://github.com/slightlyoff/ServiceWorker/issues/693
An interesting solution is the one presented in the ServiceWorker Cookbook: https://serviceworke.rs/request-deferrer.html
Basically, the solution serializes requests to IndexedDB.

I've used the following solution in a recent project with a GraphQL API: I cached all responses from API routes in an IndexedDB object store using a serialized representation of the Request as cache key. Then I used the cache as a fallback if the network was unavailable:
// ServiceWorker.js
self.addEventListener('fetch', function(event) {
// We will cache all POST requests to matching URLs
if(event.request.method === "POST" || event.request.url.href.match(/*...*/)){
event.respondWith(
// First try to fetch the request from the server
fetch(event.request.clone())
// If it works, put the response into IndexedDB
.then(function(response) {
// Compute a unique key for the POST request
var key = getPostId(request);
// Create a cache entry
var entry = {
key: key,
response: serializeResponse(response),
timestamp: Date.now()
};
/* ... save entry to IndexedDB ... */
// Return the (fresh) response
return response;
})
.catch(function() {
// If it does not work, return the cached response. If the cache does not
// contain a response for our request, it will give us a 503-response
var key = getPostId(request);
var cachedResponse = /* query IndexedDB using the key */;
return response;
})
);
}
})
function getPostId(request) {
/* ... compute a unique key for the request incl. it's body: e.g. serialize it to a string */
}
Here is the full code for my specific solution using Dexie.js as IndexedDB-wrapper. Feel free to use it!

If you are talking about form data, then you could intercept the fetch event and read the form data in a similar way as below and then save the data in indexedDB.
//service-worker.js
self.addEventListener('fetch', function(event) {
if(event.request.method === "POST"){
var newObj = {};
event.request.formData().then(formData => {
for(var pair of formData.entries()) {
var key = pair[0];
var value = pair[1];
newObj[key] = value;
}
}).then( ...save object in indexedDB... )
}
})

Another approach to provide a full offline experience can be obtained by using Cloud Firestore offline persistence.
POST / PUT requests are executed on the local cached database and then automatically synchronised to the server as soon as the user restores its internet connectivity (note though that there is a limit of 500 offline requests).
Another aspect to be taken into account by following this solution is that if multiple users have offline changes that get concurrently synchronised, there is no warranty that the changes will be executed in the right chronological order on the server as Firestore uses a first come first served logic.

According to https://w3c.github.io/ServiceWorker/#cache-put (point 4).
if(request.method !== "GET") {
return Promise.reject('no-match')
}