I tried to cache a POST request in a service worker on fetch event.
I used cache.put(event.request, response), but the returned promise was rejected with TypeError: Invalid request method POST..
When I tried to hit the same POST API, caches.match(event.request) was giving me undefined.
But when I did the same for GET methods, it worked: caches.match(event.request) for a GET request was giving me a response.
Can service workers cache POST requests?
In case they can't, what approach can we use to make apps truly offline?
You can't cache POST requests using the Cache API. See https://w3c.github.io/ServiceWorker/#cache-put (point 4).
There's a related discussion in the spec repository: https://github.com/slightlyoff/ServiceWorker/issues/693
An interesting solution is the one presented in the ServiceWorker Cookbook: https://serviceworke.rs/request-deferrer.html
Basically, the solution serializes requests to IndexedDB.
I've used the following solution in a recent project with a GraphQL API: I cached all responses from API routes in an IndexedDB object store using a serialized representation of the Request as cache key. Then I used the cache as a fallback if the network was unavailable:
// ServiceWorker.js
self.addEventListener('fetch', function(event) {
// We will cache all POST requests to matching URLs
if(event.request.method === "POST" || event.request.url.href.match(/*...*/)){
event.respondWith(
// First try to fetch the request from the server
fetch(event.request.clone())
// If it works, put the response into IndexedDB
.then(function(response) {
// Compute a unique key for the POST request
var key = getPostId(request);
// Create a cache entry
var entry = {
key: key,
response: serializeResponse(response),
timestamp: Date.now()
};
/* ... save entry to IndexedDB ... */
// Return the (fresh) response
return response;
})
.catch(function() {
// If it does not work, return the cached response. If the cache does not
// contain a response for our request, it will give us a 503-response
var key = getPostId(request);
var cachedResponse = /* query IndexedDB using the key */;
return response;
})
);
}
})
function getPostId(request) {
/* ... compute a unique key for the request incl. it's body: e.g. serialize it to a string */
}
Here is the full code for my specific solution using Dexie.js as IndexedDB-wrapper. Feel free to use it!
If you are talking about form data, then you could intercept the fetch event and read the form data in a similar way as below and then save the data in indexedDB.
//service-worker.js
self.addEventListener('fetch', function(event) {
if(event.request.method === "POST"){
var newObj = {};
event.request.formData().then(formData => {
for(var pair of formData.entries()) {
var key = pair[0];
var value = pair[1];
newObj[key] = value;
}
}).then( ...save object in indexedDB... )
}
})
Another approach to provide a full offline experience can be obtained by using Cloud Firestore offline persistence.
POST / PUT requests are executed on the local cached database and then automatically synchronised to the server as soon as the user restores its internet connectivity (note though that there is a limit of 500 offline requests).
Another aspect to be taken into account by following this solution is that if multiple users have offline changes that get concurrently synchronised, there is no warranty that the changes will be executed in the right chronological order on the server as Firestore uses a first come first served logic.
According to https://w3c.github.io/ServiceWorker/#cache-put (point 4).
if(request.method !== "GET") {
return Promise.reject('no-match')
}
Related
I am using service worker to provide a fallback page that shows the user is offline. The service worker during interception of request, fetches the same request and on error on fetching, provides response for 'offline.html' request from the cache. A small snippet of doing this is.
self.addEventListener("fetch", (event) => {
event.respondWith(
caches.match(event.request).then(() => {
return fetch(event.request).catch((err) => {
return caches.match("offline.html");
});
})
);
});
now if the offline html has other request, probably to its css files, or images, how do I load them from cache. I've tried doing the following:
self.addEventListener("fetch", (event) => {
event.respondWith(
caches.match(event.request).then(() => {
return fetch(event.request).catch((err) => {
let url = event.request.url;
if(url.endsWith('.css')) return caches.match('offline.css');
if(url.endsWith('.jpg') || url.endsWith('.png')) return caches.match('images/banner.jpg');
return caches.match("offline.html");
});
})
);
});
But is there a better way of doing this? Is there a standard way of doing this?
First off, I would recommend checking to see whether event.request.destination === 'document' before you decide whether or not to use offline.html as the fallback content. That ensure that you're not accidentally returning an HTML document to satisfy, say, a random API request that happens to fail.
Additionally, your current code includes caches.match(event.request) but then doesn't actually used the cached response, which is likely not what you intend.
That said, let's walk through what I think is your desired logic:
Your service worker attempts to make a request against the network.
If that request returns a valid response, use it, and you'd done.
If that request fails, then:
If it was a navigation request, regardless of the destination URL, use the cached offline.html for the response.
Otherwise, for non-navigation requests (like CSS or JS requests), use the cached entry matching the desired URL for the response.
Here's a service worker that implements that. You'll need to ensure that the CSS, JS, and offline.html assets are cached during service worker installation; this just includes the fetch handler logic.
self.addEventListener('install', (event) => {
event.waitUntil(
/* Cache your offline.html and the CSS and JS it uses here. */
);
});
async function fetchLogic(request) {
try {
// If the network request succeeds, just use
// that as the response.
return await fetch(request);
} catch(error) {
// Otherwise, implement fallback logic.
if (request.mode === 'navigate') {
// Use the cached fallback.html for failed navigations.
return await caches.match('offline.html');
}
// Otherwise, return a cached copy of the actual
// subresource that was requested.
// If there's a cache miss for that given URL, you'll
// end up with a NetworkError, just like you would if
// there were no service worker involvement.
return await caches.match(request.url);
}
}
self.addEventListener('fetch', (event) => {
event.respondWith(fetchLogic(event.request));
});
There's also some formal guidance in this article.
Consider this sample index.html file.
<!DOCTYPE html>
<html><head><title>test page</title>
<script>navigator.serviceWorker.register('sw.js');</script>
</head>
<body>
<p>test page</p>
</body>
</html>
Using this Service Worker, designed to load from the cache, then fallback to the network if necessary.
cacheFirst = (request) => {
var mycache;
return caches.open('mycache')
.then(cache => {
mycache = cache;
cache.match(request);
})
.then(match => match || fetch(request, {credentials: 'include'}))
.then(response => {
mycache.put(request, response.clone());
return response;
})
}
addEventListener('fetch', event => event.respondWith(cacheFirst(event.request)));
This fails badly on Chrome 62. Refreshing the HTML fails to load in the browser at all, with a "This site can't be reached" error; I have to shift refresh to get out of this broken state. In the console, it says:
Uncaught (in promise) TypeError: Failed to execute 'fetch' on 'ServiceWorkerGlobalScope': Cannot construct a Request with a Request whose mode is 'navigate' and a non-empty RequestInit.
"construct a Request"?! I'm not constructing a request. I'm using the event's request, unmodified. What am I doing wrong here?
Based on further research, it turns out that I am constructing a Request when I fetch(request, {credentials: 'include'})!
Whenever you pass an options object to fetch, that object is the RequestInit, and it creates a new Request object when you do that. And, uh, apparently you can't ask fetch() to create a new Request in navigate mode and a non-empty RequestInit for some reason.
In my case, the event's navigation Request already allowed credentials, so the fix is to convert fetch(request, {credentials: 'include'}) into fetch(request).
I was fooled into thinking I needed {credentials: 'include'} due to this Google documentation article.
When you use fetch, by default, requests won't contain credentials such as cookies. If you want credentials, instead call:
fetch(url, {
credentials: 'include'
})
That's only true if you pass fetch a URL, as they do in the code sample. If you have a Request object on hand, as we normally do in a Service Worker, the Request knows whether it wants to use credentials or not, so fetch(request) will use credentials normally.
https://developers.google.com/web/ilt/pwa/caching-files-with-service-worker
var networkDataReceived = false;
// fetch fresh data
var networkUpdate = fetch('/data.json').then(function(response) {
return response.json();
}).then(function(data) {
networkDataReceived = true;
updatePage(data);
});
// fetch cached data
caches.match('mycache').then(function(response) {
if (!response) throw Error("No data");
return response.json();
}).then(function(data) {
// don't overwrite newer network data
if (!networkDataReceived) {
updatePage(data);
}
}).catch(function() {
// we didn't get cached data, the network is our last hope:
return networkUpdate;
}).catch(showErrorMessage).then(console.log('error');
Best example of what you are trying to do, though you have to update your code accordingly. The web example is taken from under Cache then network.
for the service worker:
self.addEventListener('fetch', function(event) {
event.respondWith(
caches.open('mycache').then(function(cache) {
return fetch(event.request).then(function(response) {
cache.put(event.request, response.clone());
return response;
});
})
);
});
Problem
I came across this problem when trying to override fetch for all kinds of different assets. navigate mode was set for the initial Request that gets the index.html (or other html) file; and I wanted the same caching rules applied to it as I wanted to several other static assets.
Here are the two things I wanted to be able to accomplish:
When fetching static assets, I want to sometimes be able to override the url, meaning I want something like: fetch(new Request(newUrl))
At the same time, I want them to be fetched just as the sender intended; meaning I want to set second argument of fetch (i.e. the RequestInit object mentioned in the error message) to the originalRequest itself, like so: fetch(new Request(newUrl), originalRequest)
However the second part is not possible for requests in navigate mode (i.e. the initial html file); at the same time it is not needed, as explained by others, since it will already keep it's cookies, credentials etc.
Solution
Here is my work-around: a versatile fetch that...
can override the URL
can override RequestInit config object
works with both, navigate as well as any other requests
function fetchOverride(originalRequest, newUrl) {
const fetchArgs = [new Request(newUrl)];
if (request.mode !== 'navigate') {
// customize the request only if NOT in navigate mode
// (since in "navigate" that is not allowed)
fetchArgs.push(request);
}
return fetch(...fetchArgs);
}
In my case I was contructing a request from a serialized form in a service worker (to handle failed POSTs). In the original request it had the mode attribute set, which is readonly, so before one reconstructs the request, delete the mode attribute:
delete serializedRequest["mode"];
request = new Request(serializedRequest.url, serializedRequest);
I have built a web app using Backbone.js and it has lots of calls to a RESTful service and it works like a charm.
I tried adding a ServiceWorker to cache all the previous calls so they'll be available offline.
What I actually get is that the calls I do for the first time, dies with this error:
Failed to load resource: net::ERR_FAILED
However on page reload, I get it's cached data
My service worker fetch:
self.addEventListener('fetch', function(e) {
// e.respondWidth Responds to the fetch event
e.respondWith(
// Check in cache for the request being made
caches.match(e.request)
.then(function(response) {
// If the request is in the cache
if ( response ) {
console.log("[ServiceWorker] Found in Cache", e.request.url, response);
// Return the cached version
return response;
}
// If the request is NOT in the cache, fetch and cache
var requestClone = e.request.clone();
fetch(requestClone)
.then(function(response) {
if ( !response ) {
console.log("[ServiceWorker] No response from fetch ")
return response;
}
var responseClone = response.clone();
// Open the cache
caches.open(cacheName).then(function(cache) {
// Put the fetched response in the cache
cache.put(e.request, responseClone);
console.log('[ServiceWorker] New Data Cached', e.request.url);
// Return the response
return response;
}); // end caches.open
console.log("Response is.. ?", response)
return response;
})
.catch(function(err) {
console.log('[ServiceWorker] Error Fetching & Caching New Data', err);
});
}) // end caches.match(e.request)
); // end e.respondWith
});
edit:
I don't think there is a need for any Backbone.js web app code.
I use the fetch method from Backbone.js models and collections.
calls like
https://jsonplaceholder.typicode.com/posts/1
and
https://jsonplaceholder.typicode.com/posts/2
will replay show this error on first time. after refreshing the page, i do have this info without requesting. all from cache.
and all other request that i still didn't do, will stay error
i solved it after searching more.
Backbone.js my views in the Web app used to do:
this.listenTo(this.collection,"reset",this.render);
this.listenTo(this.collection,"add",this.addCollectionItem);
this.listenTo(this.collection,"error", this.errorRender);
while my Service worker is returning Promises.
I had to change my some code my Web app views to something like this:
this.collection.fetch({},{reset:true})
.then(_.bind(this.render, this))
.fail(_.bind(this.errorRender,this))
more or less...
The only problem I see is that when the request is not in the cache, then you do a fetch, but you do not return the result of that fetch to the enclosing then handler. You need to add a return so that you have:
return fetch(requestClone)
.then(function(response) {
None of the data provided by the return statements inside your then handler for the fetch will get transferred up the chain otherwise.
I also see that you do not return the promise provided by caches.open(cacheName).then.... This may be fine if you want to decouple saving a response in the cache from returning a result up the chain, but at the very least I'd put a comment saying that that's what I'm doing here rather than leave it to the reader to figure out whether a return statement is missing by accident, or it was done on purpose.
I am using Vue.js and Choices.js javascript plugin and I have to dynamically populate values of two select fields via ajax.
What I am trying achieve is initate a get request at page load and populate the universities select, and after a value in universities select is chosen start a new getrequest to populate the faculties select.
What is happening is that when I pick the university for the first time, everything will work normally. For example if I pick a university option with value="1" an ajax get request will be sent to /faculties?university_id=1.The console log will print onChange startedso we are sure the method is running correctly; the appropriate v-model="selectedUniversity"is updating too.
If I now change the value of the select field again, the ajax function won't be called anymore and no additional requests will be done to the server. The console.logwill still run, and the v-modelis still being updated. Does anyone understand what is going on here?
var Choices = require('choices.js');
module.exports = {
data: function() {
return {
selectedUniversity: '',
selectedFaculty: '',
universities: {},
faculties: {}
}
},
mounted: function () {
var self = this;
var universitySelect = new Choices(document.getElementById('university'));
universitySelect.ajax(function(callback) {
fetch('/universities')
.then(function(response) {
response.json().then(function(data) {
callback(data, 'id', 'name');
self.universities = data;
});
})
.catch(function(error) {
console.log(error);
});
});
},
methods: {
onChange: function () {
console.log("onChange started");
var self = this;
var url = '/faculties?university_id=' + self.selectedUniversity;
var facultySelect = new Choices(document.getElementById('faculty'));
//This part below only runs the first time when the university select is selected
facultySelect.ajax(function(callback) {
fetch(url)
.then(function(response) {
response.json().then(function(data) {
callback(data, 'id', 'name');
self.faculties = data;
});
})
.catch(function(error) {
console.log(error);
});
});
}
}
}
The Headers are set like this:
I think your request URL /faculties?university_id=1 is cached and that's why it worked on first time and second time, the response is coming from the cached response.
In your fetch API, set cache mode to ignore the cached response,
fetch(url, {cache: "no-store"}).then(....)
For complete list of cache modes for fetch() API,
https://hacks.mozilla.org/2016/03/referrer-and-cache-control-apis-for-fetch/
In case if above link is unavailable,
Fetch cache control APIs
The idea behind this API is specifying a caching policy for fetch to explicitly indicate how and when the browser HTTP cache should be consulted. It’s important to have a good understanding of the HTTP caching semantics in order to use these most effectively. There are many good articles on the web such as this one that describe these semantics in detail. There are currently five different policies that you can choose from.
“default” means use the default behavior of browsers when downloading resources. The browser first looks inside the HTTP cache to see if there is a matching request. If there is, and it is fresh, it will be returned from fetch(). If it exists but is stale, a conditional request is made to the remote server and if the server indicates that the response has not changed, it will be read from the HTTP cache. Otherwise it will be downloaded from the network, and the HTTP cache will be updated with the new response.
“no-store” means bypass the HTTP cache completely. This will make the browser not look into the HTTP cache on the way to the network, and never store the resulting response in the HTTP cache. Using this cache mode, fetch() will behave as if no HTTP cache exists.
“reload” means bypass the HTTP cache on the way to the network, but update it with the newly downloaded response. This will cause the browser to never look inside the HTTP cache on the way to the network, but update the HTTP cache with the downloaded response. Future requests can use that updated response if appropriate.
“no-cache” means always validate a response that is in the HTTP cache even if the browser thinks that it’s fresh. This will cause the browser to look for a matching request in the HTTP cache on the way to the network. If such a request is found, the browser always creates a conditional request to validate it even if it thinks that the response should be fresh. If a matching cached entry is not found, a normal request will be made. After a response has been downloaded, the HTTP cache will always be updated with that response.
“force-cache” means that the browser will always use a cached response if a matching entry is found in the cache, ignoring the validity of the response. Thus even if a really old version of the response is found in the cache, it will always be used without validation. If a matching entry is not found in the cache, the browser will make a normal request, and will update the HTTP cache with the downloaded response.
Let’s look at a few examples of how you can use these cache modes.
// Download a resource with cache busting, to bypass the cache
// completely.
fetch("some.json", {cache: "no-store"})
.then(function(response) { /* consume the response */ });
// Download a resource with cache busting, but update the HTTP
// cache with the downloaded resource.
fetch("some.json", {cache: "reload"})
.then(function(response) { /* consume the response */ });
// Download a resource with cache busting when dealing with a
// properly configured server that will send the correct ETag
// and Date headers and properly handle If-Modified-Since and
// If-None-Match request headers, therefore we can rely on the
// validation to guarantee a fresh response.
fetch("some.json", {cache: "no-cache"})
.then(function(response) { /* consume the response */ });
// Download a resource with economics in mind! Prefer a cached
// albeit stale response to conserve as much bandwidth as possible.
fetch("some.json", {cache: "force-cache"})
.then(function(response) { /* consume the response */ });
I want to use nodeJS as tool for website scrapping. I have already implemented a script which logs me in on the system and parse some data from the page.
The steps are defined like:
Open login page
Enter login data
Submit login form
Go to desired page
Grab and parse values from the page
Save data to file
Exit
Obviously, the problem is that every time my script has to login, and I want to eliminate that. I want to implement some kind of cookie management system, where I can save cookies to .txt file, and then during next request I can load cookies from file and send it in request headers.
This kind of cookie management system is not hard to implement, but the problem is how to access cookies in nodejs? The only way I found it is using request response object, where you can use something like this:
request.get({headers:requestHeaders,uri: user.getLoginUrl(),followRedirect: true,jar:jar,maxRedirects: 10,},function(err, res, body) {
if(err) {
console.log('GET request failed here is error');
console.log(res);
}
//Get cookies from response
var responseCookies = res.headers['set-cookie'];
var requestCookies='';
for(var i=0; i<responseCookies.length; i++){
var oneCookie = responseCookies[i];
oneCookie = oneCookie.split(';');
requestCookies= requestCookies + oneCookie[0]+';';
}
}
);
Now content of variable requestCookies can be saved to the .txt file and can loaded next time when script is executed, and this way you can avoid process of logging in user every time when script is executed.
Is this the right way, or there is a method which returns cookies?
NOTE: If you want to setup your request object to automatically resend received cookies on every subsequent request, use the following line during object creation:
var request = require("request");
request = request.defaults({jar: true});//Send cookies on every subsequent requests
In my case, i've used 'http'library like the following:
http.get(url, function(response) {
variable = response.headers['set-cookie'];
})
This function gets a specific cookie value from a server response (in Typescript):
function getResponseCookieValue(res: Response, param: string) {
const setCookieHeader = res.headers.get('Set-Cookie');
const parts = setCookieHeader?.match(new RegExp(`(^|, )${param}=([^;]+); `));
const value = parts ? parts[2] : undefined;
return value;
}
I use Axios personally.
axios.request(options).then(function (response) {
console.log(response.config.headers.Cookie)
}).catch(function (error) {
console.error(error)
});