Scenario:
A shared component implemented as a micro-front-end and hosted on S3...
JS bundle containing the whole app (webpacked) hosted on S3
JS bundle contains hash with the latest commit, e.g. component.{hash}.js
Question:
When we ship a new bundle, what's the best strategy for ensuring the new bundle is consumed by all clients after release, taking into account browser/CDN caching? Important note: we would like client's to get updates immediately (internal).
Examples
On release, generate a component.html file that pulls in the bundle (script tag) based on the latest hash. Ship the new component.html to S3. Clients use <link rel-'import' href='somedomain.com/component.html'> always giving them the latest shipped version.
Issue: The bundle can still take advantage of CD/browser caching, but the HTML file cannot be cached since we need it to be hot for any release. Also seems odd that we have to make two downloads to just get to a single bundle.
Ship as an NPM module that can be consumed at build time by a client.
Issue: If we have 10 clients, all 10 need to build and ship to release with the new component. Assuming package.lock won't cause issues for wildcards (don't know it well enough).
Note: Internal component; may undergo frequent changes, e.g. AB testing, etc.
It's generally important that page/app consuming your component be tested and updated with whatever new version of your component is released. Therefore, your idea of using NPM is a good one. It's generally desirable to have this lag time, so that developers of the pages/apps can verify functionality and also handle any API changes that may have occurred (intentional or otherwise).
For NPM, Semantic Versioning (SemVer) is a defacto standard. The idea is that you number your versions in such a way that bugfixes (no API changes) can be immediately updated in apps. Some developers are okay with installing the latest patch-version of your module. Many prefer to lock to a specific version, and will only release after testing like any other update.
NPM aside, what I've done in the past is use hashed or versioned URLs for the library. I've also kept a latest URL, which redirects to the latest version. For those integrating my library that don't care what version they're on, they'll always get the latest version this way. Additionally, browsers can cache the redirect target and share that cache with other pages that might specify an exact version.
Important note: we would like client's to get updates immediately (internal).
This isn't really all that possible in all cases. In most cases, using proper response headers for caching will solve it. There are edge cases though. For example, what will you do if the user loads a page and then goes offline before your JavaScript loads?
It's always tricky when deploying new pages. Some of your clients will be on old versions, some on new. Maintain backwards compatibility as best as you can, and set your caching headers as appropriate for your circumstances.
Related
We are in the process of building a new website for the company I work for which is written in React using CRA. Browser push notifications and a PWA are both required so a service worker is essential, however I beleive the service worker is responsible for some fairly major caching issues.
The website is not yet in production however I added a new page yesterday (and this has happened a few times before) and once deployed to our dev environment nobody in the office was able to access it - they were simply redirected to the homepage (as if the route didn't exist) until they cleared their cache then the route loaded with no issues.
I've read a little bit on semantic versioning however the articles all seem to use NPM rather than yarn and are versioned locally (which isn't great with a team of 8 working on this project) using npm version patch etc.
We are using MS Azure as the build and release pipeline which I assume would be the best place to set versions if this is required.
My question is what are the steps to aviod this problem and am I on the correct lines thinking versioning will mitigate?
Semantic versioning in this context doesn't make any sense, you've been reading most likely about packages (libraries, frameworks) that are published into NPM for the world to use. In CRA projects, and most other web projects too, versioning of your app happens by the build tools as they name the files based on their contents. The filenames include the hash of the contents and are versioned automatically when contents change, eg. app.iue9234980s.js becomes app.92384oujiuoisdf.js etc.
--
If you're using the default Service Worker setup provided by CRA, then you should look at src/serviceWorker.js. In the comments of that file it says
// This lets the app load faster on subsequent visits in production, and gives
// it offline capabilities. However, it also means that developers (and users)
// will only see deployed updates on subsequent visits to a page, after all the
// existing tabs open on the page have been closed, since previously cached
// resources are updated in the background.
What happens here is that the SW and the build process use Workbox SW library that is configured to use precache policy. In this policy, users get the last version that was previously cached from the browser's cache even if there's a new version available, then in the background SW updates the caches, and on another visit users get the newer version. This of course means that users might always be one version "late".
If this behaviour is not what you want, then you need to change src/serviceWorker.js and probably some configuration somewhere in CRA files. You should google something like "custom service workers with cra" for examples.
To better grasp what is happening – and especially what is correct and intended behaviour in differently configured cases – I really recommend (everyone) to read Google's primer on SWs themselves, here: https://developers.google.com/web/fundamentals/primers/service-workers
With understanding of the SWs basic principles, it is then probably useful to checkout the Workbox library https://developers.google.com/web/tools/workbox to see what it can offer for your app.
Reading and understanding the different aspects of SW is key here – it is excruciatinly easy to shoot yourself in the foot with SWs :)
i'm trying to figure out a way to make grunt write an incremental number or a hash on a file.
i have a file named config.yml which contains this:
version: 0.0.0
every time i change some files that i can specify somewhere (say all the .js and .css files), grunt should increment that number in some way.
i've seen some cache busting plugin but it's not what i am looking for as i don't want to have something like config.987234892374982.yml or config.yml?v=1.0.0 for example. i'm looking for a way to have grunt to find that number in that file, change it in a way that makes sense (incrementally ideally, or a random hash) and then save the file.
can you help in some way? thanks a lot!
I would seriously recommend against automatic version bumping for anything beyond the build number.
A version number is more than just an indication of how many times your product has been built. In essence, a version number is a semantic promise to your end user concerning compatibility with earlier releases. Node.js and npm and other systems using versioning are built around the core concept that X.Y.Z version numbers contain the following logic:
Software that doesn't have the same X version will have severely breaking changes that require those upgrading (or downgrading) to completely redesign the logic of how they use the software, and in some cases they even have to find alternatives because what they're doing right now doesn't work anymore.
Software within the same X version can relatively easily be swapped out without having to make sweeping changes to your own site or product.
Software within the same Y version can be swapped out without having to change any code, because it's supposed to be only bug fixes and security fixes. Only code that works around the things fixed needs to be changed.
Software within the same Z version has the exact same source, so if you have 2 copies of the same Z.Y.Z version, they can be used interchangeably.
This is the core contract that NPM and other package managers ask all their content providers to adhere to. In fact, it has already been shown that in some cases where content providers don't adhere to this versioning system, consumers of this content who assume semantic versioning apply have found their build to fail. Many consumers assume that if version 3.5.N works, that any version within the 3.5.X family will be interchangeable, and make use of that assumption by automatically building their code with the most recent version of the 3.5.X family.
Because of this, it's not a good idea to automatically bump versions beyond the build number. Bumping the patch version should only be done when you actually release a new version to the public, not after every build. Bumping the minor version should only be done when you have added new features to the product, but without the need for major changes to software that uses your product. Bumping the major version should only be done when you make drastic and destructive changes to your API, like moving and/or removing functions, function parameters, objects or attributes.
When we deploy new versions of our application, there is a chance that our users might still be using old front end java script code that may cause errors.
Our app is a single page app, so people do not refresh it a lot, and users sometimes leave it up for a while. So this problem is really prone to happening whenever we push new code.
We were thinking maybe putting up a popup with a refresh button that will force them reload the front end cached code.
How do you trigger that normally? There are lots of ways I can think to do it, but maybe there is a standard way of doing it?
You effectively have two independent problems.
API version mismatch between client and server
Accelerating the release of new versions of the code.
API version mismatch
Typically, you'll want to use a versioned API. Each API request should somehow indicate which version the request corresponds to. This is often done in the path of the request, which makes routing to different API servers very easy to do. For example:
http://api.example.com/1.0/foo/bar
Your web server can match on /1.0/ and route to the appropriate application server for your API.
This gives you some flexibility, allowing a rolling release process and not forcing clients to reload every single time you do a release. (You may some day want to release 50 times a day, and your clients won't be too happy about that. No need to force a reload unless you have a specific reason to.)
The easiest way to do this is make new versions of your API backwards compatible as much as possible. But, breaking changes do occur. When this happens, run two instances of your API server simultaneously. If you have a major underlying data structure change, or some other huge change that prevents this, you will need to stop the old versions at the same time as starting the new, and force a reload.
New client code releases
You'll have to decide based on your app and business conditions whether or not it's a requirement to reload each time you have a release. Again, you might some day want 50 releases in a day, most of those for minor issues that most of your customers will never see. It's common to let the user know that a new version is available and let them reload when they feel like it. You can always force a reload if you absolutely have to (due to critical security issue, or major breaking changes).
Implementation
The specifics of how you do this are up to you, but one simple way is to have a basic JSON file available, indicating the latest versions:
{
"apiVersion": "1.0.5"
"appVersion": "1.1.352"
}
More creatively, you could have this file indicate whether or not specific versions need to be killed, and thus a reload forced.
I would add ?<version> to the script's src.
<script src="myFile.js?1"></script>
You can change this version with a code push for instance, so the client's forced to take a new version, because of the querystring.
You could use a Web Worker to periodically ping a server for a new code change or version number (say from a database) and force a refresh if there's something new. That will give you more on-demand possibilities than if you are just using say, a setTimeout.
This is a case where semver really comes in handy. You want to have an automated process that updates the client code. To make sure the update is compatible, you should ensure that there is a way to compare the version strings of the new and old code and determine if it's a "major" update or not. For "minor" or "patch" updates, you can probably update on-the-fly. But for major updates, it would be smart to force a full reload, after helping the user save their work.
In terms of a process for on-the-fly updates, you could use something like AMD modules for this. You would expire the module cache, download the new code, and reinitialize relevant parts of the system (carried out by the new code).
There is also the matter of knowing when updates are available. I would recommend looking at Server Sent Events for this. But if you need Internet Explorer support, you may have to resort to a setTimeout-based polling mechanism, or similar.
for some reason all of my html seems to be 100% cached in chrome.
I am using angular 1.2 with .net web api 2 project, and my content is served in index.html.
I did not make any cache policy changes yet, but it seems to be caching everything very heavily.
none of the changes i make (to the views) are reflected until i clear browser cache.
I don't see new changes after pressing f5 or after publishing my site to the server and doing f5 on that. I have to either explicitly clear browser cache, or keep console open with "no caching while dev tools are open" setting on.
I want to prevent asking users to clear their browser cache when new versions are deployed.
Since noone is chiming in, i'll post the solution I implemented, which works, but could be better
IIS:
http features > http response headers > set custom headers > Expire Web Content: Immediately
(if you don't see http features, you'll have to add the feature to your iis server)
index.html:
<meta http-equiv="CACHE-CONTROL" content="NO-CACHE">
<meta http-equiv="CACHE-CONTROL" content="NO-STORE">
(these may cause issues in older versions of IE)
this works, but I'd prefer to strike a better balance of caching between releases.
Alternative approach would be to use a build tool such as Grunt, and generate unique filenames for your scripts in production and update links in index.html. This would be the best approach I believe, because full caching would be enabled, and browser would always make the request for new version of files since the names would be unique. (I've also seen people append ?v=666 to files, but from what i've read this is not a reliable approach)
Also if you're serving content from .Net pages (instead of basic .html) you have an option of using Bundler which does manage tags between releases.
bundles.Add(new ScriptBundle("~/bundles/angular").Include("~/Scripts/angular/angular.js")
.Include("~/Scripts/angular/angular-route.js"));
this will result in
<script src="/bundles/angular?v=20LMOimhsS0zmHdsbxeBlJ05XNyVXwPaS4nt3KuzemE1"></script>
update
additionally I've also been adding a version parameter in template includes. This prevents caching, but it may prevent caching completely so do some testing if you go that route
<div data-ng-include=" 'app/import/import-summary.html?v=' + appVersion "></div>
appVersion can be a global variable set in app.js
app.run(['$route', '$rootScope', function ($route, $rootScope) {
$rootScope.appVersion = getBuildVersion(); // this can be read from assembly file version or any other place where versions are generated by your build server
}]);
I have found I have the same issue intermittently with my production server as well.
In most instances, 80% of the code that I push to the production server has no problems. However, a new module reference, a new Angular binding, or even the smallest template edit can take several refreshes to actually clear the cache and show the latest updates.
I actually wrote a 7 paragraph question slash statement to air my grievances and hoping someone had the same issue as myself. I ended up deleting the question, as I feared it was just my experience.
I have found a weird workaround, where if I host the site from the terminal by running either 'npm start' or 'http-server' modules on my machine, the constant update between both my development and production server will force the browser to acknowledge both environments and cause the change to either appear in one environment or both.
In my opinion, there are two types of developers: modular and line-by-line asynchronous.
Modular programmers code in large chunks and then upload the entire code effort which minimizes heavy caching.
Asynchronous programmers like to view every single change they make by coding a couple of lines, uploading the small changes and then refreshing the browser to see their latest nuance of progression.
I am definitely an asynchronous programmer. I find it allows me to catch smaller mistakes as well as understand the inner workings of whichever language I am using.
Is it safe to say that asynchronous programming leads to heavy browser caching, which is leading to the problem we are all having on this particular question?
Glad to share in your blind frustration, as this has been a huge problem of mine, ever since I've been dabbling in Angular.
It may at first glance appear to be a small issue, however as you are learning Angular, you may spend hours or even a couple of days recoding the same 30 lines because there was no change in your browser. This is a very dangerous, invisible issue which can lead to large amounts of time being chipped away for development time.
When using third-party libraries such as jquery, yui-reset, swfobject and so on, do you link to the hosted versions, or download and host your own versions?
Pros and cons either way?
Hosted versions are apparently the way to go. For three main reasons (edit: I've added a fourth reason, but it's sort of a moot point):
Google/jQuery/etc servers are likely to be faster than your own
A lot of these servers use content distribution so it will be served from a server geographically close to the requester
If every site used the hosted versions, users are more likely to have the files cached in their browsers, so a trip to the server might not even be necessary
They are possibly more reliable than your own server (however if your own server goes down, this point is moot, as you probably won't be able to serve the main page, so there won't be a request to the js file anyway)
Cons would be
You have no control over the uptime/reliability of the servers (though they are more likely more reliable than your own)
Cannot make any custom mods/patches to these files (though most good frameworks allow you to extend them without needing to modify the original code)
If the hosted file does not allow you to specify the version as part of the filename (eg "jquery-1.3.2.js" rather than "jquery.js"), you probably don't want to use the hosted version, as any updates could possibly break your code
I would say the pros generally outweigh the cons.
These are all javascript libraries - You want to put a copy of it on your own server.
If you somehow use a different version then you would not have tested against the newer version and it might break your code.
I always download and host them locally, just because I'm worried about their server downtime, there's no real guarantee that their servers will be up for the rest of time. There is usually a note in the script about who it belongs to anyway.
I guess the only con would be if the person who made the script wouldn't actually want it downloaded.. But I don't see that ever happening.
Plus the requests times are much faster, instead of requesting a script hosted at google, just request it on your own server.
For production use hosted.
For development use local because if you're offline then your dev site is broke.