We are in the process of building a new website for the company I work for which is written in React using CRA. Browser push notifications and a PWA are both required so a service worker is essential, however I beleive the service worker is responsible for some fairly major caching issues.
The website is not yet in production however I added a new page yesterday (and this has happened a few times before) and once deployed to our dev environment nobody in the office was able to access it - they were simply redirected to the homepage (as if the route didn't exist) until they cleared their cache then the route loaded with no issues.
I've read a little bit on semantic versioning however the articles all seem to use NPM rather than yarn and are versioned locally (which isn't great with a team of 8 working on this project) using npm version patch etc.
We are using MS Azure as the build and release pipeline which I assume would be the best place to set versions if this is required.
My question is what are the steps to aviod this problem and am I on the correct lines thinking versioning will mitigate?
Semantic versioning in this context doesn't make any sense, you've been reading most likely about packages (libraries, frameworks) that are published into NPM for the world to use. In CRA projects, and most other web projects too, versioning of your app happens by the build tools as they name the files based on their contents. The filenames include the hash of the contents and are versioned automatically when contents change, eg. app.iue9234980s.js becomes app.92384oujiuoisdf.js etc.
--
If you're using the default Service Worker setup provided by CRA, then you should look at src/serviceWorker.js. In the comments of that file it says
// This lets the app load faster on subsequent visits in production, and gives
// it offline capabilities. However, it also means that developers (and users)
// will only see deployed updates on subsequent visits to a page, after all the
// existing tabs open on the page have been closed, since previously cached
// resources are updated in the background.
What happens here is that the SW and the build process use Workbox SW library that is configured to use precache policy. In this policy, users get the last version that was previously cached from the browser's cache even if there's a new version available, then in the background SW updates the caches, and on another visit users get the newer version. This of course means that users might always be one version "late".
If this behaviour is not what you want, then you need to change src/serviceWorker.js and probably some configuration somewhere in CRA files. You should google something like "custom service workers with cra" for examples.
To better grasp what is happening – and especially what is correct and intended behaviour in differently configured cases – I really recommend (everyone) to read Google's primer on SWs themselves, here: https://developers.google.com/web/fundamentals/primers/service-workers
With understanding of the SWs basic principles, it is then probably useful to checkout the Workbox library https://developers.google.com/web/tools/workbox to see what it can offer for your app.
Reading and understanding the different aspects of SW is key here – it is excruciatinly easy to shoot yourself in the foot with SWs :)
Related
Scenario:
A shared component implemented as a micro-front-end and hosted on S3...
JS bundle containing the whole app (webpacked) hosted on S3
JS bundle contains hash with the latest commit, e.g. component.{hash}.js
Question:
When we ship a new bundle, what's the best strategy for ensuring the new bundle is consumed by all clients after release, taking into account browser/CDN caching? Important note: we would like client's to get updates immediately (internal).
Examples
On release, generate a component.html file that pulls in the bundle (script tag) based on the latest hash. Ship the new component.html to S3. Clients use <link rel-'import' href='somedomain.com/component.html'> always giving them the latest shipped version.
Issue: The bundle can still take advantage of CD/browser caching, but the HTML file cannot be cached since we need it to be hot for any release. Also seems odd that we have to make two downloads to just get to a single bundle.
Ship as an NPM module that can be consumed at build time by a client.
Issue: If we have 10 clients, all 10 need to build and ship to release with the new component. Assuming package.lock won't cause issues for wildcards (don't know it well enough).
Note: Internal component; may undergo frequent changes, e.g. AB testing, etc.
It's generally important that page/app consuming your component be tested and updated with whatever new version of your component is released. Therefore, your idea of using NPM is a good one. It's generally desirable to have this lag time, so that developers of the pages/apps can verify functionality and also handle any API changes that may have occurred (intentional or otherwise).
For NPM, Semantic Versioning (SemVer) is a defacto standard. The idea is that you number your versions in such a way that bugfixes (no API changes) can be immediately updated in apps. Some developers are okay with installing the latest patch-version of your module. Many prefer to lock to a specific version, and will only release after testing like any other update.
NPM aside, what I've done in the past is use hashed or versioned URLs for the library. I've also kept a latest URL, which redirects to the latest version. For those integrating my library that don't care what version they're on, they'll always get the latest version this way. Additionally, browsers can cache the redirect target and share that cache with other pages that might specify an exact version.
Important note: we would like client's to get updates immediately (internal).
This isn't really all that possible in all cases. In most cases, using proper response headers for caching will solve it. There are edge cases though. For example, what will you do if the user loads a page and then goes offline before your JavaScript loads?
It's always tricky when deploying new pages. Some of your clients will be on old versions, some on new. Maintain backwards compatibility as best as you can, and set your caching headers as appropriate for your circumstances.
I have seen many techniques on the web, the most common one is a pop up toast message that says there is a new version and you are asked to reload the page (in google's Inbox for example)
As far as I understand, the only thing that is not cached is index.html, where are the references to the .js bundles with unique version hashing.
My question is - is this the best technique out there? Is it recommended to use a Service Worker for push messages on new version, or to have a request every few seconds to the server asking if there is new version? What is the full architecture?
Couldn't find any elegant solution for a good CI/CD technique for web apps (mostly Angular), that will mimc web app version updates (like in the app/play store)
Any thoughts?
I just started my adventure with frontend, most likely with web design. I've been struggling to answer one technical question and I couldn't find yet a reasonable answer.
There's so many libraries you can load, download to make your web developing faster. Therefore there is my question.
Is it better to download these libraries (e.g. Boostrap, jQuery, Angular, fonts from Google and so) and link to them (externally) from the official source or download it, upload to your server and then link to the location file (internal source) on your server?
My imagination tells me that if I would download them and upload em on my server, then link to it would make the whole website load quicker. Is that a good thinking?
Pro hosting and linking to external resources (may it be JS libraries, images or whatever):
Spreading the load: your server doesn't have to serve all content, it can concentrate on its main functionality and serve the application itself.
Spread the HTTP connections: due to more and more asynchronously working applications it is a good thing to use the maximum parallel HTTP connections per site/subdomain to deliver application data and load all necessary additional resources from other servers.
as Rafael mentioned above, CDNs scale very good and seldom go offline.
Cons
Even with fast internet connections there is a high chance that resources will be served faster when they are located on the same Intranet. That's why some companies have their own "Micro-CDNs" inside their local networks to combine the advantages of multiple servers and local availability.
External dependancy: as soon as an Internet connection becomes unavailable or a Proxy server goes down, all external resources become unavailable leaving the application in a broken state.
Sometimes it may be actually faster if you link from an external source. That's because the browser stores recent data it accesses, and many sites use Bootstrap, jQuery and the such. It might not happen frequently with less popular libraries.
Keep in mind, though, since you're downloading from external sources, you're at the mercy of their servers. If for some reason or another it gets offline, your page won't work correctly. CDNs are not supposed to go offline for that very reason, but it's good to be aware of that. Also, when/if you're offline and working on your page, you won't be able to connect during development.
It is always better to download these files locally if you are developing some application for more security so that you do not really have to depend on any third party server which hosts the CDN.
Talking about performance using CDN might be beneficial because the libraries that you require might be cached in your browser so the time to fetch the file is saved. But if the file is available locally loading these files will definately take time and space.
https://halfelf.org/2015/cdn-vs-local/
https://www.sitepoint.com/7-reasons-not-to-use-a-cdn/
I agree with Rafael's answer above, but wanted to note a few benefits of serving up these libraries locally that he or she omitted.
It is still considered best practice (until HTTP2 becomes widespread) to try to minimize the amount of downloads being made by your site by concatenating many files into a single file. SO-- if you are using three Javascript libraries/frameworks (for instance, Angular, jQuery and Moment.js), if you are using a CDN that is three separate script elements pulling down three separate .js files. However, if you host them locally, you can have your build process include a step where it bundles the three libraries together into a single file called "vendor.js" or something along those lines. This has the added bonus of simplifying dependency loading to some degree, as you can concatenate them in a certain order should the need be.
Finally, although it is a little advanced if you are just getting started, if you are considering hosting your library files with your project it would definitely be worth looking into bower (https://bower.io/docs/api/) -- it is a node build tool that allows you to define what packages need to be included in your project and then install them with a single command-- particularly useful for keeping unnecessary library files out of your version control. Good luck!
for some reason all of my html seems to be 100% cached in chrome.
I am using angular 1.2 with .net web api 2 project, and my content is served in index.html.
I did not make any cache policy changes yet, but it seems to be caching everything very heavily.
none of the changes i make (to the views) are reflected until i clear browser cache.
I don't see new changes after pressing f5 or after publishing my site to the server and doing f5 on that. I have to either explicitly clear browser cache, or keep console open with "no caching while dev tools are open" setting on.
I want to prevent asking users to clear their browser cache when new versions are deployed.
Since noone is chiming in, i'll post the solution I implemented, which works, but could be better
IIS:
http features > http response headers > set custom headers > Expire Web Content: Immediately
(if you don't see http features, you'll have to add the feature to your iis server)
index.html:
<meta http-equiv="CACHE-CONTROL" content="NO-CACHE">
<meta http-equiv="CACHE-CONTROL" content="NO-STORE">
(these may cause issues in older versions of IE)
this works, but I'd prefer to strike a better balance of caching between releases.
Alternative approach would be to use a build tool such as Grunt, and generate unique filenames for your scripts in production and update links in index.html. This would be the best approach I believe, because full caching would be enabled, and browser would always make the request for new version of files since the names would be unique. (I've also seen people append ?v=666 to files, but from what i've read this is not a reliable approach)
Also if you're serving content from .Net pages (instead of basic .html) you have an option of using Bundler which does manage tags between releases.
bundles.Add(new ScriptBundle("~/bundles/angular").Include("~/Scripts/angular/angular.js")
.Include("~/Scripts/angular/angular-route.js"));
this will result in
<script src="/bundles/angular?v=20LMOimhsS0zmHdsbxeBlJ05XNyVXwPaS4nt3KuzemE1"></script>
update
additionally I've also been adding a version parameter in template includes. This prevents caching, but it may prevent caching completely so do some testing if you go that route
<div data-ng-include=" 'app/import/import-summary.html?v=' + appVersion "></div>
appVersion can be a global variable set in app.js
app.run(['$route', '$rootScope', function ($route, $rootScope) {
$rootScope.appVersion = getBuildVersion(); // this can be read from assembly file version or any other place where versions are generated by your build server
}]);
I have found I have the same issue intermittently with my production server as well.
In most instances, 80% of the code that I push to the production server has no problems. However, a new module reference, a new Angular binding, or even the smallest template edit can take several refreshes to actually clear the cache and show the latest updates.
I actually wrote a 7 paragraph question slash statement to air my grievances and hoping someone had the same issue as myself. I ended up deleting the question, as I feared it was just my experience.
I have found a weird workaround, where if I host the site from the terminal by running either 'npm start' or 'http-server' modules on my machine, the constant update between both my development and production server will force the browser to acknowledge both environments and cause the change to either appear in one environment or both.
In my opinion, there are two types of developers: modular and line-by-line asynchronous.
Modular programmers code in large chunks and then upload the entire code effort which minimizes heavy caching.
Asynchronous programmers like to view every single change they make by coding a couple of lines, uploading the small changes and then refreshing the browser to see their latest nuance of progression.
I am definitely an asynchronous programmer. I find it allows me to catch smaller mistakes as well as understand the inner workings of whichever language I am using.
Is it safe to say that asynchronous programming leads to heavy browser caching, which is leading to the problem we are all having on this particular question?
Glad to share in your blind frustration, as this has been a huge problem of mine, ever since I've been dabbling in Angular.
It may at first glance appear to be a small issue, however as you are learning Angular, you may spend hours or even a couple of days recoding the same 30 lines because there was no change in your browser. This is a very dangerous, invisible issue which can lead to large amounts of time being chipped away for development time.
I have designed a meteor.js application and it works great on localhost and even when deployed to the internet. Now I want create a sign-up site that will spin up new instances of the application for each client who signs up on the back-end. Assuming a meteor.js application and python or javascript for the sign-up site, what high level steps need to be taken to implement this?
I am looking for a more correct and complete answer that takes the form of my poorly imagined version of this:
Use something like node or python to call a shell script that may or may not run as sudo
That script might create a new folder to hold instance specific stuff (like client files, config, and or that instances database).
The script or python code would deploy an instance of the application to that folder and on a specific port
Python might add configuration information to a tool like Pound to forward a subdomain to a port
Other things....!?
I don't really understand the high level steps that need to be taken here so if someone could provide those steps and maybe even some useful tools or tutorials for doing so I'd be extremely grateful.
I have a similar situation to you but ended up solving it in a completely different way. It is now available as a Meteor smart package:
https://github.com/mizzao/meteor-partitioner
The problem we share is that we wanted to write a meteor app as if only one client (or group of clients, in my case) exists, but that it needs to handle multiple sets of clients without them knowing about each other. I am doing the following:
Assume the Meteor app is programmed for just a single instance
Using a smart package, hook the collections on server (and possibly client) so that all operations are 'scoped' only to the instance of the user that is calling them. One way to do this is to automatically attach an 'instance' or 'group' field to each document that is being added.
Doing this correctly requires a lot of knowledge about the internals of Meteor, which I've been learning. However, this approach is a lot cleaner and less resource-intensive than trying to deploy multiple meteor apps at once. It means that you can still code the app as if only one client exists, instead of explicitly doing so for multiple clients. Additionally, it allows you to share resources between the instances that can be shared (i.e. static assets, shared state, etc.)
For more details and discussions, see:
https://groups.google.com/forum/#!topic/meteor-talk/8u2LVk8si_s
https://github.com/matb33/meteor-collection-hooks (the collection-hooks package; read issues for additional discussions)
Let me remark first that I think spinning up multiple instances of the same app is a bad design choice. If it is a stop gap measure, here's what I would suggest:
Create an archive that can be readily deployed. (Bundle the app, reinstall fibers if necessary, rezip). Deploy (unzip) the archive to a new folder when a new instance is created using a script.
Create a template of an init script and use forever or daemonize or jesus etc to start the site on reboot and keep the sites up during normal operation. See Meteor deploying to a VM by installing meteor or How does one start a node.js server as a daemon process? for examples. when a new instance is deployed populate the template with new values (i.e. port number, database name, folder). Copy the filled out template to init.d and link to the runlevel. Alternatively, create one script in init.d that executes other scripts to bring up the site.
Each instance should be listening to its own port, so you'll need a reverse proxy. AFAIK, Apache and Nginx require restarts when you change the configuration, so you'll probably want to look at Hipache https://github.com/dotcloud/hipache. Hipache uses redis to store the configuration information. Adding the new instance requires to add a key to redis. There is an experimental port of Hipache that brings the functionality to Nginx https://github.com/samalba/hipache-nginx
What about DNS updates? Once you create a new instance, do you need to add a new record to your DNS configuration?
I don't really have an answer to your question... but I just want to remind you of another potential problem that you may run into cuz I see you mentioned python, in other words, you may be running another web app on Apache/Nginx, etc... the problem is Meteor is not very friendly when it comes to co-exist with another http server, the project I'm working on was troubled by this issue and we had to move it to a stand alone server after days of hassle with guys from Meteor... I did not work on the issue, so I'm not able to give you more details, but I just looked up online and found something similar: https://serverfault.com/questions/424693/serving-meteor-on-main-domain-and-apache-on-subdomain-independently.
Just something to keep in mind...