I've been looking for ways of making my site load faster and one way that I'd like to explore is making greater use of Cloudfront.
Because Cloudfront was originally not designed as a custom-origin CDN and because it didn't support gzipping, I have so far been using it to host all my images, which are referenced by their Cloudfront cname in my site code, and optimized with far-futures headers.
CSS and javascript files, on the other hand, are hosted on my own server, because until now I was under the impression that they couldn't be served gzipped from Cloudfront, and that the gain from gzipping (about 75 per cent) outweighs that from using a CDN (about 50 per cent): Amazon S3 (and thus Cloudfront) did not support serving gzipped content in a standard manner by using the HTTP Accept-Encoding header that is sent by browsers to indicate their support for gzip compression, and so they were not able to Gzip and serve components on the fly.
Thus I was under the impression, until now, that one had to choose between two alternatives:
move all assets to the Amazon CloudFront and forget about GZipping;
keep components self-hosted and configure our server to detect incoming requests and perform on-the-fly GZipping as appropriate, which is what I chose to do so far.
There were workarounds to solve this issue, but essentially these didn't work. [link].
Now, it seems Amazon Cloudfront supports custom origin, and that it is now possible to use the standard HTTP Accept-Encoding method for serving gzipped content if you are using a Custom Origin [link].
I haven't so far been able to implement the new feature on my server. The blog post I linked to above, which is the only one I found detailing the change, seems to imply that you can only enable gzipping (bar workarounds, which I don't want to use), if you opt for custom origin, which I'd rather not: I find it simpler to host the coresponding fileds on my Cloudfront server, and link to them from there. Despite carefully reading the documentation, I don't know:
whether the new feature means the files should be hosted on my own domain server via custom origin, and if so, what code setup will achieve this;
how to configure the css and javascript headers to make sure they are served gzipped from Cloudfront.
UPDATE: Amazon now supports gzip compression, so this is no longer needed. Amazon Announcement
Original answer:
The answer is to gzip the CSS and JavaScript files. Yes, you read that right.
gzip -9 production.min.css
This will produce production.min.css.gz. Remove the .gz, upload to S3 (or whatever origin server you're using) and explicitly set the Content-Encoding header for the file to gzip.
It's not on-the-fly gzipping, but you could very easily wrap it up into your build/deployment scripts. The advantages are:
It requires no CPU for Apache to gzip the content when the file is requested.
The files are gzipped at the highest compression level (assuming gzip -9).
You're serving the file from a CDN.
Assuming that your CSS/JavaScript files are (a) minified and (b) large enough to justify the CPU required to decompress on the user's machine, you can get significant performance gains here.
Just remember: If you make a change to a file that is cached in CloudFront, make sure you invalidate the cache after making this type of change.
My answer is a take off on this: http://blog.kenweiner.com/2009/08/serving-gzipped-javascript-files-from.html
Building off skyler's answer you can upload a gzip and non-gzip version of the css and js. Be careful naming and test in Safari. Because safari won't handle .css.gz or .js.gz files.
site.js and site.js.jgz and
site.css and site.gz.css (you'll need to set the content-encoding header to the correct MIME type to get these to serve right)
Then in your page put.
<script type="text/javascript">var sr_gzipEnabled = false;</script>
<script type="text/javascript" src="http://d2ft4b0ve1aur1.cloudfront.net/js-050/sr.gzipcheck.js.jgz"></script>
<noscript>
<link type="text/css" rel="stylesheet" href="http://d2ft4b0ve1aur1.cloudfront.net/css-050/sr-br-min.css">
</noscript>
<script type="text/javascript">
(function () {
var sr_css_file = 'http://d2ft4b0ve1aur1.cloudfront.net/css-050/sr-br-min.css';
if (sr_gzipEnabled) {
sr_css_file = 'http://d2ft4b0ve1aur1.cloudfront.net/css-050/sr-br-min.css.gz';
}
var head = document.getElementsByTagName("head")[0];
if (head) {
var scriptStyles = document.createElement("link");
scriptStyles.rel = "stylesheet";
scriptStyles.type = "text/css";
scriptStyles.href = sr_css_file;
head.appendChild(scriptStyles);
//alert('adding css to header:'+sr_css_file);
}
}());
</script>
gzipcheck.js.jgz is just sr_gzipEnabled = true; This tests to make sure the browser can handle the gzipped code and provide a backup if they can't.
Then do something similar in the footer assuming all of your js is in one file and can go in the footer.
<div id="sr_js"></div>
<script type="text/javascript">
(function () {
var sr_js_file = 'http://d2ft4b0ve1aur1.cloudfront.net/js-050/sr-br-min.js';
if (sr_gzipEnabled) {
sr_js_file = 'http://d2ft4b0ve1aur1.cloudfront.net/js-050/sr-br-min.js.jgz';
}
var sr_script_tag = document.getElementById("sr_js");
if (sr_script_tag) {
var scriptStyles = document.createElement("script");
scriptStyles.type = "text/javascript";
scriptStyles.src = sr_js_file;
sr_script_tag.appendChild(scriptStyles);
//alert('adding js to footer:'+sr_js_file);
}
}());
</script>
UPDATE: Amazon now supports gzip compression. Announcement, so this is no longer needed. Amazon Announcement
Cloudfront supports gzipping.
Cloudfront connects to your server via HTTP 1.0. By default some webservers, including nginx, dosn't serve gzipped content to HTTP 1.0 connections, but you can tell it to do by adding:
gzip_http_version 1.0
to your nginx config. The equivalent config could be set for whichever web server you're using.
This does have a side effect of making keep-alive connections not work for HTTP 1.0 connections, but as the benefits of compression are huge, it's definitely worth the trade off.
Taken from http://www.cdnplanet.com/blog/gzip-nginx-cloudfront/
Edit
Serving content that is gzipped on the fly through Amazon cloud front is dangerous and probably shouldn't be done. Basically if your webserver is gzipping the content, it will not set a Content-Length and instead send the data as chunked.
If the connection between Cloudfront and your server is interrupted and prematurely severed, Cloudfront still caches the partial result and serves that as the cached version until it expires.
The accepted answer of gzipping it first on disk and then serving the gzipped version is a better idea as Nginx will be able to set the Content-Length header, and so Cloudfront will discard truncated versions.
We've made a few optimisations for uSwitch.com recently to compress some of the static assets on our site. Although we setup a whole nginx proxy to do this, I've also put together a little Heroku app that proxies between CloudFront and S3 to compress content: http://dfl8.co
Given publicly accessible S3 objects can be accessed using a simple URL structure, http://dfl8.co just uses the same structure. I.e. the following URLs are equivalent:
http://pingles-example.s3.amazonaws.com/sample.css
http://pingles-example.dfl8.co/sample.css
http://d1a4f3qx63eykc.cloudfront.net/sample.css
Yesterday amazon announced new feature, you can now enable gzip on your distribution.
It works with s3 without added .gz files yourself, I tried the new feature today and it works great. (need to invalidate you're current objects though)
More info
You can configure CloudFront to automatically compress files of certain types and serve the compressed files.
See AWS Developer Guide
Related
just noticed on several websites that the font awesome icons aren's showing in Google Chrome. The console shows the following error:
Font from origin 'http://cdn.keywest.life' has been blocked from
loading by Cross-Origin Resource Sharing policy: No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://www.keywest.life' is therefore not
allowed access.
I found the exact same issue on several other sites. This can be easily fixed by replacing the own CDN reference with:
//maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css
-however, this is not the solution, just a workaround. I would love to know the reason and the right solution.
(the cause is this: the website is using it's own CDN, provided by MaxCDN and has the reference to the font awesome fonts and these are not loaded by Chrome, if you are loading the same resource from the Bootstrapcdn resource -mentioned above- it works)
here is a n example of the issue (in the menu and the social icons in footer: http://www.keywestnight.com/fantasy-fest )
Thanks for any help/explanatioon!
Here is the working method to allow access from all domains for webfonts:
# Allow access from all domains for webfonts.
# Alternatively you could only whitelist your
# subdomains like "subdomain.example.com".
<IfModule mod_headers.c>
<FilesMatch "\.(ttf|ttc|otf|eot|woff|font.css|css)$">
Header set Access-Control-Allow-Origin "*"
</FilesMatch>
</IfModule>
The problem isn't with the CSS file, it has to do with how the font file is served. The font-awesome.min.css file has lines like
#font-face{font-family:'FontAwesome';
src:url('../fonts/fontawesome-webfont.eot?v=4.2.0');
src:url('../fonts/fontawesome-webfont.eot?#iefix&v=4.2.0')
format('embedded-opentype'),url('../fonts/fontawesome-webfont.woff?v=4.2.0')
format('woff'),url('../fonts/fontawesome-webfont.ttf?v=4.2.0')
format('truetype'),url('../fonts/fontawesome-webfont.svg?v=4.2.0#fontawesomeregular') format('svg');
font-weight:normal;
font-style:normal}
which cause the browser to request an appropriate font file (eot, woff, ttf or svg) from the same server as the CSS file. This is logical and correct.
However, when the browser requests that font file from cdn.keywest.life, it reads the headers for a Access-Control-Allow-Origin header and doesn't find one so it gives that error message. (This seems like a browser bug to me because it's coming from the same server as the CSS file).
Instead, when you use maxcdn.bootstrapcdn.com the response includes the Access-Control-Allow-Origin:* header and the browser accepts this font file. If your cdn server included this header then it would work too.
If you have an Apache server, see this answer: Font-awesome not properly displaying on Firefox / how to vend via CDN?
This issue of accessing font-awesome assets has been a problem for many people without a comprehensive explanation and resolution to the problem.
What is CORS:
Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to let a user agent gain permission to access selected resources from a server on a different origin (domain) than the site currently in use. A user agent makes a cross-origin HTTP request when it requests a resource from a different domain, protocol, or port than the one from which the current document originated.
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
The Problem:
The problem stems from how the font-awesome fonts are loaded.
#font-face{
font-family:'FontAwesome';
src:url('../fonts/fontawesome-webfont.eot?v=4.2.0');
src:url('../fonts/fontawesome-webfont.eot?#iefix&v=4.2.0') format('embedded-opentype'),url('../fonts/fontawesome-webfont.woff?v=4.2.0') format('woff'),url('../fonts/fontawesome-webfont.ttf?v=4.2.0') format('truetype'),url('../fonts/fontawesome-webfont.svg?v=4.2.0#fontawesomeregular') format('svg');
font-weight:normal;
font-style:normal
}
The fonts are loaded via the stylesheet (CSS). The situation we have here is:
The Solution:
While CORS rules have been created on your file storage e.g. S3, and access to the resource has been given to the domain in question, when CDN tries to load the fonts specified in the CSS the origin/domain specified when loading these fonts is that of the CDN but no CORS access has given to the CDN domain.
Create a CORS rule for the CDN domain.
I use a CDN that doesn't allow me to modify its response, so I modified font-awesome.min.css, replacing relative path with absolute path and it worked.
None of the answers worked for me, I had to create an edge rule on maxcnd back office (which change config file on you zone)
More info here
https://www.maxcdn.com/one/tutorial/edge-rules-recipes/
https://www.maxcdn.com/one/tutorial/create-a-rule/
The solution to this is to use another cdn for fontawesome.
https://www.cdnpkg.com/font-awesome/5.11.0
If you're like me and using the official WordPress Font Awesome plugin, you'll want to go into settings and switch from "Use a CDN" to "Use a Kit".
I'm messing around with the Darksky API and under one of the query parameters it states:
extend=hourly optional
When present, return hour-by-hour data for the next 168 hours, instead
of the next 48. When using this option, we strongly recommend enabling
HTTP compression.
I'm using Express as a node proxy which hits the Darksky api (i.e. localhost:3000/api/forecast/LATITUDE, LONGITUDE).
What does "HTTP compression" mean and how would I go about enabling it?
Here compression means the gzip compression on the express server. You can use the compression middleware to add easy gzip compression to your server.
Read more about how you can install that middleware on here.
https://github.com/expressjs/compression
An example implementation should be look like this.
var compression = require('compression')
var express = require('express')
var app = express()
// compress all responses
app.use(compression())
// add all routes
To quote from https://darksky.net/dev/docs
The Forecast Data API supports HTTP compression. We heartily recommend using it, as it will make responses much smaller over the wire. To enable it, simply add an Accept-Encoding: gzip header to your request. (Most HTTP client libraries wrap this functionality for you, please consult your library’s documentation for details.)
I'm not familiar with the Dark Sky API but I would imagine it returns a large amount of highly redundant data, which is ideal for compression. HTTP requests have a compression mechanism built in via Accept-Encoding, as mentioned above.
In your case that data will be travelling across the wire twice, once from Dark Sky to your server and then again from your server to your end user. You could compress just one of these two transmissions or both, it's up to you but it's likely you'd want both unless the end user is on the same local network as your server.
There are various SO questions about making compressed requests, such as:
node.js - easy http requests with gzip/deflate compression
The key decision for you is whether you want to decompress and recompress the data in your proxy or just stream it through. If you don't need a decompressed copy of the data in the server then it would be more efficient to skip the extra steps. You'd need to be careful to ensure all the headers are set correctly but if you just pass on the relevant headers that you receive (in both directions) it should be relatively simple to pipe through the response from Dark Sky.
I have a link to an mp3 on amazon s3 (the aws mp3 is set to public)
The audio player runs fine.
But when I try to make a visualizer, as soon as I connect to the audioplayer there is a CORS error. I don't understand why this should be so.
I have been using the MDN sample for analyserNode as the basis
https://developer.mozilla.org/en/docs/Web/API/AnalyserNode
<audio id="audio-player-console" src="${AWS_songUrl}" autoplay>
<p>Your browser does not support this audio player </p>
</audio>
Just to clarify, so long as no attempt to connect for analyserdata, the audioplayer runs the tracks without problem.
If I add crossorigin="anonymous" to the audio tag then I get nothing at all and the audio player won't play the track
<audio id="audio-player-console" src="${AWS_songUrl}" crossorigin="anonymous" autoplay>
<p>Your browser does not support this audio player </p>
</audio>
My AWS CORS configuration after 'make public' has been set
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Here is where I try to connect the audio tag to the audiocontext for a visualizer, which is one it starts failing the CORS check
canvasCtx = visualizerCanvas.getContext("2d")
audioCtx = new (window.AudioContext || window.webkitAudioContext)()
var audioSrc = audioCtx.createMediaElementSource(aPlayer)
var analyser = audioCtx.createAnalyser()
audioSrc.connect(analyser)
analyser.fftSize = 2048
var bufferLength = analyser.frequencyBinCount
var dataArray = new Uint8Array(bufferLength)
analyser.getByteTimeDomainData(dataArray)
I just don't know what I can do. I have set the AWS config to public, allow all origins and all headers
I eventually figured out what to do after hours and hours of research and experimenting. There is very little documentation, as of this writing, available online showing how to do this. I hope people will find this helpful.
Understanding CORS:
CORS is an acronym for Cross Origin Resoruce Sharing. CORS is a new standard for sharing/accessing information between different domains. CORS basically is a method of using server headers to tell the browser if it is permitted to access or interact with a specific file on another server. While you can load most things without worrying about CORS (like images, audio, videos, and even other web pages), interaction with these elements requires special permission from the server. In my case, I was attempting to read frequencies from an audio file on another server. In this instance, I was attempting to access information which required authorization from special headers on the server.
Browser support is very good but, if you are supporting older browsers, you may want to see support tables here (http://caniuse.com/#search=cors)
What I did:
Enabled the use of .htaccess (I think you can accomplish the same thing with apache2.conf or 000-default.conf but .htaccess files are much easier to edit and maintain). These files are used to set headers and settings for apache. I enabled the use of .htaccess files by going to /etc/apache2/ and edited apache2.conf. Make sure your entry matches the following:
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
I set the headers in my .htaccess file to allow access from Codepen. Create a .htaccess file in the same directory as the file you want to share. You only want to share what you have to or you might create a security risk. Type this in your .htaccess file:
Header set Access-Control-Allow-Origin: "http://websiteWantingToAccessYourFile.com".
Save your file.
Restart Apache with this command sudo service apache2 restart. Enter your password if prompted.
With the audio, I added the crossorigin="anonymous" attribute. You can read more about CORS settings (crossorigin) here (https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_settings_attributes) I imagine you can set this with ajax and xhr requests.
Different versions of apache may have different file names or standards. Check to make sure this is correct for your version. I am running Apache 2.4.18 on my Ubuntu server.
Please tell me if this can be improved. I have spent a lot of time understanding this but I am not an expert. Post your questions or suggestions in the comments. :)
I have a mostly static HTML website served from CDN (plus a bit of AJAX to the server), and do want user's browsers to cache everything, until I update any files and then I want the user's browsers to get the new version.
How do I do achieve this please, for all types of static files on my site (HTML, JS, CSS, images etc.)? (settings in HTML or elsewhere). Obviously I can tell the CDN to expire it's cache, so it's the client side I'm thinking of.
Thanks.
One way to achieve this is to make use of the HTTP Last-Modified or ETag headers. In the HTTP headers of the served file, the server will send either the date when the page was last modified (in the Last-Modified header), or a random ID representing the current state of the page (ETag), or both:
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Fri, 18 Dec 2015 08:24:52 GMT
ETag: "208f11-52727df9c7751"
Cache-Control: must-revalidate
If the header Cache-Control is set to must-revalidate, it causes the browser to cache the page along with the Last-Modified and ETag headers it received with it. On the next request, it will send them as If-Modified-Since and If-None-Match:
GET / HTTP/1.1
Host: example.com
If-None-Match: "208f11-52727df9c7751"
If-Modified-Since: Fri, 18 Dec 2015 08:24:52 GMT
If the current ETag of the page matches the one that comes from the browser, or if the page hasn’t been modified since the date that was sent by the browser, instead of sending the page, the server will send a Not Modified header with an empty body:
HTTP/1.1 304 Not Modified
Note that only one of the two mechanisms (ETag or Last-Modified) is required, they both work on their own.
The disadvantage of this is that a request has to be sent anyways, so the performance benefit will mostly be for pages that contain a lot of data, but particularly on internet connections with high latency, the page will still take a long time to load. (It will for sure reduce your traffic though.)
Apache automatically generates an ETag (using the file’s inode number, modification time, and size) and a Last-Modified header (based on the modification time of the file) for static files. I don’t know about other web-servers, but I assume it will be similar. For dynamic pages, you can set the headers yourself (for example by sending the MD5 sum of the content as ETag).
By default, Apache doesn’t send a Cache-Control header (and the default is Cache-Control: private). This example .htaccess file makes Apache send the header for all .html files:
<FilesMatch "\.html$">
Header set Cache-Control "must-revalidate"
</FilesMatch>
The other mechanism is to make the browser cache the page by sending Cache-Control: public, but to dynamically vary the URL, for example by appending the modification time of the file as a query string (?12345). This is only really possible if your page/file is only linked from within your web application, in which case you can generate the links to it dynamically. For example, in PHP you could do something like this:
<script src="script.js?<?php echo filemtime("script.js"); ?>"></script>
To achieve what you want on the client side, you have to change the url of your static files when you load them in HTML, i.e. change the file name, add a random query string like unicorn.css?p=1234, etc. An easy way to automate this is to use a task runner such as Gulp and have a look at this package gulp-rev.
In short, if you integrate gulp-rev in your Gulp task, it will automatically append a content hash to all the static files piped into the task stream and generate a JSON manifest file which maps the old files to newly renamed files. So a file like unicorn.css will become unicorn-d41d8cd98f.css. You can then write another Gulp task to crawl through your HTML/JS/CSS files and replace all the urls or use this package gulp-rev-replace.
There should be plenty of online tutorial that shows you how to accomplish this. If you use Yeoman, you can check out this static webapp generator I wrote here which contains a Gulp routine for this.
This is what the HTML5 Application Cache does for you. Put all of your static content into the Cache Manifest and it will be cached in the browser until the manifest file is changed. As an added bonus, the static content will be available even if the browser is offline.
The only change to your HTML is in the <head> tag:
<!DOCTYPE HTML>
<html manifest="cache.appcache">
...
</html>
I know that it's better to use something like AWS for static files but I am on developing stage and I prefer to have javascript/css files on localhost.
It would be great if I could get gzip working on my javascript files for testing. I am using the default gzip middleware but it's just compressing the view request.
My template looks like:
<script src='file.js' type='application/javascript'></script>
There should be a type-of-file list similar to Nginx for the django-based-server. How can I add application/javascript, text/javascript, etc for gzip compression?
You should read the GZipMiddleware documentation, where it's explained that the middleware will not compress responses when the "Content-Type header contains javascript or starts with anything other than text/".
EDIT:
To clarify what the documentation says, if the Content-Type header value contains javascript or doesn't begin with text/, then the response won't be compressed. That means both text/javascript and application/javascript will be invalid responses, since they match javascript.
Those restrictions are intentionally imposed by the middleware itself, but you can still circumvent that by wrapping the static files view handler with the gzip_page() decorator and adding it to your URL configuration manually.
During development you are using the Django built-in webserver, this server is really simple and does not have any other options than what you can see with ./manage.py help runserver
You options are either to setup a real webserver or use the staticfiles app with a custom StaticFilesStorage
But honestly, this is overkill, why would want to test gzip compression?