I have a fairly large multi-page javascript applications that uses requirejs to organize code. I am researching moving to browserify because I like the simplicity that it offers and I am already used to the node.js module system.
Currently on each page I have javascript that goes like this
<script data-main="/scripts/config/common" src="/scripts/lib/require.js">
<script data-main="/scripts/config/page-specific-js" src="/scripts/lib/require.js">
and I have a common build step and a build for each page. This way the majority of the JS is cached for every page.
Is it possible to do something similar with browserify? Is caching like this even worth it, or is it better to bundle everything into one file across all pages (considering that maybe only one page can depend on a large external library)?
You can use factor-bundle to do exactly this. You will just need to split your code up into different entry points for each file.
Suppose you have 3 pages, /a, /b, and /c. Each page corresponds to an entry point file of /browser/a.js, /browser.b.js, and /browser/c.js. With factor-bundle, you can do:
browserify -p [ factor-bundle -o bundle/a.js -o bundle/b.js -o bundle/c.js ] \
browser/a.js browser/b.js browser/c.js > bundle/common.js
any files used by more than 1 of the entry points will be factored out into bundle/common.js, while all the page-specific code will be located in the page-specific bundle file. Now on each page you can put a script tag for the common bundle and a script tag for the page-specific bundle. For example, for /a you can put:
<script src="bundle/common.js"></script>
<script src="bundle/a.js"></script>
If you don't want to use the command-line version, you can also use factor-bundle from the API:
var browserify = require('browserify');
var fs = require('fs');
var files = [ './files/a.js', './files/b.js', './files/c.js' ];
var b = browserify(files);
b.plugin('factor-bundle', {
outputs: [ 'bundle/a.js', 'bundle/b.js', 'bundle/c.js' ]
});
b.bundle().pipe(fs.createWriteStream('bundle/common.js'));
Related
I'm using Vue.js to build my frontend app, and in that page, I have (too many) Javascript external resources : Google Analytics, Typekit, Google Charts, Intercom, Raven, etc.
Is there a way to configure my Webpack config file and tell it to download the links that are in the index.html (even by adding the urls in the specific file) that would then be appended to my generated js.
The aim here is to reduce the load of JS files, and avoid issues (like not loaded libraries).
Thank you for your help!
Although it may sound like a straightforward improvement (less requests -> faster page load) it's generally not a good idea to combine all of your scripts into one bundle.
There are plenty of reasons:
CDN's are fast. Unless your bundle is also served via CDN, merging these scripts would be counterproductive.
With even the smallest code fix You have to force the client to invalidate the cache and download the whole bundle again.
Popular scripts (like GA) might have been already cached by user's browser, so they won't be downloaded at all.
Third party scripts have a tendency to a) be updated b) load other scripts. So if you bundle one specific version of such a script, it could become broken the moment after.
What can be done
Most of these scripts are not crucial to the page render, so you could
Load the scripts in a non-blocking manner.
Load the scripts after the page is fully rendered.
I understand the consequences, but I really need to bundle those scripts!
Apparently, you could not bundle remote script with webpack:
webpack is a module bundler not a javascript loader. It package files from local disk and don't load files from the web (except its own chunks).
Use a javascript loader, i. e. script.js.
var $script = require("scriptjs");
$script("//ajax.googleapis.com/ajax/libs/angularjs/1.2.9/angular.min.js", function() {
// ...
});
However, You can prepend your bundle with external scripts using some kind of task runner, like gulp or grunt.
var gulp = require('gulp');
var source = require('vinyl-source-stream');
var request = require('request');
var merge = require('merge2');
var concat = require('gulp-concat');
var buffer = require('gulp-buffer');
gulp.task('js', function() {
var jquery = request('http://code.jquery.com/jquery-latest.js').pipe(source('jquery.js'));
var main = gulp.src('main.js');
return merge(jquery, main)
.pipe(buffer())
.pipe(concat('concat.js'))
.pipe(gulp.dest('dist'));
});
Since You could integrate webpack into your building pipeline, it seems like a fairy decent approach.
I am using Webpack to bundle my clientside modules and would like to take advantage of parallel asset downloading. I would like html somewhat like this:
<script src="vendor/react.js">
<script src="vendor/underscore.js">
<script src="build/bundle.js">
Where bundle.js contains:
var React = require('react');
var _ = require('underscore');
Note that vendor/react.js and vendor/underscore.js would also be bundled by Webpack.
I know that the Webpack CommonsChunkPlugin can extract all the vendor modules and put them into a single common vendor.js file. Is it possible, though, to split that common output file into two or more output files?
After a bit more digging, I found an answer.
Soundcloud created a webpack plugin specifically for this purpose (see here) though it lacked support for splitting code in the node_modules folder. Another plugin based on it solved that problem and was trivial to set up.
I want to minimize the number of HTTP requests from the client to load scripts in the browser. This is going to be a pretty general question but I still hope I can get some answers because module management in javascript has been a pain so far.
Current situation
Right now, in development, each module is requested individually from the main html template, like this:
<script src="/libraries/jquery.js"></script>
<script src="/controllers/controllername.js"></script>
...
The server runs on Node.js and sends the scripts as they are requested.
Obviously this is the least optimal way of doing so, since all the models, collections, etc. are also separated into their own files which translates into numerous different requests.
As far as research goes
The libraries I have come across (RequireJS using AMD and CommonJS) can request modules from within the main .js file sent to the client, but require a lot of additional work to make each module compliant with each library:
;(function(factory){
if (typeof define === 'function' && define.amd) define([], factory);
else factory();
}(function(){
// Module code
exports = moduleName;
}));
My goal
I'd like to create a single file on the server that 'concatenates' all the modules together. If I can do so without having to add more code to the already existing modules that would be perfect. Then I can simply serve that single file to the client when it is requested.
Is this possible?
Additionally, if I do manage to build a single file, should I include the open source libraries in it (jQuery, Angular.js, etc.) or request them from an external cdn on the client side?
What you are asking to do, from what I can tell, is concat your js files into one file and then in your main.html you would have this
<script src="/pathLocation/allMyJSFiles.js"></script>
If my assumption is correct, then the answer would be to use one of the two following items
GULP link or GRUNT link
I use GULP.
You can either use gulp on a case by case basis, which means calling gulp from the command line to execute gulp code, or use a watch to do it automatically on save.
Besides getting gulp to work and including the gulp files you need to do what you need, I will only provide a little of what I use to get your answer.
In my gulp file I would have something like this
var gulp = require('gulp');
var concat = require('gulp-concat');
...maybe more.
Then I have the file paths I need to be reduced into one file.
var onlyProductionJS = [
'public/application.js',
'public/directives/**/*.js',
'public/controllers/**/*.js',
'public/factories/**/*.js',
'public/filters/**/*.js',
'public/services/**/*.js',
'public/routes.js'
];
and I use this info in a gulp task like the one below
gulp.task('makeOneFileToRuleThemAll', function(){
return gulp.src(onlyProductionJS)
.pipe(concat('weHaveTheRing.js'))
.pipe(gulp.dest('public/'));
});
I then run the task in my command line by calling
gulp makeOneFileToRuleThemAll
This call runs the associated gulp task which uses 'gulp-concat' to get all the files together into one new file called 'weHaveTheRing.js' and creates that file in the destination 'public/'
Then just include that new file into your main.html
<script src="/pathLocation/weHaveTheRing.js"></script>
As for including all your files into one file, including your vendor files, just make sure that your vendor code runs first. It's probably best to keep those separate unless you have a sure fire way of getting your vendor code to load first without any issues.
UPDATE
Here is my gulp watch task.
gulp.task('startTheWatchingEye', function () {
gulp.watch(productionScripts, ['makeOneFileToRuleThemAll']);
});
Then I start up my server like this (yours may differ)
npm start
// in a different terminal window I then type
gulp startTheWatchfuleye
NOTE: you can use ANY movie or show reference you wish! :)
Now just code it up, every time you make a change in the specified files GULP will run your task(s).
If you want to say run Karma for your test runner...
add the following to your gulp file
var karma = require('karma').server;
gulp.task('karma', function(done){
karma.start({
configFile: __dirname + '/karma.conf.js'
}, done);
});
Then add this task karma to your watch I stated above like this...
gulp.task('startTheWatchingEye', function(){
gulp.watch(productionScripts, ['makeOneFileToRuleThemAll', 'karma']);
});
ALSO
Your specific settings may require a few more gulp modules. Usually, you install Gulp globally, as well as each module. Then use them in your various projects. Just make sure that your project's package.json has the gulp modules you need in dev or whatever.
There are different articles on whether to use Gulp or Grunt. Gulp was made after Grunt with a few additions that Grunt was lacking. I don't know if Grunt lacks them anymore. I like Gulp a lot though and find it very useful with a lot of documentation.
Good luck!
I'd like to create a single file on the server that 'concatenates' all the modules together. If I can do so without having to add more code to the already existing modules that would be perfect.
Sure you can. You can use Grunt or Gulp to do that, more specifically grunt-contrib-concat or gulp-concat
Here's an example of a Gruntfile.js configuration to concat every file under a js directory:
grunt.initConfig({
concat: {
dist: {
files: {
'dist/built.js': ['js/**/**.js'],
},
},
},
});
Also, you can minify everything after concatenating, using grunt-contrib-minify.
Both libraries support source maps so, in the case a bug gets to production, you can easily debug.
You can also minify your HTML files using grunt-contrib-htmlmin.
There's also an extremely useful library called grunt-usemin. Usemin let's you use HTML comments to "control" which files get minified (so you don't have to manually add them).
The drawback is that you have to explicitely include them in your HTML via script tags, so no async loading via javascript (with RequireJS for instance).
Additionally, if I do manage to build a single file, should I include the open source libraries in it (jQuery, Angular.js, etc.) or request them from an external cdn on the client side?
That's debatable. Both have pros and cons. Concatenating vendors assures that, if for some reason, the CDN isn't available, your page works as intended. However the file served is bigger so you consume more bandwidth.
In my personal experience, I tend to include vendor libraries that are absolutely essential for the page to run such as AngularJS for instance.
If I understand you correctly, you could use a task runner such as Grunt to concatenate the files for you.
Have a look at the Grunt Concat plugin.
Example configuration from the docs:
// Project configuration.
grunt.initConfig({
concat: {
dist: {
src: ['src/intro.js', 'src/project.js', 'src/outro.js'],
dest: 'dist/built.js',
}
}
});
Otherwise, as you have stated, a 'module loader' system such as Require JS or Browserify may be the way to go.
I'm working on a large backbone single page application where we've all the JS files specified in order just before </body> tag. Then, for production, using grunt-usemin we concatenate and make single minified app.min.js file for production. Also all the templates are converted to JS using grunt-contrib-handlebars and concatenated into the same app.min.js file.
As the code is growing, size of app.min.js file has gone up to 1.25MB and still this application has long way to go. There are many major sections yet to be developed.
At the same time, I don't want to load 5 or more JS files and same number of templates when user visits to each screen. Rather, I wanted to have modules and load single module file and user can browse through whole of the module without loading any other JS file.
Also I want those module files to be minified and optimized and revved (for cache busting) for production
I was looking through require.js and it's optimizer. I've just started implementing one of our next module using require.js and when I came to building for deployment, it seems that r.js optimizer creates single output file which brings me back to square.
Is it not possible to do what I'm looking for using require.js, or am I missing something? Or is there any better solution of my problem than require.js?
NOTE: Though there is a benefit of using loader even though single big file is loaded at once, I don't want to load whole code when I just need one module of code.
you can build into multiple files.
inside requirejs:compile:options: do the below
modules: [
{
name: '../build1',
include: [
// 3rd party libs
'backbone', //the reference to these files are present in paths: property
'underscore'
]
},
//build 2
{
name: '../build2',
include: ['build2ReqFile1','build2ReqFile2'],
// Excludes all nested dependencies and built dependencies from "common"
exclude: ['../build1','build2UnReqFile1']
},
//build 3
{
name: '../build3',
include: ['build3ReqFile1','build3ReqFile2'],
// Excludes all nested dependencies and built dependencies from "common"
exclude: ['../build1','../build2','build3UnReqFile1']
},
]
Updating with explanation
yep this is for single page application.
Few things to understand here - whenever you define i.e. define('app/test',['app/load1','app/load2'],function(load1Ref, load2Ref){}), new script tag is added to dom and a data-xyz = name of module i.e. <script data-xyz = 'app/test' src = '' type='text/javascript'> is added. I dont remember what the name of xyz is. The returned value of the define is stored in the context data-xyz. So when ever you do require(['app/test']) it returns the value present in data-xyz of app/test script.
Now when you build, all the require and defines that are referenced are built into one single .js. SO by doing the above you can have multiple builds where in you mention which of the defines each of these individual builds should include or exclude.
So these builds are nothing all the javascript files minified and compiled into one/multiple files. Depending on which files you require you need to include these built files.
If you require 'build2ReqFile1','build2ReqFile2' then you can just include build2.js in your code, if you want backbone and underscore along with 'build2ReqFile1','build2ReqFile2' then you need to include build2.js and build1.js
Is it clear?
I have a directory like below:
/folder/b.js
/folder/jQuery.js
/folder/a.js
/folder/sub/c.js
I want to minify all these js files in one js file in order:
jQuery.js -> a.js -> b.js -> c.js
Q:
1.How can I do it via grunt-contrib-uglify?(In fact, there are lots of files, it is impractical to specify all source filepaths individually)
2.btw, How can I get unminified files when debug and get minified single file when release and no need to change script tag in html(and how to write the script tag)?
Good questions!
1) Uglify will reorder the functions in the destination file so that function definitions are on top and function execution on bottom but it seems that it will preserve the order of the function executions.
This means that the function jQuery runs to define its global functions will be put first if you make sure jQuery is mentioned first in Uglify's config in the Gruntfile.
I use this config:
uglify: {
options: {
sourceMap: true
},
build: {
files: {
'public/all.min.js': ['public/js/vendor/jquery-1.10.2.min.js', 'public/js/*.js'],
}
}
}
2) I don't think there is one definite way to accomplish this. It depends on what web framework, templating framework and what kind of requirements you have. I use express + jade and in my main jade layout I have:
if process.env.NODE_ENV === 'production'
script(src='/all.min.js')
else
script(src='/js/vendor/jquery-1.10.2.min.js')
script(src='/js/someScript.js')
script(src='/js/otherScript.js')
In my package.json I have:
"scripts": {
"postinstall": "grunt"
},
This means that when I run npm install on deploy (on Heroku) grunt is run to minify/concat files and when the app is started with NODE_ENV=production the minified client side javascript is used. Locally I get served the original client side javascripts for easy debugging.
The two downsides are:
I have to keep the two lists of script files in sync (in the Gruntfile and in the layout.js) I solve this by using *.js in the Gruntfile but this may not suite everyone. You could put the list of javascripts in the Gruntfile and create a jade-template from this but it seems overkill for most projects.
If you don't trust your Grunt config you basically have to test running the application using NODE_ENV=production locally to verify that the minification worked the way you intended.
This can be done using the following Grunt tasks:
https://github.com/gruntjs/grunt-contrib-concat concatenates
files
https://github.com/gruntjs/grunt-contrib-uglify minifies
concatenated files
EDIT
I usually run all my files through a Grunt concatenation task using grunt-contrib-concat. Then I have another task to uglify the concatenated file using grunt-contrib-uglify.
You're probably not going to like this, but the best way is to define your js source files as AMD modules and use Requirejs to manage the order in which they load. The grunt-contrib-requirejs task will recurse your dependency tree and concatenate the js files in the necessary order into one big js file. You will then use uglify (actually r.js has uglify built-in) to minify the big file.
https://github.com/danheberden/yeoman-generator-requirejs has a good example gruntfile and template js files to work from.
EDIT
I've recently started using CommonJS modules instead of AMD since it's much closer to the ES6 module spec. You can achieve the same results (1 big complied+concatenated js file) by running commonjs modules through Browserify. There are plugins for both grunt and gulp to manage the task for you.
EDIT
I'd like to add that if your site is written using ES6 that Rollup is the best new concatenating package. In addition to bundling your files, it will also perform tree shaking, removing parts of libraries you use if included via an import statement. This reduces your codebase to just what you need without the bloat of code you'll never use.
I don't think you can do this with the uglify task alone, but you have a multitude of choices which might lead to your desired outcome.
A possible workflow would be first concatenating (grunt-contrib-concat) the files in order into one single file, and put this concatenated file through uglify. You can either define the order for concat in your Gruntfile, or you use on of those plugins:
First one would be https://github.com/yeoman/grunt-usemin, where you can specify the order in your HTML file, put some comments around your script block. The Google guys made it and it's pretty sweet to use.
Second one would be https://github.com/trek/grunt-neuter, where you can define some dependencies with require, but without the bulk of require.js. It requires changes in your JS code, so might not like it. I'd go with option one.
I ran into the same issue. A quick fix is just to change the filenames - I used 1.jquery.min.js, 2.bootstrap.min.js, etc.
This might be only remotely related to your question but I wanted something similar. Only my order was important in the following way:
I was loading all vendor files (angular, jquery, and their respective related plugins) with a wildcard (['vendor/**/*.js']). But some plugins had names that made them load before angular and jquery. A solution is to manually load them first.
['vendor/angular.js', 'vendor/jquery.js', 'vendor/**/*.js]
Luckily angular and jquery handle being loaded twice well enough. Edit: Although it's not really the best practice to load such large libraries twice, causing your minified file unnecessary bloat. (thanks #Kano for pointing this out!)
Another issue was client-js the order was important in a way that it required the main app file to be loaded last, after all its dependencies have been loaded. Solution to that was to exclude and then include:
['app/**/*.js', '!app/app.js', 'app/app.js']
This prevents app.js from being loaded along with all the other files, and only then includes it at the end.
Looks like the second part of your question is still unanswered. But let me try one by one.
Firstly you can join and uglify a large number of js files into one as explained by the concat answer earlier. It should also be possible to use https://github.com/gruntjs/grunt-contrib-uglify because it does seem to have wildcards. You may have to experiment with 'expand = true' option and wildcards. That takes care of your first question.
For the second part, say you joined and uglified into big-ugly.js
Now in your html you can add following directives:
<!-- build:js:dist big-ugly.js -->
<script src="js1.js"></script>
<script src="js2.js"></script>
<!-- etc etc -->
<script src="js100.js"></script>
<!-- /build -->
And then pass it through the grunt html preprocessor at https://www.npmjs.com/package/grunt-processhtml as part of your grunt jobs.
This preprocessor will replace the entire block with
<script src="big-ugly.js"></script>
Which means that the html file with be semantically equivalent - before and after the grunt jobs; i.e. if the page works correctly in the native form (for debugging) - then the transformed page must work correctly after the grunt - without requiring you to manually change any tags.
This was #1469's answer but he didn't make it clear why this works. Use concat to put all js files into one, this module does this in the order of file names, so I put a prefix to the file names based on orders. I believe it even has other options for ordering.
concat: {
js: {
options: {
block: true,
line: true,
stripBanners: true
},
files: {
'library/dist/js/scripts.js' : 'library/js/*.js',
}
}
},
Then use uglify to create the minified ugly version:
uglify: {
dist: {
files: {
'library/dist/js/scripts.min.js': [
'library/js/scripts.js'
]
},
options: {
}
}
},
If your problem was that you had vendors which needed to be loaded in order (let's say jquery before any jquery plugins). I solved it by putting jquery in its own folder called '!jquery', effectively putting it on top of the stack.
Then I just used concat as you normally would:
concat: {
options: {
separator: ';',
},
build: {
files: [
{
src: ['js/vendor/**/*.js', 'js/main.min.js'],
dest: 'js/global.min.js'
}
]
}
},