We Keep Coding

How to bundle js files in none node environment?

I'm developing websites with several js files and I want to bundle these js files into one js file. I started looking at Webpack, but it requires node environment to run. In fact, all my js files are none-node style, and each one of them is independent. My development environment is not node, so I'm wondering how to make all my js files into one js file.

Your js files do not need to be written as CommonJS modules ("node style") in order to bundle them with webpack.
If you want you can use loaders like the imports-loader and exports-loader to make scripts not written as CommonJS modules accessible in a webpack context.
However, it sounds like you may not even need webpack for your use case.
I would recommend using a simple gulp recipe to concatenate and minify your existing JavaScript files into a single file.
const concat = require('gulp-concat')
const gulp = require('gulp')
const rename = require('gulp-rename')
const uglify = require('gulp-uglify')
gulp.task('default', function () {
// Find all the JavaScript files in the src directory.
return gulp.src('src/*.js')
// Concatenate them all together and name the resulting sciprt app.js.
.pipe(concat('app.js'))
// Minify the script to save space.
.pipe(uglify())
// Change the file extension.
.pipe(rename({ extname: '.min.js' }))
// Output to the dist directory.
.pipe(gulp.dest('dist/'))
})

Assemble every module into a single .js file

I want to minimize the number of HTTP requests from the client to load scripts in the browser. This is going to be a pretty general question but I still hope I can get some answers because module management in javascript has been a pain so far.
Current situation
Right now, in development, each module is requested individually from the main html template, like this:
<script src="/libraries/jquery.js"></script>
<script src="/controllers/controllername.js"></script>
...
The server runs on Node.js and sends the scripts as they are requested.
Obviously this is the least optimal way of doing so, since all the models, collections, etc. are also separated into their own files which translates into numerous different requests.
As far as research goes
The libraries I have come across (RequireJS using AMD and CommonJS) can request modules from within the main .js file sent to the client, but require a lot of additional work to make each module compliant with each library:
;(function(factory){
if (typeof define === 'function' && define.amd) define([], factory);
else factory();
}(function(){
// Module code
exports = moduleName;
}));
My goal
I'd like to create a single file on the server that 'concatenates' all the modules together. If I can do so without having to add more code to the already existing modules that would be perfect. Then I can simply serve that single file to the client when it is requested.
Is this possible?
Additionally, if I do manage to build a single file, should I include the open source libraries in it (jQuery, Angular.js, etc.) or request them from an external cdn on the client side?

What you are asking to do, from what I can tell, is concat your js files into one file and then in your main.html you would have this
<script src="/pathLocation/allMyJSFiles.js"></script>
If my assumption is correct, then the answer would be to use one of the two following items
GULP link or GRUNT link
I use GULP.
You can either use gulp on a case by case basis, which means calling gulp from the command line to execute gulp code, or use a watch to do it automatically on save.
Besides getting gulp to work and including the gulp files you need to do what you need, I will only provide a little of what I use to get your answer.
In my gulp file I would have something like this
var gulp = require('gulp');
var concat = require('gulp-concat');
...maybe more.
Then I have the file paths I need to be reduced into one file.
var onlyProductionJS = [
'public/application.js',
'public/directives/**/*.js',
'public/controllers/**/*.js',
'public/factories/**/*.js',
'public/filters/**/*.js',
'public/services/**/*.js',
'public/routes.js'
];
and I use this info in a gulp task like the one below
gulp.task('makeOneFileToRuleThemAll', function(){
return gulp.src(onlyProductionJS)
.pipe(concat('weHaveTheRing.js'))
.pipe(gulp.dest('public/'));
});
I then run the task in my command line by calling
gulp makeOneFileToRuleThemAll
This call runs the associated gulp task which uses 'gulp-concat' to get all the files together into one new file called 'weHaveTheRing.js' and creates that file in the destination 'public/'
Then just include that new file into your main.html
<script src="/pathLocation/weHaveTheRing.js"></script>
As for including all your files into one file, including your vendor files, just make sure that your vendor code runs first. It's probably best to keep those separate unless you have a sure fire way of getting your vendor code to load first without any issues.
UPDATE
Here is my gulp watch task.
gulp.task('startTheWatchingEye', function () {
gulp.watch(productionScripts, ['makeOneFileToRuleThemAll']);
});
Then I start up my server like this (yours may differ)
npm start
// in a different terminal window I then type
gulp startTheWatchfuleye
NOTE: you can use ANY movie or show reference you wish! :)
Now just code it up, every time you make a change in the specified files GULP will run your task(s).
If you want to say run Karma for your test runner...
add the following to your gulp file
var karma = require('karma').server;
gulp.task('karma', function(done){
karma.start({
configFile: __dirname + '/karma.conf.js'
}, done);
});
Then add this task karma to your watch I stated above like this...
gulp.task('startTheWatchingEye', function(){
gulp.watch(productionScripts, ['makeOneFileToRuleThemAll', 'karma']);
});
ALSO
Your specific settings may require a few more gulp modules. Usually, you install Gulp globally, as well as each module. Then use them in your various projects. Just make sure that your project's package.json has the gulp modules you need in dev or whatever.
There are different articles on whether to use Gulp or Grunt. Gulp was made after Grunt with a few additions that Grunt was lacking. I don't know if Grunt lacks them anymore. I like Gulp a lot though and find it very useful with a lot of documentation.
Good luck!

I'd like to create a single file on the server that 'concatenates' all the modules together. If I can do so without having to add more code to the already existing modules that would be perfect.
Sure you can. You can use Grunt or Gulp to do that, more specifically grunt-contrib-concat or gulp-concat
Here's an example of a Gruntfile.js configuration to concat every file under a js directory:
grunt.initConfig({
concat: {
dist: {
files: {
'dist/built.js': ['js/**/**.js'],
},
},
},
});
Also, you can minify everything after concatenating, using grunt-contrib-minify.
Both libraries support source maps so, in the case a bug gets to production, you can easily debug.
You can also minify your HTML files using grunt-contrib-htmlmin.
There's also an extremely useful library called grunt-usemin. Usemin let's you use HTML comments to "control" which files get minified (so you don't have to manually add them).
The drawback is that you have to explicitely include them in your HTML via script tags, so no async loading via javascript (with RequireJS for instance).
Additionally, if I do manage to build a single file, should I include the open source libraries in it (jQuery, Angular.js, etc.) or request them from an external cdn on the client side?
That's debatable. Both have pros and cons. Concatenating vendors assures that, if for some reason, the CDN isn't available, your page works as intended. However the file served is bigger so you consume more bandwidth.
In my personal experience, I tend to include vendor libraries that are absolutely essential for the page to run such as AngularJS for instance.

If I understand you correctly, you could use a task runner such as Grunt to concatenate the files for you.
Have a look at the Grunt Concat plugin.
Example configuration from the docs:
// Project configuration.
grunt.initConfig({
concat: {
dist: {
src: ['src/intro.js', 'src/project.js', 'src/outro.js'],
dest: 'dist/built.js',
}
}
});
Otherwise, as you have stated, a 'module loader' system such as Require JS or Browserify may be the way to go.

Gulp - Target all files in a folder and its subfolders

I'd like to be able to add a watch task in gulp to all of the js files in the frontend/js and any other js files below
gulp.watch('./frontend/js/**/*.js', ['browserify']);
This will only target js files one folder deep

It's supposed to match any number of subdirectories:
** If a "globstar" is alone in a path portion, then it matches zero or more directories and subdirectories searching for matches. It does not crawl symlinked directories.
https://github.com/isaacs/node-glob
Do you have symlinked directories in there?
Symlinks
I don't think you'll get gulp to natively traverse your symlinked directories. I recommend you take a look at node.js fs.readdir recursive directory search and see if any of those solutions can be applied to your use case. Nothing in the question or answers specifically addresses symlinks, so I don't know if there's a solution for you there or not. If you can get an array of dereferenced pathnames using one of those solutions, then you can just pass the array to gulp.src().

I just did some testing - and this actually works just fine for me.
I currently have the following structure -
--apps
--scripts
----test.js
----test-folder
------test2.js
------test-folder-deep
--------test3.js
--myApp
----scripts-symlinked (symlinked to apps/scripts)
----gulpfile.js
I set up my symlink folder (on Mac - from 'myApp' folder) using:
ln -s /Users/kandrews/apps/scripts ./scripts-symlinked
In my gulpfile.js I have the following:
var gulp = require('gulp'),
jshint = require('gulp-jshint');
gulp.task('jshint', function () {
gulp.src('./scripts-symlinked/**/*.js')
.pipe(jshint())
.pipe(jshint.reporter('default'));
});
gulp.task('watch', function () {
gulp.watch('./scripts-symlinked/**/*.js', ['jshint']);
});
Works perfectly. I also tried this in a sub directory as well ('scripts/symlinked-scripts') and was successful as well.

I think it’s not worth doing difficult:
gulp.watch('./frontend/js/', ['browserify']);

concat underscore templates in a directory to a single js file and load it for use with a backbone app

I'd like to combine all the underscore templates in a directory to a single js file ( possibly precompile it ? ) for use with the backbonejs part of my sails.js app.
I think I can use the plain fs module with nodejs to read the files and combine them, I'm looking at grunt to do this as well but still not sure.
Can someone help me with this ?

You can use grunt to do this. The plugin grunt-contrib-jst is what you're looking for; it has installation and usage documents here.
A simple Gruntfile.js like this should do it. (This example assumes all your source code is under a src/ subdirectory, all templates are in *.html files under src/, and you're creating an output file in the build/ subdirectory. Adjust as needed to fit your actual situation.) Run by typing grunt on the command line.
Just include the single file build/view-templates.js in your index file to load all of your Underscore view templates.
module.exports = function(grunt) {
grunt.loadNpmTasks('grunt-contrib-jst');
grunt.initConfig({
// compile view templates into single file
jst: {
compile: {
files: {
"build/view-templates.js": ["src/**/*.html"]
}
}
}
});
grunt.registerTask('default', 'jst' );
};

Working project structure that uses grunt.js to combine JavaScript files using RequireJS?

I have some projects that use RequireJS to load individual JavaScript modules in the browser, but I haven't optimized them yet. In both development and production, the app makes a separate request for each JavaScript file, and now I would like to fix that using Grunt.
I have tried to put together a simple project structure to no avail, so I'm wondering if someone can provide a working example for me. My goals are the following:
In development mode, everything works in the browser by issuing a separate request for each required module. No grunt tasks or concatenation are required in development mode.
When I'm ready, I can run a grunt task to optimize (combine) all of the JavaScript files using r.js and test that out locally. Once I'm convinced the optimized application runs correctly, I can deploy it.
Here's a sample structure for the sake of this conversation:
grunt-requirejs-example/
grunt.js
main.js (application entry point)
index.html (references main.js)
lib/ (stuff that main.js depends on)
a.js
b.js
requirejs/
require.js
text.js
build/ (optimized app goes here)
node_modules/ (necessary grunt tasks live here)
Specifically, I'm looking for a working project structure that I can start from. My main questions are:
If this project structure is flawed, what do you recommend?
What exactly needs to be in my grunt.js file, especially to get the r.js optimizer working?
If all of this isn't worth the work and there's a way to use the grunt watch task to automatically build everything in development mode every time I save a file, then I'm all ears. I want to avoid anything that slows down the loop from making a change to seeing it in the browser.

I use the grunt-contrib-requirejs task to build project based on require.js. Install it inside your project directory with:
npm install grunt-contrib-requirejs --save-dev
BTW: --save-dev will add the package to your development dependencies in your package.json. If you're not using a package.json in your project, ignore it.
Load the task in your grunt file with:
grunt.loadNpmTasks('grunt-contrib-requirejs');
And add the configuration to your grunt.initConfig
requirejs: {
production: {
options: {
baseUrl: "path/to/base",
mainConfigFile: "path/to/config.js",
out: "path/to/optimized.js"
}
}
}
Now you're able to build your require.js stuff into a single file that will be minimized with uglifyjs by running grunt requirejs
You can bundle a set of different tasks into some sort of main task, by adding this to your grunt file
grunt.registerTask('default', ['lint', 'requirejs']);
With this, you can simply type grunt and grunt will automatically run the default task with the two 'subtasks': lint and requirejs.
If you need a special production task: define it like the above
grunt.registerTask('production', ['lint', 'requirejs', 'less', 'copy']);
and run it with
grunt production
If you need different behaviors for 'production' and 'development' inside i.e. the requirejs task, you can use so called targets. In the configuration example above it's already defined as production. You can add another target if you need (BTW, you can define a global config for all targets by adding a options object on the same level)
requirejs: {
// global config
options: {
baseUrl: "path/to/base",
mainConfigFile: "path/to/config.js"
},
production: {
// overwrites the default config above
options: {
out: "path/to/production.js"
}
},
development: {
// overwrites the default config above
options: {
out: "path/to/development.js",
optimize: none // no minification
}
}
}
Now you can run them both at the same time with grunt requirejs or individually with grunt requirejs:production, or you define them in the different tasks with:
grunt.registerTask('production', ['lint', 'requirejs:production']);
grunt.registerTask('development', ['lint', 'requirejs:development']);
Now to answer your questions:
I would definitely use a subfolder in your project. In my case I use a 'src' folder for development that is build into a 'htdocs' folder for production. The project layout I prefere is:
project/
src/
js/
libs/
jquery.js
...
appname/
a.js
b.js
...
main.js // require.js starter
index.html
...
build/
... //some tmp folder for the build process
htdocs/
... // production build
node_modules/
...
.gitignore
grunt.js
package.json
see above
You can do so, but I wouldn't recommend to add requirejs to the watch task, it's a resource hungry task and it will slow down your machine noticeable.
Last but not least: Be very cautious when playing around with r.js. Especially when you want to optimize the whole project with r.js by adding a modules directive to your config. R.js will delete the output directory without asking. If it happens that it is accidentally configured to be your system root, r.js will erase your HDD. Be warned, I erased my whole htdocs folder permanently some time ago while setting up my grunt task... Always add keepBuildDir:true to your options when playing around with the r.js config.