We Keep Coding

Javascript bundling and module loading

I've recently been thrown in to clean up a project which has like 45-50 individual .js javascript files. I wonder what the best approach would be to decrease the loading size of them all. Just concatenate all files into one with npm or gulp? Install some module loader? webpack?

If you're already concatenating, minifying, and uglifying and you don't want all the files to be loaded on all the pages due to a monolithic bundle, you might be looking for something like Webpack's Commons Chunk Plugin.
This plugin walks down the tree of dependencies for each endpoint defined in your Webpack.config file and determines which modules are required across all pages. It then breaks the code into two bundles, a "common" bundle containing the modules that every page requires, which you must load with a script tag on each page:
<script src="commons.js" charset="utf-8"></script>
And an endpoint bundle for each individual page that you reference normally in a script tag placed after the commons script tag:
<script src="specificpage.bundle.js" charset="utf-8"></script>
The result is that an individual page will not have to load modules that will only ever be used on other pages.
Again, this is a Webpack plugin. I don't know if this functionality is available as a Gulp plugin, because it must have knowledge of all endpoints in order to determine which dependencies are common to them all.

I redirect you to the very good https://github.com/thedaviddias/Front-End-Checklist
In particular the following advises:
JavaScript Inline: High You don't have any JavaScript code inline
(mixed with your HTML code).
Concatenation: High JavaScript files
are concatenated.
Minification: High JavaScript files are minified (you can add the .min suffix).
You can accomplish this with a package manager such as gulp, grunt or webpack (for the most famous ones). You just need to choose what you prefer to use.
If you consider webpack, You can start with my very simple (but understanding) starter: https://github.com/dfa1234/snippets-starter

There's no much thing that you can do, basically is:
Concatenation - https://www.npmjs.com/package/gulp-concat
Minification - https://www.npmjs.com/package/gulp-minify
Instead of creating all those scripts, you can get something to re-use on yeoman, f.e. the Fountain, so it will reduce a lot of time just typing procedural code for doing the concatenation/minification.
Also if you can use some lazy load (like RequireJS or some frameworks have support to lazy load the module, like Angular) that will improve the performance of your aplication
EDIT:
If you want even more performance, you can install some compression tool in your server, for example this one for NodeJS https://www.npmjs.com/package/compression

I'm my personal opinion, if you have time, the best approach would be to read and understand the purpose of the project. Then plan a proper refactor. You are not fixing anything with concatenating, this is just a deployment step.
You should analyze which technologies are being used and if you want to maintain this code, in the long run, make a proper refactor into a much more modern stack, maybe you can take a seed project with ES6, webpack, Babel... and create a proper repository well maintained with proper modularity and dependencies resolution.
Once you have that, decreasing the load its just about adding proper tools in build time (babel, webpack, etc).
You would like to add some unit tests and continue working properly :)

Extend an application with webpack like requireJs

Existing App:
My application is using require.js.
Benefit of my application is that other people can extend my application via writing 3rd party plugins using require.js
For example: (a 3rd party plugin registration)
// registering a new plugin
{
url:'#new-page-url',
js:'plugin-folder/new-page-url.js'
}
So when #new-page-url is hit anywhere, requirejs consumes js file from plugin-folder/new-page-url.js
Please note that after I compile my application, it does not include third party source as they can be fetched on the fly with requriejs
Question:
I have been looking into webpack and since it compiles everything before distribution (bundle.js as starting file).How can a third party plugin work on the fly like above example ?

Please note that after I compile my application, it does not include third party source as they can be fetched on the fly with requriejs
Yes, that's the problem.
I'm in a situation similar to yours where a large application of mine written as a collection of AMD modules can load editing modes at runtime, which are also AMD modules. The modes are not generally bundled with the application.
AFAIK it is not possible to replicate only with Webpack the possibility of doing the equivalent of a RequireJS' require([a]) where a is a variable whose value cannot be known at build time but is determined at run time. (And for the benefit of readers who may not be familiar with RequireJS, I'll add that yes, I do mean the first argument to be [a] and not just a. RequireJS makes a distinction between the two.)
Webpack needs to know at build time which modules it is going to bundle together. ("Needs to know" means it needs to know the name and find the module's source.) So it does not support dynamically loading modules that only get to be known at runtime. If you read the documentation, or tutorial, you'll run into discussions of dynamic loading with Webpack but these do not allow doing the equivalent of the require([a]) case. Webpack can split a bundle into chunks and load chunks as needed but for this to work, Webpack still needs to know ahead of time the whole set of modules it is ever going to need. This does not allow loading at runtime a module that was unknown at build time. There's also require.context, but while it allows you to determine at run time which specific module you want, the set from which the module comes is determined at build time. If at build time you know that you're going to be using one of A, B, C, that's fine. But if you don't know at build time the name of your module, or cannot feed its source to Webpack, you're out of luck.
This being said, it should be possible to build your application's core with Webpack and have this core perform direct calls to a module loader like RequireJS or SystemJS. This is the direction I'm moving into with my own application but I've not crossed that bridge yet.

npm installs many dependencies

I bought an HTML template recently, which contains many plugins placed inside a bower_components directory and a package.js file inside. I wanted to install another package I liked, but decided to use npm for this purpose.
When I typed:
npc install pnotify
the node_modules directory was created and contained about 900 directories with other packages.
What are those? Why did they get installed along with my package? I did some research and it turned out that those were needed, but do I really need to deliver my template in production with hundreds of unnecessary packages?

This is a very good question. There are a few things I want to point out.
The V8 engine, Node Modules (dependencies) and "requiring" them:
Node.js is built on V8 engine, which is written in C++. This means that Node.js' dependencies are fundamentally written in C++.
Now when you require a dependency, you really require code/functions from a C++ program or js library, because that's how new libraries/dependencies are made.
Libraries have so many functions that you will not use
For example, take a look at the express-validator module, which contains so many functions. When you require the module, do you use all the functions it provides? The answer is no. People most often require packages like this just to use one single benefit of it, although all of the functions end up getting downloaded, which takes up unnecessary space.
Think of the node dependencies that are made from other node dependencies as Interpreted Languages
For example, JavaScript is written in C/C++, whose functions and compilers are in turn originally written in assembly. Think of it like a tree. You create new branches each time for more convenient usage and, most importantly, to save time . It makes things faster. Similarly, when people create new dependencies, they use/require ones that already exist, instead of rewriting a whole C++ program or js script, because that makes everything easier.
Problem arises when requiring other NPMs for creating a new one
When the authors of the dependencies require other dependencies from here and there just to use a few (small amount) benefits from them, they end up downloading them all, (which they don't really care about because they mostly do not worry about the size or they'd rather do this than explicitly writing a new dependency or a C++ addon) and this takes extra space. For example you can see the dependencies that the express-validator module uses by accessing this link.
So, when you have big projects that use lots of dependencies you end up taking so much space for them.
Ways to solve this
Number 1
This requires some expert people on Node.js. To reduce the amount of the downloaded packages, a professional Node.js developer could go to the directories that modules are saved in, open the javascript files, take a look at their source code, and delete the functions that they will not use without changing the structure of the package.
Number 2 (Most likely not worth your time)
You could also create your own personal dependencies that are written in C++, or more preferably js, which would literally take up the least space possible, depending on the programmer, but would take/waste the most time, in order to reduce size instead of doing work. (Note: Most dependencies are written in js.)
Number 3 (Common)
Instead of Using option number 2, you could implement WebPack.
Conclusion & Note
So, basically, there is no running away from downloading all the node packages, but you could use solution number 1 if you believe you can do it, which also has the possibility of screwing up the whole intention of a dependency. (So make it personal and use it for specific purposes.) Or just make use of a module like WebPack.
Also, ask this question to yourself: Do those packages really cause you a problem?

No, there is no point to add about 900 packages dependencies in your project just because you want to add some template. But it is up to you!
The heavyness of a template is not challenging the node.js ecosystem nor his main package system npm.
It is a fact that javascript community tend to make smallest possible module to be responsible for one task, and just one.
It is not a bad thing I guess. But it could result of a situation where you have a lot of dependencies in your project.
Nowadays hard drive memory is cheap and nobody cares any more about making efficient/small apps.
As always, it's only a matter of choice.

What is the point of delivering hundreds of packages weighing hundreds of MB for a few kB project.
There isn't..
If you intend to provide it to other developers, just gitignore (or remove from shared package) node_modules or bower_components directories. Developers simply install dependencies again as required ;)
If it is something as simple as an HTML templates or similar stuff, node would most likely be there just for making your life as a developer easier providing live reload, compiling/transpiling typescript/babel/SCSS/SASS/LESS/Coffee... ( list goes on ;P ) etc.
And in that case dependencies would most likely only be dev_dependencies and won't be required at all in production environment ;)
Also many packages come with separate production and dev dependencies, So you just need to install production dependencies...
npm install --only=prod
If your project does need many projects in production, and you really really wanna avoid that stuff, just spend some time and include css/js files your your project needs(this can be a laborious task).
Update
Production vs default install
Most projects have different dev and production dependencies,
Dev dependencies may include stuff like SASS, typescript etc. compilers, uglifiers (minification), maybe stuff like live reload etc.
Where as production version will not have those things reducing the size node_modules directory.
** No node_modules**
In some html template kind of projects, you may not need any node_modules in production, so you skip doing an npm install.
No access to node_modules
Or in some cases, when server that serves exists in node_modules itself, access to it may be blocked (coz there is no need to access these from frontend).

What are those? Why did they get installed along with my package?
Dependencies exists to facilitate code reuse through modularity.
... do I need to deliver my template in production with hundreds of unnecessary packages?
One shouldn't be so quick to dismiss this modularity. If you inline your requires and eliminate dead code, you'll lose the benefit of maintenance patches for the dependencies automatically being applied to your code. You should see this as a form of compilation, because... well... it is compilation.
Nonetheless, if you're licensed to redistribute all of your dependencies in this compiled form, you'll be happy to learn those optimisations are performed by a compiler which compile Javascript to Javascript. The Closure Compiler, as the first example I stumbled across, appears to perform advanced compilation, which means you get dead code removal and function inlining... That seems promising!

This does however have another side effect when you are required to justify the licensing of all npm modules..so when you have hundreds of npm modules due to dependencies this effort also becomes a more cumbersome task

Very old question but I happened to come across very similar situation just as RA pointed out.
I tried to work with node.js framework using vscode and the moment when I tried to install start npm using npm init -y, it generated so many different dependencies. In my case, it was vscode extension ESlint that I added to prior to running npm init -y
Uninstalling ESlint
Restarted vscode to apply that uninstallation
removed previously generated package.json and node-modules folder
do npm init -y again
This solved my problem of starting out with so many dependencies.

Concatenate NPM package into one JS file

I am trying to get Swig (the template language) working on Parse Cloud Code with Express. Parse Cloud Code is a Node/Express host that doesn't allow NPM. Ridiculous, I know. I can still load external files into code with requires statements however, so I think that there's hope I can get this working.
So my question is how do I get the whole entire Swig package into a single JS file that I can include from my Parse Express app like so:
var swig = require("./cloud/swig.js");
Worth noting that Parse breaks normal require statements so that the NPM package as-is doesn't work without modifying each and every single file in the node_modules folder to have cloud in its path (which is why my above path has cloud in it). Parse also chokes while uploading lots of small files. Concatenation is a need on this platform.
I have tried playing with browserify for hours, but no combination of anything I do makes exposes the Swig object when I load the browserified file with the require statement. I think it may be the right option since the Browserified file includes all the files from Swig, but it doesn't expose them externally.
My question is either can this be done in browserify, and if so, how? Or is there another way to concatenate a NPM repo down to one file so it can be more easily included from this platform?
Thanks so much.

Browserify is not the right tool for the job.
As the name implies, browserify is intended to be used to generate files you want to execute in the browser. It walks the require calls from an entrypoint (i.e. some JS file you pass to browserify) and bundles them in an object that maps their names to functions wrapping the modules. It does not expect a require function to already exist and doesn't make any use of it. It substitutes its own implementation of require that only does one thing: look up names from the bundle, execute the matching function and return its exports.
You could theoretically require a browserify bundle, but it would just return an empty object (although it might mess with globals). And in all likelihood it might break because the bundled modules think they are being executed in a browser. This won't do any good.
The only sane option if you want to stick with the host, is to copy over the node_modules folder from your local project folder. This may not work if your computer and the server are not 100% compatible (e.g. 32-bit vs 64-bit, Debian vs RedHat, OSX/Windows vs Linux) but this mostly depends on your exact dependencies (basically anything that is built with node-gyp can be a problem).
Node.js uses the node_modules folder when looking up dependencies in require calls automagically. If you can somehow get a node_modules folder with the right contents on the server, require("foo") will work as long as node_modules contains a module foo.

Ultimately, you are trying to use npm modules in Parse Cloud code and currently it's not possible:
https://parse.com/questions/using-npm-modules-in-cloud-code
But if you are only trying to use Swig, then as a work-around, you can consider using underscore template instead. Parse already includes underscore:
https://parse.com/docs/cloud_modules_guide#underscore

what are the advantages of using an AMD like requirejs or commonjs modules in javascript?

I've read a lot of articles on AMD solutions like RequireJS or module loaders that follow CommonJS style in Javascript.
Let's say I have an app splitted in this parts:
App definition that rely on the framework i use
Model 1 that rely on App definition and framework
Model 2 that rely on App definition, Model 1 and my framework
I may write each part as a RequireJS module or a common JS module and split my project in how many files I want but what's the advantage of writing each part as a module or splitting them in many files and then load them in the right order (to avoid dependency problems) maybe concatenatening all the files in a big one to reduce HTTP requets (as done by r.js optimizer)?

In my opinion, there are three rather important reasons:
You can create and re-use modules without polluting the global namespace. The more polluted your global namespace is, the bigger the chance of a function/variable collision. That means you define a function called "foo" and another developer defines the function "foo" = one of the functions gets overwritten.
You can structure your code into separate folders and files and requirejs will load them asynchronously when needed, so everything just works.
You can build for production. RequireJS comes with its own build tool called R.JS that will concat and uglify your javascript modules into a single (or multiple) packages. This will improve your page speed as the user will have to make less script calls and load less content (as your JS is uglified).
You can take a look at this simple demo project: https://c9.io/peeter-tomberg/requirejs (in cloud9ide).
To build your modules into a single app, all you have to do is have requirejs npm package installed and run the command: r.js -o build/build.properties.js
If there are any questions, ask away.
Edit:
In development, having all modules in separate files is just a good way to structure and manage your code. It also helps you in debugging (e.g. error on "Module.js line 17" instead of "scripts.js line 5373").
For production, you should use the build tool to concat and uglify the javascript into a single file. This will help the page load quicker as you are making less requests. Every request you make to load something slows down your page. The slower your page, the less points Google gives you. The slower the page, the more frustrated your users will be. The slower your page, the less sales you will get.
If you wish to read more about web page performance, look at http://developer.yahoo.com/performance/rules.html