We Keep Coding

Javascript bundling and module loading

I've recently been thrown in to clean up a project which has like 45-50 individual .js javascript files. I wonder what the best approach would be to decrease the loading size of them all. Just concatenate all files into one with npm or gulp? Install some module loader? webpack?

If you're already concatenating, minifying, and uglifying and you don't want all the files to be loaded on all the pages due to a monolithic bundle, you might be looking for something like Webpack's Commons Chunk Plugin.
This plugin walks down the tree of dependencies for each endpoint defined in your Webpack.config file and determines which modules are required across all pages. It then breaks the code into two bundles, a "common" bundle containing the modules that every page requires, which you must load with a script tag on each page:
<script src="commons.js" charset="utf-8"></script>
And an endpoint bundle for each individual page that you reference normally in a script tag placed after the commons script tag:
<script src="specificpage.bundle.js" charset="utf-8"></script>
The result is that an individual page will not have to load modules that will only ever be used on other pages.
Again, this is a Webpack plugin. I don't know if this functionality is available as a Gulp plugin, because it must have knowledge of all endpoints in order to determine which dependencies are common to them all.

I redirect you to the very good https://github.com/thedaviddias/Front-End-Checklist
In particular the following advises:
JavaScript Inline: High You don't have any JavaScript code inline
(mixed with your HTML code).
Concatenation: High JavaScript files
are concatenated.
Minification: High JavaScript files are minified (you can add the .min suffix).
You can accomplish this with a package manager such as gulp, grunt or webpack (for the most famous ones). You just need to choose what you prefer to use.
If you consider webpack, You can start with my very simple (but understanding) starter: https://github.com/dfa1234/snippets-starter

There's no much thing that you can do, basically is:
Concatenation - https://www.npmjs.com/package/gulp-concat
Minification - https://www.npmjs.com/package/gulp-minify
Instead of creating all those scripts, you can get something to re-use on yeoman, f.e. the Fountain, so it will reduce a lot of time just typing procedural code for doing the concatenation/minification.
Also if you can use some lazy load (like RequireJS or some frameworks have support to lazy load the module, like Angular) that will improve the performance of your aplication
EDIT:
If you want even more performance, you can install some compression tool in your server, for example this one for NodeJS https://www.npmjs.com/package/compression

I'm my personal opinion, if you have time, the best approach would be to read and understand the purpose of the project. Then plan a proper refactor. You are not fixing anything with concatenating, this is just a deployment step.
You should analyze which technologies are being used and if you want to maintain this code, in the long run, make a proper refactor into a much more modern stack, maybe you can take a seed project with ES6, webpack, Babel... and create a proper repository well maintained with proper modularity and dependencies resolution.
Once you have that, decreasing the load its just about adding proper tools in build time (babel, webpack, etc).
You would like to add some unit tests and continue working properly :)

How do you build, bundle & minify ES6-modules?

Due the fact, that ES6-modules (JavaScript-modules) are available for testing:
https://www.chromestatus.com/feature/5365692190687232
https://medium.com/dev-channel/es6-modules-in-chrome-canary-m60-ba588dfb8ab7
I wonder, how should I minify and prepare the project release-file? Earlier, I havde bundled all JavaScript-files into the single and minified file, except situations, where I have to load the JS-file dynamically via XHR or Fetch API.
As I understand, it's rather impossible to prepare a single-minified file with the ES6-modules right now or may be, I'm just misunderstanding some ways of work.
So, are the any ways to prepare my ES6-modules into single file and how I should prepare the modern JavaScript-project in 2017 year, where JavaScript-modules will be available?

I wonder, how should I minify and prepare the project release-file?
That is purpose of this action? Okay, minified files take fewer network traffic, and will be downloaded faster, but most NPM libraries provides minified dist-files already. And main question about bundling in one big file.
Why webpack do it? Of cource, due absence of support for ES-modules in browser by native, What's why webpack resolves import statements and round dependencies in synchronous manner*, and then substitute it to IIFE for scoping. And perform babel translation and polyfilling, yes.
But then native support of ES-modules is started, it's become un-useful. One of main goals when exposing your web-app to production, is minify traffic volume for your server, using CDN. Now you can do it in native way, so just import ES-modules from unpkg.org and be happy
*If not using HMR, of course, But it's not appropriate for production mode.
Live examples here: https://jakearchibald.com/2017/es-modules-in-browsers/

This blog explains how you would use the ES6 module syntax and yet still bundle your code into something that the browser will understand.
The blog explains that using SystemJs as an ES6 module polyfill and Babel along with Gulp will enable you to code you modules in ES6 yet sill be able to use it today.
https://www.barbarianmeetscoding.com/blog/2016/02/21/start-using-es6-es2015-in-your-project-with-babel-and-gulp/
Using this guide will help you write your code in ES6 while still having a normal workflow to building, minifying and bundling your code.
Keep in mind there are a lot of tools out there that will help you achieve this but I've followed this method many times and I can vouch for its validity.

Using package.json for client side packages, that could be loaded dynamically in browser

I am thinking of extending the format of package.json to include dynamic package (plugin) loading on client side and I would like to understand whether this idea contradicts with npm vision or not. In other words I want to load a bunch of modules, that share common metadata, in browser runtime. Solutions like system.js and jspm are good for modules management, but what I seek is dynamic packages management on client side.
Speaking in details I would like to add a property like "myapp-clientRuntimeDependencies" that would allow to specify dependencies that would be loaded by browser instead of standard prepackaging (npm install->browserify-like solution).
package.json example:
{
name: "myapp-package",
version: "",
myapp-clientRuntimeDependencies: {
"myapp-plugin": "file:myapp-plugin",
"myapp-anotherplugin": "file:myapp-anotherplugin"
},
peerDependencies: {
"myapp-core": "1.0.0"
}
}
The question:
Does this idea contradict with "npm" and "package.json" vision? If yes then why?
Any feedback from npm community is very much appreciated.
References:
Extending package.json: http://blog.npmjs.org/post/101775448305/npm-and-front-end-packaging
EDIT:
The question was not formulated too well, the better way to ask this is:
What is the most standard way (e.g. handled by some existing tools, likely to be supported by npm) to specify run-time dependencies between 2 dynamically loaded front-end packages in package.json?
What is the most standard way to attach metadata in JSON format to front-end packages, that are loaded dynamically?

I wouldn't say that it conflicts with the vision of package.json, however it does seem to conflict a bit with how it's typically used. As you seem to be aware, package.json is normally used pre-runtime. In order to load something from package.json into your runtime, you'd have to load the package.json into your frontend code. If you're storing configurations that you don't want visible to frontend via a simple view source, this could definitely present a problem.
One thing that didn't quite click with me on this: you said that system.js and jspm are good for module management but that you were looking for package management. In the end, packages and modules tend to be synonymous, as a package becomes a module.
I may be misunderstanding what it is that you're looking for, but from what I can gather, I recommend you take a look at code-splitting...which is essentially creating separate js files that will be loaded dynamically based on what is needed instead of bundling all your javascript into a single file. Here's some docs on how to do this with webpack (I'm sure browserify does it as well).

If I understand correctly, your question is about using the package.json file to include your own app configuration. In the example you describe, you would use such configuration to let your app know which dependencies can be loaded at runtime.
There is really nothing preventing you from inserting your own fields in the package.json file, except for the risk of conflict with names that are used by npm for other meanings. But if you use a very specific name (like in your example), you should be safe enough. Actually, many lint and build tools have done so already. It is even explicitly written in the post you refer to:
If your tool needs metadata to make it work, put it in package.json. It would be rude to do this without asking, but we’re inviting you to do it, so go ahead. The registry is a schemaless store, so every field you add is on an equal footing to all the others, and we won’t strip out or complain about new fields (as long as they don’t conflict with existing ones).
But if you want to be even safer, you could resort to use a different file (like Bower did for example).

Node.js: recursively getting building configuration from dependencies

I'm developing a JavaScript project targeting browsers using the closure compiler, but lately I've been thinking to start using browserify and follow a node-like development approach: splitting the project in a number of smaller node modules, building everything using something like grunt, using node's require system (thanks to browserify) etc.
In order to build the whole project after slicing it into some modules, I need a way to get from every module its scripts, assets, some optional configuration, and to recursively do the same on its dependencies.
Is there any standardized way to do this? Any module already taking care of this? Anything providing assistance to this work?
As requested, here an example of what i need. I'll try to pick the most complex one.
The project heavily rely on HTML5 Canvas. It's supporting FlashCanvas, but I want it to be in a separate module because I also want to build the project without FlashCanvas support, and because I'd like to release the flashcanvas module on NPM so that it can be easily integrated in other projects.
When building with the FlashCanvas support, I need to:
put flashcanvas/flashcanvas.js and flashcanvas/flashcanvas.swf in the data directory that will be shipped with the project.
Include <!--[if lt IE 9]><script src="flashcanvas/flashcanvas.js"></script><![endif]--> in the HTML page.
In case I'm compiling with closure-compiler, the FlashCanvas externs file must be passed to the compiler.
Pass an option to browserify that replaces require('canvas') with require('flashcanvas'), in order to override the functions that create the canvases.
In case FlashCanvas was depending on other modules (this is not the case for this module), take assets/configuration/whatever from the dependencies.
I believe that my problem should be quite common among who's developing with browserify: it sounds pretty normal that some modules offer assets (CSS files, images etc) that you need to include. Still I couldn't find anything to solve the issue. Is my approach the right way to structure a node/browserify project? How am I supposed to get the data listed above from my main module's Gruntfile?

Javascript compiler / dependency manager?

I have many JS files. Some of them depend on each other. Many of them depend on jQuery.
I need tool that can accept one file as parameter, fetch all its dependencies transitively and compile them into one file in proper order (based on dependencies)
Dependency information not always available inside files itself, so it would be nice to have it somewhere outside (xml file? folder structure?)
I've heard about Yahoo JS compiler, closure and so on, but I am not sure they do what I need.
Look: I have module "CustomerPage". It sould include "validation.js" and "gui.js". Both require jquery.js. And "gui.js" also requires "myFunctions.js".
I want some ant task or some script that would generate "CustomerPage.js" as result of all that files.
Tool should check dependency order, prevent double including and so on.
My project could have around 500 js files, how could I live with out of this tool?
People says "use GWT", but I need plain JS.

You might want to look at one of the AMD-style module loaders, such as RequireJS. Some of these can do what you want for precompiling, and can run in a development mode which makes it easier to debug by including all the files directly.