We Keep Coding

Bundle multiple named AMD modules with dependencies into one JS file (building a web app extension system)

I'm working on an extension system for my web app. Third-party developers should be able to extend the app by providing named AMD modules exporting constants and functions following a predefined spec and bundled into a single .js JavaScript file.
Example JavaScript bundle:
define('module1', ['exports', 'module3'], (function (exports, module3) {
exports.spec = 'http://example.com/spec/extension/v1'
exports.onRequest = function (request) { return module3.respond('Hello, World.') }
}));
define('module2', ['exports', 'module3'], (function (exports, module3) {
exports.spec = 'http://example.com/spec/extension/v1'
exports.onRequest = function (request) { return module3.respond('Foo. Bar.') }
}));
define('module3', ['exports'], (function (exports) {
exports.respond = function (message) { return { type: 'message', message: message } }
}));
In the above example module1 and module2 are extension modules (identified by the spec export) and module3 is a shared dependency (e.g. coming from an NPM package). Extension bundles will be loaded in a worker within a sandboxed iframe to seal of the untrusted code in the browser.
Example TypeScript source:
// module1.ts
import respond from 'module3'
export const spec = 'http://example.com/spec/extension/v1'
export const onRequest = (request: Request): Response => respond('Hello, World.')
// module2.ts
import respond from 'module3'
export const spec = 'http://example.com/spec/extension/v1'
export const onRequest = (request: Request): Response => respond('Foo. Bar.')
// module3.ts
import dep from 'some-npm-package'
export respond = (message: string) => dep.createMessageObject(message)
Here is my list of requirements to bundling:
All necessary dependencies (e.g. shared module, NPM package logic) must be included in the bundle
The source code needs to be transpiled to browser compatible code if necessary
The AMD format is required by the custom extension loader implementation
The AMD modules must not be anonymous as the module file names are lost while bundling
No relative paths must be used among dependencies (e.g. ./path/to/module3 instead of module3)
The result should be one JavaScript bundle, thus ONE JavaScript file and ONE sourcemaps file
What's the easiest way to do this?
This is the closest solution I found using rollup and the following rollup.config.js:
import { nodeResolve } from '#rollup/plugin-node-resolve'
import { terser } from 'rollup-plugin-terser'
import typescript from '#rollup/plugin-typescript'
export default {
input: [
'src/module1.ts',
'src/module2.ts'
],
output: {
dir: 'dist',
format: 'amd',
sourcemap: true,
amd: {
autoId: true
}
},
plugins: [
typescript(),
nodeResolve(),
terser()
]
}
From this I get the desired named AMD modules (one for each entry point and chunk) in separate .js files. Problems:
Some dependencies are referenced by ./module3 while being named module3.
The modules appear in separate JavaScript and Sourcemap files instead of being concatenated into a single bundle.
Questions:
Is there an easy fix to the above rollup.config.js config to solve the problem?
I tried to write a small rollup plugin but I failed to get the final AMD module code within it to concatenate it to a bundle. Only the transpiled code is available to me. In addition I don't know how to handle sourcemaps during concatenation.
Is there an alternative to rollup better suited to this bundling scenario?
The big picture: Am I completely on the wrong track when it comes to building an extension system? Is AMD the wrong choice?

I found a way to extend the rollup.config.js mentioned in the question with a custom concatChunks rollup plugin to bundle multiple AMD chunks within a single file and having the source maps rendered, too. The only issue I didn't find an answer to was the relative module names that kept popping up. However, this may be resolved in the AMD loader.
Here's the full rollup.config.js that worked for me:
import Concat from 'concat-with-sourcemaps'
import glob from 'glob'
import typescript from '#rollup/plugin-typescript'
import { nodeResolve } from '#rollup/plugin-node-resolve'
import { terser } from 'rollup-plugin-terser'
const concatChunks = (
fileName = 'bundle.js',
sourceMapFileName = 'bundle.js.map'
) => {
return {
name: 'rollup-plugin-concat-chunks',
generateBundle: function (options, bundle, isWrite) {
const concat = new Concat(true, fileName, '\n')
// Go through each chunk in the bundle
let hasSourceMaps = false
Object.keys(bundle).forEach(fileId => {
const fileInfo = bundle[fileId]
if (fileInfo.type === 'chunk') {
let hasSourceMap = fileInfo.map !== null
hasSourceMaps = hasSourceMaps || hasSourceMap
// Concat file content and source maps with bundle
concat.add(
fileInfo.fileName,
fileInfo.code,
hasSourceMap ? JSON.stringify(fileInfo.map) : null
)
// Prevent single chunks from being emitted
delete bundle[fileId]
}
})
// Emit concatenated chunks
this.emitFile({
type: 'asset',
name: fileName,
fileName: fileName,
source: concat.content
})
// Emit concatenated source maps, if any
if (hasSourceMaps) {
this.emitFile({
type: 'asset',
name: sourceMapFileName,
fileName: sourceMapFileName,
source: concat.sourceMap
})
}
}
}
}
export default {
input: glob.sync('./src/*.{ts,js}'),
output: {
dir: 'dist',
format: 'amd',
sourcemap: true,
amd: {
autoId: true
}
},
plugins: [
typescript(),
nodeResolve(),
terser(),
concatChunks()
]
}
Please make sure you npm install the dependencies referenced in the import statements to make this work.
Considering the big picture, i.e. the extension system itself, I am moving away from a "one AMD module equals one extension/contribution" approach, as current developer tools and JavaScript bundlers are not ready for that (as this question shows). I'll go with an approach similar to the Visual Studio Code Extension API and will use a single "default" module with an activate export to register contributions a bundle has to offer. I hope that this will make extension bundling an easy task no matter what tools or languages are being used.

Use different output.libraryTarget for a Web Worker

I have this Webpack configuration:
{
output: {
libraryTarget: "system",
...
},
...
}
I'm trying to use a Web Worker. I'm using the standard Webpack 5 syntax:
new Worker(new URL('./MyWorker', import.meta.url));
Now Webpack outputs the Web Worker as a System.js module. How can I change it to something different, like ES module, without affecting the main bundle?

You can use chunkFormat to specify what the chunk formats are, workers are by default array-push, https://webpack.js.org/configuration/output/#outputchunkformat.
You can also create multiple configs with different targets for different entries.
const config = {
//
}
module.exports = (env) => {
if (env.module) {
config.entry = //path/to/index
config.output.libraryTarget = 'module'
} else {
config.entry = //path/to/worker
config.output.libraryTarget = 'umd'
}
return config
}
Then you can separately compile your web workers or chunks different from others. You can also use chunkFileName: () => along with that.
If you want to compile it in a single run, without having to run it twice using different configs, you can also manually invoke the webpack compiler with both configs.
import {Compiler} from 'webpack'
// or import webpack from 'webpack'
compiler.run(config)
compiler.run(config2)
Then you can run both of them at the same time and compile everything.
Another possible option is enabledChunkLoadingTypes, https://webpack.js.org/configuration/output/#outputenabledchunkloadingtypes, which will allow you to specify the available loading types for chunks which webpack will automatically use based on the entry function. I've never used that myself so I don't know if that'll work but it's something you can try.

On webpack how can I import a script without evaluate it?

I'm recently working on some website optimization works, and I start using code splitting in webpack by using import statement like this:
import(/* webpackChunkName: 'pageB-chunk' */ './pageB')
Which correctly create the pageB-chunk.js, now let's say I want to prefetch this chunk in pageA, I can do it by add this statement in pageA:
import(/* webpackChunkName: 'pageB-chunk' */ /* webpackPrefetch: true */ './pageB')
Which will result in a
<link rel="prefetch" href="pageB-chunk.js">
being append to HTML's head, then the browser will prefetch it, so far so good.
The problem is the import statement I use here not just prefetch the js file, but also evaluate the js file, means the code of that js file is parsed & compile to bytecodes, the top-level code of that JS is executed.
This is a very time-consuming operation on a mobile device and I want to optimize it, I only want the prefetch part, I don't want the evaluate & execute part, because later when some user interactions happen, I will trigger the parsing & evaluate myself
↑↑↑↑↑↑↑↑ I only want to trigger the first two steps, pictures come from https://calendar.perfplanet.com/2011/lazy-evaluation-of-commonjs-modules/ ↑↑↑↑↑↑↑↑↑
Sure I can do this by adding the prefetch link myself, but this means I need to know which URL I should put in the prefetch link, webpack definitely knows this URL, how can I get it from webpack?
Does webpack have any easy way to achieve this?

UPDATE
You can use preload-webpack-plugin with html-webpack-plugin it will let you define what to preload in configuration and it will automatically insert tags to preload your chunk
note if you are using webpack v4 as of now you will have to install this plugin using preload-webpack-plugin#next
example
plugins: [
new HtmlWebpackPlugin(),
new PreloadWebpackPlugin({
rel: 'preload',
include: 'asyncChunks'
})
]
For a project generating two async scripts with dynamically generated
names, such as chunk.31132ae6680e598f8879.js and
chunk.d15e7fdfc91b34bb78c4.js, the following preloads will be injected
into the document head
<link rel="preload" as="script" href="chunk.31132ae6680e598f8879.js">
<link rel="preload" as="script" href="chunk.d15e7fdfc91b34bb78c4.js">
UPDATE 2
if you don't want to preload all async chunk but only specific once you can do that too
either you can use migcoder's babel plugin or with preload-webpack-plugin like following
first you will have to name that async chunk with help of webpack
magic comment example
import(/* webpackChunkName: 'myAsyncPreloadChunk' */ './path/to/file')
and then in plugin configuration use that name like
plugins: [
new HtmlWebpackPlugin(),
new PreloadWebpackPlugin({
rel: 'preload',
include: ['myAsyncPreloadChunk']
})
]
First of all let's see the behavior of browser when we specify script tag or link tag to load the script
whenever a browser encounter a script tag it will load it parse it
and execute it immediately
you can only delay the parsing and evaluating with help of async and
defer tag only until DOMContentLoaded event
you can delay the execution (evaluation) if you don't insert the script tag ( only preload it with link)
now there are some other not recommended hackey way is you ship your entire script and string or comment ( because evaluation time of comment or string is almost negligible) and when you need to execute that you can use Function() constructor or eval both are not recommended
Another Approach Service Workers: ( this will preserve you cache event after page reload or user goes offline after cache is loaded )
In modern browser you can use service worker to fetch and cache a recourse ( JavaScript, image, css anything ) and when main thread request for that recourse you can intercept that request and return the recourse from cache this way you are not parsing and evaluating the script when you are loading it into the cache
read more about service workers here
example
self.addEventListener('install', function(event) {
event.waitUntil(
caches.open('v1').then(function(cache) {
return cache.addAll([
'/sw-test/',
'/sw-test/index.html',
'/sw-test/style.css',
'/sw-test/app.js',
'/sw-test/image-list.js',
'/sw-test/star-wars-logo.jpg',
'/sw-test/gallery/bountyHunters.jpg',
'/sw-test/gallery/myLittleVader.jpg',
'/sw-test/gallery/snowTroopers.jpg'
]);
})
);
});
self.addEventListener('fetch', function(event) {
event.respondWith(caches.match(event.request).then(function(response) {
// caches.match() always resolves
// but in case of success response will have value
if (response !== undefined) {
return response;
} else {
return fetch(event.request).then(function (response) {
// response may be used only once
// we need to save clone to put one copy in cache
// and serve second one
let responseClone = response.clone();
caches.open('v1').then(function (cache) {
cache.put(event.request, responseClone);
});
return response;
}).catch(function () {
// any fallback code here
});
}
}));
});
as you can see this is not a webpack dependent thing this is out of scope of webpack however with help of webpack you can split your bundle which will help utilizing service worker better

Updates:
I include all the things into a npm package, check it out!
https://www.npmjs.com/package/webpack-prefetcher
After few days of research, I end up with writing a customize babel plugin...
In short, the plugin work like this:
Gather all the import(args) statements in the code
If the import(args) contains /* prefetch: true */ comment
Find the chunkId from the import() statement
Replace it with Prefetcher.fetch(chunkId)
Prefetcher is a helper class that contain the manifest of webpack output, and can help us on inserting the prefetch link:
export class Prefetcher {
static manifest = {
"pageA.js": "/pageA.hash.js",
"app.js": "/app.hash.js",
"index.html": "/index.html"
}
static function fetch(chunkId) {
const link = document.createElement('link')
link.rel = "prefetch"
link.as = "script"
link.href = Prefetcher.manifest[chunkId + '.js']
document.head.appendChild(link)
}
}
An usage example:
const pageAImporter = {
prefetch: () => import(/* prefetch: true */ './pageA.js')
load: () => import(/* webpackChunkName: 'pageA' */ './pageA.js')
}
a.onmousehover = () => pageAImporter.prefetch()
a.onclick = () => pageAImporter.load().then(...)
The detail of this plugin can found in here:
Prefetch - Take control from webpack
Again, this is a really hacky way and I don't like it, if u want webpack team to implement this, pls vote here:
Feature: prefetch dynamic import on demand

Assuming I understood what you're trying to achieve, you want to parse and execute a module after a given event (e.g click on a button). You could simply put the import statement inside that event:
element.addEventListener('click', async () => {
const module = await import("...");
});

How to include manual import() in Webpack Bundle

I am quite new to Webpack, so bear with me if thats a stupid question.
My goal is to transform my old, AMD based codebase to a ES6 Module based solution. What I am struggling with is handling dynamic import()s. So my app router works on a module basis, i.e. each route is mapped to a module path and then required. Since I know what modules will be included, I just add those dynamically imported modules to my r.js configuration and am able to build everything in a single file, with all require calls still working.
Now, I am trying to do the same with ES6 modules and Webpack. With my devmode this is no problem as I can just replace require() with import(). However I cannot get this to work with bundling. Either Webpack splits my code (and still fails to load the dynamic module anyways), or - if I use the Array format for the entry config, the dynamic module is included in the bundle but loading still fails: Error: Cannot find module '/src/app/DynClass.js'
This is how my Webpack config looks like:
const webpack = require('webpack');
const path = require('path');
module.exports = {
mode: "development",
entry: ['./main.js', './app/DynClass.js'],
output: {
filename: 'main.js',
path: path.resolve(__dirname, "../client/")
},
resolve: {
alias: {
"/src": path.resolve(__dirname, '')
}
},
module: {
rules: [
{
test: /\.tpl$/i,
use: 'raw-loader',
},
]
}
};
So basically I want to tell Webpack: "hey, there is another module (or more) that is to be loaded dynamically and I want it to be included in the bundle"
How can I do this?

So yeah, after much fiddling there seems to be a light at the end of the tunnel. Still, this is not a 100% solution and it is surely not for the faint of heart, as it is quite ugly and fragile. But still I want to share my approach with you:
1) manual parsing of my routes config
My router uses a config file looking like this:
import StaticClass from "/src/app/StaticClass.js";
export default {
StaticClass: {
match: /^\//,
module: StaticClass
},
DynClass: {
match: /^\//,
module: "/src/app/DynClass.js"
}
};
So as you can see the export is an object, with keys acting as the route id, and an object that contains the matches (regex based) and the module which should be executed by the router if the route matches. I can feed my router with both a Constructor function (or an object) for modules which are available immediatly (i.e. contained in the main chunk) or if the module value is a string, this means that the router has to load this module dynamically by using the path specified in the string.
So as I know what modules could be potentially loaded (but not if and when) I can now parse this file within my build process and transform the route config to something webpack can understand:
const path = require("path");
const fs = require("fs");
let routesSource = fs.readFileSync(path.resolve(__dirname, "app/routes.js"), "utf8");
routesSource = routesSource.substr(routesSource.indexOf("export default"));
routesSource = routesSource.replace(/module:\s*((?!".*").)*$/gm, "module: undefined,");
routesSource = routesSource.replace(/\r?\n|\r/g, "").replace("export default", "var routes = ");
eval(routesSource);
let dummySource = Object.entries(routes).reduce((acc, [routeName, routeConfig]) => {
if (typeof routeConfig.module === "string") {
return acc + `import(/* webpackChunkName: "${routeName}" */"${routeConfig.module}");`;
}
return acc;
}, "") + "export default ''";
(Yeah I know this is quite ugly and also a bit brittle so this surely could be done better)
Essentially I create a new, virtual module where every route entry which demands a dynamic import is translated, so:
DynClass: {
match: /^\//,
module: "/src/app/DynClass.js"
}
becomes:
import(/* webpackChunkName: "DynClass" */"/src/app/DynClass.js");
So the route id simply becomes the name of the chunk!
2) including the virtual module in the build
For this I use the virtual-module-webpack-plugin:
plugins: [
new VirtualModulePlugin({
moduleName: "./app/dummy.js",
contents: dummySource
})
],
Where dummySource is just a string containing the sourcecode of my virtual module I just have generated. Now, this module is pulled in and the "virtual imports" can be processed by webpack. But wait, I still need to import the dummy module, but I do not have any in my development mode (where I use everything natively, so no loaders).
So in my main code I do the following:
let isDev = false;
/** #remove */
isDev = true;
/** #endremove */
if (isDev) { import('./app/dummy.js'); }
Where "dummy.js" is just an empty stub module while I am in development mode. The parts between that special comments are removed while building (using the webpack-loader-clean-pragma loader), so while webpack "sees" the import for dummy.js, this code will not be executed in the build itself since then isDev evaluates to false. And since we already defined a virtual module with the same path, the virtual module is included while building just like I want, and of course all dependencies are resolved as well.
3) Handling the actual loading
For development, this is quite easy:
import routes from './app/routes.js';
Object.entries(routes).forEach(async ([routeId, route]) => {
if (typeof route.module === "function") {
new route.module;
} else {
const result = await import(route.module);
new result.default;
}
});
(Note that this is not the actual router code, just enough to help me with my PoC)
Well, but for the build I need something else, so I added some code specific to the build environment:
/** #remove */
const result = await import(route.module);
new result.default;
/** #endremove */
if (!isDev) {
if (typeof route.module === "string") { await __webpack_require__.e(routeId); }
const result = __webpack_require__(route.module.replace("/src", "."));
new result.default;
}
Now, the loading code for the dev environment is just stripped out, and there is another loading code that uses webpack internally. I also check if the module value is a function or string, and if it is the latter I invoke the internal require.ensure function to load the correct chunk: await __webpack_require__.e(routeId);. Remember that I named my chunks when generating the virtual module? Now thats why I still can find them now!
4) more needs to be done
Another thing I encountered is when several dynamically loaded modules have the same dependencies, webpack tries to generate more chunks with names like module1~module2.bundle.js, breaking my build. To counter this, I needed to make sure that all those shared modules go into a specific named bundle I called "shared":
optimization: {
splitChunks: {
chunks: "all",
name: "shared"
}
}
And when in production mode, I simply load this chunk manually before any dynamic modules depending on it are requested:
if (!isDev) {
await __webpack_require__.e("shared");
}
Again, this code only runs in production mode!
Finally, I have to prevent webpack renaming my modules (and chunks) to something like "1", "2" etc, but rather keep the names I just have defined:
optimization: {
namedChunks: true,
namedModules: true
}
Se yeah, there you have it! As I said this wasn't pretty but seems to work, at least with my simplified test setup. I really hope there aren't any blockers ahead of me when I do all the rest (like ESLint, SCSS etc)!

Consolidating Bundles in Webpack 2 with System.import

In Webpack 1, we do code-splitting imports with require.ensure, which can take an array of modules. These modules are combined into a single bundle and fetched with one HTTP request:
require.ensure(['module1', 'module2'], (require) => {
const module1 = require('module1');
const module2 = require('module2');
// use these modules here...
});
// ==> both modules are served as a single bundle, e.g. '5.js'
With Webpack 2, we can now use System.import for a cleaner syntax... but it seems like System.import only accepts a single module to import. Fine -- I can use Promise.all -- but then I end up with two bundles:
Promise.all([
System.import('module1'),
System.import('module2')
]).then( (module1, module2) => {
// use these modules here...
});
// ==> each module served as its own bundle, e.g. '5.js', '6.js'
Is there a way to use System.import but still combine the requested modules into a single bundle?
(Yes, in some cases I can add a new module file that in turn imports and consumes both the dependencies, and that's often the best approach, but for several of my use cases it merely adds additional boilerplate)

According to Sean T. Larkin (webpack core team), using a single file that imports both resources is your best bet (like you already discovered).
Example (untested)
bundle.js
//http://stackoverflow.com/a/36878745/402706
export {default as module1} from './modules/module1';
export {default as module2} from './modules/module2';
Import the single file
System.import('bundle').then({ module1, module2 } => {
// do stuff
}).catch(err => {
console.log("Chunk loading failed");
});
However, there's no disadvantage to using require.ensure as you have it other than,
not being able to handle async load fails via a promise. src
He mentioned future changes that, "may help with this," but I didn't press for what those might be.

I hope that answer helps you...
I had the same problem to do dynamic module loading and archieve that combining an app folder as root context of webpack 2 and the Ignore plugin because I didn't understand the ContextReplacementPlugin. But that work doesn't bundle all in a single file.
Here's the snippet:
import angular from 'angular';
var paths = ["components/app.components.js", "shared/app.shared.js"];
Promise.all(paths.map(path => System.import('../app/' + path))).then(modules => {
let dependencies = [];
modules.forEach(module => {
dependencies.push(module.default.name);
})
angular.module("app", dependencies);
angular.bootstrap(document, ['app']);
}).catch(error => {
console.error(error.message)
});
Using this structure you can fetch an api then use it for dynamic loading for example.
Then in webpack I used IgonorePlugin to avoid *.html files from webpack build:
plugins: [
new webpack.IgnorePlugin(/vertx/),
new webpack.optimize.UglifyJsPlugin({
sourceMap: true
}),
new webpack.IgnorePlugin(/[A-Za-z]*\.(html)$/i)
],
Here is the project github: https://github.com/yurikilian/angular1es6