We Keep Coding

Bundle multiple named AMD modules with dependencies into one JS file (building a web app extension system)

I'm working on an extension system for my web app. Third-party developers should be able to extend the app by providing named AMD modules exporting constants and functions following a predefined spec and bundled into a single .js JavaScript file.
Example JavaScript bundle:
define('module1', ['exports', 'module3'], (function (exports, module3) {
exports.spec = 'http://example.com/spec/extension/v1'
exports.onRequest = function (request) { return module3.respond('Hello, World.') }
}));
define('module2', ['exports', 'module3'], (function (exports, module3) {
exports.spec = 'http://example.com/spec/extension/v1'
exports.onRequest = function (request) { return module3.respond('Foo. Bar.') }
}));
define('module3', ['exports'], (function (exports) {
exports.respond = function (message) { return { type: 'message', message: message } }
}));
In the above example module1 and module2 are extension modules (identified by the spec export) and module3 is a shared dependency (e.g. coming from an NPM package). Extension bundles will be loaded in a worker within a sandboxed iframe to seal of the untrusted code in the browser.
Example TypeScript source:
// module1.ts
import respond from 'module3'
export const spec = 'http://example.com/spec/extension/v1'
export const onRequest = (request: Request): Response => respond('Hello, World.')
// module2.ts
import respond from 'module3'
export const spec = 'http://example.com/spec/extension/v1'
export const onRequest = (request: Request): Response => respond('Foo. Bar.')
// module3.ts
import dep from 'some-npm-package'
export respond = (message: string) => dep.createMessageObject(message)
Here is my list of requirements to bundling:
All necessary dependencies (e.g. shared module, NPM package logic) must be included in the bundle
The source code needs to be transpiled to browser compatible code if necessary
The AMD format is required by the custom extension loader implementation
The AMD modules must not be anonymous as the module file names are lost while bundling
No relative paths must be used among dependencies (e.g. ./path/to/module3 instead of module3)
The result should be one JavaScript bundle, thus ONE JavaScript file and ONE sourcemaps file
What's the easiest way to do this?
This is the closest solution I found using rollup and the following rollup.config.js:
import { nodeResolve } from '#rollup/plugin-node-resolve'
import { terser } from 'rollup-plugin-terser'
import typescript from '#rollup/plugin-typescript'
export default {
input: [
'src/module1.ts',
'src/module2.ts'
],
output: {
dir: 'dist',
format: 'amd',
sourcemap: true,
amd: {
autoId: true
}
},
plugins: [
typescript(),
nodeResolve(),
terser()
]
}
From this I get the desired named AMD modules (one for each entry point and chunk) in separate .js files. Problems:
Some dependencies are referenced by ./module3 while being named module3.
The modules appear in separate JavaScript and Sourcemap files instead of being concatenated into a single bundle.
Questions:
Is there an easy fix to the above rollup.config.js config to solve the problem?
I tried to write a small rollup plugin but I failed to get the final AMD module code within it to concatenate it to a bundle. Only the transpiled code is available to me. In addition I don't know how to handle sourcemaps during concatenation.
Is there an alternative to rollup better suited to this bundling scenario?
The big picture: Am I completely on the wrong track when it comes to building an extension system? Is AMD the wrong choice?

I found a way to extend the rollup.config.js mentioned in the question with a custom concatChunks rollup plugin to bundle multiple AMD chunks within a single file and having the source maps rendered, too. The only issue I didn't find an answer to was the relative module names that kept popping up. However, this may be resolved in the AMD loader.
Here's the full rollup.config.js that worked for me:
import Concat from 'concat-with-sourcemaps'
import glob from 'glob'
import typescript from '#rollup/plugin-typescript'
import { nodeResolve } from '#rollup/plugin-node-resolve'
import { terser } from 'rollup-plugin-terser'
const concatChunks = (
fileName = 'bundle.js',
sourceMapFileName = 'bundle.js.map'
) => {
return {
name: 'rollup-plugin-concat-chunks',
generateBundle: function (options, bundle, isWrite) {
const concat = new Concat(true, fileName, '\n')
// Go through each chunk in the bundle
let hasSourceMaps = false
Object.keys(bundle).forEach(fileId => {
const fileInfo = bundle[fileId]
if (fileInfo.type === 'chunk') {
let hasSourceMap = fileInfo.map !== null
hasSourceMaps = hasSourceMaps || hasSourceMap
// Concat file content and source maps with bundle
concat.add(
fileInfo.fileName,
fileInfo.code,
hasSourceMap ? JSON.stringify(fileInfo.map) : null
)
// Prevent single chunks from being emitted
delete bundle[fileId]
}
})
// Emit concatenated chunks
this.emitFile({
type: 'asset',
name: fileName,
fileName: fileName,
source: concat.content
})
// Emit concatenated source maps, if any
if (hasSourceMaps) {
this.emitFile({
type: 'asset',
name: sourceMapFileName,
fileName: sourceMapFileName,
source: concat.sourceMap
})
}
}
}
}
export default {
input: glob.sync('./src/*.{ts,js}'),
output: {
dir: 'dist',
format: 'amd',
sourcemap: true,
amd: {
autoId: true
}
},
plugins: [
typescript(),
nodeResolve(),
terser(),
concatChunks()
]
}
Please make sure you npm install the dependencies referenced in the import statements to make this work.
Considering the big picture, i.e. the extension system itself, I am moving away from a "one AMD module equals one extension/contribution" approach, as current developer tools and JavaScript bundlers are not ready for that (as this question shows). I'll go with an approach similar to the Visual Studio Code Extension API and will use a single "default" module with an activate export to register contributions a bundle has to offer. I hope that this will make extension bundling an easy task no matter what tools or languages are being used.

Use different output.libraryTarget for a Web Worker

I have this Webpack configuration:
{
output: {
libraryTarget: "system",
...
},
...
}
I'm trying to use a Web Worker. I'm using the standard Webpack 5 syntax:
new Worker(new URL('./MyWorker', import.meta.url));
Now Webpack outputs the Web Worker as a System.js module. How can I change it to something different, like ES module, without affecting the main bundle?

You can use chunkFormat to specify what the chunk formats are, workers are by default array-push, https://webpack.js.org/configuration/output/#outputchunkformat.
You can also create multiple configs with different targets for different entries.
const config = {
//
}
module.exports = (env) => {
if (env.module) {
config.entry = //path/to/index
config.output.libraryTarget = 'module'
} else {
config.entry = //path/to/worker
config.output.libraryTarget = 'umd'
}
return config
}
Then you can separately compile your web workers or chunks different from others. You can also use chunkFileName: () => along with that.
If you want to compile it in a single run, without having to run it twice using different configs, you can also manually invoke the webpack compiler with both configs.
import {Compiler} from 'webpack'
// or import webpack from 'webpack'
compiler.run(config)
compiler.run(config2)
Then you can run both of them at the same time and compile everything.
Another possible option is enabledChunkLoadingTypes, https://webpack.js.org/configuration/output/#outputenabledchunkloadingtypes, which will allow you to specify the available loading types for chunks which webpack will automatically use based on the entry function. I've never used that myself so I don't know if that'll work but it's something you can try.

Why doesn't Webpack emit chunks when using webpackMode: "weak"?

I'm transitioning a legacy app to Webpack. I'm using Webpack 5.56 (latest at time of writing).
My app is localised and I have a folder with a handful of locale files,
locales
- locale.en.ts
- locale.de.ts
- etc
Each of these locale files is an ES module and they all export (different implementations of) the same functions — getText, printNumber, etc. I have a wrapper module which dynamically imports the correct locale for the current user:
// localization.ts
interface LocaleModule {
getText(text: string): string;
// etc
}
let module: LocaleModule;
import(`locales/locale.${currentUser.language}`).then(m => {
module = m;
});
export function getText(text: string): string {
return module.getText(text);
}
I know the current user's language when the page is being rendered. I want to include the correct locale.*.js script as an initial chunk, so that:
The browser doesn't have to wait for the main chunk to load before it can start downloading the locale file.
The functions in localization.ts can be synchronous.
This seemed like it'd be a good fit for webpackMode: "weak", since I'd like to get an error in the console if the locale file is missing for whatever reason (rather than silently degrade performance). The docs seem to explicitly call out my use case:
This is useful for universal rendering when required chunks are always manually served in initial requests (embedded within the page).
Here's my code:
let module: LocaleModule;
import(
/* webpackMode: "weak" */
/* webpackChunkName: "locales/[request]" */
`./locales/locale.${currentUser.language}`
).then(m => {
module = m;
});
However, it seems webpackMode: "weak" causes Webpack to emit no chunks for the referenced modules at all. There aren't any locale files in Webpack's output folder. I can’t very well include a chunk in the HTML if it was never emitted!
What's the reason for this behaviour? Is there a clean way to get Webpack to emit chunks for dynamically imported modules but not download them asynchronously? (I know that I could use webpackMode: "lazy" and just include the chunk upfront in a script tag, but I'd like to get an error if the locale file is missing.) Or do I have an XY problem, and there’s some better way to do this which I’m unaware of?

I have similar issues and resolve this.
My local file looks like:
import dictionary from './locales/en.json'
const en = dictionary
window.__default_dictionary__ = en
module.exports = en
My locales structure looks like:
enter image description here
You must add new cacheGroup for splitChunks.cacheGroups in webpack config
locales: {
enforce: true,
reuseExistingChunk: true,
priority: 50,
chunks: 'all',
test(module) {
if (/[\\/]src\/i18n[\\/]/.test(module.resource)) return true
return false
},
name(module) {
const moduleFileName = module
.identifier()
.split('/')
.reduceRight((item) => item)
.replace('.json', '')
.replace('.js', '')
return `locales~${moduleFileName}`
},
},
Now all of your locales files will be extracted to another chunk files.
You can use any handler for load locales, for example:
loadLocaleHandler: async (locale: Locale) => {
let localeModule: { default: Dictionary } = await import(`i18n/${locale}`)
return localeModule.default
},
And for everything to work correctly you must
Add locale chunk for result html
<script src="/assets/webpack/js/runtime.js" defer="defer"></script>
<script src="/assets/webpack/js/vendors.js" defer="defer"></script>
<!-- For example it maybe value from cookie or context of app -->
<script src="/assets/webpack/js/locales~(en|it|es).chunk.js" defer="defer"></script>
<script src="/assets/webpack/js/entry.js" defer="defer"></script>
Add magic webpack code to entry point
const defaultLocale: Locale = cookies.getItem('locale') || process.env.DEFAULT_LOCALE
if (__webpack_modules__[`./src/i18n/${defaultLocale}.js`]) {
__webpack_require__(`./src/i18n/${defaultLocale}.js`)
}
Totally:
you don't need wait loading locales by runtime import for first request
you can organize locales for multilocale and multidomain app
all of your locales remain dynamic modules and can be loaded at runtime

I can't post such long comment, so it has to be an answer...
So it looks like there isn't a real link between the modules and the bundler can't resolve them compile time so they aren't emitted. The only think I changed in your code is how modules are imported and it worked out of the box:
const langCode = getLangCode();
let mod;
import("./locales/locale.en")
switch (langCode) {
case "en":
import(`./locales/locale.en.js`).then(m => {
mod = m;
console.log("loaded locale");
})
break;
case "de":
import(`./locales/locale.de.js`).then(m => {
mod = m;
console.log("loaded locale");
})
break;
default:
}
export function getText(text) {
return mod.getText(text);
}
function getLangCode() {
return "de";
}
I know the switch case is not ideal, but the bundler can't automatically guess that pattern: ./locales/locale.${langCode}.js and add all files in the directory that match .js.
The doc says the following:
'weak': Tries to load the module if the module function has already been loaded in some other way (e.g. another chunk imported it or a script containing the module was loaded). A Promise is still returned, but only successfully resolves if the chunks are already on the client. If the module is not available, the Promise is rejected. A network request will never be performed. This is useful for universal rendering when required chunks are always manually served in initial requests (embedded within the page), but not in cases where app navigation will trigger an import not initially served.
From what I understand this means the chunks are expected to be already on the page and generated through some other means.
I hope that helps you resolve your issue.

In order to use weak you have to already manually served the chunks as stated in the docs. This means that adding it in a dynamic import as comment does not create any chunks (in contradiction with lazy and lazy-once).
Is there a clean way to get Webpack to emit chunks for dynamically imported modules but not download them asynchronously?
For synchronous loading:
You can either:
Use webpackMode: "lazy" and include the chunk upfront in a script tag as you stated (the Promise returned is rejected in case of missing chunk).
You can define the locale js files as dynamic entry points and load them manually by yourself.
For your example, creating an entrypoint for each locale could be something like:
const glob = require('glob')
module.exports = {
devtool: false,
entry: {
...glob.sync('./src/locales/*').reduce((acc, module) => {
const name = module.replace('./src/locales/', '').replace('.js', '')
acc[name] = module
return acc
}, {})
}
};
This would emit locale.de.js and locale.en.js bundles and then you should somehow manually load a <script defer src="locale.<locale>.js"></script>, but that depends on how you serve your app.
For asynchronous loading:
You can use webpackMode: "lazy" along with webpackPreload: true in order to decouple main and locale chunk requests.
As stated in the docs
A preloaded chunk starts loading in parallel to the parent chunk.

Web workers inside a JS library with Rollup

I am building a negamax engine in Typescript that uses Thread.js web-workers. It is a npm library that will be imported by an application built using webpack.
I am using Rollup to build the engine - how can I export the web-worker files so they are copied into the client's build directory as a separate chunk?

There are plugins for that: Alorel/rollup-plugin-web-worker, darionco/rollup-plugin-web-worker-loader
..but I ended up doing it by scratch, using a separate build configuration for the worker(s). This simply gives me more control over the situation.
Attached is the rollup.config.worker.js that I use.
The main rollup.config.mjs imports this file, has its configuration as the first build configuration. The real build config uses #rollup/plugin-replace to inject the worker's hash to the code loading it.
/*
* Rollup config for building web worker(s)
*
* Imported by the main rollup config.
*/
import sizes from '#atomico/rollup-plugin-sizes'
import resolve from '#rollup/plugin-node-resolve'
import replace from '#rollup/plugin-replace'
import { terser } from 'rollup-plugin-terser'
import {dirname} from 'path'
import {fileURLToPath} from 'url'
const myPath = dirname(fileURLToPath(import.meta.url));
const watch = process.env.ROLLUP_WATCH;
const REGION = process.env.REGION;
if (!REGION) throw new Error("'REGION' env.var. not provided");
let loggingAdapterProxyHash;
const catchHashPlugin = {
name: 'my-plugin',
// Below, one can define hooks for various stages of the build.
//
generateBundle(_ /*options*/, bundle) {
Object.keys(bundle).forEach( fileName => {
// filename: "proxy.worker-520aaa52.js"
//
const [_,c1] = fileName.match(/^proxy.worker-([a-f0-9]+)\.js$/) || [];
if (c1) {
loggingAdapterProxyHash = c1;
return;
}
console.warn("Unexpected bundle generated:", fileName);
});
}
};
const pluginsWorker = [
resolve({
mainFields: ["esm2017", "module"],
modulesOnly: true // "inspect resolved files to assert that they are ES2015 modules"
}),
replace({
'env.REGION': JSON.stringify(REGION),
//
preventAssignment: true // to mitigate a console warning (Rollup 2.44.0); remove with 2.45?
}),
//!watch && terser(),
catchHashPlugin,
!watch && sizes(),
];
const configWorker = {
input: './adapters/logging/proxy.worker.js',
output: {
dir: myPath + '/out/worker', // under which 'proxy.worker-{hash}.js' (including imports, tree-shaken-not-stirred)
format: 'es', // "required"
entryFileNames: '[name]-[hash].js', // .."chunks created from entry points"; default is: '[name].js'
sourcemap: true, // have source map even for production
},
plugins: pluginsWorker
}
export default configWorker;
export { loggingAdapterProxyHash }
Using in main config:
replace({
'env.PROXY_WORKER_HASH': () => {
const hash= loggingAdapterProxyHash;
assert(hash, "Worker hash not available, yet!");
return JSON.stringify(hash);
},
//
preventAssignment: true // to mitigate a console warning (Rollup 2.44.0); remove with 2.45?
}),
..and in the Worker-loading code:
const PROXY_WORKER_HASH = env.PROXY_WORKER_HASH; // injected by Rollup build
...
new Worker(`/workers/proxy.worker-${PROXY_WORKER_HASH}.js?...`);
If anyone wants to get a link to the whole repo, leave a message and I'll post it there. It's still in flux.
Edit:
After writing the answer I came across this: Building module web workers for cross browser compatibility with rollup (blog, Jul 2020)
TL;DR If you wish to use EcmaScript Modules for the worker, watch out! Firefox and Safari don't have the support, as of today. source And the Worker constructor needs to be told that the worker source is ESM.

Vue cli 3 display info from the package.json

In a vue cli 3 project I want to display a version number in the webpage. The version number lies in the package.json file.
The only reference to this that I found is this link in the vue forum.
However, I can't get the proposed solution to work.
Things I tried
Use webpack.definePlugin as in the linked resource:
vue.config.js
const webpack = require('webpack');
module.exports = {
lintOnSave: true,
configureWebpack: config => {
return {
plugins: [
new webpack.DefinePlugin({
'process.env': {
VERSION: require('./package.json').version,
}
})
]
}
},
}
Then in main.ts I read process.env, but it does not contain VERSION (I tried several variants to this, like generating a PACKAGE_JSON field like in the linked page, and generating plain values like 'foo' instead of reading from package-json). It never worked, it is like the code is being ignored. I guess the process.env is being redefined later by vue webpack stuff.
The process log in main.ts contains, however, all the stuff that process usually contains in a vue-cli project, like the mode and the VUE_APP variables defined in .env files.
Try to write to process right on the configure webpack function,
like:
configureWebpack: config => {
process.VERSION = require('./package.json').version
},
(to be honest I did not have much hope with this, but had to try).
Tried the other solution proposed in the linked page,
like:
// vue.config.js
module.exports = {
chainWebpack: config => {
config.plugin('define').tap( ([options = {}]) => {
return [{
...options, // these are the env variables from your .env file, if any arr defined
VERSION: JSON.stringify(require('./package.json').version)
}]
})
}
}
But this fail silently too.
Use the config.plugin('define') syntax suggested by #Oluwafemi,
like:
chainWebpack: (config) => {
return config.plugin('define').tap(
args => merge(args, [VERSION])
)
},
Where VERSION is a local variable defined as:
const pkgVersion = require('./package.json').version;
const VERSION = {
'process.env': {
VUE_APP_VERSION: JSON.stringify(pkgVersion)
}
}
But this is not working either.
I am re-starting the whole project everytime, so that's not the reason why the process stuff does not show up.
My vue-cli version is 3.0.1.

I am adding my 2 cents, as I found a shorter way and apparently the right way (https://docs.npmjs.com/misc/scripts#packagejson-vars)
Add this in your vue.config.file before the export, not inside:
process.env.VUE_APP_VERSION = process.env.npm_package_version
And voilà!
You can then use it from a component with process.env.VUE_APP_VERSION

TLDR
The following snippet in the vue.config.js file will do the trick, and will allow you to access the version of your app as APPLICATION_VERSION:
module.exports = {
configureWebpack: config => {
return {
plugins: [
new webpack.DefinePlugin({
'APPLICATION_VERSION': JSON.stringify(require('./package.json').version),
})
]
}
},
}
TIP:
Don't even try to add some key to process.env via webpack.definePlugin: it won't work as you probably expect.
 Why my previous efforts did not work
At the end, I solved the issue via webpack.DefinePlugin. The main issue I had is that the original solution I found was using definePlugin to write to a process.env.PACKAGE_JSON variable.
This suggests that definePlugin somehow allows to add variables to process or process.env, which is not the case. Whenever I did log process.env in the console, I didn't find the variables I was trying to push into process.env : so I though the definePlugin tech was not working.
Actually, what webpack.definePlugin does is to check for strings at compile time and change them to its value right on your code. So, if you define an ACME_VERSION variable via:
module.exports = {
lintOnSave: true,
configureWebpack: config => {
return {
plugins: [
new webpack.DefinePlugin({
'ACME_VERSION': 111,
})
]
}
},
}
and then, in main.js you print console.log(ACME_VERSION), you will get 111 properly logged.
Now, however, this is just a string change at compile time. If instead of ACME_VERSION you try to define process.env.VUE_APP_ACME_VERSION...
when you log process.env the VUE_APP_ACME_VERSION key won't show up in the object. However, a raw console.log('process.env.VUE_APP_ACME_VERSION') will yield 111 as expected.
So, basically, original link and the proposed solutions were correct to some degree. However, nothing was really being added to the process object. I was logging proccess.env during my initial tries, so I didn't see anything working.
Now, however, since the process object is not being modified, I strongly suggest anyone trying to load variables to their vue app at compile time not to use it. Is misleading at best.

You can simply import your package.json file and use its variables.
import { version } from "../../package.json";
console.log(version)
If you are using Webpack, you can inject the variable in the following way:
// webpack.config.js
plugins: [
new webpack.DefinePlugin({
VERSION: JSON.stringify(require("package.json").version)
})
]
// any-module.js
console.log("Version: " + VERSION);
https://github.com/webpack/webpack/issues/237

When building the Vue app, environment variables that don't begin with the VUE_APP_ prefix are filtered out. NODE_ENV and BASE_URL environment variables are the exception.
The above information applies when the environment variables are set prior to building the Vue app and not in this situation.
In a situation where environment variables are set during the build, it's important to look at what Vue CLI is doing.
The Vue CLI uses webpack.DefinePlugin to set environment variables using the object returned from the call to resolveClientEnv.
resolveClientEnv returns
{
'process.env': {}
}
This means when configuring your environment variables at build time, you need to come upon a way to merge with the existing one.
You need to perform a deep merge of both arrays, so that value for process.env key is an object containing keys from the resolved client environment and your keys.
chainWebpack key in the default export for vue.config.js is just about one of the ways to get this done.
The arguments passed to initialize the DefinePlugin can be merged with new environment variables that you like to configure using the underlying webpack-chain API. Here is an example:
// vue.config.js
const merge = require('deepmerge');
const pkgVersion = require('./package.json').version;
const VERSION = {
'process.env': {
VERSION: JSON.stringify(pkgVersion)
}
}
module.exports = {
chainWebpack: config =>
config
.plugin('define')
.tap(
args => merge(args, [VERSION])
)
}

Your initial attempt was fine, you were just missing the JSON.stringify part:
const webpack = require('webpack');
module.exports = {
configureWebpack: config => {
return {
plugins: [
new webpack.DefinePlugin({
'process.env': {
VERSION: JSON.stringify(require('./package.json').version),
}
})
]
}
},
}
Edit: although the webpack docs recommend the 'process.env.VERSION' way (yellow panel):
new webpack.DefinePlugin({
'process.env.VERSION': JSON.stringify(require('./package.json').version),
}),

Official solutions tend to be more reliable Modes and Environment Variables | Vue CLI
TIP
You can have computed env vars in your vue.config.js file. They still need to be prefixed with VUE_APP_. This is useful for version info
process.env.VUE_APP_VERSION = require('./package.json').version
module.exports = {
// config
}

I attempted the accepted answer, and had errors. However, in the vue docs, I was able to find an answer similar (but not quite) that of #antoni's answer.
In short, just have the following in vue.config.js:
process.env.VUE_APP_VERSION = require('./package.json').version
module.exports = {
// config
}
Docs 2020-10-27:
You can access env variables in your application code:
process.env.VUE_APP_NOT_SECRET_CODE = require('./package.json').version
During build, process.env.VUE_APP_NOT_SECRET_CODE will be replaced by the corresponding value. In the case of VUE_APP_NOT_SECRET_CODE=some_value, it will be replaced by "some_value".
In addition to VUE_APP_* variables, there are also two special variables that will always be available in your app code:
NODE_ENV - this will be one of "development", "production" or "test" depending on the mode the app is running in.
BASE_URL - this corresponds to the publicPath option in vue.config.js and is the base path your app is deployed at.

The answer for this on the official VueJS forum is like so:
chainWebpack: config => {
config
.plugin('define')
.tap(args => {
let v = JSON.stringify(require('./package.json').version)
args[0]['process.env']['VERSION'] = v
return args
})
}
Description
Add this line to your vue.config.js file
module.exports = {
...
chainWebpack: config => {
config
.plugin('define')
.tap(args => {
let v = JSON.stringify(require('./package.json').version)
args[0]['process.env']['VERSION'] = v
return args
})
}
};
Then you can use this in your vue files like so:
version: function () {
return process.env.VERSION
}

A one liner alternative:
//script tag
let packageJsonInfo = require("../../package.json");
//Then you can for example, get the version no
packageJsonInfo.version