We Keep Coding

Bundle multiple named AMD modules with dependencies into one JS file (building a web app extension system)

I'm working on an extension system for my web app. Third-party developers should be able to extend the app by providing named AMD modules exporting constants and functions following a predefined spec and bundled into a single .js JavaScript file.
Example JavaScript bundle:
define('module1', ['exports', 'module3'], (function (exports, module3) {
exports.spec = 'http://example.com/spec/extension/v1'
exports.onRequest = function (request) { return module3.respond('Hello, World.') }
}));
define('module2', ['exports', 'module3'], (function (exports, module3) {
exports.spec = 'http://example.com/spec/extension/v1'
exports.onRequest = function (request) { return module3.respond('Foo. Bar.') }
}));
define('module3', ['exports'], (function (exports) {
exports.respond = function (message) { return { type: 'message', message: message } }
}));
In the above example module1 and module2 are extension modules (identified by the spec export) and module3 is a shared dependency (e.g. coming from an NPM package). Extension bundles will be loaded in a worker within a sandboxed iframe to seal of the untrusted code in the browser.
Example TypeScript source:
// module1.ts
import respond from 'module3'
export const spec = 'http://example.com/spec/extension/v1'
export const onRequest = (request: Request): Response => respond('Hello, World.')
// module2.ts
import respond from 'module3'
export const spec = 'http://example.com/spec/extension/v1'
export const onRequest = (request: Request): Response => respond('Foo. Bar.')
// module3.ts
import dep from 'some-npm-package'
export respond = (message: string) => dep.createMessageObject(message)
Here is my list of requirements to bundling:
All necessary dependencies (e.g. shared module, NPM package logic) must be included in the bundle
The source code needs to be transpiled to browser compatible code if necessary
The AMD format is required by the custom extension loader implementation
The AMD modules must not be anonymous as the module file names are lost while bundling
No relative paths must be used among dependencies (e.g. ./path/to/module3 instead of module3)
The result should be one JavaScript bundle, thus ONE JavaScript file and ONE sourcemaps file
What's the easiest way to do this?
This is the closest solution I found using rollup and the following rollup.config.js:
import { nodeResolve } from '#rollup/plugin-node-resolve'
import { terser } from 'rollup-plugin-terser'
import typescript from '#rollup/plugin-typescript'
export default {
input: [
'src/module1.ts',
'src/module2.ts'
],
output: {
dir: 'dist',
format: 'amd',
sourcemap: true,
amd: {
autoId: true
}
},
plugins: [
typescript(),
nodeResolve(),
terser()
]
}
From this I get the desired named AMD modules (one for each entry point and chunk) in separate .js files. Problems:
Some dependencies are referenced by ./module3 while being named module3.
The modules appear in separate JavaScript and Sourcemap files instead of being concatenated into a single bundle.
Questions:
Is there an easy fix to the above rollup.config.js config to solve the problem?
I tried to write a small rollup plugin but I failed to get the final AMD module code within it to concatenate it to a bundle. Only the transpiled code is available to me. In addition I don't know how to handle sourcemaps during concatenation.
Is there an alternative to rollup better suited to this bundling scenario?
The big picture: Am I completely on the wrong track when it comes to building an extension system? Is AMD the wrong choice?

I found a way to extend the rollup.config.js mentioned in the question with a custom concatChunks rollup plugin to bundle multiple AMD chunks within a single file and having the source maps rendered, too. The only issue I didn't find an answer to was the relative module names that kept popping up. However, this may be resolved in the AMD loader.
Here's the full rollup.config.js that worked for me:
import Concat from 'concat-with-sourcemaps'
import glob from 'glob'
import typescript from '#rollup/plugin-typescript'
import { nodeResolve } from '#rollup/plugin-node-resolve'
import { terser } from 'rollup-plugin-terser'
const concatChunks = (
fileName = 'bundle.js',
sourceMapFileName = 'bundle.js.map'
) => {
return {
name: 'rollup-plugin-concat-chunks',
generateBundle: function (options, bundle, isWrite) {
const concat = new Concat(true, fileName, '\n')
// Go through each chunk in the bundle
let hasSourceMaps = false
Object.keys(bundle).forEach(fileId => {
const fileInfo = bundle[fileId]
if (fileInfo.type === 'chunk') {
let hasSourceMap = fileInfo.map !== null
hasSourceMaps = hasSourceMaps || hasSourceMap
// Concat file content and source maps with bundle
concat.add(
fileInfo.fileName,
fileInfo.code,
hasSourceMap ? JSON.stringify(fileInfo.map) : null
)
// Prevent single chunks from being emitted
delete bundle[fileId]
}
})
// Emit concatenated chunks
this.emitFile({
type: 'asset',
name: fileName,
fileName: fileName,
source: concat.content
})
// Emit concatenated source maps, if any
if (hasSourceMaps) {
this.emitFile({
type: 'asset',
name: sourceMapFileName,
fileName: sourceMapFileName,
source: concat.sourceMap
})
}
}
}
}
export default {
input: glob.sync('./src/*.{ts,js}'),
output: {
dir: 'dist',
format: 'amd',
sourcemap: true,
amd: {
autoId: true
}
},
plugins: [
typescript(),
nodeResolve(),
terser(),
concatChunks()
]
}
Please make sure you npm install the dependencies referenced in the import statements to make this work.
Considering the big picture, i.e. the extension system itself, I am moving away from a "one AMD module equals one extension/contribution" approach, as current developer tools and JavaScript bundlers are not ready for that (as this question shows). I'll go with an approach similar to the Visual Studio Code Extension API and will use a single "default" module with an activate export to register contributions a bundle has to offer. I hope that this will make extension bundling an easy task no matter what tools or languages are being used.

How to bundle (minify) a Kotlin React app for deployment?

The app is created with the default template for Kotlin React apps:
uses KTS-based Gradle build scripts;
Kotlin JS plugin 1.6.10;
Kotlin wrappers for React 17.0.2.
When using ./gradlew browserProductionWebpack without any additional tweaks, it generates a build/distributions directory with:
all resources (without any modifications);
index.html (without any modifications);
Kotlin sources compiled into one minified .js file.
What I want is to:
add some hash to the generated .js file;
minify the index.html file and refer the hashed .js file in it;
minify all resources (.json localization files).
Please prompt me some possible direction to do it. Looking to webpack configuration by adding corresponding scripts into webpack.config.d, but no luck yet: tried adding required dependencies into build.gradle.kts, i.e.:
implementation(devNpm("terser-webpack-plugin", "5.3.1"))
implementation(devNpm("html-webpack-plugin", "5.5.0"))
and describing webpack scripts:
const TerserPlugin = require("terser-webpack-plugin");
const HtmlWebpackPlugin = require('html-webpack-plugin');
module.exports = {
optimization: {
minimizer: [
new TerserPlugin(),
new HtmlWebpackPlugin({
minify: {
removeAttributeQuotes: true,
collapseWhitespace: true,
removeComments: true,
},
}),
],
}
}
Any hint will be appreciated.

A couple of things to put into consideration first.
If some flexible bundling configuration is needed, most likely it won't be possible to use Kotlin-wrapped (Gradle) solutions. After checking the org.jetbrains.kotlin.gradle.targets.js.webpack.KotlinWebpackConfig it turns out that there is only a limited set of things can be configured with it.
JS-based webpack configs require a little bit of reverse engineering to find out what is generated from the Kotlin/Gradle side and how to extend it.
For simple configurations (IMHO) there is almost no need in tweaking bundling options while using the Kotlin JS plugin.
Let's start from initial webpack configs. In my case (see the short environment description in the question above) they appear in ./build/js/packages/<project-name>/webpack.config.js. These original configs will also include all contents from JS files we create inside the ./webpack.config.d folder.
Some webpack configurations require external JS dependencies. We need to declare them in the dependencies block of build.gradle.kts. In my case they are represented with:
// Bundling.
implementation(devNpm("html-webpack-plugin", "5.5.0"))
implementation(devNpm("uglifyjs-webpack-plugin", "2.2.0"))
implementation(devNpm("terser-webpack-plugin", "5.3.1"))
implementation(devNpm("copy-webpack-plugin", "9.1.0" )) // newer versions don't work correctly with npm and Yarn
implementation(devNpm("node-json-minify", "3.0.0"))
I also dropped all commonWebpackConfigs from build.gradle.kts as they are going to be performed manually on the JS level.
All webpack JS configs (inside the ./webpack.config.d folder) are divided into 3 files:
common.js (dev server configuration for both dev and production builds):
// All route paths should fallback to the index page to make SPA's routes processed correctly.
const devServer = config.devServer = config.devServer || {};
devServer.historyApiFallback = true;
development.js:
// All configs inside of this file will be enabled only in the development mode.
// To check the outputs of this config, see ../build/processedResources/js/main
if (config.mode == "development") {
const HtmlWebpackPlugin = require("html-webpack-plugin");
// Pointing to the template to be used as a base and injecting the JS sources path into.
config.plugins.push(new HtmlWebpackPlugin({ template: "./kotlin/index.html" }));
}
and production.js:
// All configs inside of this file will be enabled only in the production mode.
// The result webpack configurations file will be generated inside ../build/js/packages/<project-name>
// To check the outputs of this config, see ../build/distributions
if (config.mode == "production") {
const HtmlWebpackPlugin = require("html-webpack-plugin"),
UglifyJsWebpackPlugin = require("uglifyjs-webpack-plugin"),
TerserWebpackPlugin = require("terser-webpack-plugin"),
CopyWebpackPlugin = require("copy-webpack-plugin"),
NodeJsonMinify = require("node-json-minify");
// Where to output and how to name JS sources.
// Using hashes for correct caching.
// The index.html will be updated correspondingly to refer the compiled JS sources.
config.output.filename = "js/[name].[contenthash].js";
// Making sure optimization and minimizer configs exist, or accessing its properties can crash otherwise.
config.optimization = config.optimization || {};
const minimizer = config.optimization.minimizer = config.optimization.minimizer || [];
// Minifying HTML.
minimizer.push(new HtmlWebpackPlugin({
template: "./kotlin/index.html",
minify: {
removeAttributeQuotes: true,
collapseWhitespace: true,
removeComments: true,
},
}));
// Minifying and obfuscating JS.
minimizer.push(new UglifyJsWebpackPlugin({
parallel: true, // speeds up the compilation
sourceMap: false, // help to match obfuscated functions with their origins, not needed for now
uglifyOptions: {
compress: {
drop_console: true, // removing console calls
}
}
}));
// Additional JS minification.
minimizer.push(new TerserWebpackPlugin({
extractComments: true // excluding all comments (mostly licence-related ones) into a separate file
}));
// Minifying JSON locales.
config.plugins.push(new CopyWebpackPlugin({
patterns: [
{
context: "./kotlin",
from: "./locales/**/*.json",
to: "[path][name][ext]",
transform: content => NodeJsonMinify(content.toString())
}
]
}));
}
I use styled components, so no CSS configs are provided. In other things these configs do almost the same minification as being done out-of-the-box without any additional configs. The differences are:
JS sources use a hash in their name: it is referenced correctly from the index page HTML template;
HTML template is minified;
locales (just simple JSON files) are minified.
It can look like an overhead slightly because as mentioned in the beginning, it does almost the same with minimal differences from the out-of-the-box configs. But as advantage we're getting more flexible configs which can be tweaked easier further.

Web workers inside a JS library with Rollup

I am building a negamax engine in Typescript that uses Thread.js web-workers. It is a npm library that will be imported by an application built using webpack.
I am using Rollup to build the engine - how can I export the web-worker files so they are copied into the client's build directory as a separate chunk?

There are plugins for that: Alorel/rollup-plugin-web-worker, darionco/rollup-plugin-web-worker-loader
..but I ended up doing it by scratch, using a separate build configuration for the worker(s). This simply gives me more control over the situation.
Attached is the rollup.config.worker.js that I use.
The main rollup.config.mjs imports this file, has its configuration as the first build configuration. The real build config uses #rollup/plugin-replace to inject the worker's hash to the code loading it.
/*
* Rollup config for building web worker(s)
*
* Imported by the main rollup config.
*/
import sizes from '#atomico/rollup-plugin-sizes'
import resolve from '#rollup/plugin-node-resolve'
import replace from '#rollup/plugin-replace'
import { terser } from 'rollup-plugin-terser'
import {dirname} from 'path'
import {fileURLToPath} from 'url'
const myPath = dirname(fileURLToPath(import.meta.url));
const watch = process.env.ROLLUP_WATCH;
const REGION = process.env.REGION;
if (!REGION) throw new Error("'REGION' env.var. not provided");
let loggingAdapterProxyHash;
const catchHashPlugin = {
name: 'my-plugin',
// Below, one can define hooks for various stages of the build.
//
generateBundle(_ /*options*/, bundle) {
Object.keys(bundle).forEach( fileName => {
// filename: "proxy.worker-520aaa52.js"
//
const [_,c1] = fileName.match(/^proxy.worker-([a-f0-9]+)\.js$/) || [];
if (c1) {
loggingAdapterProxyHash = c1;
return;
}
console.warn("Unexpected bundle generated:", fileName);
});
}
};
const pluginsWorker = [
resolve({
mainFields: ["esm2017", "module"],
modulesOnly: true // "inspect resolved files to assert that they are ES2015 modules"
}),
replace({
'env.REGION': JSON.stringify(REGION),
//
preventAssignment: true // to mitigate a console warning (Rollup 2.44.0); remove with 2.45?
}),
//!watch && terser(),
catchHashPlugin,
!watch && sizes(),
];
const configWorker = {
input: './adapters/logging/proxy.worker.js',
output: {
dir: myPath + '/out/worker', // under which 'proxy.worker-{hash}.js' (including imports, tree-shaken-not-stirred)
format: 'es', // "required"
entryFileNames: '[name]-[hash].js', // .."chunks created from entry points"; default is: '[name].js'
sourcemap: true, // have source map even for production
},
plugins: pluginsWorker
}
export default configWorker;
export { loggingAdapterProxyHash }
Using in main config:
replace({
'env.PROXY_WORKER_HASH': () => {
const hash= loggingAdapterProxyHash;
assert(hash, "Worker hash not available, yet!");
return JSON.stringify(hash);
},
//
preventAssignment: true // to mitigate a console warning (Rollup 2.44.0); remove with 2.45?
}),
..and in the Worker-loading code:
const PROXY_WORKER_HASH = env.PROXY_WORKER_HASH; // injected by Rollup build
...
new Worker(`/workers/proxy.worker-${PROXY_WORKER_HASH}.js?...`);
If anyone wants to get a link to the whole repo, leave a message and I'll post it there. It's still in flux.
Edit:
After writing the answer I came across this: Building module web workers for cross browser compatibility with rollup (blog, Jul 2020)
TL;DR If you wish to use EcmaScript Modules for the worker, watch out! Firefox and Safari don't have the support, as of today. source And the Worker constructor needs to be told that the worker source is ESM.

import Stencil in Svelte

I have a Monorepo with a svelte project and a Stencil component library. On the Stencil website they very clearly describe how to integrate the library with, for example, Angular
import { defineCustomElements } from 'test-components/loader';
defineCustomElements(window);
Super easy. But now I would like to use it too in a Svelte project ..... not so super easy anymore :(
When I try to do something similar as described above I get serious errors
fbp/dist is where the Stencil files are.
When I build my Stencil project first and copy my dist into the public folder and load ./dist/fbp.js in the head of index.html it all works. But it would be a lot easier if I could include it similar as it does with Angular. Any suggestions?
Update: Added emitCss which gives
Somewhere at the end it stats: Error: Unexpected token (Note that you need plugins to import files that are not JavaScript)
UPDATE: With the fixes of #Sambor, Svelte is now able to download the web component, which unfortunately fails

I have created a new project and I manage to reproduce the same problem.
At first, I was thinking is related to typescript and I've tried bunch of plugins in rollup like : #tscc/rollup-plugin-tscc, rollup-plugin-typescript but it didn't work.
I also tried rollup-plugin-amd with same results...
Then I've tried to change the main output format and use es instead of iife.
This way it also required to change the output to a directory instead of file (because of multiple file generation).
And surprisingly this way it seems to work.
here is my code:
/// index.html
<head>
<meta charset='utf-8'>
<meta name='viewport' content='width=device-width,initial-scale=1'>
<title>Test</title>
<link rel='stylesheet' href='build/bundle.css'>
<script type="module" defer src='build/main.js'></script>
</head>
<body>
</body>
</html>
Note: main.js is imported as module.
/// main.js
import App from './App.svelte';
import { applyPolyfills, defineCustomElements } from '../my-comp/loader';
applyPolyfills().then(() => {
defineCustomElements(window);
});
const app = new App({ target: document.body });
export default app;
/// rollup.config
import svelte from 'rollup-plugin-svelte';
import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';
import livereload from 'rollup-plugin-livereload';
import { terser } from 'rollup-plugin-terser';
import postcss from 'rollup-plugin-postcss';
import autoPreprocess from 'svelte-preprocess';
import json from '#rollup/plugin-json';
const production = !process.env.ROLLUP_WATCH;
export default {
input: 'src/main.js',
output: {
sourcemap: true,
format: 'es',
name: 'app',
dir: 'public/build'
},
plugins: [
json(),
svelte({
// Enables run-time checks when not in production.
dev: !production,
// Extracts any component CSS out into a separate file — better for performance.
css: css => css.write('public/build/bundle.css'),
// Emit CSS as "files" for other plugins to process
emitCss: true,
preprocess: autoPreprocess()
}),
resolve({
browser: true,
dedupe: importee => importee === 'svelte' || importee.startsWith('svelte/')
}),
commonjs(),
postcss({
extract: true,
minimize: true,
use: [
['sass', {
includePaths: ['./node_modules']
}]
]
}),
// In dev mode, call `npm run start` once the bundle has been generated
!production && serve(),
// Watches the `public` directory and refresh the browser on changes when not in production.
!production && livereload('public'),
// Minify for production.
production && terser()
],
watch: {
clearScreen: false
}
};
function serve() {
let started = false;
return {
writeBundle() {
if (!started) {
started = true;
require('child_process').spawn('npm', ['run', 'start', '--', '--dev'], {
stdio: ['ignore', 'inherit', 'inherit'],
shell: true
});
}
}
};
}
Note: I took my config from another svelte project (you can ignore uninteresting plugins)
Now it seems to be working fine, but I think is just the starting point :) because there are some known issues with stencil itself which I come across;
core-3d1820a5.js:97 TypeError: Failed to fetch dynamically imported module: http://localhost:57231/build/my-component.entry.js
core-3d1820a5.js:863 Uncaught (in promise) TypeError: Cannot read property 'isProxied' of undefined
https://github.com/sveltejs/sapper/issues/464
https://github.com/ionic-team/stencil/issues/1981
same with react: Unable to integrate stenciljs component in React application
This is not the completely working solution, but I thought it may help you for the next steps...

I’m still having the same issue in 2020. Surprisingly, the webpack template is working fine. Switching to that for now, until this is resolved.
https://github.com/sveltejs/template-webpack

How do I generate SVG sprites with GatsbyJS

In my gatsby-browser.js file, I have two imports that look similar to:
#import npm-package/lib/icons.svg
#import npm-package/lib/icons-rich.svg
My current gatsby-node.js file is as follows
const path = require('path')
const SpriteLoaderPlugin = require('svg-sprite-loader/plugin')
exports.onCreateWebpackConfig = ({ actions, getConfig }) => {
const config = getConfig()
config.resolve.alias = {
...config.resolve.alias,
/// aliases working fine
}
config.module.rules = [
...config.module.rules,
{
test: /(icons|icons-rich).svg$/,
loader: 'svg-sprite-loader',
options: {
extract: true,
publicPath: './'
},
},
],
config.plugins = [
...config.plugins,
new SpriteLoaderPlugin()
]
actions.replaceWebpackConfig(config)
}
When I run gatsby develop, I get the following error:
Module Warning (from ./node_modules/svg-sprite-loader/lib/loader.js):
svg-sprite-loader exception. Some loaders will be applied after svg-sprite-loader in extract mode
and no files are ever output.
When i run gatsby build, I get a sprite.svg file output to the public directory but it doesn't seem like the svg sprite gets added to the html document body.
1) How do I get the gatsby develop command to process and output the svg to the proper directory
2) I suspect the issue with gatsby build is related to the svg file i'm trying include in the html document, which is in my Layout.jsx file and looks like
<Icon icon="all" iconPath="./public/sprite.svg" />
I would guess the ./public/sprite.svg is missing for some reason but I can't figure out what the correct file path is (tried everything except the right thing apparently).