We Keep Coding

Web workers inside a JS library with Rollup

I am building a negamax engine in Typescript that uses Thread.js web-workers. It is a npm library that will be imported by an application built using webpack.
I am using Rollup to build the engine - how can I export the web-worker files so they are copied into the client's build directory as a separate chunk?

There are plugins for that: Alorel/rollup-plugin-web-worker, darionco/rollup-plugin-web-worker-loader
..but I ended up doing it by scratch, using a separate build configuration for the worker(s). This simply gives me more control over the situation.
Attached is the rollup.config.worker.js that I use.
The main rollup.config.mjs imports this file, has its configuration as the first build configuration. The real build config uses #rollup/plugin-replace to inject the worker's hash to the code loading it.
/*
* Rollup config for building web worker(s)
*
* Imported by the main rollup config.
*/
import sizes from '#atomico/rollup-plugin-sizes'
import resolve from '#rollup/plugin-node-resolve'
import replace from '#rollup/plugin-replace'
import { terser } from 'rollup-plugin-terser'
import {dirname} from 'path'
import {fileURLToPath} from 'url'
const myPath = dirname(fileURLToPath(import.meta.url));
const watch = process.env.ROLLUP_WATCH;
const REGION = process.env.REGION;
if (!REGION) throw new Error("'REGION' env.var. not provided");
let loggingAdapterProxyHash;
const catchHashPlugin = {
name: 'my-plugin',
// Below, one can define hooks for various stages of the build.
//
generateBundle(_ /*options*/, bundle) {
Object.keys(bundle).forEach( fileName => {
// filename: "proxy.worker-520aaa52.js"
//
const [_,c1] = fileName.match(/^proxy.worker-([a-f0-9]+)\.js$/) || [];
if (c1) {
loggingAdapterProxyHash = c1;
return;
}
console.warn("Unexpected bundle generated:", fileName);
});
}
};
const pluginsWorker = [
resolve({
mainFields: ["esm2017", "module"],
modulesOnly: true // "inspect resolved files to assert that they are ES2015 modules"
}),
replace({
'env.REGION': JSON.stringify(REGION),
//
preventAssignment: true // to mitigate a console warning (Rollup 2.44.0); remove with 2.45?
}),
//!watch && terser(),
catchHashPlugin,
!watch && sizes(),
];
const configWorker = {
input: './adapters/logging/proxy.worker.js',
output: {
dir: myPath + '/out/worker', // under which 'proxy.worker-{hash}.js' (including imports, tree-shaken-not-stirred)
format: 'es', // "required"
entryFileNames: '[name]-[hash].js', // .."chunks created from entry points"; default is: '[name].js'
sourcemap: true, // have source map even for production
},
plugins: pluginsWorker
}
export default configWorker;
export { loggingAdapterProxyHash }
Using in main config:
replace({
'env.PROXY_WORKER_HASH': () => {
const hash= loggingAdapterProxyHash;
assert(hash, "Worker hash not available, yet!");
return JSON.stringify(hash);
},
//
preventAssignment: true // to mitigate a console warning (Rollup 2.44.0); remove with 2.45?
}),
..and in the Worker-loading code:
const PROXY_WORKER_HASH = env.PROXY_WORKER_HASH; // injected by Rollup build
...
new Worker(`/workers/proxy.worker-${PROXY_WORKER_HASH}.js?...`);
If anyone wants to get a link to the whole repo, leave a message and I'll post it there. It's still in flux.
Edit:
After writing the answer I came across this: Building module web workers for cross browser compatibility with rollup (blog, Jul 2020)
TL;DR If you wish to use EcmaScript Modules for the worker, watch out! Firefox and Safari don't have the support, as of today. source And the Worker constructor needs to be told that the worker source is ESM.

My API Key is undefined even though it's in an .env file and a global var

When trying to fetch data from an API, the key that I was using was considered "undefined". My key works because I replaced the {key=undefined} in the network console with the actual string and I was able to get the data that I needed. Any thoughts? Also, I know you shouldn't hide any API keys in a React app but I am just using this for testing purposes. If it helps to clarify things, I did use Create-React-App and they did have a major update in the last 3 months, so I wonder if that has anything to do with it.
const bartKey = process.env.REACT_API_BART_API_KEY;
console.log(`Api key: ${process.env.REACT_API_BART_API_KEY}` );
//inside class Component
async getAllStations(){
try{
const response = await fetch(`http://api.bart.gov/api/etd.aspx?cmd=etd&orig=${this.state.selectedStation}&key=${bartKey}&json=y`);
// console.log(response.json());
const data = await response.json();
console.log('Initial data: ', data);
// fetch(`http:api.bart.gov/api/etd.aspx?cmd=etd&orig=${this.state.selectedStation}&key=${bartKey}&json=y`)
// .then(response => response.json())
// .then(data => console.log(`here: ${data}`))
}catch(e){
console.log(`Error: ${e}`)
}
}

this works for me when using create-react-app:
instead of REACT_API_BART_API_KEY use REACT_APP_BART_API_KEY in your .env
Then you can call it as process.env.REACT_APP_BART_API_KEY
check this url from create-react-app docs https://facebook.github.io/create-react-app/docs/adding-custom-environment-variables

I would say a really good solution if your .env file is working weird is to make a config file. Keep the API key in there. Put that file in git ignore and it will be hidden the same way and it is sure to work.

This changes helped me solve my issue:
Please note dotenv is a zero-dependency module that loads environment variables from a .env file into process.env. So this you should at first download this package:
1 step: npm install dotenv --save-dev
Import webpack and dotenv in your webpack.config.js file and make this changes:
After make sure module.exports an function which at first generates the environment keys:
When the keys are generated use webpack.DefinePlugin() which will help you generate global constants for your app.
// other imports
const dotenv = require('dotenv');
const webpack = require('webpack');
module.exports = () => {
const env = dotenv.config().parsed;
const envKeys = Object.keys(env).reduce((prev, next) => {
prev[`process.env.${next}`] = JSON.stringify(env[next]);
return prev;
}, {});
return {
/* ... here should be your previous module.exports object attributes */
entry: ['babel-polyfill', './src/index.js'],
output: {
path: path.join(__dirname, 'build'),
filename: 'bundle.js',
},
plugins: [
new HtmlWebPackPlugin({
template: "./public/index.html",
filename: "./index.html"
}),
new webpack.DefinePlugin(envKeys)
]
}
};
Also note that on logging process.env you will still get empty object or nothing. But if you log process.env.YOU_KEY then you will get your key value stringified
Hope it helps you!

As early as possible in your application, require and configure dotenv. (https://www.npmjs.com/package/dotenv)
require('dotenv').config()
You should write that line as early as possible in your program.

Storybook webpack absolute import

In our app we are using absolute paths for import modules. We have react folder into our resolve root:
Folder structure
We are using webpack for build and develop app and it works ok, with the next options:
resolve: {
modules: [
'node_modules',
path.resolve('src')
]
},
I'm working on integration of storybook and found, that it can't find any module from this react folder.
ERROR in ./stories/index.stories.js
Module not found: Error: Can't resolve 'react/components/Button' in 'project_name/stories'
# ./stories/index.stories.js
for the next line:
import Button from 'react/components/Button';
As mark: I added resolve/modules to .storybook/webpack config and also if I try to import anything other from, for example services/xxx - it works.

Issues
react folder name conflicts with actual React package location: node_modules/react. Webpack tries to resolve to .resolution(default is node_modules) if the file does not exist in the path.
.resolution is not appropriate for this sort of usage. it is mostly used for package resolution because it can't tell source strings.
to change path selectively, use alias instead.
Solution
change your component folder's name so that it does not collide with node_modules/react. a good example is view/components/Button.
add alias to .storybook/main.js setting
// .storybook/main.js
const path = require('path');
module.exports = {
/* ... other settings goes here ... */
/**
* #param {import('webpack').Configuration} config
* */
webpackFinal: async (config, { configType }) => {
if (!config.resolve) config.resolve = {};
// this config allows to resolve `view/...` as `src/view/...`
config.resolve.alias = {
...(config.resolve.alias || {}),
view: path.resolve(__dirname, '../src/view'),
};
return config;
},
};
change storybook code in accordance with (1)
// Button.stories.jsx
import Button from 'view/components/Button';
//...

How do I generate SVG sprites with GatsbyJS

In my gatsby-browser.js file, I have two imports that look similar to:
#import npm-package/lib/icons.svg
#import npm-package/lib/icons-rich.svg
My current gatsby-node.js file is as follows
const path = require('path')
const SpriteLoaderPlugin = require('svg-sprite-loader/plugin')
exports.onCreateWebpackConfig = ({ actions, getConfig }) => {
const config = getConfig()
config.resolve.alias = {
...config.resolve.alias,
/// aliases working fine
}
config.module.rules = [
...config.module.rules,
{
test: /(icons|icons-rich).svg$/,
loader: 'svg-sprite-loader',
options: {
extract: true,
publicPath: './'
},
},
],
config.plugins = [
...config.plugins,
new SpriteLoaderPlugin()
]
actions.replaceWebpackConfig(config)
}
When I run gatsby develop, I get the following error:
Module Warning (from ./node_modules/svg-sprite-loader/lib/loader.js):
svg-sprite-loader exception. Some loaders will be applied after svg-sprite-loader in extract mode
and no files are ever output.
When i run gatsby build, I get a sprite.svg file output to the public directory but it doesn't seem like the svg sprite gets added to the html document body.
1) How do I get the gatsby develop command to process and output the svg to the proper directory
2) I suspect the issue with gatsby build is related to the svg file i'm trying include in the html document, which is in my Layout.jsx file and looks like
<Icon icon="all" iconPath="./public/sprite.svg" />
I would guess the ./public/sprite.svg is missing for some reason but I can't figure out what the correct file path is (tried everything except the right thing apparently).

How to configure multiple entry point htmls with webpack/easy-webpack?

I am trying to port my Aurelia application from System.js + JSPM to Webpack. The app has multiple entry htmls: index.html, index2.html.
As I am new to Webpack, I started with aurelia skeletons using easy-webpack, and tried to add my application specific codes gradually.
For the most basic case with single entry index.html the skeleton worked quite well. Also I added my local modules as node_modules and used the same in the app.
However, I am not able to setup the configurations properly for multiple entry htmls. This is what my webpack.config.js (showing the parts where it is modified) looks like:
const baseConfig = {
entry: {
'app': [/* this is filled by the aurelia-webpack-plugin */],
'aurelia-bootstrap': coreBundles.bootstrap,
'aurelia': coreBundles.aurelia.filter(pkg => coreBundles.bootstrap.indexOf(pkg) === -1),
'some-common-script': path.resolve('src/some-common-script.js'),
'index1-script': path.resolve('src/index1-script'), //needed exclusively in index.html
'index2-script': path.resolve('src/index2-script') //needed exclusively in index2.html
}, ...
};
switch (ENV) {
...
default:
case 'development':
process.env.NODE_ENV = 'development';
config = generateConfig(
baseConfig,
...
require('#easy-webpack/config-common-chunks-simple')
({ appChunkName: 'app', firstChunk: 'aurelia-bootstrap' }),
require('#easy-webpack/config-generate-index-html')
({ minify: false, overrideOptions: { excludeChunks: ["index2-script"] } }),
require('#easy-webpack/config-generate-index-html')
({
minify: false, overrideOptions: {
chunks: ['some-common-script', 'index2-script'],
filename: 'index2.html',
template: 'index2.html',
}
}),...
);
break;
}
module.exports = config;
The problems are as follows:
If I exclude index2-script from the default config-generate-index-html, then the order in which the scripts are added in index.html becomes app.bundle.js, aurelia-bootstrap.bundle.js, ..., instead of aurelia-bootstrap.bundle.js, app.bundle.js, ..., which causes Uncaught ReferenceError: webpackJsonp is not defined....
As per my understanding that error is caused as easy-webpack/config-common-chunks-simple (wrapper around webpack.optimize.CommonsChunkPlugin) packed the initial, and common codes into aurelia-bootstrap.bundle.js ("bootstrap" chunk). Thus, I tried without require('#easy-webpack/config-common-chunks-simple')({ appChunkName: 'app', firstChunk: 'aurelia-bootstrap' }) as well, but instead got
Uncaught TypeError: Reflect.getOwnMetadata is not a function from aurelia-metadata.js, and
Uncaught TypeError: Cannot read property 'call' of undefined from webpack:///webpack/bootstrap <hash>, which seems to be raised because of a module is not found.
I am completely confused about what I should I try next. Please suggest.