Webpack create JSON map file with dynamic modules - javascript

I have a project where I use turbolinks in conjunction with Webpack's dynamic imports.
What I have is that the initial javascript file is as small as possible and then for each page, I load the relevant javascript code:
document.addEventListener('turbolinks:load', () => {
const moduleName = document.body.dataset.route;
if (!moduleName) return;
const chunkName = moduleName.replace('.', '/');
import(`#pages/${chunkName}.js`)
.catch((e) => {console.error(e);});
});
This is a great approach as each page gets its own minimal JS file.
The only issue is that we wait until the page has fully loaded before we fetch the page's javascript which makes initial page loads feel slightly slow. Ideally, I would love to preload the assets for the page when the page loads.
I thought that maybe adding a link rel=preload would solve this issue but the thing is that I do not know which chunks I need to preload on each page. This is logic that only Weboack knows.
My webpack.config.js file looks like:
output: {
chunkFilename: 'js/chunks/[name].js?id=[chunkhash]',
},
So basically each chunk is put in the js/chunks directory and its name is 0.js, 1.js, 2.js etc.
I would love to maybe somehow generate an additional json file where webpack can build a map for me. It would basically look like this (chunk key: modules that are within it):
{
0: ['#pages/tips/index.js', '#pages/tips/show.js'],
1: ['#pages/destinations/index.js', '#pages/tips/show.js'],
}
Then, I would read the file on each page and dynamically create the link rel=preload. For example, say I render the tips/show page now, I would scan the file above for each key that contains the #pages/tips/show.js and render a link rel=preload for each file (0.js and 1.js file in this case).
I'm using Webpack Commons Chunk plugin to extract the same vendors and modules to their own chunk file.
Is doing such a thing is even possible?
Thanks!

Webpack has something that is called stats that contains a-lot of info regarding the compilation.
It also contains the separation of chunks.
You can checkout what react-loadable webpack plugin does in order to generate similar task.
Hope this helps.
EDIT
You will need to write a webpack plugin that hooks on emit phase and then you will have an access to the compilation object that has all the data.

Related

How to integrate own library with WebPack which needs to span WebWorkers and AudioWorklets

Goal
I am the author of a JavaScript library which can be consumed through AMD or ESM within various runtime environments (Browser, Node.js, Dev Servers). My library needs to spawn WebWorkers and AudioWorklets using the file it is contained. The library detects in which context it is running and sets up the required stuff for the execution context.
This works fine as long users (user=integrator of my library) do not bring bundlers like WebPack into the game. To spawn the WebWorker and AudioWorklet I need the URL to file in which my library is contained in and I need to ensure the global initialization routines of my library are called.
I would prefer to do as much as possible of the heavy lifting within my library, and not require users to do a very specialized custom setup only for using my library. Offloading this work to them typically backfires instantly and people open issues asking for help on integrating my library into their project.
Problem 1: I am advising my users to ensure my library is put into an own chunk. Users might setup the chunks based on their own setup as long the other libs don't cause any troubles or side effects in the workers. Especially modern web frameworks like React, Angular and Vue.js are typical problem children here, but also people tried to bundle my library with jQuery and Bootstrap. All these libraries cause runtime errors when included in Workers/Worklets.
The chunking is usually done with some WebPack config like:
config.optimization.splitChunks.cacheGroups.alphatab = {
chunks: 'all',
name: 'chunk-mylib',
priority: config.optimization.splitChunks.cacheGroups.defaultVendors.priority + 10,
test: /.*node_modules.*mylib.*/
};
The big question mylib now has: What is the absolute URL of the generated chunk-mylib.js as this is now the quasi-entrypoint to my library now with bundling and code splitting in place:
document.currentScript points usually to some entry point like an app.js and not the chunks.
__webpack_public_path__ is pointing to whatever the user sets it to in the webpack config.
__webpack_get_script_filename__ could be used if the chunk name would be known but I haven't found a way to get the name of the chunk my library is contained in.
import.meta.url is pointing to some absolute file:// url of the original .mjs of my library.
new URL(import.meta.url, import.meta.url) causes WebPack to generate an additional .mjs file with some hash. This additional file is not desired and also the generated .mjs contains some additional code breaking its usage in browsers.
I was already thinking to maybe create a custom WebPack plugin which can resolve the chunk my library is contained in so I can use it during runtime. I would prefer to use as much built-in features as possible.
Problem 2: Assuming that problem 1 is solved I could now spawn a new WebWorker and AudioWorklet with the right file. But as my library is wrapped into a WebPack module my initialization code will not be executed. My library only lives in a "chunk" and is not an entry and I wouldn't know that this splitting would allow mylib to run some code after the chunk was loaded by the browser.
Here I am rather clueless. Maybe chunks are not the right way of splitting for this purpose. Maybe some other setup is needed I am not yet aware of that its possible?
Maybe also this could be done best with a custom WebPack plugin.
Visual Representation of the problem: With the proposed chunking rule we get an output as shown in the blocks. Problem 1 is the red part (how to get this URL) and Problem 2 is the orange part (how to ensure my startup logic is called when the background worker/worklet starts)
Actual Project I want to share my actual project for better understanding of my use case. I am talking about my project alphaTab, a music notation rendering and playback library. On the Browser UI thread (app.js) people integrate the component into the UI and they get an API object to interact with the component. One WebWorker does the layouting and rendering of the music sheet, a second one synthesizes the audio samples for playback and the AudioWorklet sends the buffered samples to the audio context for playback.
I think the worker code should be handled as an assets instead of a source code. Maybe you could add a simple CLI to generate a ".alphaTab" folder on the root of the project and add instructions for your user to copy that to the "dist"or "public folder".
Even if come up with a Webpack specific solution, you would have to work your way around other bundlers/setups (Vite, rollup, CRA, etc).
EDIT: You would also need to add an optional parameter to the initialization for passing the script path. Not fully automated, but simpler that having to setup complex bundler configs
Disabling import.meta
Regarding import.meta.url, this link might help. It looks like you'd disable it in your webpack config by setting module.parser.javascript.importMeta to false.
Reworking Overall Architecture
For the rest, it sounds like a bit of a mess. You probably shouldn't be trying to import the same exact chunk code into your workers/worklets, since this is highly dependent on how webpack generates and consumes chunks. Even if you manage to get it to work today, it might break in the future if the webpack team changes how they internally represent chunks.
Also from a user's perspective, they just want to import the library and have it just work without fiddling with all of the different build steps.
Instead, a cleaner way would to be to generate separate files for the main library, the AudioWorklet, and the Web Worker. And since you already designed the worklet and web worker to use your library, you can just use the prebuilt, non-module library for them, and have a separate file for the entry point for webpack/other bundlers.
The most straightforward way would be to have users add your original non-module js library in with the bundle that they build, and have the es module load Web Workers and Audio Worklets using that non-module library's url.
Of course, from a user's perspective, it'd be easier if they didn't have to copy over additional files and put them in the right directory (or configure a scripts directory). The straightforward way would be to load the web worker or worklet from a CDN (like https://unpkg.com/#coderline/alphatab#1.2.2/dist/alphaTab.js), but there are restrictions from loading web workers cross origin, so you'd have to use a work around like fetching it and then loading it from a blob url (like that found here). This unfortunately makes initializing the Worker/Worklet asynchronous.
Bundling Worker code
If this isn't an option, you can bundle a library, Web Worker/Worklet code into one file by stringifying the Worker/Worklet code and loading it via a blob or data url. In your particular use case, it's a little painful from an efficiency standpoint considering how much code will be duplicated in the bundled output.
For this approach, you'd have multiple build steps:
Build the library that's used by your Web Worker and/or Audio Worklet.
Build the single library by stringifying the previous libraries/library.
This is all complicated by there being only one entry file for the library, web worker, and audio worklet. In the long term, you'd probably benefit by rewriting entry points for these different targets, but for now, we could reuse the current workflow and change the build steps by using different plugins. For the first build, we'll make a plugin that returns a dummy string when it tries to import the worker library, for the second, we'll have it return the stringified contents of that library. I'll use rollup, since that's what your project uses. The code below is mostly for illustrative purposes (which saves the worker library as dist/worker-library.js); I haven't actually tested it.
First plugin:
var firstBuildPlugin = {
load(id) {
if (id.includes('worker-library.js')) {
return 'export default "";';
}
return null;
}
}
Second plugin:
var secondBuildPlugin = {
transform(code, id) {
if (id.includes('worker-library.js')) {
return {
code: 'export default ' + JSON.stringify(code) + ';',
map: { mappings: '' }
};
}
return null;
}
}
Using these plugins, we can import the web worker/audio worklet library via import rawCode from './path/to/worker-library.js';. For your case, since you'd be reusing the same library, you may want to create a new file with an export, so the to prevent multiple bundling of the same code:
libraryObjectURL.js:
import rawCode from '../dist/worker-library.js'; // may need to tweak the path here
export default URL.createObjectURL(
new Blob([rawCode], { type: 'application/javascript' })
);
And to actually use it:
import libraryObjectURL from './libraryObjectURL.js'; // may need to tweak the path here
//...
var worker = new Worker(libraryObjectURL);
To then actually build it, your rollup.config.js would look something like:
module.exports = [
{
input: `dist/lib/alphatab.js`,
output: {
file: `dist/worker-library.js`,
format: 'iife', // or maybe umd
//...
plugins: [
firstBuildPlugin,
//...
]
}
},
{
input: `dist/lib/alphatab.js`,
output: {
file: `dist/complete-library.mjs`,
format: 'es',
//...
plugins: [
secondBuildPlugin,
//...
]
}
},
// ...
Preserving old code
Finally, for your other builds, you may still want to preserve the old paths. You can use #rollup/plugin-replace for this, by using a placeholder that will be replaced in the build process.
In your files, you could replace:
var worker = new Worker(libraryObjectURL);
with:
var worker = new Worker(__workerLibraryURL__);
and in the build process use:
// ...
// for the first build:
plugins: [
firstBuildPlugin,
replace({ __workerLibraryURL__: 'libraryObjectURL')
// ...
],
// ...
// for the second build:
plugins: [
secondBuildPlugin,
replace({ __workerLibraryURL__: 'libraryObjectURL')
// ...
],
// ...
// for all other builds:
plugins: [
firstBuildPlugin,
replace({ __workerLibraryURL__: 'new URL(import.meta.url)') // or whatever the old code was
// ...
],
You may need to use another replacement for your AudioWorklet url if it's different. In cases where the worker-library file isn't used, the imported libraryObjectURL will be tree shook out.
Future work:
You may want to look into having multiple outputs for your different targets: web worker, audio worklet, and library code. They really aren't supposed load the same exact file. This would negate the need for the first plugin (that ignores certain files), and it might make things more manageable and efficient.
More Reading:
Loading files as raw strings (you can see/use TrySound's plugin here; it's a simple plugin)
Loading strings as blob or data URLs (see https://stackoverflow.com/a/10372280/)
I found a way to solve the described problem but there are still some open pain points because the WebPack devs are rather trying to avoid vendor specific expressions and prefer to rely on "recognizable syntax constructs" to rewrite the code as they see fit.
The solution does not work in a fully local environment, but it works together with NPM:
I am launching my worker now with /* webpackChunkName: "alphatab.worker" */ new Worker(new URL('#coderline/alphatab', import.meta.url))) where #coderline/alphatab is the name of the library installed through NPM. This syntax construct is detected by WebPack and will trigger generation of a new special JS file containing some WebPack bootstrapper/entry-point which loads the library for startup. So effectively it looks after compilation like this:
For this to work, users should configure the WebPack to place the library in an own chunk. Otherwise it can happen that the library is maybe inlined into the webpack generated worker file instead of also loaded from a common chunk. It would work also without a common chunk, but it would defy the benefits of even using webpack because it duplicates the code of the library to spawn it as worker (double loading time and double disk usage).
Unfortunately this currently only works for Web Workers for now because WebPack has no support for Audio Worklet at this point.
Also there are some warnings due to cyclic dependencies produced by WebPack because there seem to be a a cycle between chunk-alphatab.js and alphatab.worker.js. In this setup it should not be a problem.
In my case there is no difference between the UI thread code and the one running in the worker. If users decide to render to an HTML5 canvas through a setting, rendering happens in the UI thread, and if SVG rendering is used it is off-loaded to a worker. The whole layout and rendering pipeline is the same on both sides.

Webpack: Get list of chunks from inside javascript

I wanted to access the list of JS chunk files from within entry point Javascript file. Is their a way I can inject it to JS as an array?
Description:
For example, I have following JS files:
node_modules
moment.js
jquery.js
public
index.html
menu.html
src
shared.js
utils.js
main.js (dependent on shared.js & uses moment)
index.js (dependent on shared.js & uses moment + jquery)
menu.js (dependent on utils.js & uses moment + jquery)
Now in webpack config I have 3 entry points say main, index, menu. For index & menu, I have related html files and I use HtmlWebPack plugin to inject the chunk files which works well.
But "main" entry point doesn't have any html files and it would be used for some other reason. Here for this entrypoint also it creates multiple dependent chunks say for example (0.js, main.chunk.js, runtime-main.chunk.js).
So for me, from "index.js" file, I wanted to get the list of all the chunks related to "main" entrypoint as array so that I can use those chunks dynamically when the code is getting executed in the browser.
Sample "index.js":
// Currently I am hardcoding this list, which makes me manually update this list whenever I add some more additional dependent files to it.
// I wanted this file list to be populated automatically while the bundling happens so that I need not maintain this list manually.
const filesList = ['/runtime-main.chunk.js','/0.js','main.chunk.js'];
filesList.forEach(file =>{
// I will do something with this file path
});
Mine is a very specific usecase where I definetely need the list of file names dynamically instead of somehow statically loading it. I know I can make use of "webpack-manifest-plugin" which generates the required mapping information as json file which I can use dynamically. But I am looking for a solution where it can somehow injected the list as array itself while bundling the code. I am definetely sure this is possible as I could achieve something similar elsewhere.
Question 2:
For the same usecase above, is their a way I can bundle all the dependency of "main" entrypoint alone as a single file which basically would be "main.chunk.js" so that i can just use that file without need for an array of files.
In advance, thank you for all your answers!!

yii2 registering JS files to a View

I have A.php view file in /views/A/ folder.
And I have A.js js file in /views/A/ folder
Please help me register js file in view file.
As I understand I must write
$this->registerJsFile('path/to/file.js'); in view file.
But (Question A) I get method registerJsFile is not found in a class message from PHPStorm.
Also (Question B) what should I write in path considering both files are in the same folder /views/A/ ?
This is not elegant, but working if you need to have your js file registered after jquery (as seen in the Yii2 doc)
<?php $this->registerJsFile(Yii::$app->request->baseUrl.'/js/youFile.js',['depends' => [\yii\web\JqueryAsset::className()]]); ?>
If you register a js file by:
$this->registerJsFile("#web/js/all.js");
This will work but you will not be able to use jQuery. Because this file all.js is loaded before jQuery. To load after jQuery we make it depend it on 'yii\web\YiiAsset' or on \yii\web\JqueryAsset . So it will be loaded after jQuery.js. Example:
$this->registerJsFile("#web/js/all.js",[
'depends' => [
\yii\web\JqueryAsset::className()
]
]);
So What is difference between \yii\web\JqueryAsset and \yii\web\YiiAsset?
In jQueryAsset the js file will load after jQuery.js and in YiiAsset the js file will load after yii.js file.
If you want to create your own custom Asset Bundle:
<?php
namespace frontend\components;
use yii;
use yii\web\AssetBundle;
class CustomAssets extends AssetBundle
{
public $css = [
"path/to/css/file.css"
];
public $js = [
"path/to/js/file.js"
];
public $depends = [
];
}
Register your js file on given possion
$this->registerJsFile('path/to/file.js', ['position' => \yii\web\View::POS_END]);
The first argument is the actual JS code we want to insert into the page. The second argument determines where script should be inserted into the page. Possible values are:
View::POS_HEAD for head section.
View::POS_BEGIN for right after opening .
View::POS_END for right before closing .
View::POS_READY for executing code on document ready event.
This will register jQuery automatically.
View::POS_LOAD for executing code on document load event. This will register jQuery automatically.
The last argument is a unique script ID that is used to identify code block and replace existing one with the same ID instead of adding a new one. If you don't provide it, the JS code itself will be used as the ID.
An external script can be added like the following:
$this->registerJsFile('http://example.com/js/main.js', ['depends' => [\yii\web\JqueryAsset::className()]]);
The arguments for registerJsFile() are similar to those for registerCssFile(). In the above example, we register the main.js file with the dependency on JqueryAsset. This means the main.js file will be added AFTER jquery.js. Without this dependency specification, the relative order between main.js and jquery.js would be undefined.
is there any specific reason to include the file manually rather than creating an asset bundle?
In any case if you've read the documentation regarding assets, you would have noticed that there's a clear distinction about source, published and external assets.
The most important part of it being that source and published assets use different options to determine whether and how a file should be published.
In your case you've got a source asset which needs to be copied over to the assets directory.
The invocation of registerJsFile as hinted in the documentation, will expect a published asset.
Here you have specifically two options, which probably the first is more quick and coherent:
move the asset in the web/ folder, as in web/js/ or whatever you prefer and keep using registerJsFile()
add a new asset bundle for source assets, specifying the various options as detailed in the above linked page.
Hope this clears things out.
A: From the docs: http://www.yiiframework.com/doc-2.0/yii-web-view.html
Your code seem correct.
Do you register the js from the view file itself? not the controller?
The registerJsFile() method is from the view class.
Its highly possible that your IDE is not finding the method, have you tried it in a apache enviroment?
B: Use a alias

Mastering external scripts loading order in Meteor (Google Maps)

I tried unsuccessfully to add a google map(externally loaded script) to a meteor app, and I noticed there were two kinds of problems:
If I do the simple thing and add the main API script to my <head></head>, then it gets rendered last.
When this happens, I am obliged to insert any scripts that depend on the API again in my template's <head> - after the main API script. (otherwise scripts complain they don't see the API blabla..)
Then the time for the actually function call comes - and now putting it inside <head> after the rest won't work. You need to use Template.MyTemplate.rendered.
Basically my question is:
What's the cleanest way to handle these kinds of things?
Is there some other variable/method I can use to make sure my Google main API file is called very first in my HTML?
I just released a package on atmosphere (https://atmosphere.meteor.com) that might help a bit. It's called session-extras, and it defines a couple functions that I've used to help with integrating external scripts. Code here: https://github.com/belisarius222/meteor-session-extras
The basic idea is to load a script asynchronously, and then in the callback when the script has finished loading, set a Session variable. I use the functions in the session-extras package to try to make this process a bit smoother. I have a few functions that have 3 or 4 different dependencies (scripts and subscriptions), so it was starting to get hairy...
I suppose I should add that you can then conditionally render templates based on whether all the dependencies are there. So if you have a facebook button, for example, with helpers that check the Session variables, you can give it a "disabled" css class and show "loading facebook..." until all the necessary scripts have loaded.
edit 03/14/2013
There is also an entirely different approach that is applicable in many cases: create your own package. This is currently possible with Meteorite (instructions), and the functionality should soon be available in Meteor itself. Some examples of this approach are:
jquery-rate-it: https://github.com/dandv/meteor-jquery-rateit
meteor-mixpanel: https://github.com/belisarius222/meteor-mixpanel
If you put a js file in a package, it loads before your app code, which is often a good way to include libraries. Another advantage of making a package is that packages can declare dependencies on each other, so if the script in question is, for example, a jQuery plugin, you can specify in the package's package.js file that the package depends on jQuery, and that will ensure the correct load order.
Sometimes it gets a little more interesting (in the Chinese curse sense), since many external services, including mixpanel and filepicker.io, have a 2-part loading process: 1) a JS snippet to be included at the end of the body, and 2) a bigger script loaded from a CDN asynchronously by that snippet. The js snippet generally (but not always!) makes some methods available for use before the bigger script loads, so that you can call its functions without having to set up more logic to determine its load status. Mixpanel does that, although it's important to remember that some of the JS snippets from external services expect you to set the API key at the end of the snippet, guaranteed to be before the bigger script loads; in some cases if the script loads before the API key is set, the library won't function correctly. See the meteor-mixpanel package for an example of an attempt at a workaround.
It's possible to simply download the bigger js file yourself from the CDN and stick it in your application; however, there are good reasons not to do this:
1) the hosted code might change, and unless you check it religiously, your code could get out of date and start to use an old version of the API
2) these libraries have usually been optimized to load the snippet quickly in a way that doesn't increase your page load time dramatically. If you include the bigger JS file in your application, then your server has to serve it, not a CDN, and it will serve it on initial page load.
It sounds like you're loading your Javascript files by linking it with HTML in your template. There's a more Meteor way of doing this:
From the Meteor Docs:
Meteor gathers all JavaScript files in your tree with the exception of
the server and public subdirectories for the client. It minifies this
bundle and serves it to each new client. You're free to use a single
JavaScript file for your entire application, or create a nested tree
of separate files, or anything in between.
So with that in mind, rather than link the gmaps.js into head, just download the un-minified version of gmaps and drop it in you application's tree.
Also from the Meteor Docs:
It is best to write your application in such a way that it is
insensitive to the order in which files are loaded, for example by
using Meteor.startup, or by moving load order sensitive code into
Smart Packages, which can explicitly control both the load order of
their contents and their load order with respect to other packages.
However sometimes load order dependencies in your application are
unavoidable. The JavaScript and CSS files in an application are loaded
according to these rules:
Files in the lib directory at the root of your application are loaded
first.
[emphasis added]
And if the sequence is still an issue, drop the js file into client/lib and it will load before all the Javascript you've written.
I've used meteor-external-file-loader and a bit of asynchronous looping to load some scripts, which will load javascript (or stylesheets) in the order you specify.
Make sure to have meteorite and add the package above >> mrt add external-file-loader
Here's the function I wrote to make use of this package:
var loadFiles = function(files, callback) {
if (!callback) callback = function() {};
(function step(files, timeout, callback) {
if (!files.length) return callback();
var loader;
var file = files.shift();
var extension = file.split(".").pop();
if (extension === "js")
loader = Meteor.Loader.loadJs(file, function() {}, timeout);
else if (extension === "css") {
Meteor.Loader.loadCss(file);
loader = $.Deferred().resolve();
}
else {
return step(files, timeout, callback);
}
loader.done(function() {
console.log("Loaded: " + file);
step(files, timeout, callback);
}).fail(function() {
console.error("Failed to load: " + file);
step(files, timeout, callback);
});
})(files, 5000, callback);
}
Then to use this, add to one of your created methods for a template like so:
Template.yourpage.created = function() {
var files = [
"//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js",
"javascripts/bootstrap.min.js"
];
loadFiles(files, function() {
console.log("Scripts loaded!");
});
}
Quick edit: Found it's just a good idea to place the functionality from the .created method in the /lib folder in Meteor.startup();
Meteor.startup(function() {
if (Meteor.isClient) {
// Load files here.
}
});
Caveat: Lots of javascript files = really long load time.... Not sure how that will be affected by the normal meteor javascript files, and the load order there. I would just make sure that there are no conflicts until users take action on the website (or that if there is, they are loaded first).

How can I split my conkeror-rc config over multiple files?

Short version: can you help me fill in this code?
var conkeror_settings_dir = ".conkeror.mozdev.org/settings";
function load_all_js_files_in_dir (dir) {
var full_path = get_home_directory().appendRelativePath(dir);
// YOUR CODE HERE
}
load_all_js_files_in_dir(conkeror_settings_dir);
Background
I'm trying out Conkeror for web browsing. It's an emacs-like browser running on Mozilla's rendering engine, using javascript as configuration language (filling the role that elisp plays for emacs). In my emacs config, I have split my customizations into a series of files, where each file is a single unit of related options (for example, all my perl-related settings might be in perl-settings.el. All these settings files are loaded automatically by a function in my .emacs that simply loads every elisp file under my "settings" directory.
I am looking to structure my Conkeror config in the same way, with my main conkeror-rc file basically being a stub that loads all the js files under a certain directory relative to my home directory. Unfortunately, I am much less literate in javascript than I am in elisp, so I don't even know how to "source" a file.
I found a suitable answer, though it isn't really what I was looking for. If you set your conkerorrc file to a directory, then all the js files in that dir will be loaded.

Categories