We Keep Coding

Reduce Javascript size in Rails app and improve performance score. Maybe issue with ESBuild?

Not specifically a Rails question, but a question within a Rails app.
In my app I am using the jsbundling-rails gem configured with esbuild.
This gem adds a build line to my package.json file. It works and compiles all my JS and runs fine. However, I found that the generated file is rather large so I started looking at ways to optimise it.
My esbuild statement at this point looks like:
"build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds"
Firstly I thought I could try making my imports conditional. Eg, only import them when they are actually required. I asked another question on how to do that here.
I learned quite a lot digging into that, but at the end of the day it's made no difference to the output of my JS weirdly.
Chrome currently says that my main JS file has 91% code unused. It looks like all the imports are still being compiled together, whether they are statically or dynamically imported. Why can this be?
I then looked further into esbuild, I spotted the --splitting flag. It sounded reasonably correct so I updated my build script to be:
"build": "esbuild app/javascript/*.* --bundle --splitting --format=esm --sourcemap --outdir=app/assets/builds"
This caused a huge amount of outputted JS files (I think they are referred to as "chunks".
I ran my app, and the JS failed to load. The console stated that
Uncaught SyntaxError: Cannot use import statement outside a module
I wasn't 100% sure why was the case but I just guessed that I needed to add type: "module" to my javascript_include_tag in my Rails application layout view.
This made the JS load (which is good :-) )
BUT... The percentage of unused JS code is still 84% of my application.js
So..... my questions are as follows:
Are my dynamic importing of modules working?
Why does static or dynamic importing appear to make no difference?
How can I effectively reduce the size of the output code and reduce the unused percentage of JS on my home page?
This all started because I ran Google's Lighthouse test on my site and it reported my Structure and Accessibility to be practically perfect but performance was < 40. I am aiming to solve this.
I look forward to hearing from you with ideas on how I can try to fix this and improve my Lighthouse Performance score.

Are my dynamic importing of modules working? Why does static or
The paradigm you want is called Propshaft. Try looking at the SProckets -> Propshaft
https://github.com/rails/propshaft/blob/main/UPGRADING.md
Why dynamic importing appear to make no difference?
Because even though ESBuild allows for ES Module syntax (import statements), it still bundles everything into "1" big file. (1 big file for each javascript_include_tag you have, of course).
How can I effectively reduce the size of the output code and reduce the unused percentage of JS on my home page?
The sprockets paradigm was built for the HTTP1 web when keeping connections open and progressive download wasn't realistic. HTTP2 changed all that and now it's more efficient to do code splitting as you want to do. But the Rails world is still very behind and most apps still use Sprockets and try to optimize/minify was much as possible.
I'd recommend you take this course of action:
(1) Try the old-style way first. Remove anything unnecessary, split your app into different sections and load different manifest files for different sections. Use minification. See how far that gets you.
(2) Start experimenting with the new Propshaft for a few weeks until you fully understand it. If you feel it is solid, migrate to that.

Understanding Laravel Mix

Understanding Laravel Mix
I am currently in the process of migrating one of my websites to Laravel in order to make it a little more maintainable in future... I have plenty of experience building API's with Laravel but I have very limited experience building websites with Laravel and resultantly I am in need of a little bit of guidance from another pro.
In short, I would very much appreciate answers to the following very simple questions if anyone can spare me a couple of mins...
File based JS & CSS instead of App based
I like to write my JS and CSS files in a particular way where each page has their own specific files relevant to the page. For example, about.php might have the following dependencies:
JS:
jquery.js
any_other_third_party_library.js
app.js (global functions)
about.js (page specific functions)
CSS:
some_third_party_library.css
app.css (global styles)
about.css (page specific styles)
On my own framework, the above would be combined and minified into one file for JS and one file for CSS. From what I understand, Laravel Mix does exactly this...
However, as far as I can see, the way to do this would be as follows:
webpack.mix.js:
// About
mix.scripts([
'resources/assets/js/app.js',
'resources/assets/js/about/about.js'
], 'public/js/about/about.js');
Very simply, what I would like to know; is the above the correct way to go about this? Is there a better, more efficient, way to automate this for each page?
What are the bootstrap.js and app.js files?
From what I can see, these files just load dependencies but this is a little confusing as some dependencies might be page specific... Please can someone explain in a little further detail what these files are for? Or at least, what the difference is between them...
Getting rid of Vue
I have no interest in using Vue in my project so I have deleted the following files:
/components/Example.vue
and the Vue code in app.js
Does this matter in any way?

You'll bundle up all your styles and scripts a single file each and serve them to the client minified.
For front end assets, call mix.sass('resources/assets/sass/app.scss'). In that entry point to your styles you will be able to import your other stylesheets as you need using Sass's #import 'about'; syntax. (Rename your other CSS files to end in .scss too).
For your back end assets, call mix.js('resources/assets/js/app.js'). Then, similarly you can import other JavaScript modules using import './about.js';. You may want to look up ES2015 modules so you can learn how to write modular JavaScript.
Read through the bootstrap.js file to see how Laravel hooks up jQuery and Vue by default. You don't need any of this, so remove whatever you don't want or delete the entire file if you don't need any of it.
Vue comes out of the box with Laravel but it's just a suggestion, you can replace it with your own front end framework of choice or rip it out and replace it with nothing. Up to you.
Edit: Long story short; use mix.sass and mix.js, read up on using Sass and ES2015 modules to write awesome code.

How to organize Vue-JS : Non Javascript way?

How to organize Vue-JS project [files and directory]: Non Javascript way?
Coming from a non-javascript background, I found Vue.js very intuitive and easy to use. My earlier experience in Javascript is with JQuery and vanilla javascript on the browser.
I created an application using Vue , Vue-components and vue router. Everything works fine.
My problem is, I have ended up writing a lot of code in a single index.html file of my project. This file contains over 10 templates that I have created and attached to different component in my app. I want to know that is there a non-javascript way to organize these templates in seperate files.
I see that there are options using webpack and browserify to modularize the project. But coming from non javascript background, I don't find them intuitive. I don't want to go node - npm way because that has its own learning curve and moreover it downloads a dozen of files and dependencies in my project which I don't understand. I am old school and more comfortable downloading the files and including them in the webpages.
So probably, you understand where I am going to. I need a solution where I could put my templates as separate files and read those files in the different components.
What I have tried :
Declaring the templates inside my components. But writing all that html inside the component is not that clean. It also, makes my JS file too huge. I am better in putting all data in the index.html instead.
Storing template as smaller chunk "homepage.html","about.html" and in my components, using $.get / $.load to read different components in ready function of the component. This works but I still have to fire an additional ajax call for each component which is not efficient.
Please refrain from suggesting the obvious node-npm [webpack and browserify] way. I know thats what is supported by Vue but this needs a learning curve and complete setup. Answer to this question would actually help other developers who hesitate going the node-npm way.
Please shout back if you need more clarifications to the question.

The options you've mentioned are your only real ones... the HTML of the template needs to be available when it's needed, so you either have to have it within your html file off the bat, or load it using AJAX or an in-browser loader like RequireJS (and this extension that allows it to load HTML https://github.com/requirejs/text).
In-file templates make sense for very small projects. As your project grows, you'll need to start using the tools that are built for this. NPM rocks and every JS package that you'll ever need can be included in your project in seconds.

I highly encourage you to try the Vue CLI
It does use node, npm, webpack and downloads dozens of files. Which you've you've explicitly asked for not to use, so let me clarify:
The Vue CLI takes care of the complexity and configures webpack for you.
You don't even have to know it's using webpack.
It's very developer friendly (it even has a built-in gui) and lowers the barrier to entry compared configuring a webpack config.
I hope you'll also find it "intuitive and easy to use".

Best practices for resolving statically javascript dependencies

It's quite sad I have to ask this question in April 2012, but I know quite few approaches to the problem.
So, I want a tool for statically compiling javascript files, resolving dependencies.
To be more precise I want such tool (let's call it sometool) which can do following:
scan a folder, find all *.js files
parse this files and find some very simple synchronous and 100% js-compliant directive. By synchronous I mean that there is no such thing like passing onload handler - it's only about static text preprocessing. By 100% js-compliant I mean that it looks it's looks like a comment from js point of view - something like // #include
resolve paths used in include directive, using some config which can be customized.
substitute all include with real output and put it into a separate folder.
I do realize that all this sounds obvious to most of you, but javascripters nowadays understand something slightly different while talking about dependency managing - see require.js, for example.

You should have a look at sprockets. It can bundle and pack JavaScript files and supports includes. You might have to write some boilerplate code though. There seems to be a script that allows sprocket to be run from command line bundled with it. And it's in Ruby. Don't know if this is a problem.

There actually is one answer, its name is: Apache ANT
ANT like Make, is a tool which can do almost any job for you. Concatenating files, stripping, reading files/foldes, minifiying, etc. etc. Instructions are done via .xml files, many jobs are already present but of course you can write your own jobs.
That way, you can have a perfect static build process. Personally I use it on severval bases. Have a look as example: https://github.com/jAndreas/typeof-NaN-2.0/tree/master/build
I'm not resolving javascript dependencies the way you described, but I do it with stylesheet dependencies there (replacing #import lines with actual files). So its a fairly easy task to do the same with javascript files. Also, last step in those build scripts are to copy the fresh generated files into my Apache website directory, its indeed one static build process and you're done.
Don't be scared about the filesize, I modified an existent config file from the HTML5 boiler plate there. I actually could shrink that file to 10% size, removing all optional builds.

How do you manage your Javascript files?

Nowadays, we have tons of Javascript libraries per page in addition to the Javascript files we write ourselves. How do you manage them all? How do you minify them in an organized way?

Organization
All of my scripts are maintained in a directory structure that I follow whenever I work on a site. The directory structure normally goes something like this:
+--root
|--javascript
|--lib
|--prototype.js
|--scriptaculous
|--scriptaculous.js
|--effects.js
|--..
|--myOwnScript.js
|--myOwnScript2.js
If, on the off chance, that I'm working on a team uses an inordinate amount of scripts, then I'll normally create a custom directory in which we'll organize scripts by relationship. This doesn't happen terribly often, though.
Compression
Though there are a lot of different compressors and obfuscators out there, I always come back to YUI Compressor.
Inclusion
Unless a site is using some form of a master page, CMS, or something that dictates what can be included on a page beyond my control, I only included the scripts necessarily for the given page just for the small performance sake. If a page doesn't require any script, there will be no script inclusions on that page.

First of all, YUI Compressor.
Keeping them organized is up to you, but most groups that I've seen have just come up with a convention that makes sense for their application.
It's generally optimal to package up your files in such a way that you have a small handful of packages which can be included on any given page for optimal caching.
You also might consider dividing your javascript up into segments that are easy to share across the team.

Cal Henderson (of Flickr fame) wrote Serving JavaScript Fast a while back. It covers asset delivery, not organization, but it might answer some of your questions.
Here are the bullet points:
Yes, you ought to concatenate JavaScript files in production to minimize the number of HTTP requests.
BUT you might not want to concatenate into one giant file; you might want to break it into logical pieces and spread the transfer cost over several pages.
gzip compression is good, but you shouldn't serve gzipped assets to IE <= 6, so you might also want to minify/compress your JavaScript.
I'll add a few bullet points of my own:
You ought to come up with a solution that works for both development and production. In development mode, it should pull in extra JavaScript files on demand; in production it should bundle everything ahead of time. Switching from one behavior to the other should be as easy as setting a flag.
Rails 2.0 handles all this through an asset cache; other web app frameworks might offer similar solutions.
As another answer suggests, placing third-party libraries in a lib directory is a good start. You can also divide your own JS files into sub-directories if it makes sense. Ideally, you'll be able to arrange them in such a way that the files in a given sub-directory can be concatenated into one file.

I will have a folder for all javascript, and a sub folder of that for 3rd party/shared libraries, and sub folders for each component of the site to keep everything organized.
For example:
/
+--/javascript/
+-- lib/
+-- admin/
+-- compnent1/
+-- compnent2/
Then run everything through a minifier/obfuscator during the build process.

I'v been using this lately:
http://code.google.com/apis/ajaxlibs/
And then have a "jscripts" folder where I keep my custom code.

In my last project, we had three kinds of JS files, all of them inside a JS folder.
Library code. A bunch of functions used on most all of the pages, so they were put together in one or a few files.
Classes. These had their own files, organized in folders as needed, but not necessarily so.
Ad hoc JS. Code that was specific to that page. These were saved in files that had the same name as the JSP pages they were supposed to run in.
The biggest effort was in having most of the code on the first two kinds, having custom code only know what to call, and when.

This might be a different approach than what you're looking for, but I've been playing around with the idea of JavaScript templates in our blog engine. In a nutshell, you assign a Javascript template to a page id using the database and it will dynamically include and minify all the JavaScript files associated with that template and create a file in a server-side cache with the template id as a file name. When a page is loaded, it calls the template file which first checks if the file exists in the cache and loads it if it does. If it doesn't exist, it creates it on the fly and includes it. I also use the template file to gzip the conglomerate JavaScript file.
The template idea would work well for site-wide JavaScript (like a JavaScript library), but it doesn't cover page-specific JavaScript. However, you can still use the same approach for page specific JavaScript by including a second file that does the same as above.