We Keep Coding

Using webpack to create & inject asset filesizes dictionary

I'm using Webpack to bundle source code and assets for a game. I also use the CompressionPlugin() to make static gzip files available so that my web server can send the precompressed files when appropriate. Some of the game assets are large so I have a loading experience up front that shows a progress bar while the assets are fetched.
Unfortunately a problem arises on Chome during loading when receiving a gzip response for an XMLHttpRequest in that the onprogress total is always 0. There are some imperfect workarounds for this such as this solution but they're not entirely appropriate for my case.
Instead I'd like to inject the compressed & decompressed file sizes of specific bundled assets into the html or javascript so that they're immediately accessible to the loading javascript code. Injecting something as follows would be perfect:
<script>
const assetSizes = {
"game.25317abd3eb6cf0fb0f1.wasm": {uncompressed: 8192, compressed: 1024},
"game.25317abd3eb6cf0fb0f1.data": {uncompressed: 8192, compressed: 1024}
};
</script>
I'm somewhat new to webpack so I'm not entirely sure how to approach this. I've considered using the WebpackManifestPlugin and implementing a custom generate option function. This would allow me to control the output of the generated manifest.json but it's still not clear to me if this the right thing to do or how I'd go about subsequently injecting this files contents ahead of my own loading javascript code.
Perhaps there is a better approach that would be more appropriate?
Update: I've been trying to progress this further and it feels like a custom Webpack plugin might be the right direction to go. If I tap into the afterEmit compiler hook it seems I have access to the filesizes I need and I can construct an appropriate dictionary:
class InjectFileSizePlugin {
apply(compiler) {
compiler.hooks.afterEmit.tap(
"InjectFileSizePlugin",
(compilation) => {
const fileSizes = {};
for (const [assetPath, assetInfo] of compilation.assetsInfo) {
if (assetInfo.related) {
fileSizes[assetPath] = {
uncompressed: assetInfo.size,
compressed: -1
};
if (assetInfo.related.gzipped) {
const gzippedAssetInfo = compilation.assetsInfo.get(
assetInfo.related.gzipped
);
if (gzippedAssetInfo) {
fileSizes[assetPath].compressed =
gzippedAssetInfo.size;
}
}
}
}
console.log(fileSizes); // <-- output is as I'd like, how to inject it now?
}
);
}
}
What's not clear though is how I can now go about injecting this fileSizes data into the bundle as afterEmit is called very late in the compilation stage after the bundle javascript has been emitted. There is an additionalPass compiler hook but I currently can't figure out how it works.

What are the purposes of vinyl-buffer and gulp-streamify in gulp?

As the documentation says, they both deal with transforming non-stream plugins to stream.
What I try to understand is, if I can use the .pipe() method on something, doesn't it mean it's a stream?
If so, what do I convert to what here?
vinyl-source-stream example:
(from: https://www.npmjs.com/package/vinyl-buffer)
var browserify = require('browserify')
var source = require('vinyl-source-stream')
var buffer = require('vinyl-buffer')
var uglify = require('gulp-uglify')
var size = require('gulp-size')
var gulp = require('gulp')
gulp.task('build', function() {
var bundler = browserify('./index.js')
return bundler.pipe()
.pipe(source('index.js'))
.pipe(buffer()) // <---------------------- why?
.pipe(uglify())
.pipe(size())
.pipe(gulp.dest('dist/'))
})
gulp-streamify example:
(from: https://www.npmjs.com/package/vinyl-source-stream)
var source = require('vinyl-source-stream')
var streamify = require('gulp-streamify')
var browserify = require('browserify')
var uglify = require('gulp-uglify')
var gulp = require('gulp')
gulp.task('browserify', function() {
var bundleStream = browserify('index.js').bundle()
bundleStream
.pipe(source('index.js'))
.pipe(streamify(uglify())) // <----------- why?
.pipe(gulp.dest('./bundle.js'))
})

One semi-useful example is to think about putting out a campfire with a bucket of water. To put out the fire you would want to completely fill up the bucket before dumping it on the fire rather putting a few drops in the bucket and then dumping lots of little drops over time on the fire. This metaphor doesn't capture everything but the big idea is this: you need a FULL bucket of water before you can put out the fire.
That "uglify" plugin works the same way. Imagine some enormous JS file you'd want to compress/uglify.
It will take a little bit of time to load the whole codebase & you definitely wouldn't want to try minifying each line as it comes in, right? Imagine you load a single line, minify it, load another line, minify it, etc etc-- it'd be a mess. You can't stream it (you need a full "bucket" of code before you can uglify it.) To uglify that file properly you'd need to load all that code first before attempting to uglify it.
Since Gulp is a "streaming" build system, you can't use uglify unless you have some mechanism to turn the stream into a buffer (& when it's done emit a stream.) Both tools you mention make this possible.
Here's the flow:
STREAM > (BUFFER) > {perform some work on the whole "buffered" file} > STREAM > {other gulp work, etc }
To your specific question, you can use .pipe() because vinyl-buffer/gulp-streamify help "convert" streams to buffers then buffers to streams. They're different approaches to accomplish essentially the same thing.

As said, most plugins work with buffers (although some of them also support streams). Examples include gulp-uglify and gulp-traceur. You can do the conversion to buffers using gulp-buffer.
via https://medium.com/#webprolific/getting-gulpy-a2010c13d3d5
gulp-uglify dosen't support stream, so you should convert stream to buffer (example uses vinyl-buffer)
gulp-streamify can wrap old plugins to support streams(example uses gulp-uglify)
Different approaches but equally satisfactory results.

What I try to understand is if I can use the .pipe() method on
something, doesn't it mean that it's a stream?
No, .pipe() can also pass buffers. this blog post explains it well:
https://medium.com/#sogko/gulp-browserify-the-gulp-y-way-bb359b3f9623
Some gulp-* plugins works by taking in buffered vinyl files objects as
input.
But vinyl-source-stream emits a streaming vinyl file object.
That’s where vinyl-buffer comes in. So we simply need to convert that
to a buffered vinyl by using vinyl-buffer, like so

What I try to understand is if I can use the .pipe() method on
something, doesn't it mean that it's a stream?
Yes! It is a stream. But it's a an object stream!
Instead of streaming a series characters, it streams a series of objects, which are the files that you sourced.
Each 'data' event in a gulp stream emits a Vinyl file object, which looks something like this:
{
cwd: '/', //<string>
base: '/test/', //<string>
path: '/test/file.js', //<string>
contents: contents //<string> | <Buffer> | <stream.Readable>
}
So gulp-buffer plugin is a Transform stream that converts the file contents from stream.Readable to Buffer.
You can see this in the source, where it saves the original content stream on line 24 and assigns a Buffer as the new file contents on line 35.
Streamify does the same thing, on line 35 and line 48.
It's ok to leave the file contents as a Buffer after Uglify is done processing it. It's always ok for the contents to be a Buffer, gulp just doesn't do this when sourcing because it's too costly.

inserting text in a gulp stream

I would like to hard-code a variable into a GULP task (for development purposes and not production), I need something like this:
gulp.task('buildJS', function() {
gulp.insert('var DEV = true;') // inserting the text somehow
.src([
'./js/a.js',
'./js/b.js',
'./js/c.js'
])
.pipe(concat('main.min.js'))
.pipe(gulp.dest('./js/'));
});
I tried to google it, but I get a bunch of bad results, also not sure what exactly to look for here, but can only describe here.
As you can see in the first line, I would like to insert some variable that comes before all other things (in this specific case, and then the rest of the files should be included, and it all gets concatenated into one file at the end, but I am not sure how to do that... Thanks for any help with this!

You can use the gulp-header plugin that allow to append text at the beginning of your files and which support templating.
You can do something like :
var header = require('gulp-header');
gulp.task('buildJS', function() {
gulp.src([
'./js/a.js',
'./js/b.js',
'./js/c.js'
])
.pipe(header('var DEV = true;'))
.pipe(concat('main.min.js'))
.pipe(gulp.dest('./js/'));
});
Or if you only want to append it on the concatened file
gulp.task('buildJS', function() {
gulp.src([
'./js/a.js',
'./js/b.js',
'./js/c.js'
])
.pipe(concat('main.min.js'))
.pipe(header('var DEV = true;'))
.pipe(gulp.dest('./js/'));
});

You may want to use gulp-header : https://www.npmjs.org/package/gulp-header
In your case, put it after the concat operation in the pipeline.

Minify Javascript programmatically in-memory

I am building a nifty little "asset-pipeline" for a express.js application, but i have a problem with the compression-step for javascript files
scripts = (fs.readFileSync(file) for file in filelist)
result = scripts.join("\n\n") # concat
upto now, things are working as expected (the logic itself is written in coffeescript). The next step after merging the JS-files would be to minify them. But here is my problem: i want to do this "hot" when i start my express-app in production mode, from within a piece of connect-middleware i wrote.
I need a solution that can minify a given blob of javascript stuff, without writing the result to disk (!), in other words: a function that does the minification and returns the result directly as a result value. (No, no webservices either.) It should be usable like this:
minified_result = awesomeMinifyFunction( result )
The raw processing performance isn't that important for me, neither is the level of compression, i need just something that works this way without hassle.
Does anyone know a suitable solution? Thanks in advance!

I'd suggest you look at one of the JavaScript based minifiers, like UglifyJS2.
npm install uglify-js
It can be used within a Node.JS application programatically:
var UglifyJS = require("uglify-js");
// you could pass multiple files (rather than reading them as strings)
var result = UglifyJS.minify([ "file1.js", "file2.js", "file3.js" ]);
console.log(result.code);
Or you could
var result = scripts.join("\n\n"); # concat
result = UglifyJS.minify(result, {fromString: true});
console.log(result.code);

You can write your own function that removes all comments/spaces/blank lines etc.
You can use a regular expression that makes use of rJSmin like:
function awesomeMinifyFunction(result)
{
pattern = (
r'([^\047"/\000-\040]+)|((?:(?:\047[^\047\\\r\n]*(?:\\(?:[^\r\n]|\r?'
r'\n|\r)[^\047\\\r\n]*)*\047)|(?:"[^"\\\r\n]*(?:\\(?:[^\r\n]|\r?\n|'
r'\r)[^"\\\r\n]*)*"))[^\047"/\000-\040]*)|(?<=[(,=:\[!&|?{};\r\n])(?'
r':[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/))*'
r'(?:(?:(?://[^\r\n]*)?[\r\n])(?:[\000-\011\013\014\016-\040]|(?:/\*'
r'[^*]*\*+(?:[^/*][^*]*\*+)*/))*)*((?:/(?![\r\n/*])[^/\\\[\r\n]*(?:('
r'?:\\[^\r\n]|(?:\[[^\\\]\r\n]*(?:\\[^\r\n][^\\\]\r\n]*)*\]))[^/\\\['
r'\r\n]*)*/)[^\047"/\000-\040]*)|(?<=[\000-#%-,./:-#\[-^`{-~-]return'
r')(?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/'
r'))*(?:(?:(?://[^\r\n]*)?[\r\n])(?:[\000-\011\013\014\016-\040]|(?:'
r'/\*[^*]*\*+(?:[^/*][^*]*\*+)*/)))*((?:/(?![\r\n/*])[^/\\\[\r\n]*(?'
r':(?:\\[^\r\n]|(?:\[[^\\\]\r\n]*(?:\\[^\r\n][^\\\]\r\n]*)*\]))[^/'
r'\\\[\r\n]*)*/)[^\047"/\000-\040]*)|(?<=[^\000-!#%&(*,./:-#\[\\^`{|'
r'~])(?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)'
r'*/))*(?:((?:(?://[^\r\n]*)?[\r\n]))(?:[\000-\011\013\014\016-\040]'
r'|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/))*)+(?=[^\000-\040"#%-\047)*,./'
r':-#\\-^`|-~])|(?<=[^\000-#%-,./:-#\[-^`{-~-])((?:[\000-\011\013\01'
r'4\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/)))+(?=[^\000-#%-,./:'
r'-#\[-^`{-~-])|(?<=\+)((?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*'
r'\*+(?:[^/*][^*]*\*+)*/)))+(?=\+)|(?<=-)((?:[\000-\011\013\014\016-'
r'\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/)))+(?=-)|(?:[\000-\011\013'
r'\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/))+|(?:(?:(?://[^'
r'\r\n]*)?[\r\n])(?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^'
r'/*][^*]*\*+)*/))*)+'
)
return result.match(pattern);
}

I'd recommend taking a look at Asset Rack, which already implements what you're building.

Loading one JS file before another

Ok so I have a .js file with about 10k lines of code. This code can be split up in
sub-object definitions
container object definitions
initialization code (after the objects have been defined)
program functionality
I would like to split this one file into 4 separate files, because it provides a better oversight. How do I go about doing this, given that they absolutely have to be declared in that order? What should I wrap up in a $(document).ready() and what not?
If I just separate the files and link them to the html in the correct order, I get undefined object errors. I was also thinking of something like this; but I don't know if that's any good...
Second JS File
function initializeContainers() {
var containerObj1 = {
bla: 'bla',
bla2: 'bla2'
},
var containerObj2 = {
bla: 'bla',
bla2: 'bla2'
};
};
First JS File
$(document).ready(function() {
function initializeSubObjects(callback) {
var subObj1 = {
somekey: 'somevalue',
someke2: 'someothervalue'
};
callback();
};
initializeSubObjects(initializeContainers);
});
I have no clue whether this is the correct way to do it?
PS: I also know you can add the script tags dynamically; but is that good practice?

In your example, you should swap the contents of your first and second file. You should only call the initializeContainers method when you know for sure the file has been loaded.
The easiest way to think about this is to load all files with definitions first (helpers, functions, classes, ...). Once all these are loaded, put the rest in the last file and start executing the code only in the last file
On a side note: If you deploy this into a production environment, you should consider bundling these files. Downloading 4 files will impact your load time, so it's better to just bundle them together and send them over as a single file. While you're at it, you probably also want to minify it.