We Keep Coding

Preprocessing an S3 asset with CDK

Is there an idiomatic way to preprocess an asset before it is uploaded to S3 with #aws-cdk/aws-s3-assets?
What I am currently doing is loading my local file (for example, a text file), modifying it, then writing it into a directory that I have in my .gitignore. I then pass the new file to new Asset(...)
The main reason I would like an alternative approach is that I have to come up with a convention for creating the temporary file names that are eventually passed into the constructor of new Asset(...), since it is possible that my construct is used in multiple places and I don't want there to be contention over the file.
I did see that there is a bundling property on the AssetOptions, but it appears to be geared towards using Docker to bundle the asset. This seems like overkill for my use case, since the preprocessing that I am talking about is basically a string replace.
Thanks in advance!

What exactly does webpack:// mean?

For years I have been building apps with Vue-CLI which bundles Webpack. I see this prefix (protocol?) webpack:// but I don't actually know what it means.
For instance, in the generated source map for an app.xxx.js.map, I see:
{"version":3,"sources":["webpack:///webpack/bootstrap","webpack:///./src/App.vue?797c",...
So what does that webpack:// mean, exactly? It's obviously not a real protocol that the browser can resolve. So resolves it? And how?
The particular problem I'm having that leads me to want to understand this is that my generated chunk-vendors.8c348425.js.map begins like this:
{"version":3,"sources":["webpack:///js/chunk-vendors.68249437.js"],...
In this case, there is no generated chunk-vendors.68249437.js file anywhere. Should there be? Or is that an abstract generated name that doesn't actually refer to a real file?
I'm seeing errors in FireFox that I'm having a bit of trouble interpreting:
I think it's saying that it's (somehow) running that non-existent chunk-vendors.68249437.js file, and that is failing to fetch some (unidentified) resource?

tl;dr - webpack:// is an arbitrary protocol name to play nicely with browsers' source map viewers. Nesting each file name under this prefix displays them in a separate dropdown (webpack-demo in this case):
Webpack (most JS bundlers, really) take some number of uncompressed source files and combine them into a single* compressed** bundle file that can be loaded by a browser in a single network request.
At a high level, source maps exist to allow developers to easily translate console messages and stack traces from their file line/column number in the bundle back to their original position in a source file. This may be something as simple as "line 1001 in the bundle was actually line 500 in foo.js", or something more tricky like "lines 1001-1021 were part of the compilation output from line 20 in MyComponent.vue". You can read the latest spec for source maps, but in essence they contain three things:
Name of the output bundle file
Name of the input file(s) used to generate the bundle
Encoded mapping data connecting each line in (1) back to a file in (2)
The simplest source map looks something like this (omitted all but the relevant parts):
{
"file": "bundle.js",
"sourceRoot": "/",
"sources": ["foo.js", "bar.js"],
"mappings": "..."
}
This mapping is generated when two source files, foo.js and bar.js, are combined into a single bundle file, bundle.js. When a browser renders this source map, it expects to find the original source files at /foo.js and /bar.js, respectively.
Instead of hosting the files individually, a source map may instead contain the files' contents itself:
{
"file": "bundle.js",
"sources": ["foo.js", "bar.js"],
"sourcesContent": ["/* foo.js contents */", "/* bar.js contents */"],
"mappings": "..."
}
When rendering this source map, the browser just reads the file contents directly, instead of making a request to another URL. File names in sources are expected to be in the same order as sourcesContent.
Alright, we're almost there.
So long as the files are inlined in the source map (sourcesContent is set), the file names are arbitrary. In other words, foo.js doesn't have to correspond to a real file named foo.js, you could just as easily call it foo/bar/baz/garply.xyz and any source map viewer would have no issue rendering it (remember, the file contents are stored by array index).
Most source map viewers (I've checked Chrome and Firefox specifically) will display sources in a separate dropdown based on their protocol prefix. The screenshot above shows a number of files nested under webpack-demo; all of their sources entries begin with webpack://webpack-demo/.... It doesn't matter what the protocol is; foobar:// works just as well, with the caveat that each unique protocol prefix will get its own dropdown in the source map filetree. In fact, you can customize this prefix by setting devtoolModuleFilenameTemplate (see here).
So, in short - webpack:// is an arbitrary, unique prefix used to nest source files in their own dropdown in the source map filetree. The filenames that follow webpack:// are usually 1:1 with actual files in the filesystem, but that's not always the case***.
* OK, code splitting is a thing, but single bundle file in the simple case.
** A bundle file doesn't have to be compressed, it can simply be every source file concatenated one after the other, but in practice bundles are usually run through one or more minficiation/uglification passes.
*** Webpack may perform one or more intermediate compression passes, combining sets of vendor files from node_modules into chunks. I'm not 100% on the specifics here, but one way to avoid this is to use a different devtool (see here).

wuic javascript optimization version number calculation

Using http://wuic.github.io/, i noticed that the version number of the js files is not the same in different envrionments (different developpers envrionments for example).
Is there a way that this version number does not depend on the environment ?
It's important because in our clustered environment, js resources can be served by any cluster nodes, so if a html is served by one node and the version number of the resource number in it is node dependant, it's problematic ...

By default WUIC computes the version number from a file thanks to its last modification date on the current file system. If your build process touch the file, the date will change even if the content is not modified.
To avoid this problem, you can add c.g.wuic.dao.contentBasedVersionNumber=true to your wuic.properties files. WUIC will computes a checksum by reading the content, and the version number will be the same for any build.

nodejs: Identify file clone

how to identify if two files are same in javascript(nodejs), one is just a renamed copy of other?
Use case: I am trying to write a script for syncing a HDD (hdd1) and its clone (hdd2). 95% only video files (size: ~1 GB, count: ~4000). Sometimes I rename the files in hdd1 and move them different folders. So while syncing, instead of delete and fresh copy from hdd1 to hdd2, I just want to rename and move the files( identified ones) in hdd2 to match its location in hdd1.

Like mentioned by mscdex, there's probably already a tool out there that does what you're looking for (like rsync).
If you're more interested in doing it from scratch as a learning experience, then what you're looking for is called a checksum or hash of a file. Generating a checksum for each file gives you a sort of finger print for a file. You can then use this to compare against the checksum or other files, and if they're the same, the checksums will match as well.
Node.js's Crypto library gives you methods for generating checksums. This blog entry walks through some of this.

Forcing the browser to reload css/js only if they have changed

There are a lot of questions and answers on SO related to my problem [I want the browser to cache js/css forever. During a new release if some of the js/css files have been updated, the browser should reload and cache them.]
This solution seemed most appropriate to me :
What is an elegant way to force browsers to reload cached CSS/JS files?
However, there is just one thing that I am unable to figure out.
The solution makes use of last_modified_time. However, I am not allowed to use it. I need to use some other mechanism.
What are the options? Is there a possibility of pre-calculating the versions during build and updating(replacing) them in jsps via build script (before deployment, so that the version numbers are not calculated on run time)? Any existing tool for this purpose? I use Java/Jsp.

We always use
file.css?[deploytimestamp]
This way the CSS file is cached for each deployment at the client. The same goes for our minified javascript. Is this an option for you?

It may not be the best way, but this is what I am doing now:
All of my js/css have a [source control = svn] revision number
References in my jsp are like /foo/path1/path2/xyz000000/foo.
Build Step 1 - Generate a map of css|js files and their revision numbers
Build Step 2 - Replace xyz000000 references in jsps with a hash of svn revisions
A rule in url rewriter to direct all /foo/path1/path2/xyz<767678>/foo. to /foo/path1/path2/foo.[js|css]
Infinitely cache the css|js files
Whenever there is a commit, the revision number changes and so do the references in .jsp

Generate an md5-hash of each css file after deployment. Use this hash instead of the timestamp in the url of the css.
file.css?[hash of file.css contents]
It may be wise to calculate the hashes once after deployment and store them to gain some performance. You could store them in a database, or even in a PHP array in a separate file that is included in your website code.