We Keep Coding

Can't resolve css url from webpack asset librarymodule

I was able to output an assets library and many other libraries that work as remote federated modules and as deep import libraries in case I am not connecting to a remote or I am not using webpack in the consumer end.
The issue is now that all my assets exports a module, that either have the raw data as uri or the string that points to the right asset. Eg: the bird.svg is outputed to dust with it's hash plus the modules that resolves to the file bird-[hash].svg.
The above is great from javascript but not so much for css. Now I can't rewrite the url() to point to the right remote path which would be sg like:
//since I don't know when the assets will change names I can't refer to it directly. So I would need to first read the string from the bird.js module. Append the publicPath and then rewrite the url.
.someClass {
background-image: url('/assets/bird.js')
}
//the above won't work for obvious reasons.
Só, the question is how can I solve this? Or is there any loader for this? I checked the resolve url loader but it does not seem to be what need.

Ok I resolved this issue, by passing additional data as variable to sass-loader. That way I can evaluate the actual name of the files, and put it as a sass map before and handle it from sass.
//I am using glob to return an object with all the assets.
//This can probably be automated better. That would be an easier way.
//But this way works for me in all 3 scenarios, node, browser and federated module.
//Also has caching ootb for the assets.
const assetsPaths = {
...glob.sync('../assets/dist/img/**.node.js').reduce(function (obj, el) {
obj['img/' + path.parse(el).name] = '../../assets/dist/' + require(el).default;
return obj
}, {}), ...glob.sync('../assets/dist/fonts/**.node.js').reduce(function (obj, el) {
obj['fonts/' + path.parse(el).name] = '../../assets/dist/' + require(el).default;
return obj
}, {})
};
//...
{
loader: 'sass-loader',
options: {
additionalData: "$assets: '" + assetsMap + "';",
sourceMap: true,
sassOptions: {
outputStyle: "compressed",
},
}
},
//...
you also need to disable url rewriting
{
loader: 'css-loader',
options: {
url: false,
}
},
then you can use assets map in your sass files:
#font-face {
font-family: 'Some Font';
src: local('Some Font');
src: url("#{map-get($assets, SomeFont)}");
}
You will need probably have your project setup sort like a mono repo and you also need to build those assets library with two bundles.
One for node so you can use the string path to your actual assets when bundling you sass/whatever.
And another for normally loading it from the browser.
update:
Instead of doing all this I just used the manifest generated from 'webpack-manifest-plugin' to build the $assets map to be used in sass.
const assetsManifest = JSON.parse(fs.readFileSync('../assets/dist/manifest.json'));
const assetsMapFn = asset => `'${asset[0]}':'${asset[1]}'`;
const assetsMap = `(
${Object.entries(assetsManifest).map(assetsMapFn).join(',')}
); `;
If anyone knows a better way to do this please reply or comment.

eslint on dynamically changing file list

I'd like to run eslint on modified files only. I've created a new run target in package.json to run eslint from command line (git diff --name-only --relative | grep -E '.*\\.(vue|js)$' | xargs eslint --ext .js,.vue). In theory, this should work fine, but there's a little transformation step happening in my project (a string replacement) when bundling the files with webpack that will throw off eslint (some non-standard markup will be expanded to JS).
What are my options and how would I go about implementing them? For instance, could I execute a particular webpack rule/loader and pipe the result to eslint? Another option I see is to include eslint into the webpack rule/loader process (instead of executing it from the command line), but how would I then filter on files that are currently modified (could this be handled by a temporary file that contains the git diff... result?)

I've got a somewhat working approach. I chose to modify webpack.base.conf.js instead of going for the command line solution to make use of the already existing string replacement loader.
The files are collected in the WebpackBeforeBuildPlugin callback function and instead of a regex based test variable, a function is used which checks against the previously collected files.
const exec = require('child_process').exec;
const WebpackBeforeBuildPlugin = require('before-build-webpack');
var modFilesList = new Set([]);
const srcPath = resolve('.');
...
rules: [{
test: function(filename) {
let relFilename = path.relative(srcPath, filename);
let lint = modFilesList.has(relFilename);
return lint
},
loader: 'eslint-loader',
include: resolve('src'),
exclude: /node_modules/,
options: {
formatter: require('eslint-friendly-formatter'),
cache: false
}
}, {
... other string replacement loader ...
}
plugins: [
...
new WebpackBeforeBuildPlugin(function(stats, callback) {
// Collect changed files before building.
let gitCmd = 'git diff --name-only --relative | grep -E ".*\\.(vue|js)$"';
const proc = exec(gitCmd, (error, stdout, stderr) => {
if (stdout) {
let files = stdout.split('\n');
modFilesList = new Set(files);
}
if (error !== null) {
console.log(`exec error: ${error}`);
}
});
callback();
})
]
The only problem at the moment is that when git file changes occur, they don't trigger a re-linting based on these file changes (i.e. new file is changed, or previously (before starting webpack-dev-server) changed file changes are discarded). I checked everything I could. The change is registered and stored in modFilesList, the test function is executed and returns true (for a new change in a previously unchanged file) or false in case the change was discarded. I also played with the cache option to no avail. It seems that at initial load, eslint-loader caches the files it will lint in future (don't know if that's a result of using a test function instead of a regex or also the case with the regex). Is anyone having an idea or has seen this before (eslint-loader not updating the file list)?
Update
This seems to be a problem with webpack (or one of the other loaders) as the eslint-loader isn't even executed when the file changed. The test function however is executed which is a bit weird. I don't fully understand how loaders work or how they play together, so there might be some other loader that is causing this...

How to include manual import() in Webpack Bundle

I am quite new to Webpack, so bear with me if thats a stupid question.
My goal is to transform my old, AMD based codebase to a ES6 Module based solution. What I am struggling with is handling dynamic import()s. So my app router works on a module basis, i.e. each route is mapped to a module path and then required. Since I know what modules will be included, I just add those dynamically imported modules to my r.js configuration and am able to build everything in a single file, with all require calls still working.
Now, I am trying to do the same with ES6 modules and Webpack. With my devmode this is no problem as I can just replace require() with import(). However I cannot get this to work with bundling. Either Webpack splits my code (and still fails to load the dynamic module anyways), or - if I use the Array format for the entry config, the dynamic module is included in the bundle but loading still fails: Error: Cannot find module '/src/app/DynClass.js'
This is how my Webpack config looks like:
const webpack = require('webpack');
const path = require('path');
module.exports = {
mode: "development",
entry: ['./main.js', './app/DynClass.js'],
output: {
filename: 'main.js',
path: path.resolve(__dirname, "../client/")
},
resolve: {
alias: {
"/src": path.resolve(__dirname, '')
}
},
module: {
rules: [
{
test: /\.tpl$/i,
use: 'raw-loader',
},
]
}
};
So basically I want to tell Webpack: "hey, there is another module (or more) that is to be loaded dynamically and I want it to be included in the bundle"
How can I do this?

So yeah, after much fiddling there seems to be a light at the end of the tunnel. Still, this is not a 100% solution and it is surely not for the faint of heart, as it is quite ugly and fragile. But still I want to share my approach with you:
1) manual parsing of my routes config
My router uses a config file looking like this:
import StaticClass from "/src/app/StaticClass.js";
export default {
StaticClass: {
match: /^\//,
module: StaticClass
},
DynClass: {
match: /^\//,
module: "/src/app/DynClass.js"
}
};
So as you can see the export is an object, with keys acting as the route id, and an object that contains the matches (regex based) and the module which should be executed by the router if the route matches. I can feed my router with both a Constructor function (or an object) for modules which are available immediatly (i.e. contained in the main chunk) or if the module value is a string, this means that the router has to load this module dynamically by using the path specified in the string.
So as I know what modules could be potentially loaded (but not if and when) I can now parse this file within my build process and transform the route config to something webpack can understand:
const path = require("path");
const fs = require("fs");
let routesSource = fs.readFileSync(path.resolve(__dirname, "app/routes.js"), "utf8");
routesSource = routesSource.substr(routesSource.indexOf("export default"));
routesSource = routesSource.replace(/module:\s*((?!".*").)*$/gm, "module: undefined,");
routesSource = routesSource.replace(/\r?\n|\r/g, "").replace("export default", "var routes = ");
eval(routesSource);
let dummySource = Object.entries(routes).reduce((acc, [routeName, routeConfig]) => {
if (typeof routeConfig.module === "string") {
return acc + `import(/* webpackChunkName: "${routeName}" */"${routeConfig.module}");`;
}
return acc;
}, "") + "export default ''";
(Yeah I know this is quite ugly and also a bit brittle so this surely could be done better)
Essentially I create a new, virtual module where every route entry which demands a dynamic import is translated, so:
DynClass: {
match: /^\//,
module: "/src/app/DynClass.js"
}
becomes:
import(/* webpackChunkName: "DynClass" */"/src/app/DynClass.js");
So the route id simply becomes the name of the chunk!
2) including the virtual module in the build
For this I use the virtual-module-webpack-plugin:
plugins: [
new VirtualModulePlugin({
moduleName: "./app/dummy.js",
contents: dummySource
})
],
Where dummySource is just a string containing the sourcecode of my virtual module I just have generated. Now, this module is pulled in and the "virtual imports" can be processed by webpack. But wait, I still need to import the dummy module, but I do not have any in my development mode (where I use everything natively, so no loaders).
So in my main code I do the following:
let isDev = false;
/** #remove */
isDev = true;
/** #endremove */
if (isDev) { import('./app/dummy.js'); }
Where "dummy.js" is just an empty stub module while I am in development mode. The parts between that special comments are removed while building (using the webpack-loader-clean-pragma loader), so while webpack "sees" the import for dummy.js, this code will not be executed in the build itself since then isDev evaluates to false. And since we already defined a virtual module with the same path, the virtual module is included while building just like I want, and of course all dependencies are resolved as well.
3) Handling the actual loading
For development, this is quite easy:
import routes from './app/routes.js';
Object.entries(routes).forEach(async ([routeId, route]) => {
if (typeof route.module === "function") {
new route.module;
} else {
const result = await import(route.module);
new result.default;
}
});
(Note that this is not the actual router code, just enough to help me with my PoC)
Well, but for the build I need something else, so I added some code specific to the build environment:
/** #remove */
const result = await import(route.module);
new result.default;
/** #endremove */
if (!isDev) {
if (typeof route.module === "string") { await __webpack_require__.e(routeId); }
const result = __webpack_require__(route.module.replace("/src", "."));
new result.default;
}
Now, the loading code for the dev environment is just stripped out, and there is another loading code that uses webpack internally. I also check if the module value is a function or string, and if it is the latter I invoke the internal require.ensure function to load the correct chunk: await __webpack_require__.e(routeId);. Remember that I named my chunks when generating the virtual module? Now thats why I still can find them now!
4) more needs to be done
Another thing I encountered is when several dynamically loaded modules have the same dependencies, webpack tries to generate more chunks with names like module1~module2.bundle.js, breaking my build. To counter this, I needed to make sure that all those shared modules go into a specific named bundle I called "shared":
optimization: {
splitChunks: {
chunks: "all",
name: "shared"
}
}
And when in production mode, I simply load this chunk manually before any dynamic modules depending on it are requested:
if (!isDev) {
await __webpack_require__.e("shared");
}
Again, this code only runs in production mode!
Finally, I have to prevent webpack renaming my modules (and chunks) to something like "1", "2" etc, but rather keep the names I just have defined:
optimization: {
namedChunks: true,
namedModules: true
}
Se yeah, there you have it! As I said this wasn't pretty but seems to work, at least with my simplified test setup. I really hope there aren't any blockers ahead of me when I do all the rest (like ESLint, SCSS etc)!

Grunt build script that reads xml file to determine which files are copied

copy: {
build: {
cwd: 'app',
src: ['**', '!**/vendors/**', '!**src/js/*.js',],
dest: 'dist',
expand: true
}
}
I am using grunt build scripts to build a distribution folder for the completed product. However, its not 100% automatic and dynamic. For example, I have a folder of xml content files. Yet, I don't use them all. Right now, the whole folder is copied over to the build version. Manually I have to go in and delete the xml files I don't want in the build version then run it. Or I could go into the grunt file and and tell it to ignore those files.
The problem is that I don't want to do that every time. A theoretical idea I had would be to have an xml file where I define elements to represent certain other files.
<bootstrap>true</bootstrap>
<extraContent>false</extraContent>
This would say that the file correlated to bootstrap and extraContent should or shouldn't be ignored in the build. I am trying to figure out if you could do this in grunt.
something like the following is how I see the logic playing out...
var bootstrap = $(xml).find("bootstrap").text()
if(bootstrap == "false"){
var url = src/bootstrap.css
//Here add the correlated filepath defined above to be ignored
}
The problem is not only writing this so grunt knows what it is, but also combining that logic with the actual "copy:{}" script I showed above

If you want to include/exclude files based on their contents you can use filter function for this. Examples can be found in the official documentation: https://gruntjs.com/configuring-tasks#custom-filter-function.
The filter property can help you target files with a greater level of detail.
In your case this could be something like this:
copy: {
build: {
cwd: 'app',
src: ['**', '!**/vendors/**', '!**src/js/*.js',],
dest: 'dist',
expand: true,
// this filter function will copy xml files only when `bootstrap` is set to 'true'
filter: filepath => {
if (require('path').extname(filepath) !== 'xml')
return true;
const xml = require('fs').readFileSync(filepath, 'utf8');
const json = require('xml2json').toJson(xml);
return json.bootstrap === 'true';
}
}
}
You can then use the process function to copy only certain contents from specific files: https://github.com/gruntjs/grunt-contrib-copy#process
This option is passed to grunt.file.copy as an advanced way to control the file contents that are copied.

Passing less variables from webpack

I'm trying to pass less variables within a webpack configuration to the less loader, naturally.
For some reason the variable is not being passed ok. I can't figure the correct syntax.
The variable has a dynamic content that is determined at the build time, in the webpack config file. This is the relevant line (I've tried many variations of it):
loader: 'style!css?-minimize!less?-minimize&{modifyVars:{"resources-path-prefix":"' + pathPrefix + '"}}'
In the above example some pathPrefix is being determined at build time and we want to pass its value into less context, where it will be used in url() css directives.
The above doesn't work - nothing is passed into less, and the default variable value defined in less applies.
Can anyone show how to correctly pass the value into the less compilation process? Thanks!

So it was tough but we finally made it work(!). Arggh - so much time invested to trying and figure out the syntax.
Here's the task: we want at build time to determine a path that should use used as the base url for misc assets in less files (background images, using url() less function).
First, we determined the path in webpack config file. Its plain JS, but the escaping pattern for the path string was absolutely nuts. We probably invested hours just on this. Amazing. Here it is:
var assetsPath = (someCondition) ? '/assets' : "\\/127.0.0.1:8080/assets";
Next is the loader configuration for less files, using the assetsPath prefix set above:
{
test: /\.less$/,
exclude: /node_modules/,
loader: 'style!css?minimize=false!less?{"modifyVars":{"assetspath":"\'' + assetsPath +'\'"}}'
}
Notice the escaping pattern above where using the assetsPath in the loader configuration.
Next, you need to make sure that an empty variable is being defined in the less files. We initialized it in our 'vars.less' file, with:
#assetspath: '';
Finally, in any relevant class, we can use the value being passed in build time like this:
background-image: url("#{assetspath}/images/facebook.png");

You can try to use the query section of the loader:
loader: 'style!css?-minimize!less?-minimize',
query: {
modifyVars: {
"resources-path-prefix": pathPrefix
}
}

It's a different approach to this situation, but we managed to make things work by converting into Base64 every resource that the CSS files loaded. We had to do this because figuring out the hostname of the resources was possible much later down the line than the webpack config file.