We Keep Coding

Webpack4 thread-loader bring little optimization for babel-loader. Is something wrong, or why?

As mentioned in Webpack docs and many information that introducing how to improve webpack build speed, thread-loader can obviously improve the build speed of babel-loader.
But when I try it in my project, it seems like that thread-loader does not bring any optimization for babel-loader.
I use speed-measure-webpack-plugin to measure build speed, and there are the code and result of my test.
my device:
Macbook Pro (15-inch, 2018, 6-core, 12-thread)
version information:
{
"webpack": "^4.41.2",
"thread-loader": "^3.0.1",
"babel-loader": "^8.2.2",
"webpack-preprocessor-loader": "^1.1.3",
}
Before using thread-loader for babel-loader, code of webpack.config.js:
// thread-loader rule:
{
test: /\.(jsx?)$/,
exclude: [
/node_modules/,
],
use: [
{
loader: 'babel-loader',
options: {
babelrc: true,
cacheDirectory: true,
rootMode: 'upward',
}
}, {
loader: 'webpack-preprocessor-loader',
options: {
params: env
}
}]
}
Time cost:
SMP ⏱ Loaders
thread-loader, and
babel-loader, and
webpack-preprocessor-loader took 15.71 secs
module count = 315
After:
// thread-loader warmup:
const threadLoader = require('thread-loader');
const threadLoaderOptions = {
poolTimeout: 2000,
name: 'thread-loader-for-babel'
}
threadLoader.warmup(threadLoaderOptions, [
'webpack-preprocessor-loader',
'babel-loader'
]);
// thread-loader rule:
{
test: /\.(jsx?)$/,
exclude: [
/node_modules/,
],
use: [
{
loader: 'thread-loader',
options: threadLoaderOptions
},
{
loader: 'babel-loader',
options: {
babelrc: true,
cacheDirectory: true,
rootMode: 'upward',
}
}, {
loader: 'webpack-preprocessor-loader',
options: {
params: env
}
}]
}
Time cost:
SMP ⏱ Loaders
thread-loader, and
babel-loader, and
webpack-preprocessor-loader took 14.64 secs
module count = 313
15.71s VS 14.64s, this effect is not obvious.
thread-loader change babel-loader work from serial execution to parallel execution, theoretically in my device with 6 cores and 12 threads, even if you factor in the worker startup time and other factors, there will be at least several times improvement, but it doesn't.
Is the thread-loader used correctly? (It should be correct, I have tried to modify the source code of thread-loader and log to ensure that multiple workers are running)
If it is used correctly, what is the reason for the effect not obvious?
I have a guess. For modules that hit the same loader, webpack 4 has built-in multi-thread processing, that is, webpack has already done thread-loader work. So thread-loader has little effect. But this guess needs to go deep into the webpack 4 source code to verify.
(I will continue to study, but it will take some time. If there are anyone that have been studied, please share it)
thanks in advance. :)

Possible reasons as to why it's not as fast, is these reasons (found on the documentation):
Loaders running in a worker pool are limited. Examples:
Loaders cannot emit files.
Loaders cannot use custom loader API (i. e. by plugins).
Loaders cannot access the webpack options.

Webpack console.log output?

Can anyone direct me in the right direction?
So i've setup the webpack-dev-server with the truffle suite demo, just to get a basis on the foundation of my app. So my config file includes index.html & app.js, yet it try to display a console.log output to from app.js nothing shows via the console?
webpack.config.js
const path = require('path');
const CopyWebpackPlugin = require('copy-webpack-plugin');
module.exports =
{
entry: './app/javascripts/app.js',
output: {
path: path.resolve(__dirname, 'build'),
filename: 'app.js',
},
plugins: [
// Copy our app's index.html to the build folder.
new CopyWebpackPlugin([
{ from: './app/index.html', to: "index.html" }
])
],
module: {
rules: [
{
test: /\.css$/,
use: [ 'style-loader', 'css-loader' ]
}
],
loaders: [
{ test: /\.json$/, use: 'json-loader' },
{
test: /\.js$/,
exclude: /(node_modules|bower_components)/,
loader: 'babel-loader',
query: {
presets: ['es2015'],
plugins: ['transform-runtime']
}
}
]
},
devServer: {
compress: true,
disableHostCheck: true, // That solved .
quiet: false,
noInfo: false,
stats: {
// Config for minimal console.log mess.
colors: true,
version: false,
hash: false,
timings: false,
chunks: false,
chunkModules: false
}
}
}
app.js
// Import libraries we need.
import { default as Web3} from 'web3';
import { default as contract } from 'truffle-contract'
// Import our contract artifacts and turn them into usable abstractions.
import metacoin_artifacts from '../../build/contracts/MetaCoin.json'
import dextre_artifacts from '../../build/contracts/Dextre.json'
console.log("starting!");
Output when running webpack
Project is running at http://localhost:8080/
webpack output is served from /
Asset Size Chunks Chunk Names
app.js 1.93 MB 0 [emitted] [big] main
index.html 19.8 kB [emitted]
webpack: Compiled successfully.
Where can view the "starting!" output, this is a real annoyance as i need to tackle errors. I've tried viewing at http://localhost:8080// and http://localhost:8080/webpack-dev-server//, but no luck.

I had this same problem. As far as I can tell, the problem is that Webpack does not actually run the generated code on the server. The process that runs on the server simply checks for file changes and re-compiles when necessary. The code is actually all running on the client. So, the only way to view output from console.log() is to view the client browser's console.
Part of the confusion here is that while normally, node runs javascript on the server, in this case, node is delegating to a separate program entirely. You can run this whole thing without node installed at all, just using a standalone installation of the webpack executable. Node's execution environment is never used, thus you can't log to it.

Since you are using webpack-dev-server the console.log is gonna be printed into the browser console. If you want to see console.log printed in your terminal you can add a console.log inside webpack.config file.
If you are not providing the webpack-dev-server a port your app should run on 80 so opening the browser there and opening the browser console and you should be able to see the "starting! log.

How to disable AMD on 4 files and load them in order with webpack

I need to disable AMD on 4 files and load video.js first before loading the other 3 files, because they depend on it. When I tried doing it in webpack.config.js like so:
const path = require('path')
const webpack = require('webpack')
module.exports = {
entry: './src/main.js',
output: {
path: __dirname + '/public',
filename: 'bundle.js'
},
devServer: {
inline: true,
contentBase: './src',
port: 3333
},
plugins: [
new webpack.DefinePlugin({
'process.env': {
'NODE_ENV': JSON.stringify('production')
}
})
],
module: {
loaders: [
{
test: /\.js$/,
exclude: /node_modules|lib/,
loader: 'babel',
query: {
presets: ['es2015', 'react', 'stage-2'],
plugins: ['transform-class-properties']
}
},
{
test: /\.json$/,
loader: 'json-loader'
},
{
test: /[\/\\]lib[\/\\](video|playlist|vpaid|overlay)\.js$/,
exclude: /node_modules|src/
loader: 'imports?define=>false'
}
]
}
}
It doesn't really work, because it just loads video.js (with disabled AMD) and completely ignores the other 3 files.
My folder structure is like so:
▾ lib/
overlay.js
playlist.js
video.js
vpaid.js
▸ node_modules/
▾ public/
200.html
bundle.js
▾ src/
App.js
index.html
main.js
LICENSE
package.json
README.md
webpack.config.js
Now, I found something that takes me 1 step back, because now even video.js doesn't load:
require('imports?define=>false!../lib/video.js')
require('imports?define=>false!../lib/playlist.js')
require('imports?define=>false!../lib/vpaid.js')
require('imports?define=>false!../lib/overlay.js')
and instead just throws these kinds of warnings:
WARNING in ./~/imports-loader?define=>false!./lib/video.js
Critical dependencies:
15:415-422 This seems to be a pre-built javascript file. Though this is possible, it's not recommended. Try to require the original source to get better results.
# ./~/imports-loader?define=>false!./lib/video.js 15:415-422
WARNING in ./~/imports-loader?define=>false!./lib/playlist.js
Critical dependencies:
10:417-424 This seems to be a pre-built javascript file. Though this is possible, it's not recommended. Try to require the original source to get better results.
# ./~/imports-loader?define=>false!./lib/playlist.js 10:417-424
WARNING in ./~/imports-loader?define=>false!./lib/vpaid.js
Critical dependencies:
4:113-120 This seems to be a pre-built javascript file. Though this is possible, it's not recommended. Try to require the original source to get better results.
# ./~/imports-loader?define=>false!./lib/vpaid.js 4:113-120
WARNING in ./~/imports-loader?define=>false!./lib/overlay.js
Critical dependencies:
10:416-423 This seems to be a pre-built javascript file. Though this is possible, it's not recommended. Try to require the original source to get better results.
# ./~/imports-loader?define=>false!./lib/overlay.js 10:416-423
So, my question is, how can I make this work in webpack.config.js so that I don't get these warnings?

I have solved the problem! To make this work you need this:
{
test: /[\/\\]lib[\/\\](video|playlist|vpaid|overlay)\.js$/,
exclude: /node_modules|src/
loader: 'imports?define=>false'
}
and this
require('script-loader!../lib/video.js')
require('script-loader!../lib/playlist.js')
require('script-loader!../lib/vpaid.js')
require('script-loader!../lib/overlay.js')
together!
Now, if you use this (instead of script-loader):
require('imports?define=>false!../lib/video.js')
require('imports?define=>false!../lib/playlist.js')
require('imports?define=>false!../lib/vpaid.js')
require('imports?define=>false!../lib/overlay.js')
It's not gonna work! (you need both imports-loader and script-loader working in unison.

Sourcemap + istanbul/isparta code coverage for a webpack + babel (for es6) + mocha (+karma)

I have a test project at https://github.com/danielbush/webpack-babel-karma-sourcemap-coverage.
It consists of two simple source files in src/, one with a deliberate error (src/lib2.js).
I want to webpack and transpile using babel from ES6 to ES5 and run the Mocha tests on this bundle using Karma.
I've already added source mapping and tested this so that I can see the line number of the original file.
For me, the stack trace looks like this:
21 02 2016 16:03:15.445:INFO [karma]: Karma v0.13.21 server started at http://localhost:9876/
21 02 2016 16:03:15.451:INFO [launcher]: Starting browser Chrome
21 02 2016 16:03:16.551:INFO [Chrome 48.0.2564 (Linux 0.0.0)]: Connected on socket /#yRS32ons0_2HGhrwAAAA with id 3072946
START:
lib1
✖ should return 1
Finished in 0.015 secs / 0.001 secs
SUMMARY:
✔ 0 tests completed
✖ 1 test failed
FAILED TESTS:
lib1
✖ should return 1
Chrome 48.0.2564 (Linux 0.0.0)
Error: SIMULATED ERROR
at foo (/home/danb/projects/so-test-project/tests.webpack.js:135:10 <- webpack:///src/lib2.js:13:8)
at Context.<anonymous> (/home/danb/projects/so-test-project/tests.webpack.js:93:31 <- webpack:///test/index.js:7:11)
On the 2nd last line above, it shows webpack:///src/lib2.js:13 which is the correct location in the original source.
To get that mapping, I did this: https://github.com/danielbush/webpack-babel-karma-sourcemap-coverage/commit/6ea992cae499dccc68488bcb3eaca642ae4ed24e
What I want to do is add coverage using something like istanbul (or isparta which uses istanbul) so that I can generate an HTML coverage report in coverage/ that shows me lines that aren't covered (potentially) -- as well as overall coverage percentages.
But I want to ensure that my stack traces are still source mapped correctly like the above.
I've tried isparta-loader and isparta-instrumenter-loader to transpile and instrument the files in question in webpack, but my line numbers are slightly wrong in the stacktrace (above). There seems to be a number of isparta-like things floating around and the documentation is not super-friendly.
To express all of this a bit more generally: I'm bundling files and tests and then testing this in the browser (karma or by more manual means), and I want my source maps (in the browser) to point to the original lines of code and at the same time I want a coverage report to refer to the original lines and files that haven't been covered.

I don't have a solution for a single karma/webpack configuration. My workaround is to have 2 karma confs - which I'm quite happy with atm. (I have other non-karma setups for running tests as well.)
One conf runs the tests in the background with accurate stack traces for my es6 code, and the other runs coverage using isparta-loader to transpile and instrument my source code. (isparta-loader uses isparta, which uses istanbul).
So my "running tests" karma conf looks a bit like this:
var webpack = require('webpack');
module.exports = function (config) {
config.set({
browsers: [ 'PhantomJS' ],
singleRun: false,
frameworks: [ 'mocha' ], // Use the mocha test framework
files: [
'tests.webpack.js'
],
preprocessors: {
'tests.webpack.js': [ 'webpack', 'sourcemap' ]
},
reporters: [ 'mocha' ],
webpack: {
devtool: 'inline-source-map',
module: {
loaders: [
// Use imports loader to hack webpacking sinon.
// https://github.com/webpack/webpack/issues/177
{
test: /sinon\.js/,
loader: "imports?define=>false,require=>false"
},
{
test: /\.js$/,
exclude: /node_modules/,
loader: 'babel-loader',
query: { presets: ['es2015'] }
}
]
}
},
webpackServer: {
noInfo: true
}
});
}
and my coverage karma conf looks a bit like this:
var webpack = require('webpack'),
path = require('path');
module.exports = function (config) {
config.set({
browsers: [ 'PhantomJS' ],
singleRun: true,
frameworks: [ 'mocha' ],
files: [
'tests.webpack.js'
],
preprocessors: {
'tests.webpack.js': [ 'webpack' ]
},
reporters: [ 'coverage' ],
webpack: {
// *optional* babel options: isparta will use it as well as babel-loader
babel: {
presets: ['es2015']
},
// *optional* isparta options: istanbul behind isparta will use it
isparta: {
embedSource: true,
noAutoWrap: true,
// these babel options will be passed only to isparta and not to babel-loader
babel: {
presets: ['es2015']
}
},
module: {
loaders: [
{
test: /sinon\.js/,
loader: "imports?define=>false,require=>false"
},
// Perform babel transpiling on all non-source, test files.
{
test: /\.js$/,
exclude: [
path.resolve('node_modules/'),
path.resolve('lib/')
],
loader: 'babel-loader',
query: { presets: ['es2015'] }
},
// Instrument source files with isparta-loader (will perform babel transpiling).
{
test: /\.js$/,
include: [
path.resolve('lib/')
],
loader: 'isparta'
}
]
}
},
webpackServer: {
noInfo: true
},
coverageReporter: {
reporters: [
{ type: 'html', dir: 'coverage/' },
{ type: 'text' }
]
}
});
}
This 2nd conf provides a text report that gives immediate coverage summary in the terminal, and the html report which gives nice source file highlighting of untested code. (There's a hack in both for sinon, which isn't related to this issue.) tests.webpack.js uses webpack's require.context to pull in my browser tests written in mocha. And various karma-* plugins are required.

You may have better luck using Istanbul's new official Babel Plugin to instrument your sources for code coverage.
In my experience, compared to isparta, it provided more accurate coverage reports and required basically no configuration apart from adding it to the list of plugins in my Babel configuration.
Sourcemaps appeared to work correctly "out of the box" when I used it on my project.

Webpack production build does not load anything

I've been working on an app using React and Webpack for a little while. My development environment works fine, and everything loads properly using webpack-dev-server.
I decided to run a production build of the application to see what the end-product might look like size-wise and observe the general output of the webpack product build.
It turns out that running webpack -p, while it does produce output (more on that in a minute), does not load anything at all when I hit the site in a browser.. A quick check of the output tells me that none of my component code is making it into the webpack -p build.
The images and HTML copy over as they exist in my src (dev) folder, however my JS bundle output is extremely small - the file (main.js) is only 246 bytes.
Here is the output from running webpack -p
$ npm run build
> project#0.1.0 build /Users/me/Development/project
> NODE_ENV=production webpack -p --bail --progress --config webpack.config.babel.js
Hash: 9e5f6974ce21c920a375
Version: webpack 1.12.10
Time: 2003ms
Asset Size Chunks Chunk Names
index.html 1.45 kB [emitted]
images/edit.svg 524 bytes [emitted]
images/search.svg 1.19 kB [emitted]
main.js 246 bytes 0, 1 [emitted] javascript, html
+ 219 hidden modules
When I run the development version of the project, the output is markedly different... I know that the dev server dependencies are in there, and the code is not minified.. And, most importantly - everything works as expected when running the dev server.
$ npm start
> project#0.1.0 start /Users/me/Development/project
> webpack-dev-server --hot --display-modules --config webpack.config.babel.js
http://localhost:3333/
webpack result is served from /
content is served from /Users/me/Development/project/dist
Hash: 1b34ed58f9e323966ada
Version: webpack 1.12.10
Time: 2745ms
Asset Size Chunks Chunk Names
index.html 1.45 kB [emitted]
images/edit.svg 524 bytes [emitted]
images/search.svg 1.19 kB [emitted]
main.js 1.54 MB 0, 1 [emitted] html, javascript
Here's my webpack.config.babel.js file:
import webpack from 'webpack';
import path from 'path';
import ModernizrWebpackPlugin from 'modernizr-webpack-plugin';
import modernizrConfig from './modernizr.config';
const appDir = path.resolve(__dirname, './src');
const distDir = path.resolve(__dirname, './dist');
const nodeModulesDir = path.resolve(__dirname, './node_modules');
const excludeDirs = /(node_modules|bower_components)/;
module.exports = {
entry: {
javascript: appDir + '/main.js',
html: appDir + '/index.html'
},
output: {
path: distDir,
filename: 'main.js',
},
devServer: {
contentBase: distDir,
inline: true,
port: 3333
},
resolve: {
extensions: ['', '.js', '.es6'],
modulesDirectories: [
'node_modules',
'./src'
]
},
plugins: [
// new webpack.optimize.CommonsChunkPlugin('common.js'),
// new ModernizrWebpackPlugin(modernizrConfig),
],
sassLoader: {
sourceMap: false,
includePaths: [
appDir,
nodeModulesDir,
nodeModulesDir + '/breakpoint-sass/stylesheets/',
nodeModulesDir + '/susy/sass'
]
},
module: {
loaders: [
{ // js/jsx
test: /\.js?$/,
exclude: excludeDirs,
loader: 'babel',
query: {
cacheDirectory: true,
presets: [
'es2015', 'react'
]
}
},
{ // html
test: /\.html$/,
exclude: excludeDirs,
loader: 'file?name=[name].[ext]'
},
{ // images
test: /\.(gif|png|jpg|jpeg|svg)$/,
exclude: excludeDirs,
loader: 'file?name=images/[name].[ext]'
},
{ // sass
test: /\.scss$/,
exclude: excludeDirs,
loader: 'style!css!sass'
}
]
}
}
I don't think I've got a particularly complex, or uncommon setup, and I've tried changing everything from es2015/es6 to commonJS already as well, with the same result.
I'm at a loss as to what the issue could possibly be here; hoping that someone can point out some obvious error I've got, or perhaps suggest config updates/changes that could resolve this problem.
Thanks for taking the time to read everything!

As I mentioned in my comment, I've been using webpack successfully for months now, and I've never come across using an HTML file as an entry point.
Usually you'll use Webpack to package up javascript, css, and sometimes images (ie: "assets") to be used by an html file. Serving that HTML file is outside the realm of Webpack's responsibility.
I would suggest using Webpack to generate only the final javascript bundle, then using some other method (ie: express, or some other web server) to serve that file and the html that consumes it.