We Keep Coding

How to integrate es6 with gulp-develop-server

I am trying to transfer an old node-express project over to be able to use es6. I have seen many posts about using gulp with es6. Most of them discuss using a syntax like this:
const gulp = require("gulp");
const babel = require("gulp-babel");
gulp.src('./index.js')
.pipe(
babel({
presets: [
["#babel/env", { modules: false }],
],
})
)
However my existing project's gulpfile does't use gulp.src at all. Instead, it uses gulp-develop-server. The gulpfile looks like this:
const gulp = require("gulp");
const devServer = require("gulp-develop-server");
const spawn = require("child_process").spawn;
const fs = require("fs");
const basedir = ".";
function serverRestart(done) {
// perform some cleanup code here
devServer.restart();
done();
}
function serverStart() {
devServer.listen({
path: basedir + "/index.js",
});
}
function serverWatch() {
serverStart();
gulp.watch(
[
basedir + "/paths/**/*",
// more directories to watch
],
serverRestart
);
}
function reload(done) {
serverWatch();
done();
}
function defaultTask() {
let p;
gulp.watch(["gulpfile.js"], killProcess);
spawnChild();
function killProcess(e) {
if (p && !p.killed) {
devServer.kill();
p.kill("SIGINT");
spawnChild();
}
}
function spawnChild() {
p = spawn("gulp", ["reload"], { stdio: "inherit" });
}
}
process.stdin.resume();
process.on("exit", handleExit.bind(null, { cleanup: true }));
process.on("SIGINT", handleExit.bind(null, { exit: true }));
process.on("uncaughtException", handleExit.bind(null, { exit: true }));
function handleExit(options, err) {
// perform some cleanup code here
if (options.cleanup) {
devServer.kill();
}
if (err) {
console.log(err.stack);
}
if (options.exit) {
process.exit();
}
}
gulp.task("serverRestart", serverRestart);
gulp.task("serverStart", serverStart);
gulp.task("serverWatch", serverWatch);
gulp.task("reload", reload);
gulp.task("default", defaultTask);
The existing flow is important because it executes needed code for setup and cleanup every time I hit save, which runs serverRestart. I've been trying a few different methods based on the other questions which recommended using gulp.src().pipe(), but I havne't had much luck integrating it with the existing pattern which uses gulp-develop-server. I am trying to not have to rewrite the whole gulpfile. Is there a simple way to integrate babel with my existing gulpfile such that I can use es6 in my source code?

There's an example with CoffeeScript in the gulp-develop-server documentation.
Using that as a model, try this:
function serverStart() {
devServer.listen({
path: "./dist/index.js",
});
}
function serverWatch() {
serverStart();
gulp.watch(
[
basedir + "/paths/**/*",
],
serverRestart
);
}
function serverRestart() {
gulp.src('./index.js')
.pipe(
babel({
presets: [
["#babel/env", { modules: false }],
],
})
)
.pipe( gulp.dest( './dist' ) )
.pipe( devServer() );
}
Other suggestions
That being said, your existing Gulp file doesn't actually really use Gulp. That is, everything is defined as a function and it doesn't leverage any of Gulp's useful features, like managing task dependencies. This is because (pre-es6), this was a very simple project. The Gulp tasks in that file are an over-elaborate way to watch files and run a server. The same could be done (with less code) using nodemon.
With the introduction of React and more complicated build processes, Gulp seems to have fallen out of favor with the community (and in my personal experience, Gulp was a time sinkhole anyhow).
If the main change you want to make is to use import, you can simply use a more recent Node version. You'll surely run into the error SyntaxError: Cannot use import statement outside a module. Simply rename the file to .mjs and it will work. This provides a way to incrementally migrate files to import syntax. Other features should automatically work (and are all backwards-compatible, anyhow). Once your project is mostly, or all, compliant, you can add "type": "module" to your package.json file, then rename all of your require-style js files to .cjs, and rename all of your .mjs files to .js, or leave them as .mjs. Read more about the rules of mixing CommonJS and Module imports in the Node.js blog post (note that some things may have changed since that article was written).

How to correctly build NestJS app for production with node_modules dependencies in bundle?

After nest build or nest build --webpack dist folder does not contain all required modules and I got Error: Cannot find module '#nestjs/core' when trying to run node main.js.
I could not find any clear instructions on https://docs.nestjs.com/ on how to correctly build app for production, so maybe I missed something?

Out of the box, nest cli does not support including the node_modules dependencies into the dist bundle.
However, there are some community examples of custom webpack configs that include the dependencies in the bundle, e.g. bundled-nest. As described in this issue, it is necessary to include the webpack.IgnorePlugin to whitelist unused dynamic libraries.

bundle-nest has been archived/discontinued:
We've concluded that it is not recommended to bundle NestJS, or actually, NodeJS web servers in general. This is archived for historical reference during the period of time when the community was attempting to tree-shake, and bundle NestJS apps. Refer to #kamilmysliwiec comment for details:
In many real-world scenarios (depending on what libraries are being used), you should not bundle Node.js applications (not only NestJS applications) with all dependencies (external packages located in the node_modules folder). Although this may make your docker images smaller (due to tree-shaking), somewhat reduce the memory consumption, slightly increase the bootstrap time (which is particularly useful in the serverless environments), it won't work in combination with many popular libraries commonly used in the ecosystem. For instance, if you try to build NestJS (or just express) application with MongoDB, you will see the following error in your console:
Error: Cannot find module './drivers/node-mongodb-native/connection' at webpackEmptyContext
Why? Because mongoose depends on mongodb which depends on kerberos (C++) and node-gyp.
Well, about mongo, you can make some exceptions (leave some modules in node_modules), can you? It's not like it's all or nothing. But still, I'm not sure you want to follow this path. I've just succeeded with bundling a nestjs application. It was a proof of concept, I'm not sure if it'll go into production. And it was hard, I might have broken something in the process, but at first glance it works. The most complex part was adminjs. It has rollup and babel as dependencies. And in the app code they unconditionally call watch for some reason (UDP noop in production). Anyways, if you'd like to follow this path you should be ready to debug/inspect your packages' code. And you might need to add workarounds as new packages are added to the project. But it all depends on your dependencies, it may be easier than in my case. For a freshly created nestjs + mysql app it was relatively simple.
The config I ended up with (it overrides the nestjs defaults):
webpack.config.js (webpack-5.58.2, #nestjs/cli-8.1.4):
const path = require('path');
const MakeOptionalPlugin = require('./make-optional-plugin');
module.exports = (defaultOptions, webpack) => {
return {
externals: {}, // make it not exclude `node_modules`
// https://github.com/nestjs/nest-cli/blob/v7.0.1/lib/compiler/defaults/webpack-defaults.ts#L24
resolve: {
...defaultOptions.resolve,
extensions: [...defaultOptions.resolve.extensions, '.json'], // some packages require json files
// https://unpkg.com/browse/babel-plugin-polyfill-corejs3#0.4.0/core-js-compat/data.js
// https://unpkg.com/browse/core-js-compat#3.19.1/data.json
alias: {
// an issue with rollup plugins
// https://github.com/webpack/enhanced-resolve/issues/319
'#rollup/plugin-json': '/app/node_modules/#rollup/plugin-json/dist/index.js',
'#rollup/plugin-replace': '/app/node_modules/#rollup/plugin-replace/dist/rollup-plugin-replace.cjs.js',
'#rollup/plugin-commonjs': '/app/node_modules/#rollup/plugin-commonjs/dist/index.js',
},
},
module: {
...defaultOptions.module,
rules: [
...defaultOptions.module.rules,
// a context dependency
// https://github.com/RobinBuschmann/sequelize-typescript/blob/v2.1.1/src/sequelize/sequelize/sequelize-service.ts#L51
{test: path.resolve('node_modules/sequelize-typescript/dist/sequelize/sequelize/sequelize-service.js'),
use: [
{loader: path.resolve('rewrite-require-loader.js'),
options: {
search: 'fullPath',
context: {
directory: path.resolve('src'),
useSubdirectories: true,
regExp: '/\\.entity\\.ts$/',
transform: ".replace('/app/src', '.').replace(/$/, '.ts')",
},
}},
]},
// adminjs resolves some files using stack (relative to the requiring module)
// and actually it needs them in the filesystem at runtime
// so you need to leave node_modules/#adminjs/upload
// I failed to find a workaround
// it bundles them to `$prj_root/.adminjs` using `rollup`, probably on production too
// https://github.com/SoftwareBrothers/adminjs-upload/blob/v2.0.1/src/features/upload-file/upload-file.feature.ts#L92-L100
{test: path.resolve('node_modules/#adminjs/upload/build/features/upload-file/upload-file.feature.js'),
use: [
{loader: path.resolve('rewrite-code-loader.js'),
options: {
replacements: [
{search: /adminjs_1\.default\.bundle\('\.\.\/\.\.\/\.\.\/src\/features\/upload-file\/components\/edit'\)/,
replace: "adminjs_1.default.bundle('/app/node_modules/#adminjs/upload/src/features/upload-file/components/edit')"},
{search: /adminjs_1\.default\.bundle\('\.\.\/\.\.\/\.\.\/src\/features\/upload-file\/components\/list'\)/,
replace: "adminjs_1.default.bundle('/app/node_modules/#adminjs/upload/src/features/upload-file/components/list')"},
{search: /adminjs_1\.default\.bundle\('\.\.\/\.\.\/\.\.\/src\/features\/upload-file\/components\/show'\)/,
replace: "adminjs_1.default.bundle('/app/node_modules/#adminjs/upload/src/features/upload-file/components/show')"},
],
}},
]},
// not sure what babel does here
// I made it return standardizedName
// https://github.com/babel/babel/blob/v7.16.4/packages/babel-core/src/config/files/plugins.ts#L100
{test: path.resolve('node_modules/#babel/core/lib/config/files/plugins.js'),
use: [
{loader: path.resolve('rewrite-code-loader.js'),
options: {
replacements: [
{search: /const standardizedName = [^;]+;/,
replace: match => `${match} return standardizedName;`},
],
}},
]},
// a context dependency
// https://github.com/babel/babel/blob/v7.16.4/packages/babel-core/src/config/files/module-types.ts#L51
{test: path.resolve('node_modules/#babel/core/lib/config/files/module-types.js'),
use: [
{loader: path.resolve('rewrite-require-loader.js'),
options: {
search: 'filepath',
context: {
directory: path.resolve('node_modules/#babel'),
useSubdirectories: true,
regExp: '/(preset-env\\/lib\\/index\\.js|preset-react\\/lib\\/index\\.js|preset-typescript\\/lib\\/index\\.js)$/',
transform: ".replace('./node_modules/#babel', '.')",
},
}},
]},
],
},
plugins: [
...defaultOptions.plugins,
// some optional dependencies, like this:
// https://github.com/nestjs/nest/blob/master/packages/core/nest-application.ts#L45-L52
// `webpack` detects optional dependencies when they are in try/catch
// https://github.com/webpack/webpack/blob/main/lib/dependencies/CommonJsImportsParserPlugin.js#L152
new MakeOptionalPlugin([
'#nestjs/websockets/socket-module',
'#nestjs/microservices/microservices-module',
'class-transformer/storage',
'fastify-swagger',
'pg-native',
]),
],
// to have have module names in the bundle, not some numbers
// although numbers are sometimes useful
// not really needed
optimization: {
moduleIds: 'named',
}
};
};
make-optional-plugin.js:
class MakeOptionalPlugin {
constructor(deps) {
this.deps = deps;
}
apply(compiler) {
compiler.hooks.compilation.tap('HelloCompilationPlugin', compilation => {
compilation.hooks.succeedModule.tap(
'MakeOptionalPlugin', (module) => {
module.dependencies.forEach(d => {
this.deps.forEach(d2 => {
if (d.request == d2)
d.optional = true;
});
});
}
);
});
}
}
module.exports = MakeOptionalPlugin;
rewrite-require-loader.js:
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions#escaping
function escapeRegExp(string) {
return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
}
function processFile(source, search, replace) {
const re = `require\\(${escapeRegExp(search)}\\)`;
return source.replace(
new RegExp(re, 'g'),
`require(${replace})`);
}
function processFileContext(source, search, context) {
const re = `require\\(${escapeRegExp(search)}\\)`;
const _d = JSON.stringify(context.directory);
const _us = JSON.stringify(context.useSubdirectories);
const _re = context.regExp;
const _t = context.transform || '';
const r = source.replace(
new RegExp(re, 'g'),
match => `require.context(${_d}, ${_us}, ${_re})(${search}${_t})`);
return r;
}
module.exports = function(source) {
const options = this.getOptions();
return options.context
? processFileContext(source, options.search, options.context)
: processFile(source, options.search, options.replace);
};
rewrite-code-loader.js:
function processFile(source, search, replace) {
return source.replace(search, replace);
}
module.exports = function(source) {
const options = this.getOptions();
return options.replacements.reduce(
(prv, cur) => {
return prv.replace(cur.search, cur.replace);
},
source);
};
The supposed way to build the app is:
$ nest build --webpack
I didn't bother with source maps, since the target is nodejs.
It's not a config you can just copy-paste, you should figure out what's needed for your project yourself.
One more trick here, but well, you probably won't need it.
UPD adminjs seems to come with prebuilt bundles, so this config may be significantly simpler.

Requiring an optimized module after optimizing with RequireJS

Upon starting my server, I'm attempting to utilize the RequireJS optimizer to combine all my RequireJS modules into a single file. Once the optimizer is finished, I am attempting to utilize the modules but it doesn't seem to be working.
var path = require('path');
var fs = require('fs');
var requirejs = require('requirejs');
requirejs.config({
baseUrl: __dirname,
nodeRequire: require
});
requirejs.optimize({
baseUrl: path.join(__dirname, 'foo'),
dir: 'build',
modules: [{
name: 'main',
include: [ 'src/bar' ]
}]
}, function (data) {
console.log(fs.readFileSync(path.join(__dirname, 'build', 'main.js'), 'utf-8'));
var main = requirejs('/build/main.js'));
var bar = requirejs('src/bar');
}, function (error) {
console.log(error);
});
The output from the console.log is the concatenated files as expected, but bar is undefined.
If I run the following script after the previous script, bar is defined.
var requirejs = require('requirejs');
requirejs.config({
baseUrl: __dirname,
nodeRequire: require
});
var main = requirejs('/build/main.js'));
var bar = requirejs('src/bar');
console.log(bar);
Can anyone offer any insight into what may be preventing the first script from working?
Thanks,
Jake

Your path for baseUrl in requirejs.config was wrong. I've also added removeCombined in the optimizer configuration to prevent loading files that have been combined into main.
However, that not all that was needed. You see I also save the value from requirejs.config in r and then I use this r to load the modules. This is the same thing you would do if you used RequireJS contexts: you'd save the value of the requirejs.config call for each context so that you can use one function to load things from one context and the other to load things from the other context. I have to admit I don't know why this makes the code work. It smells like a bug to me.
Another thing I discovered is that requirejs.optimize swallows any exception thrown in its completion callback. This is definitely a bug. If it ha not been swallowing exceptions, you'd have had a better idea of what was going on.
Here is the code:
var path = require('path');
var fs = require('fs');
var requirejs = require('requirejs');
var r = requirejs.config({
baseUrl: path.join(__dirname, 'build'),
nodeRequire: require
});
requirejs.optimize({
baseUrl: path.join(__dirname, 'foo'),
dir: 'build',
// You should do this to make sure that you are not accidentally loading
// files that have *not* been combined.
removeCombined: true,
modules: [{
name: 'main',
include: [ 'src/bar' ]
}]
}, function (data) {
// console.log(fs.readFileSync(path.join(__dirname, 'build', 'main.js'), 'utf-8'));
var main = r('main');
console.log(main);
var bar = r('src/bar');
console.log(bar);
}, function (error) {
console.log(error);
});

Grunt build not exposing the globals I need

When I run my project locally with my grunt:server task, the project works as I expect. However, after building which takes all the vendor code and puts it into one file, two of my needed module aren't avialable, and the project doesn't work.
Here is my requirejs configuration:
requirejs.config
baseUrl: './js'
shim:
'underscore':
exports: '_'
'backbone':
deps: ['underscore', 'jquery']
exports: 'Backbone'
'stack':
deps: ['d3.global']
exports: 'stack'
'highlight':
exports: 'hljs'
paths:
'underscore': '../components/underscore/underscore'
'backbone': '../components/backbone/backbone'
'jquery': '../components/jquery/jquery'
'd3': '../components/d3/d3'
'd3.global': '../components/d3.global/d3.global'
'stack': '../components/stack/stack'
'highlight': '../components/highlightjs/highlight.pack'
require ['app/vendors'],->
console.log("Backbone", Backbone)
console.log("_", _)
console.log("$", $)
console.log("d3", d3)
console.log("stack", stack)
console.log("hljs", hljs)
app/vendors looks like
define [
'underscore'
'jquery'
'backbone'
'text'
'd3.global'
'stack'
'highlight'
], ->
When I run the project locally via grunt, I see all the globals printed out. However, when I build the project, Backbone Underscore and JQuery print out, while stack fails (hljs is also not available, and if I remove stack from app/vendors, it doesn't fix highlight, so its probably not an order thing).
the requirejs optimizer is called with the following configuration:
requirejs:
compile:
options:
baseUrl: 'js/'
appDir: './<%= yeoman.tmp_dist %>/'
dir: './<%= yeoman.dist %>/'
wrap: true
removeCombined: true
keepBuildDir: true
inlineText: true
mainConfigFile: '<%= yeoman.tmp_dist %>/js/main.js'
# no minification, is done by the min task
optimize: "none"
modules: [
{ name: 'app/vendors', exclude: [] }
{ name: 'app/app', exclude: ['app/vendors'] }
{ name: 'main', exclude: ['app/app', 'app/vendors'] }
Could there be something wrong with the stack and highlight files that I need to fix in order to make requirejs optimization and uglify work with them?
I installed highlightjs via bower by adding "highlightjs": "~8.0" to my bower.json file and running bower install. I downloaded stack.js from mbostock's stack project. I'm using v0 at the moment, with minor changes to make it work in this project. The source for all these are in the components directory of my github project.
BOUNTY If anyone is willing to clone the repo themselves, and try running the project with grunt server and grunt build to help me track down the problem, I'd greatly appreciate it. At the moment I have the vendor scripts in the github repo itself, so all you should need is compass and bower to run it.

This is due to wrap: true in the r.js config. Here's a simple configuration that isolates the issue:
main.js
define([ 'legacy' ], function(legacy) {
var greeting = 'hi';
console.log(greeting, legacy.foo);
});
legacy.js
var globalThing = { foo: 1, bar: 2 };
build.json
{
"name": "main",
"optimize": "none",
"out": "main-built.js",
"shim": { "legacy": { "exports": "globalThing" } },
"wrap": true
}
Let's run r.js (r.js -o build.json) and consider the result (formatted by me):
(function() { // this immediately-invoked function expression (IIFE)
// is here because r.js has "wrap: true" in the config
var globalThing = { foo: 1, bar: 2 };
// code generated from the "shim" entry in the config
define('legacy', function(global) {
return function() {
var ret, fn;
// since global.globalThing is undefined,
// that's where it goes wrong
return ret || global.globalThing;
};
}(this));
define('main', [ 'legacy' ], function(legacy) {
var greeting = 'hi';
console.log(greeting, legacy.foo);
});
})(); // end of the IIFE
As you can see from the code above, globalThing isn't global any more. The same happens with the stack and highlight libraries in your project as they use var and function declarations to define their globals.
To tackle this issue, we have a couple of options. The first is to consider whether you really need wrap: true in the config. If you drop it, the globals will get global again and everything should start working as expected. The second option is to try adding wrapShim: true to the config. You can read about nuances of using this option here. If we try it with our sample configuration, we'll get something like this:
(function() {
(function(root) {
define('legacy', [], function() {
return function() {
var globalThing = { foo: 1, bar: 2 };
return root.globalThing = globalThing;
}.apply(root, arguments);
});
})(this);
define('main', [ 'legacy' ], function(legacy) {
var greeting = 'hi';
console.log(greeting, legacy.foo);
});
})();
Looks good to me.

Why does my Handlebars not have a compile method?

I am setting up a Backbone project with Handlebars and I am having an issue with Handlebars not finding the compile method. Here is my config file:
require.config({
hbs: {
templateExtension: '.hbs'
},
paths: {
backbone: "libs/backbone/backbone",
handlebars: 'libs/handlebars/handlebars.amd',
hbs: 'libs/requirejs-hbs/hbs',
jquery: 'libs/jquery/jquery',
jqueryMockAjax: 'libs/jquery-mockjax/jquery.mockjax',
text: 'libs/requirejs-text/text',
templates: 'templates/',
underscore: 'libs/underscore/underscore'
},
shim: {
backbone: {
deps: [
'underscore',
'jquery'
],
exports: 'Backbone'
},
hbs: {
deps: ['handlebars'],
exports: 'hbs'
},
jqueryMockAjax: {
deps: [ 'jquery' ],
exports: '$.mockjax'
},
underscore: {
exports: '_'
}
}
});
require(['app'], function(App) {
'use strict';
var app = new App();
app.render();
});
Here is the app.js that I am trying to render:
define(function(require) {
var Backbone = require('backbone');
var testTemplate = require('hbs!templates/test');
var router = Backbone.View.extend({
el: $('body'),
template: testTemplate,
render: function() {
return $(this.el).html(this.template());
}
});
return router;
});
When Handlebars calls the hbs.js file on line 25 it cannot find the compile function
define(["handlebars"], function(Handlebars) {
var buildMap = {},
templateExtension = ".hbs";
return {
// http://requirejs.org/docs/plugins.html#apiload
load: function (name, parentRequire, onload, config) {
// Get the template extension.
var ext = (config.hbs && config.hbs.templateExtension ? config.hbs.templateExtension : templateExtension);
if (config.isBuild) {
// Use node.js file system module to load the template.
// Sorry, no Rhino support.
var fs = nodeRequire("fs");
var fsPath = config.dirBaseUrl + "/" + name + ext;
buildMap[name] = fs.readFileSync(fsPath).toString();
onload();
} else {
// In browsers use the text-plugin to the load template. This way we
// don't have to deal with ajax stuff
parentRequire(["text!" + name + ext], function(raw) {
// Just return the compiled template
****HERE onload(Handlebars.compile(raw));
});
}
},
// http://requirejs.org/docs/plugins.html#apiwrite
write: function (pluginName, name, write) {
var compiled = Handlebars.precompile(buildMap[name]);
// Write out precompiled version of the template function as AMD
// definition.
write(
"define('hbs!" + name + "', ['handlebars'], function(Handlebars){ \n" +
"return Handlebars.template(" + compiled.toString() + ");\n" +
"});\n"
);
}
};
});
The Handlebars variable gives me the Handlebars environment, but it has an extra layer in it, so I have to change that line to Handlebars.default.compile(raw). Where is that default object coming from and how do I get rid of it? I wouldn't worry about it, but if I pull down this project somewhere else I am always going to have to remember to do that.

I just encountered this myself, using Handlebars for this first time. It's likely that you're using the "runtime" build of Handlebars. I included this one in my requirements, mistakenly assuming it was the minified version or something.
But in fact the runtime version is significantly smaller as it excludes the template compiler, and is for use only with pre-compiled templates. If you're compiling the template client-side then you need the full version from http://builds.handlebarsjs.com.s3.amazonaws.com/handlebars-v3.0.3.js (Note: This link may be out of date; you're probably better off going directly to handlebarsjs.com and looking for the current download for the "full version", as opposed to runtime.)
Otherwise, you can follow the instructions on the Handlebars website to run the template compiler. You need node for this. The template compiler produces a JavaScript file containing the pre-compiled template code which you need to link to your page along with the Handlebars runtime build.

Here is how I fixed this issue, although I do not understand completely why the configuration above did not work. The hbs plugin had a folder with all the dependencies it needed in it, like handlebars. When I referred to the handlebars copy contained in the hbs directory, then everything worked like it was supposed to. I do not understand why the vanilla copy of handlebars did not work. I was not using the handlebars runtime, it was the full version, but still there was an issue. After I resolved this, then my template stuff just worked.