We Keep Coding

Using RequireJS with node to optimize creating single output file does not include all the required files

I use the FayeJS and the latest version has been modified to use RequireJS, so there is no longer a single file to link into the browser. Instead the structure is as follows:
/adapters
/engines
/mixins
/protocol
/transport
/util
faye_browser.js
I am using the following nodejs build script to try and end up with all the above minified into a single file:
var fs = require('fs-extra'),
requirejs = require('requirejs');
var config = {
baseUrl: 'htdocs/js/dev/faye/'
,name: 'faye_browser'
, out: 'htdocs/js/dev/faye/dist/faye.min.js'
, paths: {
dist: "empty:"
}
,findNestedDependencies: true
};
requirejs.optimize(config, function (buildResponse) {
//buildResponse is just a text output of the modules
//included. Load the built file for the contents.
//Use config.out to get the optimized file contents.
var contents = fs.readFileSync(config.out, 'utf8');
}, function (err) {
//optimization err callback
console.log(err);
});
The content of faye_browser.js is:
'use strict';
var constants = require('./util/constants'),
Logging = require('./mixins/logging');
var Faye = {
VERSION: constants.VERSION,
Client: require('./protocol/client'),
Scheduler: require('./protocol/scheduler')
};
Logging.wrapper = Faye;
module.exports = Faye;
As I under stand it the optimizer should pull in the required files, and then if those files have required files, it should pull in those etc..., and and output a single minified faye.min.js that contains the whole lot, refactored so no additional serverside calls are necessary.
What happens is faye.min.js gets created, but it only contains the content of faye_browser.js, none of the other required files are included.
I have searched all over the web, and looked at a heap of different examples and none of them work for me.
What am I doing wrong here?

For anyone else trying to do this, I mist that on the download page it says:
The Node.js version is available through npm. This package contains a
copy of the browser client, which is served up by the Faye server when
running.
So to get it you have to pull down the code via NPM and then go into the NPM install dir and it is in the "client" dir...

npm global packages: Reference content files from package

I'm in the process of building an npm package which will be installed globally. Is it possible to have non-code files installed alongside code files that can be referenced from code files?
For example, if my package includes someTextFile.txt and a module.js file (and my package.json includes "bin": {"someCommand":"./module.js"}) can I read the contents of someTextFile.txt into memory in module.js? How would I do that?

The following is an example of a module that loads the contents of a file (string) into the global scope.
core.js : the main module file (entry point of package.json)
//:Understanding: module.exports
module.exports = {
reload:(cb)=>{ console.log("[>] Magick reloading to memory"); ReadSpellBook(cb)}
}
//:Understanding: global object
//the following function is only accesible by the magick module
const ReadSpellBook=(cb)=>{
require('fs').readFile(__dirname+"/spellBook.txt","utf8",(e,theSpells)=>{
if(e){ console.log("[!] The Spell Book is MISSING!\n"); cb(e)}
else{
console.log("[*] Reading Spell Book")
//since we want to make the contents of .txt accesible :
global.SpellBook = theSpells // global.SpellBook is now shared accross all the code (global scope)
cb()//callBack
}
})
}
//·: Initialize :.
console.log("[+] Time for some Magick!")
ReadSpellBook((e)=>e?console.log(e):console.log(SpellBook))
spellBook.txt
ᚠ ᚡ ᚢ ᚣ ᚤ ᚥ ᚦ ᚧ ᚨ ᚩ ᚪ ᚫ ᚬ ᚭ ᚮ ᚯ
ᚰ ᚱ ᚲ ᚳ ᚴ ᚵ ᚶ ᚷ ᚸ ᚹ ᚺ ᚻ ᚼ ᚽ ᚾ ᚿ
ᛀ ᛁ ᛂ ᛃ ᛄ ᛅ ᛆ ᛇ ᛈ ᛉ ᛊ ᛋ ᛌ ᛍ ᛎ ᛏ
ᛐ ᛑ ᛒ ᛓ ᛔ ᛕ ᛖ ᛗ ᛘ ᛙ ᛚ ᛛ ᛜ ᛝ ᛞ ᛟ
ᛠ ᛡ ᛢ ᛣ ᛤ ᛥ ᛦ ᛧ ᛨ ᛩ ᛪ ᛫ ᛬ ᛭ ᛮ ᛯ
If you require it from another piece of code, you will see how it prints to the console and initializes by itself.
If you want to achieve a manual initalization, simply remove the 3 last lines (·: Initialize :.) and use reload() :
const magick = require("core.js")
magick.reload((error)=>{ if(error){throw error}else{
//now you know the SpellBook is loaded
console.log(SpellBook.length)
})

I have built some CLIs which were distributed privately, so I believe I can illuminate a bit here.
Let's say your global modules are installed at a directory called $PATH. When your package will be installed on any machine, it will essentially be extracted at that directory.
When you'll fire up someCommand from any terminal, the module.js will be invoked which was kept at $PATH. If you initially kept the template file in the same directory as your package, then it will be present at that location which is local to module.js.
Assuming you edit the template as a string and then want to write it locally to where the user wished / pwd, you just have to use process.cwd() to get the path to that directory. This totally depends on how you code it out.
In case you want to explicitly include the files only in the npm package, then use files attribute of package.json.
As to particularly answer "how can my code file in the npm package locate the path to the globally installed npm folder in which it is located in a way that is guaranteed to work across OSes and is future proof?", that is very very different from the template thingy you were trying to achieve. Anyway, what you're simply asking here is the global path of npm modules. As a fail safe option, use the path returned by require.main.filename within your code to keep that as a reference.

When you npm publish, it packages everything in the folder, excluding things noted in .npmignore. (If you don't have an .npmignore file, it'll dig into .gitignore. See https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package) So in short, yes, you can package the text file into your module. Installing the module (locally or globally) will get the text file into place in a way you expect.
How do you find the text file once it's installed? __dirname gives you the path of the current file ... if you ask early enough. See https://nodejs.org/docs/latest/api/globals.html#globals_dirname (If you use __dirname inside a closure, it may be the path of the enclosing function.) For the near-term of "future", this doesn't look like it'll change, and will work as expected in all conditions -- whether the module is installed locally or globally, and whether others depend on the module or it's a direct install.
So let's assume the text file is in the same directory as the currently running script:
var fs = require('fs');
var path = require('path');
var dir = __dirname;
function runIt(cb) {
var fullPath = path.combine(__dirname, 'myfile.txt');
fs.readFile(fullPath, 'utf8' , function (e,content) {
if (e) {
return cb(e);
}
// content now has the contents of the file
cb(content);
}
}
module.exports = runIt;
Sweet!

Angular2 bundling and minification

ASP.NET (or gulp) will take care of bundling and minification. However, the problem i came across is rather different. Per Angular2's tutorials, the view HTML is embedded within the component itself. There is a way, using TypeScript, to separate that into a .ts and .html files. Here's how:
...
/// <reference path="./view-declaration.d.ts" />
...
import {html} from '/app.html!text';
...
#Component({
...
template: html
})
...
And faking .html as a module in view-declaration.d.ts file:
declare module '/app.html!text' {
var html:string;
return default html;
}
This is using SystemJS with its text plugin. This will not generate System.register for .html files which means HTMLs can't be bundled along the transpiled .js files.
So questions is, how do you separate the HTML from JavaScript, but also be able to bundle them properly?
Also to note, this approach is just the same as setting the templateUrl on your component. Both of which defeat the purpose of bundling and reduction of server hits per component. The offered solution from Angular2 is to use string and set template on a component. This is pretty far from reality of junior developers and code reviews (yea not gonna get the whole code base in order to run and see if browser complains about a non-closed tag!).

This is gulp plugin which I think can solve your issue.
look at:
https://github.com/ludohenin/gulp-inline-ng2-template
the advantage you keep having a clean html file using templateUrl during development. And this task can be part of your production\staging -minified - gulp build task.
e.g (Part of my build tasks!)
var inlineNg2Template = require('gulp-inline-ng2-template');
gulp.task('build-prod', ['build.lib'], function () {
var tsProject = typescript.createProject('./tsconfig.json', { typescript: require('typescript') });
var tsSrcInlined = gulp.src([webroot + '**/*.ts'], { base: webroot })
.pipe(inlineNg2Template({ base: webroot }));
return eventStream.merge(tsSrcInlined, gulp.src('Typings/**/*.ts'))
.pipe(sourcemaps.init())
.pipe(typescript(tsProject))
.pipe(sourcemaps.write())
.pipe(gulp.dest(webroot));
});

As it turns out, you'll need to use templateUrl for development, but replace that with template during the bundling and minification. This is the gulp task for the job:
var gulp = require('gulp'); //install 'gulp' via npm
var uglify = require('gulp-uglify'); //install 'gulp-uglify' via npm
var concat = require('gulp-concat'); //install 'gulp-concat' via npm
var replace = require('gulp-replace'); //install 'gulp-replace' via npm
var fs = require("fs"); //already available in Visual Studio and of course NodeJS
gulp.task('bundle:js', function () {
return gulp
.src([
"file1.js",
"file2.js"
])
.pipe(replace(/templateUrl.*\'/g, function (matched) {
var fileName = matched.match(/\/.*html/g).toString();
var fileContent = fs.readFileSync(fileName, "utf8");
return 'template:\'' + fileContent.replace(/\r\n/g, '') + '\'';
}))
.pipe(concat('file.min.js'))
.pipe(gulp.dest('my bundle folder'))
.pipe(uglify())
.pipe(gulp.dest('my bundle folder'));
});
This is to match .html files and the template URL, but it can be tweaked if needed.

Your template file app.html.ts can export the html template as a string.
export const htmlTemplate = `
<p>My app</p>
`;
Then your component (app.component.ts) can import the template inline.
import { Component } from '#angular/core';
import { htmlTemplate } from './app.html';
#Component({
selector: 'my-app',
template: htmlTemplate,
})
...
This approach:
allows for in-lining of templates at compilation, avoiding the additional network request that using "templateUrl" adds and allowing templates to be minified and bundled with other js
allows templates to live in external files for code clarity and scaling
can be easily migrated to a plain HTML file once Typescript "import file as string" functionality is in place
allows Webstorm html syntax highlighting to still work correctly
See blog from Angular University

How to set up gulp to bundle several files into one?

This seems like a very simple question, but spent the last 3 hours researching it, discovering it can be slow on every save on a new file if not using watchify.
This is my directory tree:
gulpfile.js
package.json
www/
default.htm
<script src="toBundleJsHere/file123.js"></script>
toBundletheseJs/
componentX/
file1.js
componentY/
file2.js
componentZ/
file3.js
toPutBundledJsHere/
file123.js
Requirements.
On every creation or save of a file within the folder toBundleTheseJs/ I want this file to be rebundled into toBundleJsHere/
What do I need to include in my package.json file?
And whats the minimum I need to write into my gulp file?
This should be as fast as possible so think I should be using browserify and watchify. I want to understand the minimum steps so using package manager like jspm is overkill a this point.
thanks

First you should listen to changes in the desired dir:
watch(['toBundletheseJs/**/*.js'], function () {
gulp.run('bundle-js');
});
Then the bundle-js task should bundle your files. A recommended way is gulp-concat:
var concat = require('gulp-concat');
var gulp = require('gulp');
gulp.task('bundle-js', function() {
return gulp.src('toBundletheseJs/**/*.js')
.pipe(concat('file123.js'))
.pipe(gulp.dest('./toPutBundledJsHere/'));
});

The right answer is: there is no legit need for concatenating JS files using gulp. Therefore you should never do that.
Instead, look into proper JS bundlers that will properly concatenate your files organizing them according to some established format, like commonsjs, amd, umd, etc.
Here's a list of more appropriate tools:
Webpack
Rollup
Parcel
Note that my answer is around end of 2020, so if you're reading this in a somewhat distant future keep in mind the javascript community travels fast so that new and better tools may be around.

var gulp = require('gulp');
var concat = require('gulp-concat');
gulp.task('js', function (done) {
// array of all the js paths you want to bundle.
var scriptSources = ['./node_modules/idb/lib/idb.js', 'js/**/*.js'];
gulp.src(scriptSources)
// name of the new file all your js files are to be bundled to.
.pipe(concat('all.js'))
// the destination where the new bundled file is going to be saved to.
.pipe(gulp.dest('dist/js'));
done();
});

Use this code to bundle several files into one.
gulp.task('scripts', function() {
return gulp.src(['./lib/file3.js', './lib/file1.js', './lib/file2.js']) //files separated by comma
.pipe(concat('script.js')) //resultant file name
.pipe(gulp.dest('./dist/')); //Destination where file to be exported
});

Compiling dynamically required modules with Browserify

I am using Browserify to compile a large Node.js application into a single file (using options --bare and --ignore-missing [to avoid troubles with lib-cov in Express]). I have some code to dynamically load modules based on what is available in a directory:
var fs = require('fs'),
path = require('path');
fs.readdirSync(__dirname).forEach(function (file) {
if (file !== 'index.js' && fs.statSync(path.join(__dirname, file)).isFile()) {
module.exports[file.substring(0, file.length-3)] = require(path.join(__dirname, file));
}
});
I'm getting strange errors in my application where aribtrary text files are being loaded from the directory my compiled file is loaded in. I think it's because paths are no longer set correctly, and because Browserify won't be able to require() the correct files that are dynamically loaded like this.
Short of making a static index.js file, is there a preferred method of dynamically requiring a directory of modules that is out-of-the-box compatible with Browserify?

This plugin allows to require Glob patterns: require-globify
Then, with a little hack you can add all the files on compilation and not executing them:
// Hack to compile Glob files. Don´t call this function!
function ಠ_ಠ() {
require('views/**/*.js', { glob: true })
}
And, for example, you could require and execute a specific file when you need it :D
var homePage = require('views/'+currentView)

Browserify does not support dynamic requires - see GH issue 377.
The only method for dynamically requiring a directory I am aware of: a build step to list the directory files and write the "static" index.js file.

There's also the bulkify transform, as documented here:
https://github.com/chrisdavies/tech-thoughts/blob/master/browserify-include-directory.md
Basically, you can do this in your app.js or whatever:
var bulk = require('bulk-require');
// Require all of the scripts in the controllers directory
bulk(__dirname, ['controllers/**/*.js']);
And my gulpfile has something like this in it:
gulp.task('js', function () {
return gulp.src('./src/js/init.js')
.pipe(browserify({
transform: ['bulkify']
}))
.pipe(rename('app.js'))
.pipe(uglify())
.pipe(gulp.dest('./dest/js'));
});