I'm writing a javascript library that contains a core module and several
optional submodules which extend the core module. My target is the browser
environment (using Browserify), where I expect a user of my module will only
want to use some of my optional submodules and not have to download the rest to
the client--much like custom builds work in lodash.
The way I imagine this working:
// Require the core library
var Tasks = require('mymodule');
// We need yaks
require('mymodule/yaks');
// We need razors
require('mymodule/razors');
var tasks = new Tasks(); // Core mymodule functionality
var yak = tasks.find_yak(); // Provided by mymodule/yaks
tasks.shave(yak); // Provided by mymodule/razors
Now, imagine that the mymodule/* namespace has tens of these submodules. The
user of the mymodule library only needs to incur the bandwidth cost of the
submodules that she uses, but there's no need for an offline build process like
lodash uses: a tool like Browserify solves the dependency graph for us and
only includes the required code.
Is it possible to package something this way using Node/npm? Am I delusional?
Update: An answer over here seems to suggest that this is possible, but I can't figure out from the npm documentation how to actually structure the files and package.json.
Say that I have these files:
./lib/mymodule.js
./lib/yaks.js
./lib/razors.js
./lib/sharks.js
./lib/jets.js
In my package.json, I'll have:
"main": "./lib/mymodule.js"
But how will node know about the other files under ./lib/?
It's simpler than it seems -- when you require a package by it's name, it gets the "main" file. So require('mymodule') returns "./lib/mymodule.js" (per your package.json "main" prop). To require optional submodules directly, simply require them via their file path.
So to get the yaks submodule: require('mymodule/lib/yaks'). If you wanted to do require('mymodule/yaks') you would need to either change your file structure to match that (move yaks.js to the root folder) or do something tricky where there's a yaks.js at the root and it just does something like: module.exports = require('./lib/yaks');.
Good luck with this yak lib. Sounds hairy :)
Related
I have a nodejs project which uses large dictionary lists (millions of entries), stored in js files, that look like this:
module.exports = ["entry1", "entry2", "entry3", "entry4", "entry5", etc.];
and then I use them from the other files like this:
var values = require('./filePath');
This works great and it works in the browser too (using browserify), however bundling takes ages - about 10 minutes.
I use the following command to create the bundle:
browserify "./src/myModule.js" --standalone myModule -t [ babelify --presets [ es2015 stage-2 ] --plugins ["transform-es2015-classes", {"loose": true}]
I have tried to avoid parsing of my dictionary js files using --noparse ["path1", "path2", "path3", etc.] but it did not make any difference.
Ideally I would like to just speed up the browserify\babelify process, however if that's not possible I would be very happy to find another way (ie. avoid require) to store and use my lists, so that they don't slow the process down but that crucially work in node and in the browser too.
You can bundle the data files separately, so you'll only need to rebundle them when they change. This is possible using the --require -r and --external -x options.
To create the data bundle, do something like this:
browserify -r ./path/to/data.js -r ./path/to/other/data.js > data-bundle.js
The resulting data-bundle.js will define the require function globally which can be used to obtain any file you listed in the command above. Just make sure you include this bundle in a script tag before your main bundle.
It would be nice to be able to --require a glob pattern, but unfortunately browserify does not support this. If you try to use the shell to expand a pattern, the -r option will only apply to the first, which sucks. You can probably write a shell script that builds a command from an ls or something, to avoid having to list all of the data files explicilty, but that's beyond the scope of the question, I think.
To create your main bundle without rebuilding the data files, simply add an option like this to your command:
-x './path/to/data/*.js'
This tells browserify to basically ignore them and let them be pulled in through the global require function created by your other bundle. As you can see, this does support glob patterns, so it's a bit easier.
Update:
To make the two bundles into one, just put something like this at the end of a shell script that starts with the browserify command that builds your main bundle:
cat data-bundle.js main-bundle.js > bundle.js
rm main-bundle.js
Unfortunately this will always have to write a copy of data-bundle.js to disk, which may be the ultimate cause of the slowdown, as I mentioned in the comments below. Worth giving a shot, though.
If even that doesn't work, there are some other, much more hacky approaches you might take. I'll pass on going into those for now though, because I don't think they're worth it unless you absolutely must have it as one file and have no other way of doing it. :\
If you have files with data - just load them in separate way and don't include them into build process
Format your big data files as JSON
On the server use:
let fs = require('fs');
let yourContent = JSON.parse(fs.readFileSync('path/to/file'));
On client use:
let request = require("client-request"); // do npm install client-request
var options = {
uri: "http://.com/path/to/file",
json: true
}
var req = request(options, function callback(err, response, body) {
console.log(response.statusCode)
if (body) {
let yourContent = body
}
})
Or use any other library which makes HTTP request which you prefer
Im working with project that is written in Node CommonJS modules. My point was to make this project accessible from a browser. I decided to use Rollup.js, so when the bundle is created you can include script in browser and use functions from library (thanks to iife format). I needed to install plugins for Rollup to convert CommonJS modules into ES6 modules, so browser can understand it.
Everything is fine, except that this project uses bson library from npm. This bson library is required in one of my modules which uses some of it's functions. After I create a bundle and include it into index.html an error appears in console which says: "require is not defined". When I look inside the created bundle there are some requires.
var Map = require('./map'),
Long = require('./long'),
Double = require('./double'),
Timestamp = require('./timestamp'),
ObjectID = require('./objectid'),
BSONRegExp = require('./regexp'),
Symbol$1 = require('./symbol'),
Int32 = require('./int_32'),
Code = require('./code'),
Decimal128 = require('./decimal128'),
MinKey = require('./min_key'),
MaxKey = require('./max_key'),
DBRef = require('./db_ref'),
Binary = require('./binary');
I have created simple code in Plunker to illustrate you my config and simplified structure.
https://plnkr.co/edit/YuiVJxhwhjUQ0Flw0Mg3?p=preview
In this plunker there are two simple modules, which one requires bson library, and second requires this first module. There is also Rollup config file, where I use plugins (if there is no globals plugin there is an error: Uncaught ReferenceError: Buffer is not defined).
I'm really confused. Am I misunderstanding something? Why isn't it converted into ES6 modules just like other of my code?
Here is link to bson library: https://www.npmjs.com/package/bson
Let's see:
In order for rollup to bundle your application you need to transform commonjs modules into es modules first, this is done with rollup-plugin-commonjs (https://github.com/rollup/rollup-plugin-commonjs) which you are already using, maybe you could explicitly set in the options to include the bson package. I usually use it like this, just in case:
commonjs({
include: ['node_modules/**']
})
If some library uses node global modules you will need to include them in the browser, thats why you need rollup-plugin-node-globals.
Finally, if you take a look at bson github repository there is a folder called browser_build, which contains the UMD definition of the library, so if you require 'bson/browser_build' instead of 'bson' it should work, and you may not need to use globals plugin.
Take a look at js module formats (cjs, umd, iife, es, amd) it is worth it.
Is there a way to invoke Browserify (via Gulp) so that it includes a different file when requireing a module with a given name?
Briefly, the end result I would like is for my Browserify entry point, main.js:
var myPlatformSpecificImplmentation = require('./platform');
// go to town
to use the contents of ./path/to/platform-a.js when I run gulp js:platform-a and ./path/to/platform-b.js when I run gulp js:platform-b.
If I were using RequireJS, this would be as simple as modifying the paths option accordingly:
paths: {
platform: './path/to/platform-a'
}
It would be great if I could somehow generate these modules dynamically via gulp's built-in streaming mechanism. In that case, I could, say, pipe a file into gulp-template and on into Browserify.
Thanks
One solution would be to use my pathmodify plugin like so:
gulpfile.js
var
pathmod = require('pathmodify'),
paths = {a: '/path/to/platform-a.js', b: '/path/to/platform-b.js'};
function platform (version) {
return function () {
return browserify('./main')
.plugin(pathmod(), {mods: [
pathmod.mod.id('app/platform', paths[version])
]})
.bundle()
.pipe(...);
};
}
gulp.task('js:platform-a', platform('a'));
gulp.task('js:platform-b', platform('b'));
main.js
var myPlatformSpecificImplmentation = require('app/platform');
I've illustrated this with your require() string changed to app/platform because that allows the simplest implementation of pathmodify without collisions with other ./platform relative paths in other files. But this can be implemented with pathmodify without risking collision (by testing the parent module [main.js in this case] pathname). If it's important to you to keep the ./platform string I'll illustrate that.
Or you could use a transform. Take a look at makeRequireTransform() in benbria/browserify-transform-tools if you don't want to roll your own.
It would be great if I could somehow generate these modules dynamically via gulp's built-in streaming mechanism. In that case, I could, say, pipe a file into gulp-template and on into Browserify.
That's not out of the question, but it's not really easy to do. To do it without touching disk, you'd need to do something like create / gulp.src() a vinyl file and run it through whatever gulp plugins, then convert it to a stream to feed to browserify.
I want to use gulp to build bundles of JavaScript files.
For example I have the following structure in my project:
/vendor/vendor1/vendor1.js
/vendor/vendor2/vendor2.js
/js/includes/include1.js
/js/includes/include2.js
/js/bundle1.js
/js/bundle2.js
There are vendor includes (1-2), local includes (3-4), and bundle files (5-6).
Vendor includes are just third-party JavaScript libraries installed with bower or composer. They can be CommonJS, AMD or just a plain-old jQuery plugins.
I want to specify dependencies inside of a bundle files like this:
/js/bundle1.js
(function() {
// Vendor includes.
include('vendor1');
include('vendor2');
// Local includes.
include('includes/include1.js');
include('includes/include2.js');
// Some code here.
})();
I want gulp to process this source file and create a final distribution file (bundle) ensuring that all dependencies (includes) are merged together in a single file. So I can include foo.js from my HTML and all dependencies will be available to it.
I want to have a clear and robust system to manage all dependencies inside of a project and build distribution files.
How can I achieve this?
What format should I use for my own scripts (AMD, CommonJS, other)?
How do I specify dependencies in my source bundle files?
How do I build distribution?
Your question is posed as if there's a single answer, but there isn't. The problem you're trying to solve is one that many people have solved in many different ways, and you've identified two of the major options: AMD and CommonJS. There are other ways, but given that you might be new to Javascript dependency management as well as gulp, I'd recommend going with something that's relatively straightforward (even though this subject is inherently not straightforward).
I think the easiest route for you might be:
use CommonJS to express the dependencies
use browserify to resolve them into bundles
in browserify, use the "UMD" method so that you get a single bundle that will work for apps that use either AMD or CommonJS or are not using either of these dependency management systems
The statement in gulp to run browserify as such might look something like:
var browserify = require('gulp-browserify');
gulp.src('bundles/bundle1.js', {read: false})
.pipe(browserify({
'standalone': true
})
.pipe(rename('bundle1Output.js'))
.pipe(gulp.dest('dist'));
That should give you a dist/bundle1Output.js file.
There is a gulp plugin for this:
https://www.npmjs.com/package/gulp-include
It should do what you want, except that in your bundle file instead of this:
(function() {
// Vendor includes.
include('vendor1');
include('vendor2');
// Local includes.
include('includes/include1.js');
include('includes/include2.js');
// Some code here.
})();
You would have to write:
//=require vendor1/**/*.js
//=require vendor2/**/*.js
//=require includes/include1.js
//=require includes/include2.js
// Some code here
I am just getting use to CoffeeScript and I have gotten stuck with classes.
I want to have my files structured like in node so that I can require a JavaScript file containing a class like this.
Test = require "test.js"
Test.start()
Where start is a method of the Test Class.
Is this possible?
Is this possible?
Not exactly like in Node. There is no synchronous require in browser environments. Yet, you could try one of the many asynchronous libraries to do that, have a look at AMD. The most famous implementation is require.js.
I've found that the simplest way to use CommonJS modules (the ones that Node.js uses) in browser environments is to use Browserify. I personally also prefer the CommonJS module definitions to the the AMD ones, but that's just personal taste.
Also, take into account that in order to export your classes so that require 'test' will give you the class constructor directly, you would have to assign your class to module.exports:
# In test.coffee
module.exports = class Test
#start = -> console.log 'start!'
Then compile that file to test.js and you're ready to use it:
Test = require './test'
Test.start()
In Node.js this will Just Work. In browsers, you will need to process the files first with Browserify (or some other tool) to get it working (it will create the proper require function as well as some exports and module.exports variables for CommonJS modules to work correctly).
Take a look on stitch, hem (which is inspired by stitch, but has more neat features) and browserify.
Personally, I prefer hem. You can do something like this with it:
# app/lib/model.coffee
module.exports = class Model
...
.
# app/lib/util.coffee
helper1 = -> ...
helper2 = -> ...
module.export = {helper1, helper2}
.
# app/index.coffee
Model = require 'lib/model'
{helper1} = require 'lib/util'
# do whatever you want with required stuff
...
Hem takes care of compiling CoffeeScript on the fly and bundling all the needed code together (it also supports npm modules and arbitrary js libs as external dependencies for your code, see the docs for more details).