With out libraries, I'm trying to learn data structures.
I have these dependencies
jquery.js->jqueryui.js
(underscores.js, jquery.js) -> backbone.js
Bascially, jqueryui depends upon jquery. Bacbkone depends on both underscore and jquery. Jquery and underscore are not related.
I want to create a dependency tree have you to "shed light" on these relations.
I was told this is how it is done on this posted question. Particularly this comment.
As long as you don't have a circular dependency you can always build a
dependency forest, which consists only of directed trees and/or sole
nodes. On the trees you can simply use DFS. You then start by adding
all roots or single nodes into a queue and add other resources to the
queue when their dependency has been loaded. (note that if a resource
has several dependencies you cannot model your dependencies as a
forest, but it stays acyclic and you can use a similar approach). –
Zeta
... so I do have resources with multiple dependencies, so I can't use a dependency forest.
...further discussion has suggested a directed acyclic graph.
A directed acyclic graph. Every path from the starting point can be
done in parallel, however if a node has more than one incident edges
you have to wait for all dependencies being loaded. By the way, I
would represent example 3 as P:[U-underscore, U-jquery]
S:[U-underscore, U-backbone-js] S:[U-jquery, U-backbone.js], showing
the original dependency, but they are equivalent
Can I use a dependency tree? If not what data structure is suggested to model complex dependencies...and and finally how do I implement it?
I believe that I have solved this problem, albeit its a long time ago. Let the dependencies be described like this:
Module A
Module X
Module Y
Module B
Module X
Module A
Module C
Module A
Module B
Which means that Module A depends on Module X and Module Y - and so forth.
Iterate over the leaves in this forest and for each leaf with no dependencies (look that up at the base), put the leaf in the load queue and remove it from the forest, so first pass yields:
Module A
Module B
Module A
Module C
Module A
Module B
Queue: Module X, Module Y.
Second pass:
Module B
Module C
Module B
Queue: Module X, Module Y, Module A.
...and so forth. Modules found in the same pass can be loaded in parallel, so a way to represent this could be:
[[Module X, Module Y], [Module A], [Module B], [Module C]]
Which means that Module X and Module Y should be loaded first and that they can be loaded in parallel. The rest must be loaded sequentially.
My biggest concern with the approach above is that it has complexity O(n^2). It should be possible to improve. Detecting cycles can easily be done with a lookup map.
Considering your asking about modular code without a library but using libraries as the modules (jQuery, jQuery-ui, etc.), It seems as this question is actually two questions.
How can you make sense of the many external libraries you have (sans depending on linear loading with script tags)
How do you implement modular dependencies
To answer the first one, it is a bit complicated. Most modular dependency designed require some wrapping around the modules to keep track of them. Most JS libraries do not use such systems in the assumption that they will be loaded via script tags (linear loading). Some systems such as require.js will provide a modified version to be compatible while others attempt to inject script tags into the page in a predefined order. The more popular solutions are to use a build tool that will concatenate the different library file you have into one large file that will order them in the correct order.
Most libraries are nice and will contain their code inside to prevent clobbering other scripts that also load. Even jQuery offers a noConflict() method to prevent the $ syntax from clobbering what other libraries might expect (for example Zepto).
The answer to handling dependencies with external libraries depends on either modifying the libraries to comply with what ever modular system you have or to have a external system such as a build environment that will (for lack of a better term) compile them into the correct order.
That's the bad news. The good news is code you control can in fact use a dependency tree that works very well. require.js is one prime example of doing this. Although a system such as CommonJS or AMD can be very powerful you can implement a simple dependency API on your own.
Taking from Rye.js as the example of how to implement your own modular system for your code:
(function(global) {
var module_list = {};
global.require = function (module) {
return module_list[module];
};
global.define = function (module, fn) {
modules[module] = fn();
};
})(this);
Then you can define your own modules:
define("hello_world", function() {
function helloWorld() {
alert("Hello World!");
}
return {
sayHello: helloWorld
};
});
Then in another module that depends on that one:
define("greetings", function() {
var hello = require("hello_world");
function sayAll() {
hello.sayHello();
alert("Good-Bye!");
}
return {
sayAll: sayAll
};
});
The data structure I've shown you in my previous answer,
deps = {
"U-jqueryui": ["U-jquery"],
"group1": ["U-underscore", "U-jquery"],
"U-backbone.js": ["group1"]
}
is represents a DAG:
U-jquerui U-backbone.js
| |
| v
| group1
| / |
v L v
U-jquery U-underscore.js
From this, you can extract a dependency tree, like
root
| |
v v
U-jquery U-underscore.js
for group1. The collection of all possible trees is then called a forest.
so I do have resources with multiple dependencies, so I can't use a dependency forest.
No, you don't. Having resources with multiple dependencies only means you need a tree instead of a queue. Where it become no (multi-/poly-) tree any more is when you have a diamond shape in your DAG diagram, for example if you had a module U-myapp that relies on U-jquerui and U-backbone.js - it relies on U-jquery "twice".
However, I don't think we really need to use any algorithms for a whole graph.
If not what data structure is suggested to model complex dependencies
The one I've already shown you - just modeling direct dependencies. It allows you to represent a DAG (including polyforests) even if you don't need it yet.
and and finally how do I implement it?
I'd go with memoized promises (sorry, I like them). Every promise represents the fact that a module has been loaded and executed; and we memoize (cache) them by module name so we don't start them multiple times. The code then is rather simple:
var modulepromises = {};
function loadAndExecute(name) {
if (name in modulepromises) // memoized?
return modulepromises[name]; // we're on it already!
var dependencies = [];
if (name in deps)
dependencies = deps[name].map(loadAndExecute);
// all promises for direct dependencies (indirect deps are included) combined
// will be fulfilled already if empty array
var depsExecuted = Promise.when(dependencies);
if (name.slice(0, 2) == "U-") // if a group, don't load anything
var file = ajaxPromiseForModulesource(name);
return modulepromises[name] = /*memoize!*/ depsExecuted.then(function() {
if (file)
return file.then(function(code) {
execute(code); // eval?
return "module "+name+" executed";
});
return "group loaded";
});
}
Related
I'm trying to create a build pipeline which doesn't bundle the files together, but instead uses <script type="module">. This will let me just recompile files as they change without rebundling, greatly improving build times during development.
Our project uses ES6, so this is generally easy.
There is however one snag: third-party modules that only have CommonJS builds (such as react).
There are a few ways around this. For now, I have a transform that changes the import name from react to /node_modules/react and my server is smart enough to then go find the appropriate dist file from node_modules and serve it up. This all works fine.
The problem is that it gets confused when I try to do something like:
import { Component } from 'react';
That won't work how it currently is (because it gets confused by there not being a default). However, this will work:
import * as React from 'react';
const { Component } = React;
I could manually do this for all files and packages, but a) that would make it unnecessarily ugly (with Redux and other things, there are half a dozen different packages in many files we'd have to do this to and b) there are lots of files, I don't want to manually change them all.
Is there a Babel transform plugin that can automatically make this kind of conversion? It seems that this isn't a completely novel approach, so I'm hoping there is a plugin that'll do it for me that my Google-fu failed to find.
I managed to get this working. I ended up having to write my own Babel plugin which makes the changes that I was looking for. Basically, for the different versions of the import statement, it changes it around in order to work better with UMD and CJS modules.
For example, something like this:
import A, { a, b, c } from 'a';
was transformed into something like this:
import * from __babel_A from 'a';
const A = __babel_A.default && (__babel_A.default.default || __babel_A.default) || __babel_A;
const { a, b, c } = __babel_A.default || __babel_A;
This format came about based on the way most things export. Many (like React) would put everything into an object with it's name on either module.exports (if I provided a fake one) or this or window. Others ended up not grouping things together (like ReactRTE) and had its own "default", which is where the default.default bit came from.
In addition to this transform, when my server serves up the dist versions of the third-party files, it'd wrap them up in a way that would then let me do an export default. That looked like this:
const module = {};
const keys = Object.keys(this || window);
const toExport = (function __auto_wrapper() {
${fileCode}
return module.exports || Object.keys(this).reduce((r, k) => keys.includes(k) ? r : Object.assign(r, { [k]: this[k].default || this[k] }), {});
}).call(this || window);
export default Object.keys(toExport).length === 1 ? Object.entries(toExport)[0][1] : toExport;
Again, the different ways are based on what the different projects output. Some will use module.exports when given it, others won't consider module.exports a real thing since I (purposely) didn't initialize export as an object (if I did, it'd try to use require() which I didn't want). In my case, ReactRTE had just a CJS module instead of an UMD, so I also had to do replace on its code to replace require('react') and require('react-dom') with references to the objects on window instead.
This all works as I wanted (completely unbundled code loading in the browser). The only slight side-effect is that React and friends are all available on window (which they normally wouldn't be if bundled properly), but that's pretty minor.
I'm finding some problems when loading certain .js module via require.js
Problem is: I only need to load certain module in some pages, not the entire website. Therefore I'm placing code this way:
if($('.module-news').length > 0 + $('.module-ticket-euromillones-multiple').length + $('.module-news-page').length > 0) {
require('_modules/news-grid').init();
}
This code search in the HTML if a class="module-news" exists (news page). If so, then load the module with the javascript.
That is NOT working. The IF is evaluating correctly, but module news-grid is ALWAYS loading no matter .module-news exists or not.
I found out that if I change the string of the module path for a variable, then requires behaves correctly, but that makes no sense. This how, following code works:
var name = "_modules/news-grid";
if($('.module-news').length > 0 + $('.module-ticket-euromillones-multiple').length + $('.module-news-page').length > 0) {
require(name).init();
}
Is this a known issue with require? Am I missing something? (maybe in the requirejs.config settings?
Help appreciated
Solution
You should be using the regular RequireJS idiom for calling require:
if (...) {
require(['_modules/news-grid'], function (news_grid) {
news_grid.init();
});
}
Why It Fails
You are using RequireJS' feature which lets you write require calls in the CommonJS format var module = require("module name"). You cannot use it for loading modules conditionally.
The call require('module name') (with a string as the first argument rather than a dependency array) is a convenience. It will return a module but if and only if the module is already loaded. If the module is not already loaded, then the call will fail. The only reason you don't have to worry about pre-loading the modules before calling require('module name') is that RequireJS does it for you.
define(function (require) {
var foo = require('foo');
});
is interpreted by RequireJS as:
define(['require', 'foo'], function (require) {
var foo = require('foo');
});
It scans the function for calls to require in the CommonJS idiom and generates a list of modules that it loads before executing the module. Conditionals are completely transparent to this process. Any require call with a string as the first parameter will be detected and the module it means to load will be added to the dependencies loaded before define's callback is called...
RequireJS won't detect those cases where something else than a string is passed to require. If you write:
var name = "foo";
var foo = require(name);
RequireJS will not know that you want to load foo and will not add foo to the list of dependencies. Is this a solution? No, because remember what I said earlier require('module name') will return a module, and not fail, only if the module is already loaded.
If you want your module to be loaded conditionally, then abandon the CommonJS idiom.
I'm using RequireJS to manage an app that is implemented as an undetermined number of classes split across files.
The app has some some utility classes, and then some consumer classes which use the utility classes. Each consumer class uses RequireJS to load only the utility classes it needs, and is responsible for its own Require config.
Consumer classes aren't aware of each other, and the app as a whole doesn't know the needs of each consume class. It merely knows the consumer class exists, and loads it. The problem is, if two different consumer classes define the same path but with different names, the first one wins, and subsequent one times out.
For example, this is some utility.js
define( function(){
return function(msg){
console.log(msg);
};
});
This is consumer1.js
require.config({
paths: {
ut1: '/utility'
}
});
require(['ut1'], function(utility){
utility('hello from client1');
})
And this is consumer2.js
require.config({
paths: {
ut2: '/utility'
}
});
require(['ut2'], function(utility){
utility('hello from client2');
})
If used independently each consumer class works, but when called together, the second one returns an error "Load timeout for modules: ut2".
As part of the modular nature of the app design, consumer classes cannot know each other's content, so I cannot enforce naming rules, nor can I have a central app-level require.config. How can I stop one consumer from breaking another's require config?
I supposed I could write a global helper that managed names and paths, grouping common paths and returning the first path name only, but I'd prefer a simpler solution within Require itself.
You should use the map option:
map: {
consumer1: {
ut1: '/utility'
},
consumer2: {
ut2: '/utility'
}
}
And then you don't need a paths configuration for /utility. What this does is tell RequireJS that when consumer1 requires ut1, it should load the /utility module instead. When consumer2 requires ut2, RequireJS should load the /utility module instead.
I'm trying to include IOUtil.js and ChannelReplacement.js in my add-on, using the Cu.import(...) function. These two both use xpcom_generateQI, which I'm trying to obtain from the XPCOM jsm, but the two scripts cant access it.
const {Cc, Ci, Cu, Cr} = require("chrome");
Cu.import("resource://gre/modules/XPCOMUtils.jsm");
const xpcom_generateQI = XPCOMUtils.generateQI;
Cu.import(self.data.url("IOUtil.js"));
Cu.import(self.data.url("ChannelReplacement.js"));
gives me xpcom_generateQI is not defined.
How do I access a function which is defined in main.js?
Issues
Don't use Cu.import for local SDK modules. Don't write JS code modules for SDK add-ons, the SDK uses CommonJS-style modules together with the require() facility which also comes with proper cleanup for free, which cannot be said for JS code modules and Cu.import (you'd need to properly Cu.unload everything and likely kill some references yourself).
That https-everywhere stuff are neither JS code modules nor SDK modules, but uses the subscript loader. Either convert it to SDK code modules, or use the subscript loader yourself.
It is OK to import built-in JS Code modules in different scopes/modules. There is not actually a need to make available xpcom_generateQI from main (although it can be done; well, get to that).
To be future proof, you should bind your xpcom_generateQI shortcut properly, as in XPCOMUtils.generateQI.bind(XPCOMUtils). Otherwise, if the implementation changes and requires a proper this, your stuff will break.
To export something from any CommonJS module, you need to put it into the exports module. See the first link.
To import something, use require() (first link again).
Be aware of circular references, where Module A imports Module B imports Module A. Right now this kinda works (but only kinda, because some stuff might not be available from Module A when Module B imports it like this, as Module A is not fully loaded). Better avoid it.
Example 1 (circular)
So here is a example with circular require (main imports modules imports main)
main.js
function someFunction() {
console.log("some function called");
}
exports.someFunction = someFunction;
var mod = require("./module");
mod.anotherFunction();
module.js
const { someFunction } = require("./main");
exports.anotherFunction = function() {
someFunction();
}
Now, because of circular references this is a fragile construct. If works right now, but when modules get more complex or the SDK changes, it might break... Better put someFunction into a third module.
Example 2 (avoiding circular imports)
main.js
var mod = require("./module");
mod.anotherFunction();
// Or call someFunction directly
var { someFunction } = require("./utils");
someFunction();
module.js
const { someFunction } = require("./utils");
exports.anotherFunction = function() {
someFunction();
}
utils.js
function someFunction() {
console.log("some function called");
}
exports.someFunction = someFunction;
There are no circles anymore. If you wanted to reuse xpcom_generateQI, you'd put it as a property of exports in utils.js (in this example) and later require it with require("./utils").
https-everywhere
The https-everywhere stuff needs to be either converted or loaded using the subscript loader. I would recommend against the subscript loader, because in all likelihood the verbatim https-everywhere code does not clean up after itself. I'd actually also recommend against just converting it by throwing some stuff in (exports.xzy = ...). This code is not meant to be run in the SDK. Better create your own implementation and just borrow ideas from https-everywhere where needed.
I'm trying to load underscore.js with require.js like this:
require(["libs/underscore-1.2.3.js"], function(_) {
...
});
But this doesn't work because underscore.js exports a module name: define('underscore', function() { ... }).
Without renaming lib/underscore-1.2.3.js, how can I load it with require.js?
Alright, after some more googling, I've found: https://github.com/documentcloud/underscore/pull/338#issuecomment-3245213
Where
#dvdotsenko all AMD loaders allow mapping a module ID to a partial path, usually the configuration is called 'paths', so to do what you want:
requirejs.config({
paths:
underscore: 'js/libs/underscore-1.2.3.min'
}
});
require(['underscore'], function () {});
Since underscore is used by other higher-level modules, like backbone, a common dependency name needs to be used to communicate a common dependency on underscore, and it makes sense to call that dependency 'underscore'. The paths config gives a way to do the mapping to a specific URL you want to use for that dependency.
This doesn't answer my question (ie, I still don't know how I'd go about loading underscore if all I had was a URL), but at least it's a functional workaround.
While this doesn't strike me as the most ideal solution, you can require your external files, and then require their registered module names in the inner block.
JSFiddle Example
require(
['require','http://documentcloud.github.com/underscore/underscore-min.js'],
function(require){
require(['underscore'],function(_){
var a = _.intersection([1,2,3],[2,3,4]);
document.write("Underscore is available in the closure : " + a);
})
}
)
It might not look pretty, but that might be a recommended pattern for loading up initial assets so that they can be required intuitively in dependent modules.