I'm dealing and wondering with context issues (and possibilities) in our microservice backend and I'm trying to better understand what exactly is shared across NodeJS forked cluster instances (for example using PM2).
The situation - lets assume I have an NPM package named that exports one instance of itself as in:
class Content {
constructor(){
if(!Content.Registry) Content.Registry = {};
}
registerStatic(key,obj) {
Content.Registry[key] = obj;
}
getStatic(key){
return Content.Registry[key];
}
}
module.exports = new Content();
and my Node application includes this module at several points and registers the instances of several objects using the static context methods.
Now, I know that the require registry from Node caches the instances when you require a package (or to maybe go with a better wording that suits the node documentation - it will 'probably' bring out the same cached instance).
Questions asked:
can I count on the package being cached? I require it as an NPM module and not as a file (require('package-name'))
if even though the instance returned from the require registry won't be the same - will the static context be the same across the same PID? (Content.Registry)
to complicate things a bit more - when running this whole thing over PM2 - will every forked child PID will have a different package instance AND static context? will the static context be preserved? I have already noticed that Node shares some resources (the sharing of sockets via IPC is documented, however we found that even file handles are shared when trying to write to the same file from different PM2 instances - logic says each one will try to acquire a lock handle on the file but only one will succeed. in reality they all write together - this was done using winston)
Thanks for any help!
EDIT: would there even be any way to test for this? could I somehow access within the NPM package some node/javascript functionality that would give me some UUID of the package's instance? memory address?
Related
I have been studying NodeJS for about a year and today I found something strange for me. We all know that in order to use a module (function, object or variable) in another module we must export and import it except for the native modules like String, Number, Promise, etc. I installed an external package for unit testing called Jest.
The strange thing here is that I have created two test modules called: logic.js and logic.test.js, in none I have imported the Jest module, however I can access all its methods. Let's show some code:
logic.js
module.exports.add = function(a, b){
return a + b;
}
logic.test.js
const lib = require('../logic')
test('Add - Should return the sum of two numbers', () => {
const result = lib.add(1,3);
expect(result).toBe(4);
});
As you can see in logic.test.js I have access to expect and test methods and i have not impoted nothing about Jest.
The questions here are:
How is this posible?
Its a good practice to do this with my modules
As Jonas W stated in the comments, they make use of the global variable that is common to all your application.
The use of the global variable is very simple
test.js
global.myObject = 'MyMan'
app.js
require('./test')
console.log(myObject)
Loading app.js will render MyMan
You might say that I actually import the test module and that Jest does not.
The thing is that you execute your node application using node yourFile.js but you instanciate your jests tests with the jest command line.
It's the jest command line that handles the binding between its framework (the expect and test methods.) and your script.
Is it a good practice?
I would say no. Except if you plan to make a library like Jest that have its own command line launcher and that you want to give tools like that to the users of your library.
The power of Node lives into the module organization, don't be afraid to use them.
I'm working on a multi-page site using AngularJS, and I want to write a utility that can be included in more than one page. I've looked at services and providers, and all the examples I find are single-page examples. I'm not sure how to generalize this to multiple apps used on different pages.
This is what I want to have for my two different pages/apps.
in app1.js:
var app1 = angular.module('app1',['myUtil'])
app1.controller('ctrl1',function ctrl1($scope,myUtil){...})
in app2.js:
var app2 = angular.module('app2',['myUtil'])
app2.controller('ctrl2',function ctrl2($scope,myUtil){...})
in myUtil.js:
??? Provider? Service? Module?
All the examples I have found for providers and services show them as being attached to a single app. Is this possible with AngularJS? If so, what am I missing?
The answer from zero298 is a nice answer as it's a great way of organising and reusing the utility module you create.
If you want a less broad and more "codey" answer, then one way of doing it would be to have some kind of utility module that houses whatever services you want to put in it, and then you can pass that in as a dependency for all apps that use it. This will all depend on your build process as to how you import/organise the files, but as a very basic example you could have a "utilsmodule" module with a "utils" service:
myUtils.js:
angular.module('utilsmodule', []);
// Service could be in another file
angular.module('utilsmodule').service('myutil', function() {
return {
myUtilFunction : function() {
return "This is from myutil";
}
};
});
Then in your app files you can pass in the module by name, which will give the app access to the 'myutil' service.
app1.js:
var app1 = angular.module('app1',['utilsmodule'])
app1.controller('ctrl1',function ctrl1($scope,myutil){...})
Then you would import the myUtils.js file before the app1.js file so that the "utilsmodule" module is registered with angular before your app is created. You do the same with app2 and the utility module should be available to both.
Example Plunker
This may be a bit too broad. However, what I would suggest you do is create a library module dedicated to the feature/utility that you want to make available to your projects.
I would suggest using npm to organize all of this. Give this feature module it's own package.json and add whatever code you need to make it run. In your consumer projects, add the library module as a dependency.
A good method to get this working locally (as well as quickly since you don't have to constantly push to the npm registry) is to use the npm link utility.
If your consumer projects are already npm oriented, the workflow would be as follows:
Create a new directory to contain your utility library module lets call it my-utility
cd to the new directory
npm init to create a package.json in this library
npm link to make the library available locally
cd to any of the consumer projects
npm link my-utility so that the consumer projects create a symlink to the local folder that contains your utility module
After that is setup, depending on how your consumer projects build or use dependencies, you can use your new utility library in your top level projects.
I am trying to get some of the NPM functionality into my Node.js programs. In particular, I would like to be able to analyze the available node modules on my system. "Module" here means "module identifier", thus either an identifier like "fd", or a file path; in other words, anything that can be put into a require() call and does load a module.
This question is split into three sub-problems:
1) Get a list of all core modules
2) Get a list of all loaded modules
3) Get a list of all installed and available modules.
Question 1 is answered by zeke's node-core-module-names list. Loading another module to find the core modules list is not elegant and it may outdate itself, but it is an option and does work. The (ordered) list is thus ['assert', 'buffer', 'child_process', ..., 'zlib'].
The 2. question can be answered by a call of Object.keys(require.cache), which returns a list of file paths.
What I cannot elegantly solve by now is the 3. question. There is the shell command npm ls which returns a tree graph. But is there anything usable and better?
Thank's for listening!
Tom
Here is something I have found tinkering around and I think this should be valid.
The V8 code has a standard set of bindings which you have seen in Node. They include but are not limited to:
fs
path
http
etc.
Also, there is a global variable by name of process. This exposes process level information and functionality, but also lets you get your hands on some V8 code through a function inside of the process variable called bindings.
The bindings(...) functions allows you to interface into exposed C++ libraries created by Node or you can create your own NodeJS modules by following the V8 developer guide (beyond the scope of this answer, read more here).
A funny little line I saw in the Node.cc file included a static check for checking for bindings for a keyword natives. This returns, it seems, a list of system level modules that you are looking for, and then some.
So that being said, I went into Node REPL and plugged in two lines (which I am sure can be shortened in a more elegant, expressive manner). Also note that I am pruning out anything starting with an underscore (_) so to preserve private functions or bindings:
var natives = process.binding('natives');
for (var key in natives) {if (key.indexOf('_') !== 0) {console.log(key);}}
npm list has various output options/flags, including json and parseable (which outputs a list of paths)
Try this:
var exec = require('child_process').exec;
var cmd = 'npm ls --json';
exec(cmd, function(error, stdout, stderr) {
var treeObject = JSON.parse(stdout);
});
The above requires no external packages but might need more code to work around the buffer limit: https://github.com/nodejs/node/issues/4236
Alternatively npm can also be used programmatically, perhaps through global-npm:
var npm = require('global-npm');
npm.load({}, function (err) {
npm.commands.list(null, function(err, treeObject) {
var firstLevelDependenciesArray = Object.keys(treeObject.dependencies);
});
});
Given
3 Node.js projects Main - Framework - Repositories
Main has the two other projects connected via npm link.
In a test i wrapped the require in a method. I've got some problems resolving linked projects (details see below)
Simplified code looks like this:
module.export.resolve = function(file){
[...]//Some more logik to handle relative pathes
return require(file)
}
This works fine in most scenarios. I also worked out to get handled with relatives pathes (looking up for caller and apply pathes based on this path)
Now this is in Project Framework which is linked (npm link) to Project Main. Project main has also Project Repositories linked.
Now in Project Main i have:
require('ProjectRepositories/foo') // Works as expected
myRequire.resolve('ProjectRepositories/foo') // Returns MODULE_NOT_FOUND "Cannot find module 'ProjectRepositories/foo'
I assume the problem is that Repositories Project ist not linked in the Framework Project. But is there an other way than linking them ?
I'd prefer to have less dependencies. Any hints on that?
You are absolutely correct in that the reason why the Project Framework resolve does not work is because the requireFn being used from within that project only knows about the modules installed in that framework. This is because when you require a javascript file, node evaluates the script within the context of the module, and not the context of the current project (this is how dependency modules' requires work when from your top-level script).
What you can do, however, is provide a way for the framework resolver to use a user-specified require function to do its work, once it has transformed the paths.
module.exports.resolve = function(file, resolver) {
//Some more logik to handle relative pathes
resolver = typeof resolver == 'function' ? resolver : require;
return resolver(file)
}
Now in your code, you could do
myRequire.resolve('ProjectRepositories/foo', require);
So now your Project Main require will be used to resolve the file.
You can also take this a step further if you want and have the module be stateful and remember the resolver it's supposed to use.
var _requireFn = require;
module.exports = {
resolve: resolve,
setRequireFn: setRequireFn
};
function resolve(path) {
return _requireFn(path);
}
function setRequireFn(requireFn) {
_requireFn = requireFn;
}
On another note, I would be careful about using the term resolve because in node that's semantically used for looking up the correct file path to be required, a la require.resolve.
Finally, in terms of minimizing dependencies, I'd recommend including your subprojects in npm using github repos. This has worked pretty well for me in the past, unless your two subrepos are in a constant state of flux. See the install docs for more info.
How can I use the modules loaded from other Node process from another Node process.
Example I run:
node my_modules
which load MyModule
then I will run another nodejs process:
node grab_modules
which will run GrabModule
GrabModule will attempt to use the functions inside MyModule
Is this possible? And if this is possible how?
What you want is probably dnode:
From the README of dnode:
The server (which hosts the functions to be run):
var dnode = require('dnode');
var server = dnode({
zing : function (n, cb) { cb(n * 100) }
});
server.listen(5050);
The client (which calls functions on the server and gets their results in a callback)
var dnode = require('dnode');
dnode.connect(5050, function (remote) {
remote.zing(66, function (n) {
console.log('n = ' + n);
});
});
It depends on what you are trying to do.
If you simply want to reuse the same module (MyModule) from two separate node processes, that's quite easy. You just need to put require('MyModule') in GrabModule and MyModule is accessible when you run grab_module.
If you want to 'share' MyModule including its global variables among two processes, it is much more complex. You need to define a inter-process protocol between two processes (typically REST over socket), and use that protocol to access the module in one process from another process.
1) To use a module (implementation) not an instance (module loaded somewhere of the process using require) in different process, you only need to require that module wherever you need.
If you run two process, for example, process A that use 'MyModule' and process B that use 'GrabModule', but you only need that 'GrabModule', in process B, can access to the exported properties of 'MyModule' then you only need to use require('path to MyModule').
2) On the other hand, if you need that a process B, can access to a module's state (a module that has been executed, because you use require in somewhere) of a process A, then you need to use a IPC (Inter-process communication) that allows to exchange data between process A and process B, and build or use the same protocol in both, over it.
Depending if your process are in the same machine or in different one, to can use some IPC build in the same OS, as nodejs offer with child fork (http://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options) or use an IPC built in some network channel.
For example, you can use the publish/subscribe messaging system of Redis (http://redis.io/topics/pubsub)
What about this:
my_modules will work as the program with public api (rest api, xml-rpc, ...)
and grab_modules will connect to api and call functions from my_modules
If you also require interoperability with other languages and/or high speed, ZeroMQ is also an option. While originally being a plain C library, I have had good experiences with the NodeJS bindings.
There a ZeroMQ binding for almost all popular languages, see http://zeromq.org/bindings:_start