I just recently heard of the package.json file but all my small projects have always worked without it.
I've installed many npm modules globally and always use var x = require("express");"just for example" without even initializing the package.json and putting express as a dependency.
Is it really important
First of all, I strongly doubt require("express") will work out of the box without having the express package installed in your project's node_modules folder, but just globally. There are ways to accomplish this, though.
But even if you accomplished to require packages from the global installation folder, what happens if you have multiple packages requiring different versions of the same package? There may be breaking changes between major versions of packages. So if you install a new version of package xy globally, older projects of yours expecting a different version of package xy may stop working.
On just your local machine, you can achieve all that, still without a package.json though.
The main purpose of the package.json comes clear, when you want to share your project with other people. Aside from some basic information (like a package name and some description), it will also list the dependencies which need to be installed for the project to be runable. While the necessary dependencies may exist on your machine, how will you make sure, they are also installed on a different machine without having them listed somewhere? And the place for listing the dependencies of a package is -- surprise surprise --- the package.json file ...
They are global, not in your project so when you do the deploy, you will must have to install all global for each server.
Yuu can install packages-cli global, but project dependencies ( also dev on dev dependencies) is better have its own package.json so you can deploy.
Also if you share your project, how someone will know what packages is needed.
The better is to have for each project its own package.json on its root folder, even if you always use the same libs.
I'm developing a web app which uses a private package.
In some circumstances, I rather to use the local version of the package or different version of that package.
Is there any solution to indicate to use different version of package in package.json?
i.e:
npm install --local
while my package.json looks like:
...
"dependencies": {
...
"my_package": if(local) "address_to_local_package/" else "5.6.1"
...
}
npm does not accommodate this (and honestly, probably should not). This seems like the type of thing that is usually handled at runtime via the NODE_ENV environment variable or similar mechanism. The module changes behavior depending on whether NODE_ENV is set to "production" or "development". (But that's just convention. You can use values like "local" if you want.) So rather than installing different versions of the module, there's a single version that behaves differently based on the value of that environment variable.
If you really want different code bases installed entirely, it will take some effort but you can write a postinstall script for npm to run. Your module then becomes nothing more than a script and then the postinstall figures out what to actually install based on environment variables or maybe a command line flag. This seems brittle to me, though. I'd think hard about if you're solving the right problem here if you go this route. NODE_ENV seems more elegant and conventional.
I bought an HTML template recently, which contains many plugins placed inside a bower_components directory and a package.js file inside. I wanted to install another package I liked, but decided to use npm for this purpose.
When I typed:
npc install pnotify
the node_modules directory was created and contained about 900 directories with other packages.
What are those? Why did they get installed along with my package? I did some research and it turned out that those were needed, but do I really need to deliver my template in production with hundreds of unnecessary packages?
This is a very good question. There are a few things I want to point out.
The V8 engine, Node Modules (dependencies) and "requiring" them:
Node.js is built on V8 engine, which is written in C++. This means that Node.js' dependencies are fundamentally written in C++.
Now when you require a dependency, you really require code/functions from a C++ program or js library, because that's how new libraries/dependencies are made.
Libraries have so many functions that you will not use
For example, take a look at the express-validator module, which contains so many functions. When you require the module, do you use all the functions it provides? The answer is no. People most often require packages like this just to use one single benefit of it, although all of the functions end up getting downloaded, which takes up unnecessary space.
Think of the node dependencies that are made from other node dependencies as Interpreted Languages
For example, JavaScript is written in C/C++, whose functions and compilers are in turn originally written in assembly. Think of it like a tree. You create new branches each time for more convenient usage and, most importantly, to save time . It makes things faster. Similarly, when people create new dependencies, they use/require ones that already exist, instead of rewriting a whole C++ program or js script, because that makes everything easier.
Problem arises when requiring other NPMs for creating a new one
When the authors of the dependencies require other dependencies from here and there just to use a few (small amount) benefits from them, they end up downloading them all, (which they don't really care about because they mostly do not worry about the size or they'd rather do this than explicitly writing a new dependency or a C++ addon) and this takes extra space. For example you can see the dependencies that the express-validator module uses by accessing this link.
So, when you have big projects that use lots of dependencies you end up taking so much space for them.
Ways to solve this
Number 1
This requires some expert people on Node.js. To reduce the amount of the downloaded packages, a professional Node.js developer could go to the directories that modules are saved in, open the javascript files, take a look at their source code, and delete the functions that they will not use without changing the structure of the package.
Number 2 (Most likely not worth your time)
You could also create your own personal dependencies that are written in C++, or more preferably js, which would literally take up the least space possible, depending on the programmer, but would take/waste the most time, in order to reduce size instead of doing work. (Note: Most dependencies are written in js.)
Number 3 (Common)
Instead of Using option number 2, you could implement WebPack.
Conclusion & Note
So, basically, there is no running away from downloading all the node packages, but you could use solution number 1 if you believe you can do it, which also has the possibility of screwing up the whole intention of a dependency. (So make it personal and use it for specific purposes.) Or just make use of a module like WebPack.
Also, ask this question to yourself: Do those packages really cause you a problem?
No, there is no point to add about 900 packages dependencies in your project just because you want to add some template. But it is up to you!
The heavyness of a template is not challenging the node.js ecosystem nor his main package system npm.
It is a fact that javascript community tend to make smallest possible module to be responsible for one task, and just one.
It is not a bad thing I guess. But it could result of a situation where you have a lot of dependencies in your project.
Nowadays hard drive memory is cheap and nobody cares any more about making efficient/small apps.
As always, it's only a matter of choice.
What is the point of delivering hundreds of packages weighing hundreds of MB for a few kB project.
There isn't..
If you intend to provide it to other developers, just gitignore (or remove from shared package) node_modules or bower_components directories. Developers simply install dependencies again as required ;)
If it is something as simple as an HTML templates or similar stuff, node would most likely be there just for making your life as a developer easier providing live reload, compiling/transpiling typescript/babel/SCSS/SASS/LESS/Coffee... ( list goes on ;P ) etc.
And in that case dependencies would most likely only be dev_dependencies and won't be required at all in production environment ;)
Also many packages come with separate production and dev dependencies, So you just need to install production dependencies...
npm install --only=prod
If your project does need many projects in production, and you really really wanna avoid that stuff, just spend some time and include css/js files your your project needs(this can be a laborious task).
Update
Production vs default install
Most projects have different dev and production dependencies,
Dev dependencies may include stuff like SASS, typescript etc. compilers, uglifiers (minification), maybe stuff like live reload etc.
Where as production version will not have those things reducing the size node_modules directory.
** No node_modules**
In some html template kind of projects, you may not need any node_modules in production, so you skip doing an npm install.
No access to node_modules
Or in some cases, when server that serves exists in node_modules itself, access to it may be blocked (coz there is no need to access these from frontend).
What are those? Why did they get installed along with my package?
Dependencies exists to facilitate code reuse through modularity.
... do I need to deliver my template in production with hundreds of unnecessary packages?
One shouldn't be so quick to dismiss this modularity. If you inline your requires and eliminate dead code, you'll lose the benefit of maintenance patches for the dependencies automatically being applied to your code. You should see this as a form of compilation, because... well... it is compilation.
Nonetheless, if you're licensed to redistribute all of your dependencies in this compiled form, you'll be happy to learn those optimisations are performed by a compiler which compile Javascript to Javascript. The Closure Compiler, as the first example I stumbled across, appears to perform advanced compilation, which means you get dead code removal and function inlining... That seems promising!
This does however have another side effect when you are required to justify the licensing of all npm modules..so when you have hundreds of npm modules due to dependencies this effort also becomes a more cumbersome task
Very old question but I happened to come across very similar situation just as RA pointed out.
I tried to work with node.js framework using vscode and the moment when I tried to install start npm using npm init -y, it generated so many different dependencies. In my case, it was vscode extension ESlint that I added to prior to running npm init -y
Uninstalling ESlint
Restarted vscode to apply that uninstallation
removed previously generated package.json and node-modules folder
do npm init -y again
This solved my problem of starting out with so many dependencies.
meteor/packages seems completely redundant considering there is packages.json. What does meteor/packages do then? Why should I not delete it?
It's unclear in your question whether you are asking about the file .meteor/packages or the packages/ sub-directory in your project folder. So here an explanation of both:
.meteor/packages stores the list of meteor (atmosphere) packages you have added to your project using meteor add; the corresponding versions are stored in .meteor/versions. This file is needed, e.g., for collaboration: by adding this file to version control you tell others which packages need to be installed, i.e., avoid the need for them to also run meteor add, etc.
packages/ stores local packages. This is only needed if you have or want local packages, e.g., when you have developed packages yourself that you did not publish on atmosphere, or packages you are actively working on. Now that meteor uses ES6 and supports ES6 modules, there is less of a need for local packages, but in the past it was very useful to be able to encapsulate certain behaviors into a package (which you'd now do into a class). Of course, if you want to use the same component in multiple meteor apps, packages (local or not) are still the way to go.
Hope this helps.
I'm developing a node module my-module which in turn depends on another module other-module. other-module is thus a dependency explicitly listed in my module's package.json.
As my module modifies the behaviour of other-module just by being required, it is important that other-module is loaded only once and that this, one-and-only 'instance' is the one referenced throughout any application that requires both my and other.
I expected this to hold true according to node's Module Caching Policy but what I've come across while writing a simple test app is this:
If my-module is npm installed before other-module then the latter is brought in as a dependency of the former. npm installing other-module afterwards brings it into the node_modules hierarchy a second time. Then, when my module requires other-module, node loads my module's 'local' copy and when the app requires it a second time node loads it again, (this time the version that was installed due to the second npm install). This obviously is not the intended result.
If my-module is npm installed after other-module then I end up with only one copy of other-module in node_modules and my test app works as expected.
This behaviour got me looking through node's relevant policies again and sure enough I came across the 'Module Caching Caveats':
Modules are cached based on their resolved filename. Since modules may resolve to a different filename based on the location of the calling module (loading from node_modules folders), it is not a guarantee that require('foo') will always return the exact same object, if it would resolve to different files.
At this point it looks like that my module may or may not behave as expected depending on the order of npm installs.
Are there any best practices I'm missing? Is there any way to avoid this mess without changing the way my module works?
The short answer: You can not.
As you pointed out node will load the required module from the most local place. This is as far as I know unique to a package manager and it enables you to don't care about the exact dependency tree of your modules. Node and npm will figure that out for you. In my opinion this is something really good.
Dependency hell is simply avoided by giving your modules the opportunity to require an exact version of what they need.
I think what you are trying to do, unless I do not understand your question fully, is not good node practice. Modules are loaded and assigned to a local variable. Global state should be avoided, as this can lead to rather awkward and untestable code. Additionally if you would succeed to inject your modified module into other's people code, there could be no guarantee that their code would still work. This would be as in the old Prototype.js_ days when it was ok to hack around with JavaScript's built-in globals like String or Array, which led to some disastrous code.
However keep in mind that this writing is just the opinion of one person. If you don't find more answers here, post your question otherwhere like the node's IRC channel.
I had the similar problem while developing tests with jest.
The following statement would allow you to load the same module again in different context:
jest.resetModules();