Multiple package.json files in one package - javascript

I have a site running on php that uses Node for a few development tasks. Currently, our team relies on puppet to install the node modules that we use for these development tasks, however, I was thinking of moving to using a package.json file in the root of the site instead.
My problem is this:
My php app is actually 3 separate sites that we have mixed into one (1 public facing site, 1 internal site, 1 client facing site) and its a likely scenario that some installations of the app depend on different node packages (or different versions of the same pacakge). While I know it would be nice to split these apps apart and have each one manage its own dependencies, short of this, is there some way to have multiple package.json files in one site?
In other words:
/package.json
/site1/package.json
/site2/package.json
/site3/package.json
Or perhaps, is there a way to have the one package.json handle dependencies for more than one app?
I don't know if this makes sense, let me know if I'm just way out in left field here. Thanks!

As far as I have seen, package.json deals with a single directory's dependencies (and a single instance of node_modules).
Your best bet is to split your applications into separate directories.
Barring that, you could just add all of the dependencies to a single package.json file. It will install into the node_modules folder and any app within that directory will have access to all of the installed modules.

Related

Why do we need multiple node modules folders inside each project?

I'm new to JS development and was wondering given that the size of node modules is an issue of concern where one has to even go on deleting the folder when many projects pile up and is not used, why can't we have a global node modules folder, like m2 for maven and then reference it in our projects?
Why do we need multiple node modules folders inside each project?
You can install packages globally and use them with the -g flag when using npm install.
But this isn't done often, and often isn't recommended, because the same package can have many, many different versions. SomeLibrary version 5.6.1 can be significantly different from SomeLibrary version 5.7.1. If you have multiple projects, they'll very often have different dependencies, and different dependency versions. If you try to source them all from the same global install, you'll usually run into problems very quickly. Having a separate node_modules folder for each project solves this problem for you; often, package versioning for each project you have will "just work" without any extra configuration.
There are also managers like Yarn which can cache package downloads, so you don't re-download huge numbers of duplicate megabytes over multiple projects
Variety of reasons. Just FYI, you can install npm packages globally using the -g flag when installing.
Here's some reasons:
Different project may require different versions of a package.
A single machine can consist of multiple different environment, i.e
containers or virtual machines.
Sharing code between projects/environments introduces potential security risks.
For the above reasons, and others, you might not actually want a projects
packages to leave the scope of that project.
since java script is not compiled, we will need to have all the plugins in each deployment verses maven is compiled so we can compile it to machine code with the plugin globally installed

npm installs many dependencies

I bought an HTML template recently, which contains many plugins placed inside a bower_components directory and a package.js file inside. I wanted to install another package I liked, but decided to use npm for this purpose.
When I typed:
npc install pnotify
the node_modules directory was created and contained about 900 directories with other packages.
What are those? Why did they get installed along with my package? I did some research and it turned out that those were needed, but do I really need to deliver my template in production with hundreds of unnecessary packages?
This is a very good question. There are a few things I want to point out.
The V8 engine, Node Modules (dependencies) and "requiring" them:
Node.js is built on V8 engine, which is written in C++. This means that Node.js' dependencies are fundamentally written in C++.
Now when you require a dependency, you really require code/functions from a C++ program or js library, because that's how new libraries/dependencies are made.
Libraries have so many functions that you will not use
For example, take a look at the express-validator module, which contains so many functions. When you require the module, do you use all the functions it provides? The answer is no. People most often require packages like this just to use one single benefit of it, although all of the functions end up getting downloaded, which takes up unnecessary space.
Think of the node dependencies that are made from other node dependencies as Interpreted Languages
For example, JavaScript is written in C/C++, whose functions and compilers are in turn originally written in assembly. Think of it like a tree. You create new branches each time for more convenient usage and, most importantly, to save time . It makes things faster. Similarly, when people create new dependencies, they use/require ones that already exist, instead of rewriting a whole C++ program or js script, because that makes everything easier.
Problem arises when requiring other NPMs for creating a new one
When the authors of the dependencies require other dependencies from here and there just to use a few (small amount) benefits from them, they end up downloading them all, (which they don't really care about because they mostly do not worry about the size or they'd rather do this than explicitly writing a new dependency or a C++ addon) and this takes extra space. For example you can see the dependencies that the express-validator module uses by accessing this link.
So, when you have big projects that use lots of dependencies you end up taking so much space for them.
Ways to solve this
Number 1
This requires some expert people on Node.js. To reduce the amount of the downloaded packages, a professional Node.js developer could go to the directories that modules are saved in, open the javascript files, take a look at their source code, and delete the functions that they will not use without changing the structure of the package.
Number 2 (Most likely not worth your time)
You could also create your own personal dependencies that are written in C++, or more preferably js, which would literally take up the least space possible, depending on the programmer, but would take/waste the most time, in order to reduce size instead of doing work. (Note: Most dependencies are written in js.)
Number 3 (Common)
Instead of Using option number 2, you could implement WebPack.
Conclusion & Note
So, basically, there is no running away from downloading all the node packages, but you could use solution number 1 if you believe you can do it, which also has the possibility of screwing up the whole intention of a dependency. (So make it personal and use it for specific purposes.) Or just make use of a module like WebPack.
Also, ask this question to yourself: Do those packages really cause you a problem?
No, there is no point to add about 900 packages dependencies in your project just because you want to add some template. But it is up to you!
The heavyness of a template is not challenging the node.js ecosystem nor his main package system npm.
It is a fact that javascript community tend to make smallest possible module to be responsible for one task, and just one.
It is not a bad thing I guess. But it could result of a situation where you have a lot of dependencies in your project.
Nowadays hard drive memory is cheap and nobody cares any more about making efficient/small apps.
As always, it's only a matter of choice.
What is the point of delivering hundreds of packages weighing hundreds of MB for a few kB project.
There isn't..
If you intend to provide it to other developers, just gitignore (or remove from shared package) node_modules or bower_components directories. Developers simply install dependencies again as required ;)
If it is something as simple as an HTML templates or similar stuff, node would most likely be there just for making your life as a developer easier providing live reload, compiling/transpiling typescript/babel/SCSS/SASS/LESS/Coffee... ( list goes on ;P ) etc.
And in that case dependencies would most likely only be dev_dependencies and won't be required at all in production environment ;)
Also many packages come with separate production and dev dependencies, So you just need to install production dependencies...
npm install --only=prod
If your project does need many projects in production, and you really really wanna avoid that stuff, just spend some time and include css/js files your your project needs(this can be a laborious task).
Update
Production vs default install
Most projects have different dev and production dependencies,
Dev dependencies may include stuff like SASS, typescript etc. compilers, uglifiers (minification), maybe stuff like live reload etc.
Where as production version will not have those things reducing the size node_modules directory.
** No node_modules**
In some html template kind of projects, you may not need any node_modules in production, so you skip doing an npm install.
No access to node_modules
Or in some cases, when server that serves exists in node_modules itself, access to it may be blocked (coz there is no need to access these from frontend).
What are those? Why did they get installed along with my package?
Dependencies exists to facilitate code reuse through modularity.
... do I need to deliver my template in production with hundreds of unnecessary packages?
One shouldn't be so quick to dismiss this modularity. If you inline your requires and eliminate dead code, you'll lose the benefit of maintenance patches for the dependencies automatically being applied to your code. You should see this as a form of compilation, because... well... it is compilation.
Nonetheless, if you're licensed to redistribute all of your dependencies in this compiled form, you'll be happy to learn those optimisations are performed by a compiler which compile Javascript to Javascript. The Closure Compiler, as the first example I stumbled across, appears to perform advanced compilation, which means you get dead code removal and function inlining... That seems promising!
This does however have another side effect when you are required to justify the licensing of all npm modules..so when you have hundreds of npm modules due to dependencies this effort also becomes a more cumbersome task
Very old question but I happened to come across very similar situation just as RA pointed out.
I tried to work with node.js framework using vscode and the moment when I tried to install start npm using npm init -y, it generated so many different dependencies. In my case, it was vscode extension ESlint that I added to prior to running npm init -y
Uninstalling ESlint
Restarted vscode to apply that uninstallation
removed previously generated package.json and node-modules folder
do npm init -y again
This solved my problem of starting out with so many dependencies.

is it ok to have more than one package.json in a project?

Is it ok to have more than one package.json in a project? I'm working in a .NET MVC solution and it has a package.json at the root level. I need to integrate Jasmine/Karma into the solution and this is my first time doing this type of integration. I found a sample project for Jasmine/Karma on the web and I was able to get this running locally. This project has it's own package.json.
It seems like it would be useful to maintain the package.json file for the sample Jasmine/Karma sample project separately from the package.json at the root level of the solution to provide more flexibility and to allow the same properties to be used differently based on context.
Would this be valid or generally considered an ok practice? Or do I need to figure out how to merge the contents of package.json from the sample project into the package.json at the root level of the .NET MVC solution?
In terms of keeping things simple, it is easier to only have to maintain a single package.json file, but there is nothing inherently wrong with multiple package.json files within a repo. Some companies do use mono-repos, for which it would make total sense to have multiple package.json files.
Multiple package.json files give you a lot of flexibility to run different/incompatible versions of dependencies. As a practical example, on one of the projects that I work on we have 2 package.json files, one for the main application code and then another one for our BDD tests. We use chimp for our BDD tests and had issues with running that on the latest node version, whereas we don't want to keep the rest of the app from upgrading just because of this. We also found that the single package.json file got very messy when it was combined at first.
I would say its typically bad practice to have more than one package.json. I would only expect to have to npm install once, and having to deal with two sets of dependency management could lead to issues down the line.

What is node_modules directory in AngularJS?

I am exploring AngularJS tutorial project and found it has node_modules directory inside, which size if 60 megabytes.
Does simple clientside javascript project really need so huge corpus of unknown data?
I tried to delete this directory and project still works. I suspect it somehow relates with node.js and it's npm but how? Suppose I need to run my project on some conventional web server (not node.js), then how to know, which files/directories are unneeded?
Many javascript libraries require to use bower to install them. If I use bower, does this mean I need to keep node_modules?
The node_modules directory is only for build tools.
The package.json file in the app root defines what libraries will be installed into node_modules when you run npm install.
Very often with an angular app, on your dev machine or on a build server, you use other Javascript libraries from npm (a node.js package manager) to build your angular app. Tasks can be concatenating resources, using CSS preprocessors like LESS or SASS, minification, replacing of values, etc. etc. The most common tools for managing and running these tasks are called grunt and gulp, which are installed through npm as well.
When you deploy your app, you only distribute the resulting build, not any of the source files or build tools.
It is of course possible to write an AngularJS app without building anything.
edit from comments: When you dive into Angular more, there are more advanced techniques of using libraries installed by npm even in the client app, you then selectively choose the ones you need, not the whole 50MB+ thing. I'd recommend staying with the basic approaches until you get a good grasp on them though.
NPM is the node package manager, which installs packages locally into a project, specifically, into the node_modules folder. From there the package code can be included into a project, yes, can is the important word there.
The browser has no way to include modules in code (yet), so you need to use a library that can expose node's commonJS style modules. Browserify and Webpack are two popular methods of doing so.
Angular complicates this by introducing its own module system, which more closely resembles AMD-style modules. There are ways around this, so that you can use node-style modules, maybe your project uses those.
Using npm for managing dependencies is a great idea, its a fantastic package manager. It is likely in your case though that the project is only built using node and that the node_modules folder contains dependencies only related to the build.

Is there any way to reorganize node_modules?

There is big and deep node_modules directory. And there are many sub-folders with the same modules that are located in different subdirectories. Sometimes the same versions of the modules, there are sometimes differences in minor versions.
Is there a tool for reorganization of node_modules to remove duplicates, put them in the root directory and somehow still a bunch of modules to optimize this?
The NPM hierarchy is actually fairly complex and they've done a lot of work to optimize it. The most you're losing here is a little disk space. If you really need to prune the package structure for your app then you can take a look at npm dedupe which is built right into npm and does exactly what you're asking for (consolidates duplicates as much as possible).
I know a lot of people are against checking in node_modules directory but in production applications we've found that checking in node_modules makes it so much easier to pin down fatal changes in our application. Sure you usually know that your application broke after the last npm update but if your app is rather large like ours is then that simply isn't granular enough to solve problems quickly and efficiently. So in large production applications (i.e. not libraries being published to NPM) I use npm dedupe to simplify the package structure before checking them in.
If you're writing code that others will consume (via NPM or otherwise) then checking in node_modules isn't the best idea and you should avoid doing so by adding node_modules to your version control system's ignore list. You should also make sure you're dependencies in package.json are specific version numbers with as few ranges as possible (please don't put asterisks in place of the version numbers in your production apps :/).
If you follow those basic patterns then you can just forget about the node_modules directory and let npm take care of you.

Categories