We Keep Coding

Alternatives to structure and use javascript libraries without npm

We have been working on our projects all tied to a single lerna repository, as:
lerna
|----- app1 - application 1
|----- app2 - application 2
|----- appN - application N
|----- commondb (common database libraries for app1, app2 to appN)
|----- commonux (common ux libraries for app1, app2 to appN)
|----- commonauth (common authentication libraries for app1, app2 to appN)
As the code grew a lot, lerna is really full of packages (40+ packages) and too much code.
We're now trying to split lerna into smaller pieces and we're looking for alternatives. Doing that, applications would need a way to import common libraries as we do today.
Certainly NPM seens to be a solution (making each common package independent and publishing it on NPM), but we want to keep our code in our environment without third party services or clouds (we have our own git server instance).
What are the current options to manage javascript libraries that we can make use of? What would be the recommended one in such a scenario?

Your decision can be greatly affected by answering the following - do you want your apps to be running the same version of the shared libraries? Or do want autonomy within the libraries, and to be able to publish and manage different versions of the libraries, where it is the responsibility on the consumer app to manage which version of the library it is using?
If it is the former, my suggestion would be to stick with a Mono-repo approach, maybe consider something like NX, where it has some nice tooling for only linting, testing, building and deploying only the affected modules, whilst sharing a common single package.json and therefore common libraries shared across multiple apps and libs
Otherwise you are looking at potentially managing multiple repos, multiple versions of each library, multiple pipelines, multiple workspace configs.

You could simply use pnpm workspaces: protocol which allows you to create a monorepo structure but without even requiring you to publish when used this way: "foo": "workspace:*". So if you wish to use this approach, then you could have a monorepo structure and just keep local packages, it will use the local code without downloading anything. I used this concept in a Vue pnpm Workspace demo. With this approach you could stay with local packages and/or go with Lerna to publish if you also want to (I prefer Lerna-Lite which is what I use) since they now both support pnpm/yarn workspace: protocol. This approach is flexible and allows you to switch to publishing (to npm or a private registry) at any point in the time in the future if you wish to, in other words there's no major code refactoring to add Lerna after the fact when using a pnpm workspace structure.

npm installs many dependencies

I bought an HTML template recently, which contains many plugins placed inside a bower_components directory and a package.js file inside. I wanted to install another package I liked, but decided to use npm for this purpose.
When I typed:
npc install pnotify
the node_modules directory was created and contained about 900 directories with other packages.
What are those? Why did they get installed along with my package? I did some research and it turned out that those were needed, but do I really need to deliver my template in production with hundreds of unnecessary packages?

This is a very good question. There are a few things I want to point out.
The V8 engine, Node Modules (dependencies) and "requiring" them:
Node.js is built on V8 engine, which is written in C++. This means that Node.js' dependencies are fundamentally written in C++.
Now when you require a dependency, you really require code/functions from a C++ program or js library, because that's how new libraries/dependencies are made.
Libraries have so many functions that you will not use
For example, take a look at the express-validator module, which contains so many functions. When you require the module, do you use all the functions it provides? The answer is no. People most often require packages like this just to use one single benefit of it, although all of the functions end up getting downloaded, which takes up unnecessary space.
Think of the node dependencies that are made from other node dependencies as Interpreted Languages
For example, JavaScript is written in C/C++, whose functions and compilers are in turn originally written in assembly. Think of it like a tree. You create new branches each time for more convenient usage and, most importantly, to save time . It makes things faster. Similarly, when people create new dependencies, they use/require ones that already exist, instead of rewriting a whole C++ program or js script, because that makes everything easier.
Problem arises when requiring other NPMs for creating a new one
When the authors of the dependencies require other dependencies from here and there just to use a few (small amount) benefits from them, they end up downloading them all, (which they don't really care about because they mostly do not worry about the size or they'd rather do this than explicitly writing a new dependency or a C++ addon) and this takes extra space. For example you can see the dependencies that the express-validator module uses by accessing this link.
So, when you have big projects that use lots of dependencies you end up taking so much space for them.
Ways to solve this
Number 1
This requires some expert people on Node.js. To reduce the amount of the downloaded packages, a professional Node.js developer could go to the directories that modules are saved in, open the javascript files, take a look at their source code, and delete the functions that they will not use without changing the structure of the package.
Number 2 (Most likely not worth your time)
You could also create your own personal dependencies that are written in C++, or more preferably js, which would literally take up the least space possible, depending on the programmer, but would take/waste the most time, in order to reduce size instead of doing work. (Note: Most dependencies are written in js.)
Number 3 (Common)
Instead of Using option number 2, you could implement WebPack.
Conclusion & Note
So, basically, there is no running away from downloading all the node packages, but you could use solution number 1 if you believe you can do it, which also has the possibility of screwing up the whole intention of a dependency. (So make it personal and use it for specific purposes.) Or just make use of a module like WebPack.
Also, ask this question to yourself: Do those packages really cause you a problem?

No, there is no point to add about 900 packages dependencies in your project just because you want to add some template. But it is up to you!
The heavyness of a template is not challenging the node.js ecosystem nor his main package system npm.
It is a fact that javascript community tend to make smallest possible module to be responsible for one task, and just one.
It is not a bad thing I guess. But it could result of a situation where you have a lot of dependencies in your project.
Nowadays hard drive memory is cheap and nobody cares any more about making efficient/small apps.
As always, it's only a matter of choice.

What is the point of delivering hundreds of packages weighing hundreds of MB for a few kB project.
There isn't..
If you intend to provide it to other developers, just gitignore (or remove from shared package) node_modules or bower_components directories. Developers simply install dependencies again as required ;)
If it is something as simple as an HTML templates or similar stuff, node would most likely be there just for making your life as a developer easier providing live reload, compiling/transpiling typescript/babel/SCSS/SASS/LESS/Coffee... ( list goes on ;P ) etc.
And in that case dependencies would most likely only be dev_dependencies and won't be required at all in production environment ;)
Also many packages come with separate production and dev dependencies, So you just need to install production dependencies...
npm install --only=prod
If your project does need many projects in production, and you really really wanna avoid that stuff, just spend some time and include css/js files your your project needs(this can be a laborious task).
Update
Production vs default install
Most projects have different dev and production dependencies,
Dev dependencies may include stuff like SASS, typescript etc. compilers, uglifiers (minification), maybe stuff like live reload etc.
Where as production version will not have those things reducing the size node_modules directory.
** No node_modules**
In some html template kind of projects, you may not need any node_modules in production, so you skip doing an npm install.
No access to node_modules
Or in some cases, when server that serves exists in node_modules itself, access to it may be blocked (coz there is no need to access these from frontend).

What are those? Why did they get installed along with my package?
Dependencies exists to facilitate code reuse through modularity.
... do I need to deliver my template in production with hundreds of unnecessary packages?
One shouldn't be so quick to dismiss this modularity. If you inline your requires and eliminate dead code, you'll lose the benefit of maintenance patches for the dependencies automatically being applied to your code. You should see this as a form of compilation, because... well... it is compilation.
Nonetheless, if you're licensed to redistribute all of your dependencies in this compiled form, you'll be happy to learn those optimisations are performed by a compiler which compile Javascript to Javascript. The Closure Compiler, as the first example I stumbled across, appears to perform advanced compilation, which means you get dead code removal and function inlining... That seems promising!

This does however have another side effect when you are required to justify the licensing of all npm modules..so when you have hundreds of npm modules due to dependencies this effort also becomes a more cumbersome task

Very old question but I happened to come across very similar situation just as RA pointed out.
I tried to work with node.js framework using vscode and the moment when I tried to install start npm using npm init -y, it generated so many different dependencies. In my case, it was vscode extension ESlint that I added to prior to running npm init -y
Uninstalling ESlint
Restarted vscode to apply that uninstallation
removed previously generated package.json and node-modules folder
do npm init -y again
This solved my problem of starting out with so many dependencies.

How to know if a package from Node Package Management (NPM) is tested

I am relatively new to Node.js and NPM, and I have a kind of naive question. I would like to know if there is a way to know if package published on NPM is tested, and if there is away could we automate that process, and if there is tool or framwork that tell me this packages is tested. Also, does NPM require developers to provide a test for their packages.
Thank you

NPM is just a package manager. As they say in their site,
It's a way to reuse code from other developers, and also a way to
share your code with them, and it makes it easy to manage the
different versions of code.
NPM does not require developers to provide a test for their packages.
Best to use a package that has more stars and downloads cos that vouch for the package.
P.S: Never forget that a developer can pull his/her code from npm anytime :)

There is no way to know absolutely for sure, but usually a good indicator is if the author/maintainer has a test script set in the module's package.json. npm does not require modules to have tests.

NPM doesn't require package developers to write tests for their code.
To understand if a specific package is tested, the best you can do is browse the source code of the package: does it have tests? Just unit tests or other types like integration tests and the like? Are these tests ready to run with straightforward commands? Do these tests offer good code coverage of the package? Do they actually test relevant cases?
To automate a process that tells you if a package has been tested, this process will have to make multiple checks within the source code of the package, as there are multiple conventions on how to write, name and structure tests within a Node.js codebase (not to mention the amount of available testing frameworks that can be used). My concern with this approach is how complicated (if possible) would it be to automatically determine if a package is well tested, without actually having a human look at the tests.

What is the perfect workflow to work on A, B & C in parallel where A depends on B and B on C?

I wonder what is the perfect workflow if one needs to work on project A, B and C in parallel where A depends on B and B depends on C.
Currently I have everything in one repository which speeded up the early development. So my working directory looks like this:
/A/
/A/B
/A/B/C
So A is the project that is driving the development but it also means B and C are in parallel evolvement.
However, I want to release the projects B and C individually as well as they are quite useful for others.
However, I'm torn on how to do this without ruining my development speed. npm is great for distribution and dependency management but during development you definitely don't want to move temporally versions across the internet just to get the files to update in a different folder on your machine :)
On the other hand you also don't want to manually copy them over. Heck, all this I have to switch directories to work on B and now copy it over to /A/B is scary and seems error prone.
So, git submodule seemed like the answer as it's essentially enabling exactly that: You would keep your directory layout just as it is. Then when you make changes to files in B you can directly test them out in A without having to copy something over. And when you think it's ready you can just commit and push from the three different folders. Everything goes into three different repositories automatically. Seems like heaven yet everybody hates git submodule for various reasons.
I have pretty much the same problem when working on grunt plugins. I have the grunt plugin in it's own repository and then when I'm working on it I have to copy it over in one of the projects where I use it to drive the development. And then at the end of the day I copy it back to the grunt plugin working directory, make commits and push them. Thank god, I'm not the author of thousands of grunt plugins so I can deal with that but for this project I'm currently working on, I would definitely like to find a better solution.
So I wonder, what is the answer?

Note that Git submodules point to a specific version of the external repository, i.e. to a specific commit.
To update all Git submodules to the latest version, you’d still have to run a command:
git submodule foreach git pull origin master
Depending on your situation, you could possibly use npm instead of Git submodules. In that case, you simply list your dependencies in package.json, then run npm install in the root of the repository to fetch them. If a dependency is updated and a new version is published, you just run npm update again and it will match the version requirements set in the package.json file. You could also use npm to point to a specific commit, much like how Git submodules work:
{
"devDependencies": {
"dependency-a": "git://github.com/the-user-name/the-project-name.git#b3c24169432a924d05020c85f852d4a1736e66d4"
}
}
Or, if you want to use a bleeding edge version of a dependency, like the master branch of a given Git repository, you could use:
{
"devDependencies": {
"dependency-a": "git://github.com/the-user-name/the-project-name.git#master"
}
}

I've tried all number of these workflows and have decided that git submodules are not the answer. The main reason for me being that they couple projects together at a scm level and it is not intuitive as to where exactly in the revision history this coupling happens. Especially in a detached head scenario.
I rely on npm for depency managment with a combination of npm link and git URLs. npm link is great when I want to test something locally. git URLs are perfect to resolve dependencies during pull requests. Eventually I always want a published module with a version number I can match to a git tag for future reference and issue tracking. I use npm version to do this in one step. Allows for projects to depend on each other at varying levels of coupling during my development cycle.

Would using git subtree be the answer here? This article Alternatives To Git Submodule: Git Subtree did a good job explaining the pros and cons of using git subtree versus git submodules.

Node.js / npm - anyway to tell if a package is pure JS or not?

I've noticed that in trying to get seemingly simple node packages to install with npm (e.g. nerve, a "micro-framework") I often run into some form of dependency pain. After some digging, I tracked down the problem with nerve to the bcrypt module, which is apparently written in C/C++ and has to be compiled after the package manager downloads it.
Unfortunately, it seems like if you want this to work on Windows, the answer is (from one of the bcrypt issues threads) "install a Linux VM". So earlier today I did just that, and started running into other dependencies (you need certain unnamed apt packages installed before you can even think about building, despite GCC being installed), then eventually after seeing yet another C compiler error (about some package or other not being able to find "Arrays.c" I think), I actually gave up, and switched from nerve to express instead. Ironically, the larger and more complicated express installs with npm on Linux and Windows without a single issue.
So, my question is: is there any filter / dependency tracking available that lets you see if a package has additional dependencies besides node core? Because to me the allure of node is "everything in Javascript", and this kind of stuff dispels the illusion quite unpleasantly. In fact, despite having done more than my time working with C/C++, whenever I see a requirement to "make" something these days I generally run in the other direction screaming. :)

The first solution doesn't tell you if a dependency makes the package impure or not. Much better to search for gyp generated output:
find node_modules/ | grep binding.gyp || echo pure

Look out for the "scripts" field in the package.json.
If it contains something like
"scripts": {
"install": "make build",
}
and a Makefile in the root directory, there's a good possibility that the package has some native module which would have to be compiled and built. Many packages include a Makefile only to compile tests.
This check on the package documents does not exclude the possibility that some dependency will have to be compiled and built. That would mean repeating this process for each dependency in the package.json, their dependencies and so on.
That said many modules have been updated to install, without build on Windows, express for one. However that cannot be assured of all packages.
Using a Linux VM seems to be the best alternative. Developing Node.js applications on Window gives you step by step instructions on installing a VM, Node.js and Express.

Node is not "everything javascript" , since one way to extend node core is to write c/c++ plugins.
So Node is more a javascript wrapper around c/c++ modules using V8.
How could you write efficient database drivers in pure javascript for instance ? it would be possible but slow.
as for the filters , it is up to the author to document his package. there is no automatic filter.