Often while using npm I've come across errors that appear to mean nothing - Visual Studio projects failing to build, build tools (eg: python.exe / CL.exe) not being available on the command line etc.
Some examples of packages I've seen fail to build many times:
kerberos
node-gyp
bcrypt
These throw big error messages with stack traces etc to the console during npm install, clearly having failed completely; however, NPM carries along happy as Larry and 9 times out of 10 my Javascript application and all its dependencies work fine.
Does npm install re-build every single dependency recursively, using whatever compilers are available on the local machine?
If so, and considering the huge number of dependencies even simple packages can have, how am I able to do ANYTHING without a full suite of programming languages and compilers installed?
Why is it that these dependencies failing doesn't necessarily mean my final project will be unusable?
If a dependency failing to build is "ok", why bother having the dependency at all?
I haven't been able to find clear answers on any of this, due to the overwhelming number of resources found when searching for terms like "npm build fail".
npm will succeed if those dependencies are actually marked as optional. The ws module is an example of this where they have optional dependencies on two compilable addons. If they fail to build, then ws just uses pure js fallback implementations.
The reason that addons are sometimes added as optional dependencies is that the they (more often than not) perform faster than the pure js alternatives, even for something as "simple" as UTF-8 validation or XOR'ing the contents of a Buffer.
Related
I have been following various blogs and videos on setting up and configuring eslint and prettier for vscode and development. But every article fails to explain why do we need to separately install eslint as an npm package and vs code extension?
what difference it will make if I install either of the ones?
why do we need to separately install eslint as npm package and vscode extension?
Short answer: you don't.
Long answer:
Installing ESLint/Prettier as extension, allows you to format/check your code inside the VSCode.
However, installing them also as dependencies brings extra benefits:
VSCode will use exact same package as installed. So you will not spot the situation when VSCode says OK, but your CI server says: NOT OK
You will get control over the versions, and can update whenever you want
You will be able to use different versions for different projects. This is especially important, when you can't migrate old project, but want to use the latest possibilities for the new one
You will be able to access Prettier/ESlint through the script block of the package.json, and be able to write custom commands with parameters exactly as you need
You will be able to pair them with Husky or NPM hooks to automatically verify/format the code
From my experience, if you can install something locally - install it as package dependency (except CLI like create-react-app or angular-cli that helps you start the app). This will make your life a bit predictable.
These programs can format your code (ESLint and Prettier) and detect specific syntax (ESLint).
When installed as an extension in your IDE (vscode for example), you can get:
squiggly lines in real time;
and format-on-save.
But someone who starts up your project on their own environment might not have these extensions installed (might not even have the same IDE) and thus might not get these.
When installed as npm packages (and included somewhere in the pipeline, either in the npm start, or in your continuous deployment, or...)
you won't get real time squiggly lines,
but you can still get auto-formatting (though not necessarily on save, depending on the configuration),
you can get blocking rules (meaning instead of just seeing errors / warnings, you can actually block the pipelines until the dev fixes said errors / warnings)
you can insure anyone who starts the project, from any IDE, gets the packages included
Regarding Zero Installs, the Yarn 2 documentation says:
While not a feature in itself, the term "Zero Install" encompasses a lot of Yarn features tailored around one specific goal - to make your projects as stable and fast as possible by removing the main source of entropy from the equation: Yarn itself. [...]
I read the whole story, but didn't really understand that fully.
What is the difference between Yarn 2 Zero Installs and Yarn 2 normal install?
The difference is that using Yarn normal install, you won't need to commit the node_modules (all your deps), whereas, using Zero-Install you will have to take care of all your dependencies.
This certainly makes your dependency on remote repositories less, however, it requires more responsibility, as said in the docs:
Note that, by design, this setup requires that you trust people
modifying your repository. In particular, projects accepting PRs from
external users will have to be careful that the PRs affecting the
package archives are legit (since it would otherwise be possible to a
malicious user to send a PR for a new dependency after having altered
its archive content).
After all, zero-install is a great feature. It solves the "I cloned/switched branch and now a dependency is missing" problem, it speeds up CI significantly and it lowers our dependence on our on-prem npm registry.
I bought an HTML template recently, which contains many plugins placed inside a bower_components directory and a package.js file inside. I wanted to install another package I liked, but decided to use npm for this purpose.
When I typed:
npc install pnotify
the node_modules directory was created and contained about 900 directories with other packages.
What are those? Why did they get installed along with my package? I did some research and it turned out that those were needed, but do I really need to deliver my template in production with hundreds of unnecessary packages?
This is a very good question. There are a few things I want to point out.
The V8 engine, Node Modules (dependencies) and "requiring" them:
Node.js is built on V8 engine, which is written in C++. This means that Node.js' dependencies are fundamentally written in C++.
Now when you require a dependency, you really require code/functions from a C++ program or js library, because that's how new libraries/dependencies are made.
Libraries have so many functions that you will not use
For example, take a look at the express-validator module, which contains so many functions. When you require the module, do you use all the functions it provides? The answer is no. People most often require packages like this just to use one single benefit of it, although all of the functions end up getting downloaded, which takes up unnecessary space.
Think of the node dependencies that are made from other node dependencies as Interpreted Languages
For example, JavaScript is written in C/C++, whose functions and compilers are in turn originally written in assembly. Think of it like a tree. You create new branches each time for more convenient usage and, most importantly, to save time . It makes things faster. Similarly, when people create new dependencies, they use/require ones that already exist, instead of rewriting a whole C++ program or js script, because that makes everything easier.
Problem arises when requiring other NPMs for creating a new one
When the authors of the dependencies require other dependencies from here and there just to use a few (small amount) benefits from them, they end up downloading them all, (which they don't really care about because they mostly do not worry about the size or they'd rather do this than explicitly writing a new dependency or a C++ addon) and this takes extra space. For example you can see the dependencies that the express-validator module uses by accessing this link.
So, when you have big projects that use lots of dependencies you end up taking so much space for them.
Ways to solve this
Number 1
This requires some expert people on Node.js. To reduce the amount of the downloaded packages, a professional Node.js developer could go to the directories that modules are saved in, open the javascript files, take a look at their source code, and delete the functions that they will not use without changing the structure of the package.
Number 2 (Most likely not worth your time)
You could also create your own personal dependencies that are written in C++, or more preferably js, which would literally take up the least space possible, depending on the programmer, but would take/waste the most time, in order to reduce size instead of doing work. (Note: Most dependencies are written in js.)
Number 3 (Common)
Instead of Using option number 2, you could implement WebPack.
Conclusion & Note
So, basically, there is no running away from downloading all the node packages, but you could use solution number 1 if you believe you can do it, which also has the possibility of screwing up the whole intention of a dependency. (So make it personal and use it for specific purposes.) Or just make use of a module like WebPack.
Also, ask this question to yourself: Do those packages really cause you a problem?
No, there is no point to add about 900 packages dependencies in your project just because you want to add some template. But it is up to you!
The heavyness of a template is not challenging the node.js ecosystem nor his main package system npm.
It is a fact that javascript community tend to make smallest possible module to be responsible for one task, and just one.
It is not a bad thing I guess. But it could result of a situation where you have a lot of dependencies in your project.
Nowadays hard drive memory is cheap and nobody cares any more about making efficient/small apps.
As always, it's only a matter of choice.
What is the point of delivering hundreds of packages weighing hundreds of MB for a few kB project.
There isn't..
If you intend to provide it to other developers, just gitignore (or remove from shared package) node_modules or bower_components directories. Developers simply install dependencies again as required ;)
If it is something as simple as an HTML templates or similar stuff, node would most likely be there just for making your life as a developer easier providing live reload, compiling/transpiling typescript/babel/SCSS/SASS/LESS/Coffee... ( list goes on ;P ) etc.
And in that case dependencies would most likely only be dev_dependencies and won't be required at all in production environment ;)
Also many packages come with separate production and dev dependencies, So you just need to install production dependencies...
npm install --only=prod
If your project does need many projects in production, and you really really wanna avoid that stuff, just spend some time and include css/js files your your project needs(this can be a laborious task).
Update
Production vs default install
Most projects have different dev and production dependencies,
Dev dependencies may include stuff like SASS, typescript etc. compilers, uglifiers (minification), maybe stuff like live reload etc.
Where as production version will not have those things reducing the size node_modules directory.
** No node_modules**
In some html template kind of projects, you may not need any node_modules in production, so you skip doing an npm install.
No access to node_modules
Or in some cases, when server that serves exists in node_modules itself, access to it may be blocked (coz there is no need to access these from frontend).
What are those? Why did they get installed along with my package?
Dependencies exists to facilitate code reuse through modularity.
... do I need to deliver my template in production with hundreds of unnecessary packages?
One shouldn't be so quick to dismiss this modularity. If you inline your requires and eliminate dead code, you'll lose the benefit of maintenance patches for the dependencies automatically being applied to your code. You should see this as a form of compilation, because... well... it is compilation.
Nonetheless, if you're licensed to redistribute all of your dependencies in this compiled form, you'll be happy to learn those optimisations are performed by a compiler which compile Javascript to Javascript. The Closure Compiler, as the first example I stumbled across, appears to perform advanced compilation, which means you get dead code removal and function inlining... That seems promising!
This does however have another side effect when you are required to justify the licensing of all npm modules..so when you have hundreds of npm modules due to dependencies this effort also becomes a more cumbersome task
Very old question but I happened to come across very similar situation just as RA pointed out.
I tried to work with node.js framework using vscode and the moment when I tried to install start npm using npm init -y, it generated so many different dependencies. In my case, it was vscode extension ESlint that I added to prior to running npm init -y
Uninstalling ESlint
Restarted vscode to apply that uninstallation
removed previously generated package.json and node-modules folder
do npm init -y again
This solved my problem of starting out with so many dependencies.
I'm working on node.js app that is written in Typescript, which means it needs to be compiled to JS before running. As I'm coming from java/jvm background where you ship prebuilt package to server and it gets run there I'm a bit afraid of the way of deployment where you push code to git and it's being built/compiled on server first and then run.
I don't like it for two main reasons:
dev dependencies need to be installed on server
deployment depends on external resources availability (npm etc).
I found NAR https://github.com/h2non/nar which is more or less what I wanted but it has some drawbacks (doesn't work with some deps that have native extensions).
My question is: is there any other "sane" way of doing deployment node.js deployment than this risky combination of npm install and tsc on server? Or should I let that sink in and do it that way?
To be honest I don't believe there are no more sane/reliable options for that.
What you can do (but there are probably other perfectly valid approaches) is building your project locally (or on a CI service), and only deploy this built version when you consider it as valid (tests, etc.).
This way, if something bad happens, like npm that fails, or a compilation error, you don't deploy anything, and you have time to resolve the situation.
For example, I used to have a gulp task (but it can be anything else: Grunt, a simple npm script...) that clone a production repository and build the project into this directory.
That way, I can check that my build is valid. If it is, I make a new commit and push it to the production repo, that is served the way you need (on a Heroku instance for example).
Pros
Clear separation of dev and non-dev dependencies
Deployment only when you know that the build is valid
No built files on source control on the development repository
No "live" dependency on external tasks like npm install or tsc build
Cons
You have two separated git repositories (one with the source code, one with the built version of your project)
Production process is a little bit heavier than simply committing to master
(From comment) Doesn't properly handle the case of npm package that relies on native extensions that have to be (re)built
is there any other "sane" way of doing deployment node.js deployment than this risky combination of npm install and tsc on server
package.json + npm install + tsc is the way to do it. Nothing risky about it.
More
Just use an npm script : https://github.com/TypeStrong/ntypescript#npm-scripts
I've noticed that in trying to get seemingly simple node packages to install with npm (e.g. nerve, a "micro-framework") I often run into some form of dependency pain. After some digging, I tracked down the problem with nerve to the bcrypt module, which is apparently written in C/C++ and has to be compiled after the package manager downloads it.
Unfortunately, it seems like if you want this to work on Windows, the answer is (from one of the bcrypt issues threads) "install a Linux VM". So earlier today I did just that, and started running into other dependencies (you need certain unnamed apt packages installed before you can even think about building, despite GCC being installed), then eventually after seeing yet another C compiler error (about some package or other not being able to find "Arrays.c" I think), I actually gave up, and switched from nerve to express instead. Ironically, the larger and more complicated express installs with npm on Linux and Windows without a single issue.
So, my question is: is there any filter / dependency tracking available that lets you see if a package has additional dependencies besides node core? Because to me the allure of node is "everything in Javascript", and this kind of stuff dispels the illusion quite unpleasantly. In fact, despite having done more than my time working with C/C++, whenever I see a requirement to "make" something these days I generally run in the other direction screaming. :)
The first solution doesn't tell you if a dependency makes the package impure or not. Much better to search for gyp generated output:
find node_modules/ | grep binding.gyp || echo pure
Look out for the "scripts" field in the package.json.
If it contains something like
"scripts": {
"install": "make build",
}
and a Makefile in the root directory, there's a good possibility that the package has some native module which would have to be compiled and built. Many packages include a Makefile only to compile tests.
This check on the package documents does not exclude the possibility that some dependency will have to be compiled and built. That would mean repeating this process for each dependency in the package.json, their dependencies and so on.
That said many modules have been updated to install, without build on Windows, express for one. However that cannot be assured of all packages.
Using a Linux VM seems to be the best alternative. Developing Node.js applications on Window gives you step by step instructions on installing a VM, Node.js and Express.
Node is not "everything javascript" , since one way to extend node core is to write c/c++ plugins.
So Node is more a javascript wrapper around c/c++ modules using V8.
How could you write efficient database drivers in pure javascript for instance ? it would be possible but slow.
as for the filters , it is up to the author to document his package. there is no automatic filter.