How can I cleanup least frequently used node packages? - javascript

My system has 1000s of node packages installed mostly via yarn and few via npm. Now I want to cleanup least frequently used node packages, how do I proceed? Is there any package which can track the usage of these over time and help me identify those to be cleaned up?

Maybe you could do two things.
First you could use the npm prune command to unbuild packages that are not listed in the packages.json dependencies. This way you can get rid of those that the package manager cant audit, search for vulnerabilities and fix.
Second, you could use webpack in order to list dependencies and set how they will be loaded, in order to optimize your app.
Hope this help you, or at least give you some ideas to do your research :)

Related

npm install changes my package-lock.json, will that interfere with the remote code?

I know there are numerous issues about this, and I discovered the command npm ci that is supposed to not change package-lock.json, but when I run npm ci it fails:
ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
Also tried another solution involving deleting my node_modules directory and running npm i again, but that's not changing the outcome.
I'm a junior dev working with a team remotely.. I was given a task, so I created a new branch on Gitlab, pulled it down to my local machine and ran npm i to get up-to-speed...
But it keeps changing my package-lock.json DRAMATICALLY(it adds like 20,000 lines of code)
Committing that to the team's project seems insane to me. Anyone have advice?
Update: OP wound up needing to make use of yarn which their team and project made use of. If anyone finds a yarn.lock in their project root, this is an indication that yarn is involved and any package-lock.json, if there is one, is possibly outdated.
TL;DR: it sounds like the package-lock.json needs some updates and resolution, which is done primarily with npm install.
It sounds like the package-lock.json no longer "agrees" with your package.json. It also sounds like others on your team are avoiding committing the changes to your package-lock.json; this is a bad practice in my experience as it only deepens any divergence in their contents. Over time they can be out of sync when dependencies of dependencies may publish a bug fix release and potentially un-publish a previous version.
As a junior dev on this team, I would bring this up to your development/team lead and ask their preferred approach here. If there was a major dependency intentionally removed and that has a lot of its own dependencies, it could cause a large removal of lines from package-lock.json and look severe to one less accustomed to it.
Extra context:
During an install, npm installs the dependencies and development dependencies listed in your package.json. In the process it's possible and increasingly likely over time, that some of the dependencies of those dependencies, which are needed to execute, will overlap and often with conflicting versions. The execution of npm i(nstall) will attempt to reconcile all these competing versions of sub-dependencies for you.
In a Continuous Integration context, it is desirable for the alternate command npm ci to be used, which explicitly installs only what's resolved already, from package-lock.json (the formerly known as "shrinkwrap"). This is meant to reduce the "but it works on my machine!" moments. As a developer, I've found it to be preferable to use npm install, as this alerts the developers more quickly to any dependency resolution issues and keeps the package-lock.json up to date.
Since this post got so many views I thought I'd come back and post what I found.
Yarn and NPM both update and install packages and dependencies, but the difference is:
yarn creates a file called yarn.lock
npm install creates a file called package-lock.json.
I didn't know this at the time, so when I cloned the project repo to my local machine, I ran npm i which created the package-lock.json. My teammates were already using yarn, however.
So make sure you use the one already being used.

The best way to resolve vulnerabilities in package-lock.json?

I am warned about vulnerabilities in the packages listed in the package-lock.json file of my Node.Js project.
I can follow the advice here and reinstall all the packages with npm install <package-name>, however, I also use other npm projects that use the older versions of those packages, which will not get reinstalled with a simple npm install.
Does it mean I have to go to package-lock.json and manually change all the dependencies to the latest version?
What if they break?
Isn't there a proper way of doing the updates that ensures you won't break the other packages dependent on the old versions?
If the issue is on a package you directly depend upon, you should update it directly and save it to the package.json + lock its version in package-lock.json in the process by doing something like npm install your-dependency#latest --save[-dev]. But beware: there might be breaking changes that will break your code (for example in case the dependency had a major version update inbetween with some deprecations and breaking changes).
But if the issue is from a dependency of one of your dependencies, the very very best way to solve it is to raise an issue (potentially with a PR to help them) with the maintainer of the parent package, then when they provide an update, update the dependency itself in your project.
You can use npm audit to resolve some issues as well (probably not all, and if a sub-dependency version is specifically required by a dependency, it will not update it because it could break things), but the single best way to solve the issue for you and for everybody else is to get the maintainer of the module you want to update its dependencies, when/if they can.
Reinstalling everything will not solve the issue if the dependency is still vulnerable. Installing does not magically fix stuff, people do :-) However, what you may want to do is use npm outdated to list all the packages that have newer versions available and try to update them, one by one, and see if your vulnerabilities are resolved after that (npm audit).
One more thing: it's usually a bad practice to go and change stuff around manually in package-lock.json. This file should be only auto-generated by one of your npm install (or similar) scripts. This file is what is used by npm to resolve the list of exact dependency/subdependency versions on a fresh install, and it is really the single best way to ensure all the people who use or work on this project have the exact same version of all their dependencies, so it better be correct. Always commit your package-lock.json!

Is bad practice to run "bower install" when deploying in production?

I am working on an application that uses bower.js; it's the first time I use bower, so please correct me if you see anything evidently wrong in my problem description.
------------------------------------------------
Coming from a Ruby background, I expect a package manager to have a .lock file, tracked under git, that tells me exactly which are the versions currently in use. This doesn't seem to happen with bower (am I right?).
A couple of days ago I deleted and re-cloned my repository, and ran bower install, thinking that such command would just install the required versions of the js components.
Then, today I did a one-line fix in a javascript file, compiled application.js using grunt watch, and realised that application.js was automatically filled up with tons of new code from bower component updates I wasn't aware of.
I found out that our bower components were under .gitignore, and that bower install, that I had run a few days ago, had actually updated two components without me noticing it.
When I realised what was happening, I immediately looked into our deployment procedure, which I paste here:
bundle install --path ${SNAP_CACHE_DIR}/.bundle
npm install -g bower grunt-cli
bower cache clean && bower install && bower list
bundle exec cap [our application name] deploy
Is this dangerous? Will bower install update all the components, that are likely not updated in my local version and are not tracked by git, ending up having completely different js code in production?
Is this dangerous? Will bower install update all the components, that are likely not updated in my local version and are not tracked by git, ending up having completely different js code in production?
Yes, this may happen and can cause problems. Although the impact will be limited as long as your dependency versions are specified as e.g. "~1.2.3", which will lock the major/minor version and only allow patch level updates.
In contrast to bower, the package manager normally used in node.js environments - npm - has a feature/command called npm shrinkwrap, which creates an npm-shrinkwrap.json file which locks down your dependency versions so that it is safe to run npm install afterwards.
This is probably what you would want.
However, bower as it stands does not have this feature yet - there is a discussion about it going on on Github e.g. here.
I think there currently are the following options to solve this problem in your situation:
Un-ignore and commit your bower_components (very ugly because of the huge amount of noise this produces in git).
Specify your dependency versions down to the patch level, e.g. "1.2.3" instead of "~1.2.3".
Culprit: If your dependencies have sub-dependencies, they might still be specified on the minor-version level, which means that even if your direct dependencies have a predictable version, your transitive dependencies may not.
Stop using bower and use npm instead (interface-/usability-wise, they are almost identical imho) and use npm shrinkwrap to lock down your dependencies.
Cheers, Alex

Is there any way to reorganize node_modules?

There is big and deep node_modules directory. And there are many sub-folders with the same modules that are located in different subdirectories. Sometimes the same versions of the modules, there are sometimes differences in minor versions.
Is there a tool for reorganization of node_modules to remove duplicates, put them in the root directory and somehow still a bunch of modules to optimize this?
The NPM hierarchy is actually fairly complex and they've done a lot of work to optimize it. The most you're losing here is a little disk space. If you really need to prune the package structure for your app then you can take a look at npm dedupe which is built right into npm and does exactly what you're asking for (consolidates duplicates as much as possible).
I know a lot of people are against checking in node_modules directory but in production applications we've found that checking in node_modules makes it so much easier to pin down fatal changes in our application. Sure you usually know that your application broke after the last npm update but if your app is rather large like ours is then that simply isn't granular enough to solve problems quickly and efficiently. So in large production applications (i.e. not libraries being published to NPM) I use npm dedupe to simplify the package structure before checking them in.
If you're writing code that others will consume (via NPM or otherwise) then checking in node_modules isn't the best idea and you should avoid doing so by adding node_modules to your version control system's ignore list. You should also make sure you're dependencies in package.json are specific version numbers with as few ranges as possible (please don't put asterisks in place of the version numbers in your production apps :/).
If you follow those basic patterns then you can just forget about the node_modules directory and let npm take care of you.

package.json generation / npm unused packages

I'm introducing unit testing in my project and for this, I need to make myself a package.json file.
First question is, which unit testing suite are you using? I'm looking forward mocha which seem to be pretty much standard for Node.js projects.
Second question is: Is there any magical way of generating a package.json file? (for dependencies and versions)
Third question is: I've been testing a lot of npm packages while developing my project and now I'm stuck with a lot of probably unused packages. Is there any way to tell which one are useless? (I saw npm list installed which is useful though)
I am using Mocha.
npm init
npm ls will list "extraneous" next to ones that are not in your package.json. But, it sounds like you don't have a package.json yet.
Basically, your workflow is very backward. Here is how it is intended to work:
Start a new project with npm init. It has no dependencies.
Oh, I want to start using a package, say express? Add it to package.json under dependencies, then run npm install.
Oh, I want to start using a package for development, say mocha? Add it to package.json under devDependencies, then run npm install.
You seem to have some existing code with manually-installed packages (via npm install <packageName>), which is a mess. I suggest starting over and following the above workflow.
To answer the third question:
npm prune
will remove all installed modules that are no longer mentioned in your package.json.
And you should really have asked 3 separate questions.
I am also using Mocha. It has code coverage, BDD, TDD, runs in browser. It is pretty complete and also heavily maintained by I think one of the most brilliant javascript/node.js programmers named TJ.
It is almost impossible to guess which version(s) to use. Because npm does not know which version breaks which dependencies. You could probably install all dependencies using something like node-detective. Then you can just install them using npm.js from within javascript. Maybe I would like to tackle this in the future.
I would also probably delete all dependencies , next install needed dependencies back using step(2). But also disc-space is not such a big case anymore with the current HDs.
P.S: I think I also agree with Domenic
I am using vows. It's pretty good, but not perfect. I have found unit testing in node to often be challenging because of async callbacks to dbs & such, and have mostly been testing top level functionality.
Here's your magic: Managing Node.js Dependencies with Shrinkwrap.
The only way to know what packages you are using is to know. You can't generate this programmatically. My advice would be to remove packages aggressively, then retest all functionality - if it breaks, you'll know you need to reinstall one of your packages.
Answering your third question, you can use Sweeper to list unused dependencies, and them remove them from your package.json. Just npm install -g sweeper then on your project directory call sweeper on the command line.

Categories