The best way to resolve vulnerabilities in package-lock.json? - javascript

I am warned about vulnerabilities in the packages listed in the package-lock.json file of my Node.Js project.
I can follow the advice here and reinstall all the packages with npm install <package-name>, however, I also use other npm projects that use the older versions of those packages, which will not get reinstalled with a simple npm install.
Does it mean I have to go to package-lock.json and manually change all the dependencies to the latest version?
What if they break?
Isn't there a proper way of doing the updates that ensures you won't break the other packages dependent on the old versions?

If the issue is on a package you directly depend upon, you should update it directly and save it to the package.json + lock its version in package-lock.json in the process by doing something like npm install your-dependency#latest --save[-dev]. But beware: there might be breaking changes that will break your code (for example in case the dependency had a major version update inbetween with some deprecations and breaking changes).
But if the issue is from a dependency of one of your dependencies, the very very best way to solve it is to raise an issue (potentially with a PR to help them) with the maintainer of the parent package, then when they provide an update, update the dependency itself in your project.
You can use npm audit to resolve some issues as well (probably not all, and if a sub-dependency version is specifically required by a dependency, it will not update it because it could break things), but the single best way to solve the issue for you and for everybody else is to get the maintainer of the module you want to update its dependencies, when/if they can.
Reinstalling everything will not solve the issue if the dependency is still vulnerable. Installing does not magically fix stuff, people do :-) However, what you may want to do is use npm outdated to list all the packages that have newer versions available and try to update them, one by one, and see if your vulnerabilities are resolved after that (npm audit).
One more thing: it's usually a bad practice to go and change stuff around manually in package-lock.json. This file should be only auto-generated by one of your npm install (or similar) scripts. This file is what is used by npm to resolve the list of exact dependency/subdependency versions on a fresh install, and it is really the single best way to ensure all the people who use or work on this project have the exact same version of all their dependencies, so it better be correct. Always commit your package-lock.json!

Related

How to update multiple packages and dependencies with major version changes in Node/npm/React?

I have an old React project with multiple outdated packages both in package.json and in dependencies in package-lock.json. Many of this packages need to be updated to next major version or even to the many major version change (for example React from 16.14 to 18.2). Since this is major version changes npm update and npm outdated would not work. For updating single top-level package I have this and this and this answers with npm install the-package#version, but that will not update packages, that dependent on this package. There is also npm-check-updates utility that will update all packages to the latest version, but according to this it is better to update and test single package after major version changes. There is also multiple packages that required update to major versions changes and not listed in package.json and only listed in package-lock.json since they used only as the dependencies of some other packages (and in come cases I must update to the versions bigger than listed in the depency itself due to security scan requirements). This answer suggests manually updating package-lock.json and running npm install.
What is the best way of updating multiple packages and dependencies with major version changes including packages that are only listed as dependencies of some other packages (an may be referenced to lower major versions then required)? Is there a way to update dependency tree for single package (not all packages, not only this package)?
I have tried npm audit and npm audit fix --force but it work strange - do not update all packages, sometimes do not upgrade to the latest version (which I have to use by the security scan) and sometimes even downgrade package version. I have tried to update packages by the install --save the-package#version but this do not update dependencies and work only for package.json. So far I see the possible way like this
Update top-level packages with npm install and test
Manually check dependencies, update with npm install and test
Manually check dependency tree for packages from package-lock.json, update packages that dependent on them if they still references to old version manually change package-llock.json and test changes
This requires a lot of manual work for each package and manual editing of package-lock.json and probably not the best way. There is also one possible solution for dependencies of dependensies as mentioned in this answers using override section in package.json, but it is recommended only for security fixes. Is there a better way with more automation?

npm install changes my package-lock.json, will that interfere with the remote code?

I know there are numerous issues about this, and I discovered the command npm ci that is supposed to not change package-lock.json, but when I run npm ci it fails:
ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
Also tried another solution involving deleting my node_modules directory and running npm i again, but that's not changing the outcome.
I'm a junior dev working with a team remotely.. I was given a task, so I created a new branch on Gitlab, pulled it down to my local machine and ran npm i to get up-to-speed...
But it keeps changing my package-lock.json DRAMATICALLY(it adds like 20,000 lines of code)
Committing that to the team's project seems insane to me. Anyone have advice?
Update: OP wound up needing to make use of yarn which their team and project made use of. If anyone finds a yarn.lock in their project root, this is an indication that yarn is involved and any package-lock.json, if there is one, is possibly outdated.
TL;DR: it sounds like the package-lock.json needs some updates and resolution, which is done primarily with npm install.
It sounds like the package-lock.json no longer "agrees" with your package.json. It also sounds like others on your team are avoiding committing the changes to your package-lock.json; this is a bad practice in my experience as it only deepens any divergence in their contents. Over time they can be out of sync when dependencies of dependencies may publish a bug fix release and potentially un-publish a previous version.
As a junior dev on this team, I would bring this up to your development/team lead and ask their preferred approach here. If there was a major dependency intentionally removed and that has a lot of its own dependencies, it could cause a large removal of lines from package-lock.json and look severe to one less accustomed to it.
Extra context:
During an install, npm installs the dependencies and development dependencies listed in your package.json. In the process it's possible and increasingly likely over time, that some of the dependencies of those dependencies, which are needed to execute, will overlap and often with conflicting versions. The execution of npm i(nstall) will attempt to reconcile all these competing versions of sub-dependencies for you.
In a Continuous Integration context, it is desirable for the alternate command npm ci to be used, which explicitly installs only what's resolved already, from package-lock.json (the formerly known as "shrinkwrap"). This is meant to reduce the "but it works on my machine!" moments. As a developer, I've found it to be preferable to use npm install, as this alerts the developers more quickly to any dependency resolution issues and keeps the package-lock.json up to date.
Since this post got so many views I thought I'd come back and post what I found.
Yarn and NPM both update and install packages and dependencies, but the difference is:
yarn creates a file called yarn.lock
npm install creates a file called package-lock.json.
I didn't know this at the time, so when I cloned the project repo to my local machine, I ran npm i which created the package-lock.json. My teammates were already using yarn, however.
So make sure you use the one already being used.

Package doesn't relate on its internal node_modules packages dependencies, uses external scope instead

I can't figure out why project dependency babel-polifyll doesn't relate to its internal packages from the own node_modules root (they're present there actually). Instead, it trying to get some dependency packages from the same level as the common project node_modules scope...
I understand that this is not a bug in babel-polifyll, probably something happens in node/yarn environment for sure. Anyway, I can't spot what exactly wrong...
Error:
Babel-polifyll reqiure imports that links from external scope:
This is an NPM thing, it by default tries to install all packages at the root level, so dependencies can be shared between components. This way when you build your code, if you have three packages using same component, then you output file will only need to have one copy of the built sub-component in it. This keeps the file size down a lot.
NPM will put a dependancy as a child when two packages require different versions of a library and normally you would never notice when this happens, unless you go looking.
This is great when it works, which it does most of the time, but can be a bit of a pig to sort out when it goes wrong.
I just had a look and the latest version of core-js is v3.2.1, but babel-polyfil requires v2.6.5. So the quick fix might be to force install the older version.
npm i -D core-js#2.6.5
This might break something else, if that happens try doing
rm -rf node-modules package-lock.json
npm i
Hopefully that will the force what ever needs the newer version of core-is to install as a child dep and everything will then work for you.

What's the point of having a "compatible version" (^version) declared in package.json if package-lock.json locks it?

I know the main advantages of package-lock.json and I agree with that. It not only locks the downloaded version in the last install, but also the uri... and that's required on most cases for being possible to replicate the most similar project as possible.
But one thing that seems weird to me is that package.json has the feature of declaring a dependency like dependency: ^1.0.0, that should make npm to download the most recent and compatible version of that package in each installation.
I'm working at a project that I actually need this. Otherwise every time my dependency releases a patch, it will be required to make a new commit updating package.json only changing the version, so my pipeline can also overwrite package-lock.json.
In short, it seems that while package.json uses a feature... package-lock.json prevents that one.
Am I missing something?
The point of package-lock.json is to accurately represent the tree as it actually exists at a point in time, so that someone cloning the project gets exactly the same tree you had.
If you want to upgrade that dependency to a newer version, just use npm update and then commit the updated package-lock.json. Other members of your team will get that update as part of the normal process of picking up the latest.
More in the npmjs.com page on package locks.
Let's consider as scenario where you and I are on a team and our project uses nifty-lib, with package.json saying "nifty-lib": "^0.4.0", and we don't share package-lock.json. Perhaps I've been working on the project a couple of months longer than you have and I got nifty-lib v0.4.0 when I installed it. But when you picked it up and installed, you got v0.4.1 (a bugfix update which, sadly, introduced a new bug). At some point, you notice what seems like a bug in our project, but I can't replicate it. We spin in place for a while trying to figure out why it happens to you and not to me. In the end, we realize it's because it's actually a bug in nifty-lib that they introduced in v0.4.1. Hopefully we then get 0.4.2 or something (or if there isn't one, we fix the bug and do a PR, meanwhile rolling back to 0.4.0 across the project).
If we'd been sharing package-lock.json, we wouldn't have spun in place wondering why the problem happened to you and not to me, because you would have had the same version of nifty-lib as me. As part of our normal cycle, we'd do npm update periodically, and if a new bug showed up in our tests, we'd know from the commit history that it was because of a bug in a dependency.
Now, for "me" and "you" read "dev" and "production". :-)
Which is why package-lock.json locks the version, but package.json lets you say "this or better". package-lock.json keeps your team unified on versions, but you can intentionally update with npm update, which shows up in the commit history so you can track down regressions to it.
As I mentioned in a comment above, the short answer is it makes updateing your dependencies easier.
However, another way I like to think about the two files is: package.json is the file the human reads, while package-lock.json is the file the computer reads.
NPM is a package / dependency manager. So, in your package.json file, you write out "these libraries are needed for my library to work." As a feature, you have a range of versions you can list a dependency at. This helps when you run npm update on a specific package. It'll look to see what is the latest version that matches within your *package.json**, and updates you lockfile.
The package-lock.json lockfile is useful because it verbosely describes what your node_modules/ folder looks like so it can be accurately recreated when someone else installs your library. Additionally, since this file is generated automatically, you don't have to worry about maintaining it.
Of course, all of this just happens to be how NPM (and conversely how most package managers) handle this. That is, there isn't a technical reason why we couldn't have one file to describe both the version range that would be allowed when running updates, and a verbose lockfile portion that pins versions to allow for a recreatable dependency tree.
Basically, it is just a convenience. You have one file to succinctly list what dependencies your projects needs. It is readable and easily updatable. The other file, the lockfile, is automatically generated and ensures each npm install gives you the exact same node_modules/ folder as before.

Is bad practice to run "bower install" when deploying in production?

I am working on an application that uses bower.js; it's the first time I use bower, so please correct me if you see anything evidently wrong in my problem description.
------------------------------------------------
Coming from a Ruby background, I expect a package manager to have a .lock file, tracked under git, that tells me exactly which are the versions currently in use. This doesn't seem to happen with bower (am I right?).
A couple of days ago I deleted and re-cloned my repository, and ran bower install, thinking that such command would just install the required versions of the js components.
Then, today I did a one-line fix in a javascript file, compiled application.js using grunt watch, and realised that application.js was automatically filled up with tons of new code from bower component updates I wasn't aware of.
I found out that our bower components were under .gitignore, and that bower install, that I had run a few days ago, had actually updated two components without me noticing it.
When I realised what was happening, I immediately looked into our deployment procedure, which I paste here:
bundle install --path ${SNAP_CACHE_DIR}/.bundle
npm install -g bower grunt-cli
bower cache clean && bower install && bower list
bundle exec cap [our application name] deploy
Is this dangerous? Will bower install update all the components, that are likely not updated in my local version and are not tracked by git, ending up having completely different js code in production?
Is this dangerous? Will bower install update all the components, that are likely not updated in my local version and are not tracked by git, ending up having completely different js code in production?
Yes, this may happen and can cause problems. Although the impact will be limited as long as your dependency versions are specified as e.g. "~1.2.3", which will lock the major/minor version and only allow patch level updates.
In contrast to bower, the package manager normally used in node.js environments - npm - has a feature/command called npm shrinkwrap, which creates an npm-shrinkwrap.json file which locks down your dependency versions so that it is safe to run npm install afterwards.
This is probably what you would want.
However, bower as it stands does not have this feature yet - there is a discussion about it going on on Github e.g. here.
I think there currently are the following options to solve this problem in your situation:
Un-ignore and commit your bower_components (very ugly because of the huge amount of noise this produces in git).
Specify your dependency versions down to the patch level, e.g. "1.2.3" instead of "~1.2.3".
Culprit: If your dependencies have sub-dependencies, they might still be specified on the minor-version level, which means that even if your direct dependencies have a predictable version, your transitive dependencies may not.
Stop using bower and use npm instead (interface-/usability-wise, they are almost identical imho) and use npm shrinkwrap to lock down your dependencies.
Cheers, Alex

Categories