Best way to audit dependencies in Yarn 2 (berry)? - javascript

I'm looking for a way to audit dependencies for vulnerabilities in Yarn 2. In Yarn 1.x it was possible the same was as in npm, by running yarn audit instead of npm audit. But with Yarn 2 there is no such command. And according to this issue on the berry github, it won't be implemented (project maintainers prefer it was done via plugin).
I have tried running npm install --package-lock-only && npm audit but the install chokes on some of my local packages (which I am listing in package.json using the link: url type).
It wouldn't be a complex plugin to build and I'm up for some fun doing so, but it would not be so muchfun as just installing something and then going about my day. I have looked around but always end up at the same couple of vapourware / abandonware repos.
But I still guess that I'm just not finding them. Or there's an undocumented trick for making it easy. Hence my question :)
PS, yeah I can delete the local packages using link: temporarily while I run the npm install and npm audit commands above, but it's not exactly the kind of thing I want to try and automate for CI.

Update (28-Oct-2020):
Yarn 2 just merged long-awaited yarn npm audit enhancement.
PR - https://github.com/yarnpkg/berry/pull/1892
Docs - https://yarnpkg.com/cli/npm/audit
I was experimenting with Yarn 2 recently, and I saw that you can do this with #efrem/auditdeps utility:
yarn dlx #efrem/auditdeps [--level=(low|moderate|high|critical)] [--production]
The output is not as pretty as from npm audit, but you get more details in JSON format and you can pipe it to other tools or to any custom reformatting script to get exactly what you want.

I'd try https://snyk.io/ not free for huge team commercial use, but it could get you started on daily runs etc.
( I'm not affiliated in any way with Snyk.io )

Related

Where to install DataTables in slim4 by Daniel Opitz

I purchased the eBooks by Daniel Opitz. I am on page 226 of the books because I want to implement data tables in my project. In the books it reads.
DataTables Setup
DataTables.net is a very flexible table plug-in for jQuery.
You have to setup jQuery for Webpack first.
To install the DataTables Core library, run:
npm install datatables.net
npm install datatables.net-dt
To install the DataTables Bootstrap 5 styles, run:
npm install datatables.net-bs5
npm install datatables.net-responsive-bs5
npm install datatables.net-select-bs5
Add a new webpack entry in webpack.config.js:
In my naivete, I can't figure out on my own where I should be running the npm install. I was hoping this would be the fastest route to get help.
I have enjoyed learning the skeleton that he published. Most things have been straight forward and not to hard to figure out. This one has me stumped.
I did some more searching looking for the package.json location.
https://odan.github.io/slim4-skeleton/frontend.html
https://odan.github.io/slim4-skeleton/directory-structure.html
I looked over the directory structure and still not obvious to me where the npm should be executed.
Most package managers work by using a configuration file found in the root directory for the tooling you're trying to use. Generally, the assumption is these go in the root of the project. For instance:
/composer.json # PHP's Composer
/package.json # Javascript's NPM
/requirements.txt # Python's PIP
These are typically created by the package manager:
composer init
npm init
pip freeze > requirements.txt
Here are a some articles that can help explain this process:
https://getcomposer.org/doc/01-basic-usage.md
https://docs.npmjs.com/creating-a-package-json-file
https://learnpython.com/blog/python-requirements-file/
I'll also note many package managers will use a lock file, which records what was the last resolved and installed packages. The files above, in other words, depict what should be, while the lock file depicts what was (last) installed.
Your lock file should be handled with care and committed to the your project's repo. When you remove it, you are telling that package manager to recalculate the installed packages list and you may have subtly different versions (leading to potential bugs).
/composer.lock
/package-lock.json

Good way to switch between 2 versions of the same dependency in package.json?

Turns out you can't have comments in JSON files, and it's a bit awkward to have people refer to some documentation telling them what line to copy/paste in and where in order to achieve this.
I think I can make a python script to copy/paste in one of two package.json files depending on what flags they pass in, but that feels overcomplicated.
I think I can include both dependencies (under different names) but that would create a requirement for both to be available, which is not good either.
Looking for ideas/thoughts on a good way to accomplish this. I have a release and dev version of the same dependency and I often need to swap between the two. Would like to improve the workflow beyond just having a notepad on the side with the two lines pasted in it...
yarn and npm already do this job, why not use them?
Releases
Tag the dev versions when you release them
yarn publish --tag dev dep
npm publish --tag dev dep
Then reference the tag at install time yarn install dep#dev.
Local
For local dependencies, npm and yarn provide the "link" command.
In your dependency dir run yarn link
In you project dir run yarn link dep
You could document the commands as scripts so easy to run or view.
"scripts" : {
"dep:local": "yarn link dep",
"dep:dev": "yarn install dep#dev",
"dep:latest": "yarn install dep#latest"
}

What is the step-by-step procedure for installing any npm module with Aurelia CLI?

I wanted to install jquery and found instructions here:
http://aurelia.io/hub.html#/doc/article/aurelia/framework/latest/contact-manager-tutorial/4
I then wanted to install "moment.js" and found instructions here ( I am not using typescript) :
How to import Moment-Timezone with Aurelia/Typescript
To install both of these with the Aurelia CLI the procedure is to install the respective npm module and then to manually modify aurelia.json in some way so the app recognizes it.
In the case of moments the instructions then say to place an import at the top of app.js , but this is not the case for JQuery.
First off , is there any way the changes to aurelia.json can be automated ( like a regular node.js package.json) so I don't need to manually do it and second, how do I know what modifications I am expected to make to aurelia.json ( or app.js or any other file) for the module I want to install?
With a basic node.js app its pretty simple , just npm install. With Aurelia its much more confusing.
Edit: There is also JSPM which I've read is used for front end libraries like the ones I mentioned above. However, the links with instructions for installation that I posted are not using JSPM.
Edit
I found some of the answers here:
http://aurelia.io/hub.html#/doc/article/aurelia/framework/latest/the-aurelia-cli/6
The CLI is still under development. I think the automatic adding of a package might some day be included in the CLI itself, for example with an install command.
The extra registration is required to register the package correctly for usage with RequireJS (http://requirejs.org/). And if the plugin exists of more than just 1 file, this registration is a bit more complex then just adding the name of the plugin.
There is an experimental CLI task here https://github.com/martonsagi/aurelia-cli-pacman that will do the automation for you.
Which can be installed by running:
npm install aurelia-cli-pacman -D
The above will install the package manager and register/ include itself in the tasks in your current project (be sure to run it with install, because npm won't run the post install script if you run it the i shorthand). Next, you can run the following command to install an extension:
npm i aurelia-interactjs -S
au pacman i aurelia-interactjs
The only downside for many might be that currently there aren't that many registry entries, but I think the author of the package would be very happy if you help him out by creating a pull to extend the registry. Would take you some time to figure out what would be the correct install/ import settings, but you will help out someone else and make them happy when they hit the same problem you experience :-).
JSPM has a same sort of issue around this, only is more matured/ the registry is bigger and/ or authors added specific information for JSPM installations to their package.json. For example: To install the above plugin with JSPM it will use the following highlighted section https://github.com/eriklieben/aurelia-interactjs/blob/master/package.json#L72,L86. The same is currently not possible with aurelia-cli, because the installation is done by NPM instead of through JSPM that redirects it to NPM.
If the author of the plugin didn't specify the JSPM section in the package.json, you would most likely and up with the same sort of issues. JSPM has a similar registry (https://github.com/jspm/registry/tree/master/package-overrides/npm) as aurelia-cli-pacman.

Is bad practice to run "bower install" when deploying in production?

I am working on an application that uses bower.js; it's the first time I use bower, so please correct me if you see anything evidently wrong in my problem description.
------------------------------------------------
Coming from a Ruby background, I expect a package manager to have a .lock file, tracked under git, that tells me exactly which are the versions currently in use. This doesn't seem to happen with bower (am I right?).
A couple of days ago I deleted and re-cloned my repository, and ran bower install, thinking that such command would just install the required versions of the js components.
Then, today I did a one-line fix in a javascript file, compiled application.js using grunt watch, and realised that application.js was automatically filled up with tons of new code from bower component updates I wasn't aware of.
I found out that our bower components were under .gitignore, and that bower install, that I had run a few days ago, had actually updated two components without me noticing it.
When I realised what was happening, I immediately looked into our deployment procedure, which I paste here:
bundle install --path ${SNAP_CACHE_DIR}/.bundle
npm install -g bower grunt-cli
bower cache clean && bower install && bower list
bundle exec cap [our application name] deploy
Is this dangerous? Will bower install update all the components, that are likely not updated in my local version and are not tracked by git, ending up having completely different js code in production?
Is this dangerous? Will bower install update all the components, that are likely not updated in my local version and are not tracked by git, ending up having completely different js code in production?
Yes, this may happen and can cause problems. Although the impact will be limited as long as your dependency versions are specified as e.g. "~1.2.3", which will lock the major/minor version and only allow patch level updates.
In contrast to bower, the package manager normally used in node.js environments - npm - has a feature/command called npm shrinkwrap, which creates an npm-shrinkwrap.json file which locks down your dependency versions so that it is safe to run npm install afterwards.
This is probably what you would want.
However, bower as it stands does not have this feature yet - there is a discussion about it going on on Github e.g. here.
I think there currently are the following options to solve this problem in your situation:
Un-ignore and commit your bower_components (very ugly because of the huge amount of noise this produces in git).
Specify your dependency versions down to the patch level, e.g. "1.2.3" instead of "~1.2.3".
Culprit: If your dependencies have sub-dependencies, they might still be specified on the minor-version level, which means that even if your direct dependencies have a predictable version, your transitive dependencies may not.
Stop using bower and use npm instead (interface-/usability-wise, they are almost identical imho) and use npm shrinkwrap to lock down your dependencies.
Cheers, Alex

package.json generation / npm unused packages

I'm introducing unit testing in my project and for this, I need to make myself a package.json file.
First question is, which unit testing suite are you using? I'm looking forward mocha which seem to be pretty much standard for Node.js projects.
Second question is: Is there any magical way of generating a package.json file? (for dependencies and versions)
Third question is: I've been testing a lot of npm packages while developing my project and now I'm stuck with a lot of probably unused packages. Is there any way to tell which one are useless? (I saw npm list installed which is useful though)
I am using Mocha.
npm init
npm ls will list "extraneous" next to ones that are not in your package.json. But, it sounds like you don't have a package.json yet.
Basically, your workflow is very backward. Here is how it is intended to work:
Start a new project with npm init. It has no dependencies.
Oh, I want to start using a package, say express? Add it to package.json under dependencies, then run npm install.
Oh, I want to start using a package for development, say mocha? Add it to package.json under devDependencies, then run npm install.
You seem to have some existing code with manually-installed packages (via npm install <packageName>), which is a mess. I suggest starting over and following the above workflow.
To answer the third question:
npm prune
will remove all installed modules that are no longer mentioned in your package.json.
And you should really have asked 3 separate questions.
I am also using Mocha. It has code coverage, BDD, TDD, runs in browser. It is pretty complete and also heavily maintained by I think one of the most brilliant javascript/node.js programmers named TJ.
It is almost impossible to guess which version(s) to use. Because npm does not know which version breaks which dependencies. You could probably install all dependencies using something like node-detective. Then you can just install them using npm.js from within javascript. Maybe I would like to tackle this in the future.
I would also probably delete all dependencies , next install needed dependencies back using step(2). But also disc-space is not such a big case anymore with the current HDs.
P.S: I think I also agree with Domenic
I am using vows. It's pretty good, but not perfect. I have found unit testing in node to often be challenging because of async callbacks to dbs & such, and have mostly been testing top level functionality.
Here's your magic: Managing Node.js Dependencies with Shrinkwrap.
The only way to know what packages you are using is to know. You can't generate this programmatically. My advice would be to remove packages aggressively, then retest all functionality - if it breaks, you'll know you need to reinstall one of your packages.
Answering your third question, you can use Sweeper to list unused dependencies, and them remove them from your package.json. Just npm install -g sweeper then on your project directory call sweeper on the command line.

Categories