Alternatives to structure and use javascript libraries without npm - javascript

We have been working on our projects all tied to a single lerna repository, as:
lerna
|----- app1 - application 1
|----- app2 - application 2
|----- appN - application N
|----- commondb (common database libraries for app1, app2 to appN)
|----- commonux (common ux libraries for app1, app2 to appN)
|----- commonauth (common authentication libraries for app1, app2 to appN)
As the code grew a lot, lerna is really full of packages (40+ packages) and too much code.
We're now trying to split lerna into smaller pieces and we're looking for alternatives. Doing that, applications would need a way to import common libraries as we do today.
Certainly NPM seens to be a solution (making each common package independent and publishing it on NPM), but we want to keep our code in our environment without third party services or clouds (we have our own git server instance).
What are the current options to manage javascript libraries that we can make use of? What would be the recommended one in such a scenario?

Your decision can be greatly affected by answering the following - do you want your apps to be running the same version of the shared libraries? Or do want autonomy within the libraries, and to be able to publish and manage different versions of the libraries, where it is the responsibility on the consumer app to manage which version of the library it is using?
If it is the former, my suggestion would be to stick with a Mono-repo approach, maybe consider something like NX, where it has some nice tooling for only linting, testing, building and deploying only the affected modules, whilst sharing a common single package.json and therefore common libraries shared across multiple apps and libs
Otherwise you are looking at potentially managing multiple repos, multiple versions of each library, multiple pipelines, multiple workspace configs.

You could simply use pnpm workspaces: protocol which allows you to create a monorepo structure but without even requiring you to publish when used this way: "foo": "workspace:*". So if you wish to use this approach, then you could have a monorepo structure and just keep local packages, it will use the local code without downloading anything. I used this concept in a Vue pnpm Workspace demo. With this approach you could stay with local packages and/or go with Lerna to publish if you also want to (I prefer Lerna-Lite which is what I use) since they now both support pnpm/yarn workspace: protocol. This approach is flexible and allows you to switch to publishing (to npm or a private registry) at any point in the time in the future if you wish to, in other words there's no major code refactoring to add Lerna after the fact when using a pnpm workspace structure.

Related

Why do we need multiple node modules folders inside each project?

I'm new to JS development and was wondering given that the size of node modules is an issue of concern where one has to even go on deleting the folder when many projects pile up and is not used, why can't we have a global node modules folder, like m2 for maven and then reference it in our projects?
Why do we need multiple node modules folders inside each project?
You can install packages globally and use them with the -g flag when using npm install.
But this isn't done often, and often isn't recommended, because the same package can have many, many different versions. SomeLibrary version 5.6.1 can be significantly different from SomeLibrary version 5.7.1. If you have multiple projects, they'll very often have different dependencies, and different dependency versions. If you try to source them all from the same global install, you'll usually run into problems very quickly. Having a separate node_modules folder for each project solves this problem for you; often, package versioning for each project you have will "just work" without any extra configuration.
There are also managers like Yarn which can cache package downloads, so you don't re-download huge numbers of duplicate megabytes over multiple projects
Variety of reasons. Just FYI, you can install npm packages globally using the -g flag when installing.
Here's some reasons:
Different project may require different versions of a package.
A single machine can consist of multiple different environment, i.e
containers or virtual machines.
Sharing code between projects/environments introduces potential security risks.
For the above reasons, and others, you might not actually want a projects
packages to leave the scope of that project.
since java script is not compiled, we will need to have all the plugins in each deployment verses maven is compiled so we can compile it to machine code with the plugin globally installed

the difference between nx and Lerna ? (monorepos)

hello i'm in new in monorepos world and i'm now searching about different monorepos tools
and i found two intersting tools Nx(nrwl) and lerna but i didn't understand the real differnce between these two tools so any help and thanks
2022 Update
As pointed out by my great fellow at the comment section, Lerna is now being held by the Nx team.
The best solution nowadays is to use more native approaches for publishing (as NPM/YARN workspaces) or go straight to Nx.
Good references:
Lerna is dead — Long Live Lerna
Integrating Nx and Lerna
2021 Answer (slightly outdated but still valid)
Although both are great tools to work with mono repo, they're quite different in their purpose.
Differences
Lerna is focused on linking multiple packages from the same project and managing npm publishing, and that's about it.
Nx is more focused on managing development workflow for multiple packages. It means it can scaffold packages, and for every package, you can define configurations on how to run and build them, in a similar manner to Webpack.
Nx can also work to spawn multiple processes at once. For example: run frontend and backend at the same time, without the need to open two different terminals. Similar to docker-compose.
Where they thrive
Lerna fits better for open source projects with multiple packages (because you can easily publish your packages).
Nx fits better for handling complex workflows with multiple packages.
How to choose
If you don't intend to publish your packages, Nx might be a better fit.
If you do intend to publish then, but you don't have a complex workflow, Lerna is definitely the way to go.
If you want both (publish and complex workflow), neither one seems great, but you should probably choose Nx, and manage the publishing manually. Or, maybe, use Lerna and configure the workflow manually with Webpack.
Someone can wonder if you could choose both, but I don't think they can work well together. At the time I'm writing this, Nx doesn't seem to care about being compatible with Lerna, and it Nx still lacks support for easy publishing.
An important disclaimer about terminology
I used the name "package" here because "package" is what you can publish on the internet (for example, as npm is a node package manager).
Lerna uses the name "project" as the wrapper folder of all your packages. Usually, a git repository is a "Lerna project" with several "Lerna packages".
Nx uses the name "project" to designate what Lerna calls "packages", and uses the name "workspace" to designate what Lerna calls "project" (i.e., the wrapper folder which holds all the projects).

npm installs many dependencies

I bought an HTML template recently, which contains many plugins placed inside a bower_components directory and a package.js file inside. I wanted to install another package I liked, but decided to use npm for this purpose.
When I typed:
npc install pnotify
the node_modules directory was created and contained about 900 directories with other packages.
What are those? Why did they get installed along with my package? I did some research and it turned out that those were needed, but do I really need to deliver my template in production with hundreds of unnecessary packages?
This is a very good question. There are a few things I want to point out.
The V8 engine, Node Modules (dependencies) and "requiring" them:
Node.js is built on V8 engine, which is written in C++. This means that Node.js' dependencies are fundamentally written in C++.
Now when you require a dependency, you really require code/functions from a C++ program or js library, because that's how new libraries/dependencies are made.
Libraries have so many functions that you will not use
For example, take a look at the express-validator module, which contains so many functions. When you require the module, do you use all the functions it provides? The answer is no. People most often require packages like this just to use one single benefit of it, although all of the functions end up getting downloaded, which takes up unnecessary space.
Think of the node dependencies that are made from other node dependencies as Interpreted Languages
For example, JavaScript is written in C/C++, whose functions and compilers are in turn originally written in assembly. Think of it like a tree. You create new branches each time for more convenient usage and, most importantly, to save time . It makes things faster. Similarly, when people create new dependencies, they use/require ones that already exist, instead of rewriting a whole C++ program or js script, because that makes everything easier.
Problem arises when requiring other NPMs for creating a new one
When the authors of the dependencies require other dependencies from here and there just to use a few (small amount) benefits from them, they end up downloading them all, (which they don't really care about because they mostly do not worry about the size or they'd rather do this than explicitly writing a new dependency or a C++ addon) and this takes extra space. For example you can see the dependencies that the express-validator module uses by accessing this link.
So, when you have big projects that use lots of dependencies you end up taking so much space for them.
Ways to solve this
Number 1
This requires some expert people on Node.js. To reduce the amount of the downloaded packages, a professional Node.js developer could go to the directories that modules are saved in, open the javascript files, take a look at their source code, and delete the functions that they will not use without changing the structure of the package.
Number 2 (Most likely not worth your time)
You could also create your own personal dependencies that are written in C++, or more preferably js, which would literally take up the least space possible, depending on the programmer, but would take/waste the most time, in order to reduce size instead of doing work. (Note: Most dependencies are written in js.)
Number 3 (Common)
Instead of Using option number 2, you could implement WebPack.
Conclusion & Note
So, basically, there is no running away from downloading all the node packages, but you could use solution number 1 if you believe you can do it, which also has the possibility of screwing up the whole intention of a dependency. (So make it personal and use it for specific purposes.) Or just make use of a module like WebPack.
Also, ask this question to yourself: Do those packages really cause you a problem?
No, there is no point to add about 900 packages dependencies in your project just because you want to add some template. But it is up to you!
The heavyness of a template is not challenging the node.js ecosystem nor his main package system npm.
It is a fact that javascript community tend to make smallest possible module to be responsible for one task, and just one.
It is not a bad thing I guess. But it could result of a situation where you have a lot of dependencies in your project.
Nowadays hard drive memory is cheap and nobody cares any more about making efficient/small apps.
As always, it's only a matter of choice.
What is the point of delivering hundreds of packages weighing hundreds of MB for a few kB project.
There isn't..
If you intend to provide it to other developers, just gitignore (or remove from shared package) node_modules or bower_components directories. Developers simply install dependencies again as required ;)
If it is something as simple as an HTML templates or similar stuff, node would most likely be there just for making your life as a developer easier providing live reload, compiling/transpiling typescript/babel/SCSS/SASS/LESS/Coffee... ( list goes on ;P ) etc.
And in that case dependencies would most likely only be dev_dependencies and won't be required at all in production environment ;)
Also many packages come with separate production and dev dependencies, So you just need to install production dependencies...
npm install --only=prod
If your project does need many projects in production, and you really really wanna avoid that stuff, just spend some time and include css/js files your your project needs(this can be a laborious task).
Update
Production vs default install
Most projects have different dev and production dependencies,
Dev dependencies may include stuff like SASS, typescript etc. compilers, uglifiers (minification), maybe stuff like live reload etc.
Where as production version will not have those things reducing the size node_modules directory.
** No node_modules**
In some html template kind of projects, you may not need any node_modules in production, so you skip doing an npm install.
No access to node_modules
Or in some cases, when server that serves exists in node_modules itself, access to it may be blocked (coz there is no need to access these from frontend).
What are those? Why did they get installed along with my package?
Dependencies exists to facilitate code reuse through modularity.
... do I need to deliver my template in production with hundreds of unnecessary packages?
One shouldn't be so quick to dismiss this modularity. If you inline your requires and eliminate dead code, you'll lose the benefit of maintenance patches for the dependencies automatically being applied to your code. You should see this as a form of compilation, because... well... it is compilation.
Nonetheless, if you're licensed to redistribute all of your dependencies in this compiled form, you'll be happy to learn those optimisations are performed by a compiler which compile Javascript to Javascript. The Closure Compiler, as the first example I stumbled across, appears to perform advanced compilation, which means you get dead code removal and function inlining... That seems promising!
This does however have another side effect when you are required to justify the licensing of all npm modules..so when you have hundreds of npm modules due to dependencies this effort also becomes a more cumbersome task
Very old question but I happened to come across very similar situation just as RA pointed out.
I tried to work with node.js framework using vscode and the moment when I tried to install start npm using npm init -y, it generated so many different dependencies. In my case, it was vscode extension ESlint that I added to prior to running npm init -y
Uninstalling ESlint
Restarted vscode to apply that uninstallation
removed previously generated package.json and node-modules folder
do npm init -y again
This solved my problem of starting out with so many dependencies.

Jam vs Bower, what's the difference?

There are two package managers for client-side Javascript, but how do they compare? Could someone explain which one excels at what?
Jam
Bower
As others mentioned in comments already there are a few alternatives in this space beyond just Jam and Bower.
Component
Ender
Volo
Both aim to provide a way to package up your assets and manage the dependencies between them for the client. Both Bower and Jam appear to have made their debut in 2012 -- Sept and May respectively.
Both are available through node/npm and if all you want to do is resolve dependencies between public libraries like backbone, underscore, jquery, etc. for your application then either solution will work and allow you some basic options to control version, where to put it in your project, and checking for updates.
As for what's different -- Bower is a bit lower level than Jam which makes it more usable to a wider audience. You can create bower components for more than just JavaScript libraries. Jam focuses more solely on AMD style JavaScript libraries. With Jam, you can specify your dependencies in the package.json file you would use with npm components already whereas Bower has chosen component.json by convention. The limitation with Bower is that it only fetches your dependencies, you still need a build system if you want to use Require.js or other solutions which Jam has chosen so you get for free. Bower is getting support from Twitter and a few other projects (Ender, Yeoman).
Apologies if this is incorrect, but one additional limitation of Jam is that it does not allow you to create your own components for distribution in a private repository. This is something Bower allows you to configure as an endpoint in .bowerrc but I have as yet found a way to do that in Jam. Perhaps I haven't searched well enough but it appears there is at least one fork for private-jam.
A few other good reads:
http://yeoman.io/packagemanager.html
http://dailyjs.com/2013/01/28/components/
Using the following breakdown of some of the popular package managers you can decide what you want to use in you development. It compares factos as
whether the manager uses package.json or other form of descriptor
what features does it support (scaffolding, compilation, having central registry)
speed
form of packages support (js only, js and css, js html and css)
module types supported
and of course some notes based on personal point of view
https://github.com/wilmoore/frontend-packagers

Managing 3rd Party Javascript Libraries Across Projects

The company I work for develops and maintains multiple web-based products. We have been pushing the teams to standardize and reuse 3rd party libraries of all kinds across prod. dev. We use Ivy to manage jar file dependencies for Java. We use Python eggs in combination with custom build RPMs to manage Python dependencies.
I was wondering if there is a similar way to manage Javascript dependencies (e.g. jquery, raphaeljs, etc.).
Perhaps you would like to check RequireJS or Ender.
[Update]
As of today, 31 March 2017, there are a lot of different package managers and/or tools you could use to manage front-end libraries (npm, bower, rollup, etc.), but I currently found Yarn and Webpack 2 most interesting.

Categories