what is benfit of using `prestart` over `&&` in `package.json` command - javascript

I think the title is self explain but again:
what is the benefit of using the pre script of npm packege.json for example prestart over just concatenate commands with && in the start script?
{
prestart: "parcel build",
start "nodemon server.js"
}
vs
{
start: "parcel build && nodemon server.js"
}
It is more cross platform ?
can it handle two async endless process like two servers (build + api) ?
something else?
edit:
I found benefit for postInstall. Heroku and such delete devDependency after npm install so in postinstall I can put build process before Heroku delete the code that do that.

prestart runs before start as the name suggests, therefore running a command in prestart and a command in start runs the two commands in sequence, not parallel. Running commands in start with && runs them sequentially, but inside the same step.
The two methods are pretty much the same, at least in terms of results. However, there might be compatibility issues with && on certain versions of Windows.
If you want to run commands in parallel, you can use & inside start, instead of &&.

In addition to the other answers, it should be noted that the prestart hook in package.json is not supported by the yarn package manager. So in this respect, using && allows for easier migration between npm and yarn (provided that your shell can interpret the &&).
The reason for this is to increase maintainability by avoiding implicit dependency chains:
In particular, we intentionally don't support arbitrary pre and post hooks for user-defined scripts (such as prestart). This behavior, inherited from npm, caused scripts to be implicit rather than explicit, obfuscating the execution flow. It also led to surprising executions with yarn serve also running yarn preserve.
https://yarnpkg.com/advanced/lifecycle-scripts

These methods are more for clarity in code, for separation of logical steps.
About compatability. As I understand npm runs all scripts in the local shell, so on most linux systems it will be some sh clone, and on windows it will be cmd. So there may be situation where && will not be supported by the shell. But it's unlikely and do you really need to support such behaviour, considering users could install bash on any platform node.js could be installed on and set nom to use it? I personally use bash in npm scripts and document in the README.
If you want to run multiple long-running processes use something like pm2 https://github.com/Unitech/PM2/ in production. When you're developing, usually it's helpful to run processes in multiple terminals to see logs, use supervisor https://github.com/petruisfan/node-supervisor to restart processes on errors and changes.
Also I usually write .sh scripts for maintenance, like deploy and periodic, but manual tasks and run them using npm - you could add any named scripts in scripts section.

That is because when you run a script like this - npm install && npm start node will run both commands in the same process, so if one of the commands will send any non-zero exit code (in case of an exception) you'll not be able to determine which command failed. Running them in separate scripts with "pre" keyword will execute them in separate processes, so it is more accurate.

Related

how to resolve JavaScript heap out of memory for angular production build? [duplicate]

When I run an Ionic 3 project using the ionic serve command, then I am getting this error:
For a non-Angular general answer for those who land on this question from Google:
Most times when you face this error it’s probably because of a memory leak, an addition/version upgrade of a library or a difference in how Node.js manages memory between versions (e.g. Node.js version <= 10 and Node.js version > 10).
Usually just increasing the memory allocated to Node.js will allow your program to run but may not actually solve the real problem and the memory used by the node process could still exceed the new memory you allocate. I'd advise profiling memory usage in your Node.js process when it starts running or updating to Node.js > 10.
I had a memory leak.
Here is a great article on debugging memory leaks in Node.js.
That said, to increase the memory, in the terminal where you run your Node.js process:
export NODE_OPTIONS="--max-old-space-size=8192"
where values of max-old-space-size can be: [2048, 4096, 8192, 16384] etc
More examples for further clarity:
export NODE_OPTIONS="--max-old-space-size=5120" # Increase to 5 GB
export NODE_OPTIONS="--max-old-space-size=6144" # Increase to 6 GB
export NODE_OPTIONS="--max-old-space-size=7168" # Increase to 7 GB
export NODE_OPTIONS="--max-old-space-size=8192" # Increase to 8 GB
# and so on...
# formula:
export NODE_OPTIONS="--max-old-space-size=(X * 1024)" # Increase to X GB
# Note: it doesn't have to be multiples of 1024.
# max-old-space-size can be any number of memory megabytes (MB) you have available.
See the current value of max-old-space-size (in MB)
To see the current (not exact but very close) value of max-old-space-size (in MB), run in your terminal
node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
In my case, I fixed this problem by installing Node.js, version 12.10.0.
I had the same issue on CentOS server 7, but this solved my problem:
node --max-old-space-size=X node_modules/#angular/cli/bin/ng build --prod
Where X = (2048 or 4096 or 8192 o..) is the value of memory.
Just type this in the terminal:
export NODE_OPTIONS="--max-old-space-size=8192"
The error occurs when you exceed the default maximum memory allowed for Node.js. All this does is increase the maximum memory allowed.
I got the same error when I execute ng build command in Visual Studio Code. But I can build successfully when I execute the same thing on the Windows command line in the following sequence.
Step 1.
set NODE_OPTIONS=--max_old_space_size=4096
Step 2.
ng build
Try this solution which was pointed out in an old message on the forum: 3.7.0: iOS build with --prod not working
Open node_modules/#ionic/app-scripts/bin/ionic-app-scripts.js
Change the first line from:
#!/usr/bin/env node
to
#!/usr/bin/env node --max-old-space-size=4096
Try values 1024 and 2048, but for a relatively large app you may need 4096.
Windows
From the control panel go to System → Advanced system settings → Environment Variables → New (user or system)
Or this can be done in PowerShell with:
$env:NODE_OPTIONS="--max-old-space-size=8192"
You can also increase this number, if necessary. We've seen folks need to increase this up to 14 GB for some larger projects!
Linux/macOS
export NODE_OPTIONS=--max-old-space-size=8192
In my case it was a recursion that was causing React to use up all memory.
This happened when I was refactoring my code and didn't notice this.
const SumComponent = () => {
return (
<>
<SumComponent />
</>
)
}
In other Node.js applications this might look like:
const someFunction = () => {
...
someFunction();
...
}
I got the same error message when I executed the following statements in Visual Studio Code. But I can build successfully when I execute the same thing in on the Windows command line.
npm install -g increase-memory-limit
increase-memory-limit
set NODE_OPTIONS=--max_old_space_size=4096
ng build -c deploy --build-optimizer --aot --prod --sourceMap
Updating from Node.js 12 to Node.js 14 solved the problem for me.
Update Now Node.js 16 is available, and I recommend updating to the latest available version of Node.js.
For some reasons all the previous answers didn't really work for me. I did the following to fix my issue:
I had to first delete the node_modules folder
reinstall Node.js on my PC and
then npm install
Adding parameter --build-optimizer resolved the issue in my case:
node --max_old_space_size=4096 ./node_modules/#angular/cli/bin/ng build --prod --build-optimizer
I am not sure why adding only --build-optimizer solves the issue, but as per the Angular documentation it should be used with ahead-of-time (AOT) compilation enabled, so the updated command should be like below:
--build-optimizer=true --aot=true
Angular build documentation
export NODE_OPTIONS="--max-old-space-size=6144" #it will increase to 6gb.
-------If Not Solved try this 2nd step-------------
2) Just update your node version to the latest one will solve this issue.
-------If Not Solved try this 3rd step-------------
3)Just run this command in your windows terminal.
set NODE_OPTIONS=--max_old_space_size=4096
node --max_old_space_size=4096 node_modules/#angular/cli/bin/ng build --baseHref=/baseUrl/ --prod=true
For me, I had a syntax error (which didn't show up) and caused this error.
Run this command in your project folder. Use serve instead of build
node --max_old_space_size=8000 node_modules/#angular/cli/bin/ng serve --prod --port=4202
Replace the line
"start": "ng serve -o --port 4300 --configuration=en" with
"start": "node --max_old_space_size=5096 node_modules/#angular/cli/bin/ng serve -o --port 4300 --configuration=en"
NOTE:
port--4300 is not constant depends upon which port you selects.
--max_old_space_size=5096 too not constant; any value 1024,2048,4096 etc
I faced the same problem on Angular. Then I wrote
"serve": "node --max_old_space_size=8192 ./node_modules/#angular/cli/bin/ng serve"
this script to package.json scripts and for me this problem solved.
And run project this command:
npm run serve
Instead of using ng build, I have executed below command in terminal to fix this issue.
node --max_old_space_size=8192 ./node_modules/#angular/cli/bin/ng build --prod
Then do ng serve.
This is how my terminal look like
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS D:\ProjectPath\Project1> node --max_old_space_size=8192 ./node_modules/#angular/cli/bin/ng build --prod
For me it was a problem with a Firebase package.
Only add "#firebase/database": "0.2.1" for your package.json file. Reinstall node_modules and it works.
For me, the issue was having an extra node_modules folder that I renamed to node_modules_old and running an npm install to generate a fresh node_modules. Somehow the build must have still been picking up the node_modules_old folder, so I moved node_modules_old out of the directory to fix the issue.
I deleted the existing Node.js module and ran the below commands to fix my issue:
npm install -all
npm audit fix
Run this command:
export NODE_OPTIONS="--max-old-space-size=2048"
To check how much you have already:
> node
> v8.getHeapStatistics()
{
total_heap_size: 6049792,
total_heap_size_executable: 524288,
total_physical_size: 5477720,
total_available_size: 1094444024,
used_heap_size: 4141728,
heap_size_limit: 1098907648,
malloced_memory: 8192,
peak_malloced_memory: 582752,
does_zap_garbage: 0,
number_of_native_contexts: 2,
number_of_detached_contexts: 0
}
and then heap_size_limit: 1098907648
Please check your Node.js version:
node -v
If it’s 10.1.1 something, then you need to update your root level Node.js version via the below commands:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
source ~/.nvm/nvm.sh
nvm ls
nvm install 12.18.1
Once done, please restart your terminal or Visual Studio.
It's working 100$.
For Ionic users, please add the below code in your package.json
"ionic:build": "node --max-old-space-size=16384 ./node_modules/#ionic/app-scripts/bin/ionic-app-scripts.js build",
Another non-Angular answer (I was facing the same issue building a React application on AWS Amplify).
As mentioned by Emmanuel, it seems that it comes from the difference in the way memory is handled by Node.js v10 vs. Node.js v12.
I tried to increase memory with no avail. But using Node.js v12 did it.
Check how you can add nvm use $VERSION_NODE_12 to your build settings as explained by richard
frontend:
phases:
preBuild:
commands:
- nvm use $VERSION_NODE_12
- npm ci
build:
commands:
- nvm use $VERSION_NODE_12
- node -v
- npm run-script build
I guess there are plenty of ways to reach this error!
On my side, I had a loop in my package.json. Project A had a dependency on project B, that had a dependency on project A.
Just run this command:
export NODE_OPTIONS="--max-old-space-size=8192"
If you are developing on Windows and running into this issue while publishing, upgrade Node.js through the official site.
The memory usage handling does increase with each newer version of Node.js, although I did not find exact numbers on what the increase is.
That was the only solution that worked for me. It took a whole weekend and more for me to solve this issue.
I am using the latest stable version of Node.js v-14.17. I was having the same issue with new Angular Ionic projects and tried most of the previous answers without success.
Finally after upgrading to Node.js 16.4.2 LTS, it fixed this issue.
For me I got this error because I lost access to the output path for the dist folder set in my angular.json file. After I reconnected to the remote path with updated credentials the error went away.

NPM post-install, CA certificates, and Windows

TLDR;
Is it possible, in a Windows environment, to set NODE_EXTRA_CA_CERTS in a way that works with NPM packages' post-install scripts, without making system changes, configuration file changes, or changes that require admin-level permissions?
Details
This has been driving me nuts.
I added image optimization to our Webpack build process via imagemin, imagemin-webpack, and the various imagemin format-specific plugins.
All of the imagemin plugins have one thing in common -- during post-install, they:
a. Attempt to download a pre-built EXE.
b. If (a) fails, they attempt to build the EXE from source.
I.T. snoops on our traffic, so (a) fails due to the "self-signed certificate in chain" error when attempting to fetch the remote EXE. (b) fails because our studio is Windows-based, and we don't have all the various build tools installed to make that happen. It's also not reasonable to have them installed on every machine where npm install might be run.
I did some digging (thanks S.O.), found our company's CA certificate, added it to the repo, and was able to get (a) working with the following commands:
> SET NODE_EXTRA_CA_CERTS=%cd%\ca.cert
> npm install
I thought I was home free at that point -- all I'd have to do is add this npm script to our package.json:
{
"preinstall": "SET NODE_EXTRA_CA_CERTS=%cd%\\ca.cert"
}
But that doesn't work. I'm guessing it's because there's a separate process involved, and the environment variable doesn't carry over to the other process.
Note that this does work, but is absolutely awful:
{
"preinstall": "SET NODE_EXTRA_CA_CERTS=%cd%\\ca.cert&& npm install imagemin-gifsicle imagemin-mozjpeg imagemin-optipng imagemin-svgo"
}
Is there a way to set this environment variable automatically in a way that works with NPM packages' post-install scripts?
I'd like this to be transparent to other team members so that they can just continue to npm install without any additional steps, system changes, or configuration file changes if at all possible. Some team members are not developers, so while they're used to running npm install, I don't want to introduce any additional complications. I super appreciate any help in advance!
Have you tried
npm config set cafile "<path to your certificate file>"

Sane approach to build and deploy typescript/node.js application

I'm working on node.js app that is written in Typescript, which means it needs to be compiled to JS before running. As I'm coming from java/jvm background where you ship prebuilt package to server and it gets run there I'm a bit afraid of the way of deployment where you push code to git and it's being built/compiled on server first and then run.
I don't like it for two main reasons:
dev dependencies need to be installed on server
deployment depends on external resources availability (npm etc).
I found NAR https://github.com/h2non/nar which is more or less what I wanted but it has some drawbacks (doesn't work with some deps that have native extensions).
My question is: is there any other "sane" way of doing deployment node.js deployment than this risky combination of npm install and tsc on server? Or should I let that sink in and do it that way?
To be honest I don't believe there are no more sane/reliable options for that.
What you can do (but there are probably other perfectly valid approaches) is building your project locally (or on a CI service), and only deploy this built version when you consider it as valid (tests, etc.).
This way, if something bad happens, like npm that fails, or a compilation error, you don't deploy anything, and you have time to resolve the situation.
For example, I used to have a gulp task (but it can be anything else: Grunt, a simple npm script...) that clone a production repository and build the project into this directory.
That way, I can check that my build is valid. If it is, I make a new commit and push it to the production repo, that is served the way you need (on a Heroku instance for example).
Pros
Clear separation of dev and non-dev dependencies
Deployment only when you know that the build is valid
No built files on source control on the development repository
No "live" dependency on external tasks like npm install or tsc build
Cons
You have two separated git repositories (one with the source code, one with the built version of your project)
Production process is a little bit heavier than simply committing to master
(From comment) Doesn't properly handle the case of npm package that relies on native extensions that have to be (re)built
is there any other "sane" way of doing deployment node.js deployment than this risky combination of npm install and tsc on server
package.json + npm install + tsc is the way to do it. Nothing risky about it.
More
Just use an npm script : https://github.com/TypeStrong/ntypescript#npm-scripts

Why does NPM success when submodules fail to build?

Often while using npm I've come across errors that appear to mean nothing - Visual Studio projects failing to build, build tools (eg: python.exe / CL.exe) not being available on the command line etc.
Some examples of packages I've seen fail to build many times:
kerberos
node-gyp
bcrypt
These throw big error messages with stack traces etc to the console during npm install, clearly having failed completely; however, NPM carries along happy as Larry and 9 times out of 10 my Javascript application and all its dependencies work fine.
Does npm install re-build every single dependency recursively, using whatever compilers are available on the local machine?
If so, and considering the huge number of dependencies even simple packages can have, how am I able to do ANYTHING without a full suite of programming languages and compilers installed?
Why is it that these dependencies failing doesn't necessarily mean my final project will be unusable?
If a dependency failing to build is "ok", why bother having the dependency at all?
I haven't been able to find clear answers on any of this, due to the overwhelming number of resources found when searching for terms like "npm build fail".
npm will succeed if those dependencies are actually marked as optional. The ws module is an example of this where they have optional dependencies on two compilable addons. If they fail to build, then ws just uses pure js fallback implementations.
The reason that addons are sometimes added as optional dependencies is that the they (more often than not) perform faster than the pure js alternatives, even for something as "simple" as UTF-8 validation or XOR'ing the contents of a Buffer.

Node.js / npm - anyway to tell if a package is pure JS or not?

I've noticed that in trying to get seemingly simple node packages to install with npm (e.g. nerve, a "micro-framework") I often run into some form of dependency pain. After some digging, I tracked down the problem with nerve to the bcrypt module, which is apparently written in C/C++ and has to be compiled after the package manager downloads it.
Unfortunately, it seems like if you want this to work on Windows, the answer is (from one of the bcrypt issues threads) "install a Linux VM". So earlier today I did just that, and started running into other dependencies (you need certain unnamed apt packages installed before you can even think about building, despite GCC being installed), then eventually after seeing yet another C compiler error (about some package or other not being able to find "Arrays.c" I think), I actually gave up, and switched from nerve to express instead. Ironically, the larger and more complicated express installs with npm on Linux and Windows without a single issue.
So, my question is: is there any filter / dependency tracking available that lets you see if a package has additional dependencies besides node core? Because to me the allure of node is "everything in Javascript", and this kind of stuff dispels the illusion quite unpleasantly. In fact, despite having done more than my time working with C/C++, whenever I see a requirement to "make" something these days I generally run in the other direction screaming. :)
The first solution doesn't tell you if a dependency makes the package impure or not. Much better to search for gyp generated output:
find node_modules/ | grep binding.gyp || echo pure
Look out for the "scripts" field in the package.json.
If it contains something like
"scripts": {
"install": "make build",
}
and a Makefile in the root directory, there's a good possibility that the package has some native module which would have to be compiled and built. Many packages include a Makefile only to compile tests.
This check on the package documents does not exclude the possibility that some dependency will have to be compiled and built. That would mean repeating this process for each dependency in the package.json, their dependencies and so on.
That said many modules have been updated to install, without build on Windows, express for one. However that cannot be assured of all packages.
Using a Linux VM seems to be the best alternative. Developing Node.js applications on Window gives you step by step instructions on installing a VM, Node.js and Express.
Node is not "everything javascript" , since one way to extend node core is to write c/c++ plugins.
So Node is more a javascript wrapper around c/c++ modules using V8.
How could you write efficient database drivers in pure javascript for instance ? it would be possible but slow.
as for the filters , it is up to the author to document his package. there is no automatic filter.

Categories