I've been trying for some time to cache node_modules on a Docker build. I've tried several approaches including the one here, but without success.
My main reason to cache is because it takes 30+ minutes to build my image, which is way too much.
My Dockerfile:
# This image will be based on the oficial nodejs docker image
FROM node:4.2.1
RUN npm install -g jspm#0.17.0-beta.7 && \
npm install -g gulp && \
npm install -g tsd
# Use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
ADD package.json /src/package.json
RUN cd /src && npm install
# Put all our code inside that directory that lives in the container
ADD . /src
# Set in what directory commands will run
WORKDIR /src
# Install dependencies
RUN cd /src && \
tsd reinstall -so && \
jspm install && \
gulp build -p
# Tell Docker we are going to use this port
EXPOSE 3000
# The command to run our app when the container is run
CMD ["npm", "run", "start-production"]
I do not have a .dockerignore file. I added one before but it still didn't cache my node_modules.
So, how to I cache my node_modules? Feel free to suggest modifications to the Dockerfile.
Thanks!
I'm not sure whether it is the root of the error, but try to specify the destination folder in the ADD command and not the destination file.
ADD package.json /src
Moreover, you can use COPY instead of ADD (ADD can work with url and archives but you don't need it here).
You can also specify your working directory earlier in the file.
Try with this code :
# This image will be based on the official nodejs docker image
FROM node:4.2.1
RUN npm install -g jspm#0.17.0-beta.7 && \
npm install -g gulp && \
npm install -g tsd
# Set in what directory commands will run
WORKDIR /src
# Use changes to package.json to force Docker not to use the cache
# when we change our application’s nodejs dependencies:
COPY package.json ./
RUN npm install
# Put all our code inside that directory that lives in the container
COPY . ./
# Install dependencies
RUN tsd reinstall -so && \
jspm install && \
gulp build -p
# Tell Docker we are going to use this port
EXPOSE 3000
# The command to run our app when the container is run
CMD ["npm", "run", "start-production"]
Related
Is there any way to use the package installed globally instead of install the same when we run npm i.
I have the following context:
I've created a docker image with one package already installed (install statement in the Dockerfile). When I run the container with the volumen which has the javascript project and I run npm i it seems the package is installed again.
Could it be possible use the global package instead of instal it again?
If you dont want to install again during the docker build the node_modules, you can create a base image with your node_modules.
FROM node:18-alpine3.16
WORKDIR /usr/src/app
COPY package* ./
RUN npm install
docker build -t myimage .
You can tag and push it to registry (dockerhub)
Or use it locally as your new base image.
FROM myimage
WORKDIR /usr/src/app #Here are now already your node_modules
COPY . . # copy your code inside the image, or map the folder if you are in
# development
RUN npm run build # if you need a build
CMD ["mycommand","myargument"]
Note this quick, but you need to rebuild your base image if you add or update the node modules.
Consider also to use npm ci instead of npm install. In this way you keep same versions of your node modules each time you reinstall them.
Take care of security issues of the node modules as well of your base images.
run docker scan myimage, periodically to get information of your node_modules or base images need updates.
During development, it is absolutly fine to map your code folders with the docker image. While developing is not even necessary to copy your code in the image. Just map it in your WORKDIR.
After researching it seems the follow code solves my problem
WORKDIR /dir
RUN npm install package
RUN npm cache add package
RUN rm -rf *
WORKDIR /
I made 2 docker images from 2 folder in this repo. The dockerfile was:
FROM node:13-alpine
EXPOSE 5000
RUN mkdir /app
RUN mkdir -p /home/app
COPY . /home/app
RUN npm install
CMD ["npm","run","start"]
But the docker logs showed that the package.json file was not found and hence the containers closed, I can't even enter the terminal of the containers to see the available files.
The error output is in the below image.
To reproduce the error, clone the repo and run sudo docker-compose -f docker-compose.yaml up
You didn't specify the working directory in your dockerfile and that's why the package.json cannot be found.
Ref: https://docs.docker.com/engine/reference/builder/#workdir
add WORKDIR before RUN npm install
WORKDIR /home/app
RUN npm install
I just found about pnpm today and it helped solve my issue with npm timing out on my installation which is amazing.
I've a problem with pnpm tho in the docker image.
Previously with just npm I had unprivileged user like so
FROM node:14.17.3-slim
# build args
ARG NPM_AUTH_TOKEN
ARG HOME=/home/app
ARG NPMRC_PATH=$HOME/.npmrc
# setup unprivileged user
RUN useradd -Umrd $HOME app
WORKDIR $HOME
USER app
# copy configuration
COPY --chown=app:app "bin" "bin"
COPY --chown=app:app "package.json" "package-lock.json" "webpack.config.js" ".babelrc" ./
RUN ./bin/write_npmrc && \
npm ci --production=false
ENV NODE_ENV=development
VOLUME ["$HOME/config", "$HOME/log", "$HOME/src"]
CMD ["npm", "start"]
EXPOSE 9000
But if I switch to the pnpn I'm no longer able to proceed with building the image due to Permission Denied and I need to use root user.
FROM node:14.17.3-slim
# build args
ARG NPM_AUTH_TOKEN
ARG HOME=/home/app
ARG NPMRC_PATH=$HOME/.npmrc
RUN apt-get update && apt-get install -y curl \
&& rm -rf /var/lib/apt/lists/*
RUN curl -f https://get.pnpm.io/v6.16.js | node - add --global pnpm
WORKDIR $HOME
# copy configuration
COPY "bin" "bin"
COPY "package.json" "pnpm-lock.yaml" "webpack.config.js" ".babelrc" ./
RUN ./bin/write_npmrc && \
pnpm install --frozen-lockfile
ENV NODE_ENV=development
VOLUME ["$HOME/config", "$HOME/log", "$HOME/src"]
CMD ["pnpm", "start"]
EXPOSE 9000
Is there a way so I can keep
# setup unprivileged user
RUN useradd -Umrd $HOME app
WORKDIR $HOME
USER app
With pnpm instead?
You just need to change to your non-privileged user after installing the system packages.
Example:
FROM node:14.17.3-slim
# build args
ARG NPM_AUTH_TOKEN
ARG HOME=/home/app
ARG NPMRC_PATH=$HOME/.npmrc
RUN apt-get update && apt-get install -y curl \
&& rm -rf /var/lib/apt/lists/*
RUN curl -f https://get.pnpm.io/v6.16.js | node - add --global pnpm
# setup unprivileged user
RUN useradd -Umrd $HOME app && \
chown -R app:app $HOME
WORKDIR $HOME
USER app
# copy configuration
COPY --chown=app:app "bin" "bin"
COPY --chown=app:app "package.json" "pnpm-lock.yaml" "webpack.config.js" ".babelrc" ./
RUN ./bin/write_npmrc && \
pnpm install --frozen-lockfile
ENV NODE_ENV=development
VOLUME ["$HOME/config", "$HOME/log", "$HOME/src"]
CMD ["pnpm", "start"]
EXPOSE 9000
As per the yarn installation for yarn v2, they want you to install using npm install -g yarn. So I ran sudo npm install -g yarn on Ubuntu 20.04. But after I do that, it says command not found.
❯ sudo npm install -g yarn
> yarn#1.22.10 preinstall /usr/local/lib/node_modules/yarn
> :; (node ./preinstall.js > /dev/null 2>&1 || true)
❯ yarn --version
zsh: command not found: yarn
sudo npm install -g npm
then
sudo npm install -g yarn
Then reboot your system. That did it for me.
Before a reboot only sudo yarn worked. I tried looking at file permissions but everything seemed in order and the files were executable as expected. Nevertheless after a reboot it worked.
If you go to /usr/local/bin after the installation there's a link there to where yarn.js lives, as expected, and file permissions for it were also correct.
/usr/local/bin is added to $PATH, so it's surprising that it doesn't see the new cmd right away, but perhaps it didn't reload or map it until after the reboot? I don't know. But I just spent a good hour trying to figure this out so I'm posting what worked for me to spare other the hassle.
TL;DR
If you are managing node via nvm, then probably the path to yarn binary is not included in the $PATH variable. You should add this -
# Add this at the end (or after the $NVM_DIR initialization)
# in your profile - .bashrc | .zshrc | .profile, etc
export PATH="`yarn global bin`:$PATH"
at the end of your profile file (.zshrc for me) or at least after the $NVM_DIR initialization.
I have recently faced this issue and while searching for a solution, I landed up here.
Here is what my environment looks like:
OS: Ubuntu 20.04
Shell: zsh
NodeJS: managing it via nvm, and NOT apt.
After going through all the answers, I was not keen on uninstalling anything. So I tried to dig a bit deeper.
I installed yarn via npm install -g yarn command. So the first thing I wanted to verify was the location of the yarn binary. To do this, I ran the command where yarn which lists the installation path for the yarn binary.
$ where yarn
/home/<user_name>/.nvm/versions/node/v16.11.1/bin/yarn
Then it hit me. In my .zshrc file, I had added the yarn global bin command (which spills out the directory of all the global packages installed by yarn) at the top like so:
# Top of my .zshrc file
export PATH="`yarn global bin`:$HOME/bin:/usr/local/bin:$PATH"
and as per the installation instruction of nvm, the $NVM_DIR (the variable which holds the nvm directory path) was added at the end of my .zshrc file.
So when I was starting up my shell, it was actually trying to load the yarn command (present inside the nvm directory) even before loading the $NVM_DIR path.
To solve this, I tweaked my .zshrc file and moved the yarn global bin command after the $NVM_DIR like this:
# Top of my .zshrc file
export PATH="$HOME/bin:/usr/local/bin:$PATH"
# ...
#
# Something in between
#
# ...
# Bottom of my .zshrc file
export NVM_DIR="${HOME}/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
# Here is where I have added the path to yarn global
export PATH="`yarn global bin`:$$PATH"
I hope that this would be of help.
This solved it for me:
corepack enable
(if you get "Internal Error: EACCES: permission denied", run it with sudo)
This is also recommended by the Yarn documentation: https://yarnpkg.com/getting-started/install
Uninstall cmdtest:
sudo apt remove cmdtest
Then, run these commands:
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update
sudo apt install yarn
If you want to avoid reboot, use
/usr/local/lib/node_modules/yarn/bin/yarn --version
The yarn documentation is missing a step, you need to restart your computer between this installation and running yarn --version.
This worked for me
I recently had a similar situation and here is how I solved it.
First I troubleshoot the current npm installation:
npm config -list
I had a ~/.npmrc file that had a different prefix:
PREFIX=/opt/homebrew
That made my npm installation look for globally installed packages under /opt/homebrew.
In my case, I'm using a different npm installation (not with homebrew anymore). A simple fix is to remove this custom PREFIX from the ~/.npmrc file and the problem was solved.
Now npm looks for globally installed packages under /usr/local/bin/.
I installed yarn with npm install -g yarn on git bash and I tested it with yarn -v that show the version of the installed yarn, but when I used yarn start it gives me this error
C:\Users\{username}\AppData\Roaming\npm/node_modules/node/bin/node: line 1: This: command not found
These are simple steps that I used to fix my problem on Windows 10:
Uninstall node.js
Restart your computer
Delete your C:\Program Files\nodejs and C:\Users\{username}\AppData\Roaming\npm
Install node.js again and check it with node -v
Start your vs code as an admin and write npm install
Write yarn start
I have 0 experience with GitLab Continuous Integration and I need to setup a job to run ESLint on .js files.
I've already read the GitLab CI and Pipeline documentations, along with some Git Hooks, but I still have no idea how to setup this, so any detailed and starting from the very beginning answer is appreciated.
First you need to setup your CI and have some runners available so they can run your continuous integration jobs. The easiest way for this is to use gitlab-ci-multi-runner (project is here along with documentation) along with the docker executor that will run your CI jobs in docker containers. Once you have configured some runners, add them to your Gitlab project so they're available to run jobs.
Once that's taken care of, you need to add a .gitlab-ci.yml file to your project. This file is used to describe the jobs that need to run during continuous integration etc. Here is an example (assuming you install eslint using npm)
image: node:latest
stages:
- lint
eslint:
stage: lint
script:
# Install ESLint in this docker container
- npm install -g eslint
# Configure ESLint (will read your .eslintrc file)
- eslint --init
# Run ESLint
- eslint <your_js_file>
Add your .gitlab-ci.yml file, commit and push the changes. The CI pipeline should start and run the above steps.
If you'd like to have comments to your PRs here is an example with eslint and pronto. (we have ruby app so we check ruby code style too )
image: 'circleci/ruby:2.5.1-node-browsers'
codestyle:
script:
- sudo apt -y install cmake
# install eslint dependencies
- sudo npm install -g eslint
- sudo npm install -g eslint-plugin-babel
- sudo npm install -g eslint-plugin-react
- sudo npm install -g eslint-plugin-import
- sudo npm install -g babel-eslint
- sudo npm install -g eslint-config-airbnb
- sudo npm install -g eslint-plugin-jsx-a11y
# install pronto runners
- gem install pronto --no-ri
- gem install pronto-rubocop --no-ri
- gem install rubocop-rspec --no-ri
- gem install pronto-eslint_npm --no-ri
# run linters
- vendor/ruby/bin/pronto run -f gitlab -c origin/dev --exit-code
You can also run linters separately:
- vendor/ruby/bin/pronto run -r eslint_npm -f gitlab -c origin/dev --exit-code
This piece -f gitlab -c origin/dev tells linters to check only changed lines of code.
Also if you use pronto-eslint_npm and want to check files in specific folder add
.pronto_eslint_npm.yml that will contain needed regex. (in my case it has next line)
files_to_lint: app\/frontend\/\S*(\.js|\.es6|\.jsx)$