We Keep Coding

how to combine all yarn files with gulp

I have yarn up and running, have figured out a bit how it works, and made my inroads into figuring out gulp, after having discovered how to install version 4 instead of the default version that throws deprecation errors.
Now I have installed 3 packages with yarn, and it has downloaded a LOT of dependencies. No problem, one can use a gulp file to combine those into one javascript(or so i'm told)
The only thing is, how do I do that whilst maintaining the yarn dependencies as yarn builds those up? How would I format my gulp task for combining the yarn libaries i've added?
My gulp task currently looks like this:
//Concatenate & Minify JS
gulp.task('scripts', function() {
return gulp.src('assets/javascript/*.js')
.pipe(concat('all.js'))
.pipe(gulp.dest('assets/dist'))
.pipe(rename('all.min.js'))
.pipe(uglify())
.pipe(gulp.dest('assets/dist/js'));
});
And this concatenates my scripts as it should, but when I wanted to add the yarn folder it hit me that yarn manages dependencies and what not so everything has it's correct dependency and such. I doubt I can just add them all to the same file and hope all is well.(or can I?)
I run this task with yarn run watch
I've added the following packages: html5shiv, jquery, modernizr
What would be the correct way to add the yarn files in in assets/node_modules?

After long searching I found https://pawelgrzybek.com/using-webpack-with-gulpjs/
which gave me the following solution:
Execute the command:
sudo yarn add global gulp webpack webpack-stream babel-core babel-loader babel-preset-latest
create a webpack.config.js file
enter in it:
module.exports = {
output: {
filename: 'bundle.js', // or whatever you want the filename to be
},
module: {
rules: [
{
test: /\.(js|jsx)$/,
exclude: /(node_modules)/,
loader: 'babel-loader',
query: {
presets: [
['latest', { modules: false }],
],
},
},
],
},
};
Then create a gulpfile.js
var gulp = require('gulp');
var webpack = require('webpack');
var webpackStream = require('webpack-stream');
var webpackConfig = require('./webpack.config.js');
gulp.task('watch', watchTask);
gulp.task('default', defaultTask);
gulp.task('scripts', function() {
return gulp.src('assets/javascript/*.js')
.pipe(webpackStream(webpackConfig), webpack)
.pipe(gulp.dest('./assets/js')); // Or whereever you want your js file to end up.
});
function watchTask(done) {
// Wherever you stored your javascript files
gulp.watch('assets/javascript/*.js', gulp.parallel('scripts'))
done();
}
function defaultTask(done) {
// place code for your default task here
done();
}
Then in the directory execute yarn watch and have it run in the background where you can throw an eye on it now and then.

Gulp not creating CSS folder on production build

I have a static site that uses two gulpfiles to compile. One is gulpfile.dev.js which compiles all the files correctly and uses a static server served by browsersync. The gulpfile.prod.js is the the exact same as the dev file, just without browsersync server. When I compile my file using DeployHQ to move files on my server it doesn't compile the necessary css folder.
My gulpfile.prod.js file:
//DH
var gulp = require('gulp'), //Task runner
uglify = require('gulp-uglify'), //Minimizies JS
sass = require('gulp-ruby-sass'), //Compiles to CSS
imagemin = require('gulp-imagemin'), // Minimize images
concat = require('gulp-concat'),
concatCss = require('gulp-concat-css'),
gutil = require('gulp-util'),
autoprefixer = require('gulp-autoprefixer'),
cssnano = require('gulp-cssnano'),
pug = require('gulp-pug'),
htmlmin = require('gulp-htmlmin');
// Define paths for sources and destination.
var paths = {
src: {
js: './src/js/*.js',
sass: './src/sass/**/*.sass',
html: './views/*.pug'
},
dest: {
js: './app/build/js',
css: './app/build/css',
html: './app'
}
};
var autoprefixerOptions = {
browsers: ['last 2 versions', '> 5%', 'Firefox ESR']
};
//Error look-outs in gulp
function errorLog(error){
console.error.bind(error);
this.emit('end');
}
// Scripts
gulp.task('scripts', function(){
gulp.src(paths.src.js)
.pipe(uglify())
.on('error', errorLog)
.pipe(concat('main.js'))
.pipe(gulp.dest(paths.dest.js))
});
// PUG -> HTML
gulp.task('pug', function buildHTML() {
return gulp.src(paths.src.html)
.pipe(pug())
.pipe(gulp.dest(paths.dest.html))
.pipe(htmlmin({collapseWhitespace: true}));
});
// Styles
gulp.task('styles', function(){
return sass(paths.src.sass)
.on('error', errorLog)
.pipe(autoprefixer(autoprefixerOptions))
.pipe(cssnano())
.pipe(gulp.dest(paths.dest.css))
});
//Run task
gulp.task('default', [
'scripts',
'pug',
'styles']);
These are the bash commands DeployHQ executes to serve the files correctly in the right directory. Most of it is just removing files that I don't need to be on the server. I build the files needed to be served from app/ then move the files to the root html directory. nam run prod translates to gulp default --gulpfile gulpfile.prod.js
cd /var/www/site.com/html
npm install
npm run prod
rm -rf node_modules/ src/ views/
cd app
cp -a build/ /var/www/site.com/html
cp index.html /var/www/site.com/html
cd ..
rm -rf app/
rm .gitlab-ci.yml gulpfile.dev.js gulpfile.prod.js package.json .gitignore README.md
Edit:
Running the prod file locally compiles everything as its supposed to, but on the server it does not. It does not compile the css folder.
Edit 2:
I've changed my styles task to reflect gulp-ruby-sass's documentation to this:
gulp.task('styles', function(){
sass(paths.src.sass)
.on('error', errorLog)
.pipe(autoprefixer(autoprefixerOptions))
.pipe(cssnano())
.pipe(gulp.dest(paths.dest.css))
});
The CSS folder still does not compile.

Run a npm script from gulp task

How to run a npm script command from inside a gulp task?
package.json
"scripts":
{
"tsc": "tsc -w"
}
gulpfile.js
gulp.task('compile:app', function(){
return gulp.src('src/**/*.ts')
.pipe(/*npm run tsc*/)
.pipe(gulp.dest('./dist'))
.pipe(connect.reload());
});
I want to do this because running npm run tsc does not give me any error but if I use gulp-typescript to compile .ts then I get bunch of errors.

You can get the equivalent using gulp-typescript
var gulp = require('gulp');
var ts = require('gulp-typescript');
gulp.task('default', function () {
var tsProject = ts.createProject('tsconfig.json');
var result = tsProject.src().pipe(ts(tsProject));
return result.js.pipe(gulp.dest('release'));
});
gulp.task('watch', ['default'], function() {
gulp.watch('src/*.ts', ['default']);
});
Then on your package.json
"scripts": {
"gulp": "gulp",
"gulp-watch": "gulp watch"
}
Then run
npm run gulp-watch
Alternatively using shell
var gulp = require('gulp');
var shell = require('gulp-shell');
gulp.task('default', function () {
return gulp.src('src/**/*.ts')
.pipe(shell('npm run tsc'))
.pipe(gulp.dest('./dist'))
.pipe(connect.reload());
});
gulp-shell has been blacklisted you can see why here
Another alternative would be setting up webpack.

Wasted about 1 hour on this simple thing, looking for a ~complete answer, so adding another here:
If you question is only on typescript (tsc), see https://stackoverflow.com/a/36633318/984471
Else, see below for a generic answer.
The question title is generic, so a generic example is given below first, then the answer.
Generic example:
Install nodejs, if you haven't, preferably LTS version, from here: https://nodejs.org/
Install below:
npm install --save-dev gulp gulp-run
File package.json has below contents (other contents can be there):
{
"name": "myproject",
"scripts": {
"cmd1": "echo \"yay! cmd1 command is run.\" && exit 1",
}
}
Create a file gulpfile.js with below contents:
var gulp = require('gulp');
var run = require('gulp-run');
gulp.task('mywatchtask1', function () {
// watch for javascript file (*.js) changes, in current directory (./)
gulp.watch('./*.js', function () {
// run an npm command called `test`, when above js file changes
return run('npm run cmd1').exec();
// uncomment below, and comment above, if you have problems
// return run('echo Hello World').exec();
});
});
Run the task mywatchtask1 using gulp?
gulp mywatchtask1
Now, gulp is its watching for js file changes in the current directory
if any changes happen then the npm command cmd1 is run, it will print yay! cmd1 command is run. everytime the one of the js file changes.
For this question: as another example:
a) package.json will have
"tsc": "tsc -w",
instead of the below:
"cmd1": "echo \"yay! cmd1 command is run.\" && exit 1",
b) and, gulpfile.js will have:
return run('npm run tsc').exec();
instead of below:
return run('npm run cmd1').exec();
Hope that helps.

You can try to implement it using childprecess node package or
use https://www.npmjs.com/package/gulp-run
var run = require('gulp-run');
gulp.task('compile:app', function(){
return gulp.src(['src/**/*.js','src/**/*.map'])
.pipe(run('npm run tsc'))
.pipe(gulp.dest('./dist'))
.pipe(connect.reload());
});

What is wrong with this code that combines multiple js files to one?

I have this node.js code that tries to minify and combine multiple js files to a single js file.
var concat = require('gulp-concat');
var gulp = require('gulp');
gulp.task('scripts', function() {
//gulp.src(['./lib/file3.js', './lib/file1.js', './lib/file2.js'])
gulp.src(['./js/*.js'])
.pipe(concat('all.js'))
.pipe(uglify())
.pipe(gulp.dest('./dist/'))
});
All my js files are located in js folder. My node.js file is above the js folder. I am expecting the single minified file to appear in dist folder. I see nothing and get no error message when I run the code. What could have gone wrong?

Gulpfile.js:
"use strict";
var concat = require('gulp-concat');
var gulp = require('gulp');
var uglify = require('gulp-uglify'); // Add gulp-uglify module to your script
gulp.task('scripts', function() {
gulp.src('./js/*.js')
.pipe(concat('all.js'))
.pipe(uglify())
.pipe(gulp.dest('./dist/'));
});
Check package.json dependencies
Run npm install to verify that all dependencies correctly loaded. I think this was your issue:
{
"dependencies": {
"gulp-concat": "2.x",
"gulp": "3.x",
"gulp-uglify": "1.x"
}
}

How to automatically zip files with Node.js and npm

Is there a way to automatically zip certain files at the build time with Node.js and npm?
For example, I have a project, that file structure looks like this:
Project/
--lib/
--node_modules/
--test/
--index.js
--package.json
I want to be able to zip lib folder, certain modules from node_modules and index.js into some zip archive to upload it on the AWS Lambda, for example. I do not need test folder or test Node.js modules (mocha and chai) to be zipped. I have even created a bash script for generating zip file, but is there a way to automatically execute this script, when 'npm install' is called?
This should be a standard problem and it should have a standard solution, but I was unable to discover such.
UPDATE
thanks to michael, decided to use gulp. This is my script, in case some one else will need it for AWS Lambda:
var gulp = require('gulp');
var clean = require('gulp-clean');
var zip = require('gulp-zip');
var merge = require('merge-stream');
gulp.task('clean', function () {
var build = gulp.src('build', {read: false})
.pipe(clean());
var dist = gulp.src('dist', {read: false})
.pipe(clean());
return merge(build, dist);
});
gulp.task('build', function() {
var index = gulp.src('index.js')
.pipe(gulp.dest('build'));
var lib = gulp.src('lib/**')
.pipe(gulp.dest('build/lib'));
var async = gulp.src('node_modules/async/**')
.pipe(gulp.dest('build/node_modules/async'));
var collections = gulp.src('node_modules/collections/**')
.pipe(gulp.dest('build/node_modules/collections'));
var underscore = gulp.src('node_modules/underscore/**')
.pipe(gulp.dest('build/node_modules/underscore'));
var util = gulp.src('node_modules/util/**')
.pipe(gulp.dest('build/node_modules/util'));
var xml2js = gulp.src('node_modules/xml2js/**')
.pipe(gulp.dest('build/node_modules/xml2js'));
return merge(index, lib, async, collections, underscore, util, xml2js);
});
gulp.task('zip', ['build'], function() {
return gulp.src('build/*')
.pipe(zip('archive.zip'))
.pipe(gulp.dest('dist'));
});
gulp.task('default', ['zip']);

I realize this answer comes years too late for the original poster. But I had virtually the same question about packaging up a Lambda function, so for posterity, here's a solution that doesn't require any additional devDependencies (like gulp or grunt) and just uses npm pack along with the following package.json (but does assume you have sed and zip available to you):
{
"name": "my-lambda",
"version": "1.0.0",
"scripts": {
"postpack": "tarball=$(npm list --depth 0 | sed 's/#/-/g; s/ .*/.tgz/g; 1q;'); tar -tf $tarball | sed 's/^package\\///' | zip -#r package; rm $tarball"
},
"files": [
"/index.js",
"/lib"
],
"dependencies": {
"async": "*",
"collections": "*",
"underscore": "*",
"util": "*",
"xml2js": "*"
},
"bundledDependencies": [
"async",
"collections",
"underscore",
"util",
"xml2js"
],
"devDependencies": {
"chai": "*",
"mocha": "*"
}
}
Given the above package.json, calling npm pack will produce a package.zip file that contains:
index.js
lib/
node_modules/
├── async/
├── collections/
├── underscore/
├── util/
└── xml2js/
The files array is a whitelist of what to include. Here, it's just index.js and the lib directory.
However, npm will also automatically include package.json, README (and variants like README.md, CHANGELOG (and its variants), and LICENSE (and the alternative spelling LICENCE) unless you explicitly exclude them (e.g. with .npmignore).
The bundledDependencies array specifies what packages to bundle. In this case, it's all the dependencies but none of the devDependencies.
Finally, the postpack script is run after npm pack because npm pack generates a tarball, but we need to generate a zip for AWS Lambda.
A more detailed explanation of what the postpack script is doing is available at https://hackernoon.com/package-lambda-functions-the-easy-way-with-npm-e38fc14613ba (and is also the source of the general approach).

If you're UNIX-based you could also just use the zip command in one of your scripts:
"scripts": {
"zip": "zip -r build.zip build/"
"build": "build",
"build-n-zip": "build && zip
}
The above creates a build.zip at the root, which is a zipped up version of the /build folder.
If you wanted to zip multiple folders/files, just add them to the end:
"scripts": {
"zip": "zip -r build.zip build/ some-file.js some-other-folder/"
}
Note
If a build.zip already exists in the folder, the default behaviour is for zip to add files to that existing archive. So many people who are continuously building will probably want to delete the build.zip first:
"scripts": {
"zip": "rm -f build.zip && zip -r build.zip build",
"build": "build",
"build-n-zip": "yarn build && yarn zip"
}

I would go with gulp using gulp-sftp, gulp-tar and gulp-gzip and an alias as command. Create a file called .bash_aliases in your users home folder containing
alias installAndUpload='npm install && gulp runUploader'
After a reboot you can call both actions at once with this alias.
A gulp file could look something like this
var gulp = require('gulp');
var watch = require('gulp-watch');
var sftp = require('gulp-sftp');
var gzip = require('gulp-gzip');
gulp.task('runUploader', function () {
gulp.src('.path/to/folder/to/compress/**')
.pipe(tar('archive.tar'))
.pipe(gzip())
.pipe(gulp.dest('path/to/folder/to/store')) // if you want a local copy
.pipe(sftp({
host: 'website.com',
user: 'johndoe',
pass: '1234'
}))
});
Of course, you can also add gulp-watch to automatically create the tar/zip and upload it whenever there is a change in the directory.

You should take a look to npm scripts.
You'll still need a bash script laying around in your repository, but it will be automatically triggered by some npm tasks when they are executed.

npm-pack-zip worked for me.
npm install --save-dev npm-pack-zip
To publish the whole lambda using aws I used this node script in package.json:
"publish": "npm-pack-zip && aws lambda update-function-code --function-name %npm_package_name% --zip-file fileb://%npm_package_name%.zip && rm %npm_package_name%.zip"

You can use Zip-Build, this little package will use the data in your package.json file and create a compressed file named project-name_version.zip.
Disclaimer: I am a developer of this library.
How to use zip-build
Just install in your project as dev dependency with:
$ npm install --save-dev zip-build
Then modify the build script in your package.json, adding && zip-build at the end, like this:
"scripts": {
"build": your-build-script && zip-build
}
If your build directory is named different than build and your desired directory for compressed files is named different than dist, you can provide the directory names as arguments for zip-build:
"scripts": {
"build": your-build-script && zip-build build-dirname zip-dirname
}

If you need automate tasks take a look to Grunt or Gulp.
In the case of Grunt needed plugins:
https://www.npmjs.com/package/grunt-zip
https://www.npmjs.com/package/grunt-aws-lambda

Check out my gist at https://gist.github.com/ctulek/6f16352ebdfc166ce905
This uses gulp for all the tasks you mentioned except creating the lambda function initially (it only updates the code)
It assumes every lambda function is implemented in its own folder, and you need to define your AWS credential profile.