As the documentation says, they both deal with transforming non-stream plugins to stream.
What I try to understand is, if I can use the .pipe() method on something, doesn't it mean it's a stream?
If so, what do I convert to what here?
vinyl-source-stream example:
(from: https://www.npmjs.com/package/vinyl-buffer)
var browserify = require('browserify')
var source = require('vinyl-source-stream')
var buffer = require('vinyl-buffer')
var uglify = require('gulp-uglify')
var size = require('gulp-size')
var gulp = require('gulp')
gulp.task('build', function() {
var bundler = browserify('./index.js')
return bundler.pipe()
.pipe(source('index.js'))
.pipe(buffer()) // <---------------------- why?
.pipe(uglify())
.pipe(size())
.pipe(gulp.dest('dist/'))
})
gulp-streamify example:
(from: https://www.npmjs.com/package/vinyl-source-stream)
var source = require('vinyl-source-stream')
var streamify = require('gulp-streamify')
var browserify = require('browserify')
var uglify = require('gulp-uglify')
var gulp = require('gulp')
gulp.task('browserify', function() {
var bundleStream = browserify('index.js').bundle()
bundleStream
.pipe(source('index.js'))
.pipe(streamify(uglify())) // <----------- why?
.pipe(gulp.dest('./bundle.js'))
})
One semi-useful example is to think about putting out a campfire with a bucket of water. To put out the fire you would want to completely fill up the bucket before dumping it on the fire rather putting a few drops in the bucket and then dumping lots of little drops over time on the fire. This metaphor doesn't capture everything but the big idea is this: you need a FULL bucket of water before you can put out the fire.
That "uglify" plugin works the same way. Imagine some enormous JS file you'd want to compress/uglify.
It will take a little bit of time to load the whole codebase & you definitely wouldn't want to try minifying each line as it comes in, right? Imagine you load a single line, minify it, load another line, minify it, etc etc-- it'd be a mess. You can't stream it (you need a full "bucket" of code before you can uglify it.) To uglify that file properly you'd need to load all that code first before attempting to uglify it.
Since Gulp is a "streaming" build system, you can't use uglify unless you have some mechanism to turn the stream into a buffer (& when it's done emit a stream.) Both tools you mention make this possible.
Here's the flow:
STREAM > (BUFFER) > {perform some work on the whole "buffered" file} > STREAM > {other gulp work, etc }
To your specific question, you can use .pipe() because vinyl-buffer/gulp-streamify help "convert" streams to buffers then buffers to streams. They're different approaches to accomplish essentially the same thing.
As said, most plugins work with buffers (although some of them also support streams). Examples include gulp-uglify and gulp-traceur. You can do the conversion to buffers using gulp-buffer.
via https://medium.com/#webprolific/getting-gulpy-a2010c13d3d5
gulp-uglify dosen't support stream, so you should convert stream to buffer (example uses vinyl-buffer)
gulp-streamify can wrap old plugins to support streams(example uses gulp-uglify)
Different approaches but equally satisfactory results.
What I try to understand is if I can use the .pipe() method on
something, doesn't it mean that it's a stream?
No, .pipe() can also pass buffers. this blog post explains it well:
https://medium.com/#sogko/gulp-browserify-the-gulp-y-way-bb359b3f9623
Some gulp-* plugins works by taking in buffered vinyl files objects as
input.
But vinyl-source-stream emits a streaming vinyl file object.
That’s where vinyl-buffer comes in. So we simply need to convert that
to a buffered vinyl by using vinyl-buffer, like so
What I try to understand is if I can use the .pipe() method on
something, doesn't it mean that it's a stream?
Yes! It is a stream. But it's a an object stream!
Instead of streaming a series characters, it streams a series of objects, which are the files that you sourced.
Each 'data' event in a gulp stream emits a Vinyl file object, which looks something like this:
{
cwd: '/', //<string>
base: '/test/', //<string>
path: '/test/file.js', //<string>
contents: contents //<string> | <Buffer> | <stream.Readable>
}
So gulp-buffer plugin is a Transform stream that converts the file contents from stream.Readable to Buffer.
You can see this in the source, where it saves the original content stream on line 24 and assigns a Buffer as the new file contents on line 35.
Streamify does the same thing, on line 35 and line 48.
It's ok to leave the file contents as a Buffer after Uglify is done processing it. It's always ok for the contents to be a Buffer, gulp just doesn't do this when sourcing because it's too costly.
Related
I want to combine two gzipped files: dist/public/scripts/vendors.js.gz and dist/public/scripts/scripts.js.gz. I am currently trying to do it like this:
gulp.task('scriptCombination', ['compressScripts'], () => {
var jsDest = 'dist/public/scripts';
gulp.src(['dist/public/scripts/vendors.js.gz', 'dist/public/scripts/scripts.js.gz'], { base: '.' })
.pipe(concat('new-file.js.gz'))
.pipe(gulp.dest(jsDest));
});
Right now it seems like the first file is zipping correctly, but the second file always seems to get messed up. Upon return and unzip, the files will look good at first, but scrolling down it seems like things get really messy. new-file.js.gz ends up looking something like this when it is unzipped in the browser:
oadprogress","reset","queuecomplete"];e.forEach(p,function(o){var n=t.callbacks[o]||e.noop;s.on(o,function(){n.apply(null,arguments),t.$$phase||t.$root.$$phase||t.$apply()})})}}}}])}"object"==typeof module&&module.exports?module.exports=o(require("angular"),require("dropzone")):"function"==typeof define&&define.amd?define(["angular","dropzone"],o):o(e.angular,e.Dropzone)}(this);
‹�����í=ksÛ¶–ß÷W0œ\/¹Ah;¶¡ÂzœWë½Mš‰tîf²Z„-Æ©’”EÖ߃'¤(ÇN;s7Ó©…×p^À9�Oçù¸N‹Üyìà ü¦xž¥ãsÏ_žŠ¢,ÍϽj\Ì0žâ¼Fq]—éɼÆ×eæ//âÒIó‰H:8ÃõQ|ò’Õ> ùoê(¤àîe\'^(˜ŠŒ‘èÚ‹iŽ¿d·¶<
ý´Ï«ƒ¦·
(good and minified for the first file, bad for the second)
How can I combine these two files in gulp?
EDIT: As I wrote IN THE TITLE AND FROM THE BEGINNING, this is not about command-line parameters and is thus NOT A DUPLICATE. //EDIT
I have a Sass setup with an indefinite number of uniquely-designed pages (page_1, page_2, etc), each having their own sass/pages/page_1/page_1.scss file.
The pages all belong to the same website, and each page's sass file #imports the same set of files from a sass/includes folder.
With a basic gulp task watching sass/**/*, every page's styles get compiled anytime I make a change to any page's styles. Obviously this doesn't scale well.
I tried using gulp-watch, but it doesn't catch if changes are made to one of the included .scss files. It only catches changes made to the files that actually get compiled into an equivalent .css.
For the purposes of having my gulpfile be as DRY as possible, the best solution I could come up with was to maintain a basic array of folder names in gulpfile.js, and to loop through and watch each of them separately, using the same sass-compiling task for each folder.
var pageFolderNames = [
'page_1',
'page_2'
// etc
];
Then for the gulp task, I have:
gulp.task('watch_pages', function()
{
// Get array length
var numPages = pageFolderNames.length;
// Add a new watch task for each individual page
for (var i = 0; i < numPages; i++)
{
gulp.watch('sass/pages/' + pageFolderNames[i] + '/**/*.scss', ['sass_page']);
}
});
The (simplified) task that compiles sass:
// Task: Compile page-specific Sass
gulp.task('sass_page', function()
{
return gulp.src('sass/pages/' + pageFolderNames[i] +'/**/*.scss')
.pipe(plumber(plumberErrorHandler))
.pipe(sass(...))
.pipe(gulp.dest('css/pages/' + pageFolderNames[i]));
});
This approach (I know my JS-fu is weaksauce) results in an error:
'sass_page' errored after 71 μs
ReferenceError: i is not defined
Is there any way to pass parameters, such as i, to gulp tasks to get this working? Alternately, is there a better way to accomplish what I'm trying to do? I have a sneaking suspicion there is. :-/
I found out there is an on change event for gulp watch. So this might be what you're looking for:
var pagesDir = 'sass/pages/';
gulp.task('watch_pages', function() {
gulp.watch(pagesDir + '**/*')
.on("change", function(file) {
// absolute path to folder that needs watching
var changedDest = path.join(__dirname, pagesDir);
// relative path to changed file
var changedFile = path.relative(changedDest, file.path);
// split the relative path, get the specific folder with changes
var pageFolder = changedFile.split('\\')[0];
gulp.src(path.join(pagesDir, pageFolder) +'/**/*.scss')
.pipe(plumber(plumberErrorHandler))
.pipe(sass(...))
.pipe(gulp.dest('css/pages/' + pageFolder));
console.log(changedDest);
console.log(changedFile);
console.log(pageFolder);
});
});
Also, this way you don't need to declare the folder variables. If you add directories within the path being watched, it should pick it up and name the destination folder accordingly.
Theoretically the gulp task to compile sass should work within the watch task. I played around with the paths, and it seems to spitting them out. Let me know what happens, I can modify if necessary.
The required packages:
var gulp = require("gulp"),
path = require("path"),
rimraf = require("rimraf");
BTW, since you already have access to the file path, you can perhaps target the specific scss file instead of the whole directory.
As Brian answered, the best approach is to have one watcher. In the same way as the principle of delegation with dom event listeners. it's better, even in our case it can not really matter. We need our thing done. if the consumed resources and performance doesn't bother us. It's ok. But stay as Brian answered. one watcher is the best approach. then you've got to get the file that was changed. and from there you get your page folder. So for that i will not add a thing. Except that you don't necessarily need to use the .on("change". You can directly set the same parameter for your function as this example show:
watch('./app/tempGulp/json/**/*.json', function (evt) {
jsonCommentWatchEvt = evt
gulp.start('jsonComment')
})
evt here is what Brian set as file.
What i want to add, is about how to pass a parameter from your watcher to your task. For example about the i in your initial work. A way of doing that, that i see and use, is to set a global variable that hold the data wanted to be passed. So it's accessible in both blocks. You set the value in watcher just before starting the task. And then you use it in the task. A good practice, is to save it in a task local variable just at start. That way you avoid the problem that it can change by another watch handling triggering.
For a lively example. check my answer out here : https://stackoverflow.com/a/49733123/7668448
I have a directory assets with a number of images inside. Inside my JavaScript code, I have an array, with the same list of images, that I use to preload everything. It means I have twice the same information, the list of images, both in my JavaScript and in my filesystem. I would like to have a gulp task to list all those images, and populate the array in JavaScript, with something like:
var listFiles = require("list-files-in-folder");
var images = listFiles("./assets/");
Do you know of any plugin who does that? Or a simple way to implement it maybe?
Assuming you're talking about nodejs, take a look at this
var glob = require("glob")
// options is optional
glob("**/*.js", options, function (er, files) {
// files is an array of filenames.
// If the `nonull` option is set, and nothing
// was found, then files is ["**/*.js"]
// er is an error object or null.
})
I have an illustrator file I'm regularly updating, that's then processed and distributed to /public/ though a Gulp task.
However, Illustrator appends _[0-9]+_ to some key tag IDs, and I want to perform something like .pipe(thisSVGFile.contents().replace('/_[0-9]+_/',"")) whenever Gulp Watch comes across a change.
How can I achieve this without producing my own Gulp plugin?
gulp-replace supports regular expressions.
var replace = require('gulp-replace');
gulp.task('replace', function () {
return gulp.src('./public/file.svg')
.pipe(replace(/_[0-9]+_/g, ''))
.pipe(gulp.dest('public'));
});
I am building a nifty little "asset-pipeline" for a express.js application, but i have a problem with the compression-step for javascript files
scripts = (fs.readFileSync(file) for file in filelist)
result = scripts.join("\n\n") # concat
upto now, things are working as expected (the logic itself is written in coffeescript). The next step after merging the JS-files would be to minify them. But here is my problem: i want to do this "hot" when i start my express-app in production mode, from within a piece of connect-middleware i wrote.
I need a solution that can minify a given blob of javascript stuff, without writing the result to disk (!), in other words: a function that does the minification and returns the result directly as a result value. (No, no webservices either.) It should be usable like this:
minified_result = awesomeMinifyFunction( result )
The raw processing performance isn't that important for me, neither is the level of compression, i need just something that works this way without hassle.
Does anyone know a suitable solution? Thanks in advance!
I'd suggest you look at one of the JavaScript based minifiers, like UglifyJS2.
npm install uglify-js
It can be used within a Node.JS application programatically:
var UglifyJS = require("uglify-js");
// you could pass multiple files (rather than reading them as strings)
var result = UglifyJS.minify([ "file1.js", "file2.js", "file3.js" ]);
console.log(result.code);
Or you could
var result = scripts.join("\n\n"); # concat
result = UglifyJS.minify(result, {fromString: true});
console.log(result.code);
You can write your own function that removes all comments/spaces/blank lines etc.
You can use a regular expression that makes use of rJSmin like:
function awesomeMinifyFunction(result)
{
pattern = (
r'([^\047"/\000-\040]+)|((?:(?:\047[^\047\\\r\n]*(?:\\(?:[^\r\n]|\r?'
r'\n|\r)[^\047\\\r\n]*)*\047)|(?:"[^"\\\r\n]*(?:\\(?:[^\r\n]|\r?\n|'
r'\r)[^"\\\r\n]*)*"))[^\047"/\000-\040]*)|(?<=[(,=:\[!&|?{};\r\n])(?'
r':[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/))*'
r'(?:(?:(?://[^\r\n]*)?[\r\n])(?:[\000-\011\013\014\016-\040]|(?:/\*'
r'[^*]*\*+(?:[^/*][^*]*\*+)*/))*)*((?:/(?![\r\n/*])[^/\\\[\r\n]*(?:('
r'?:\\[^\r\n]|(?:\[[^\\\]\r\n]*(?:\\[^\r\n][^\\\]\r\n]*)*\]))[^/\\\['
r'\r\n]*)*/)[^\047"/\000-\040]*)|(?<=[\000-#%-,./:-#\[-^`{-~-]return'
r')(?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/'
r'))*(?:(?:(?://[^\r\n]*)?[\r\n])(?:[\000-\011\013\014\016-\040]|(?:'
r'/\*[^*]*\*+(?:[^/*][^*]*\*+)*/)))*((?:/(?![\r\n/*])[^/\\\[\r\n]*(?'
r':(?:\\[^\r\n]|(?:\[[^\\\]\r\n]*(?:\\[^\r\n][^\\\]\r\n]*)*\]))[^/'
r'\\\[\r\n]*)*/)[^\047"/\000-\040]*)|(?<=[^\000-!#%&(*,./:-#\[\\^`{|'
r'~])(?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)'
r'*/))*(?:((?:(?://[^\r\n]*)?[\r\n]))(?:[\000-\011\013\014\016-\040]'
r'|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/))*)+(?=[^\000-\040"#%-\047)*,./'
r':-#\\-^`|-~])|(?<=[^\000-#%-,./:-#\[-^`{-~-])((?:[\000-\011\013\01'
r'4\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/)))+(?=[^\000-#%-,./:'
r'-#\[-^`{-~-])|(?<=\+)((?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*'
r'\*+(?:[^/*][^*]*\*+)*/)))+(?=\+)|(?<=-)((?:[\000-\011\013\014\016-'
r'\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/)))+(?=-)|(?:[\000-\011\013'
r'\014\016-\040]|(?:/\*[^*]*\*+(?:[^/*][^*]*\*+)*/))+|(?:(?:(?://[^'
r'\r\n]*)?[\r\n])(?:[\000-\011\013\014\016-\040]|(?:/\*[^*]*\*+(?:[^'
r'/*][^*]*\*+)*/))*)+'
)
return result.match(pattern);
}
I'd recommend taking a look at Asset Rack, which already implements what you're building.