November 5, 2014 at 14:32Advanced Gulp and Browserify: Interesting Tricks
A couple of weeks ago, I started a cycle on how I did a non-profit musical project (the first post is in “I'm PR”, I won’t put any links), but, unfortunately, I got carried away in the first article, and instead of talking about how he did it specifically, he began to recall effective tricks from other projects. Apparently, this, together with the prescribed emphasis on the project itself, led to the fact that a UFO came flying behind me and the post.
However, everything that was in the article was at least little known, and half of it was generally unique, as I am sure, and each of these tips can significantly facilitate the work with gulp, so I really would be sorry if this material was irretrievably gone would.
Therefore, I tried to remove all references to the project and re-publish (with revisions and corrections) an article that, in fact, no one had yet seen. If you are a grunt fan, at least read the second part: the fact that you do not like gulp does not mean that you do not like browserify.
Summary:
I switched to gulp while working on one video portal. Now they are doing well, they are developing, and it seems that my build system still stands with them.
Then I switched to stylus .
Why? Extremely fast.
I bring many numbers from memory, so I can lie a little, but the proportions are the same, I remember them exactly.
Introductory:
Everything indicates that there is a stop in the work with the hard drive, and in only one task out of three, everything can be solved with some kind of cache.
A study of the problem also indicated that styles compile the longest.
The solution to the problem was as follows:
As a result, the assembly of the entire project began to take 3-4 seconds, of individual file types - about a second.
watchdog worked out generally generally instantly, of course.
Since then I have been using the gulp + browserify stack.
Of the advantages of a browser browser - wrapping widgets in a closure, plus essentially the simplest code validation - it will not miss code that cannot parse.
However, the failed code forced the collector to crash. This was not the case, of course.
At first the solution was something like:
But, as it turned out, this is not very good: the stream "freezes" and does not restart on updating.
Why is this happening? The fact is that the task does not end, but remains hanging in this state, and in gulp, it seems that restarting is impossible without the end of the previous execution.
I have been looking for a good solution for this issue for a long time, but over time I wrote my own function to solve this problem:
It also closes the stream (this.end ()), causing the completion of the task.
If desired, you can add here, for example, growl alerts, but I personally have enough.
The function requires a pre-installed npm-package color and gives a very beautiful output. If you do not want to put extra packages, you can remove the methods from the flowers.
The most important thing here is in the last line.
When we execute this.end (), the specific gulp task terminates. Yes, this is a little crap in memory, but the watchdog task will be able to restart your style assembly when you update them.
It looks like this:
If everything is neatly arranged in folders like:
I congratulate you.
But I personally have everything lying around like this:
And it's convenient, much more convenient than before. Why? Yes, because I had the opportunity to arbitrarily structure the files that are included in each other without changing anything in the collector.
You write @require in styles, layouts and inclusions in templates and browserify for scripts, everything just works.
In the end, it's all going to index.html, app.js and style.css - the very base for any project.
How did I get this?
In all projects, I try to stick to a similar scheme:
What kind of glob path is this?
This is a selection of all files that do not begin with underscores. At any depth. Accordingly, if you name the file src / lib / _some_lib.js, it will not be compiled on its own. But require it with pleasure picks up.
Now I do not use this technique, because I switched to a circuit with inclusions of everything and everything in the code, I write mostly from memory, so I can lie a little.
But it is very interesting, and in due time I did not find it anywhere.
When I needed to solve a problem such as “glue all CoffeeScript files and js files from the vendor folder, and then from the main folder”, at first I was upset because I did not know what to do. Why such a sequence - I think it is clear - vendor scripts must be loaded first, and if you do it somehow else, everything will mix up.
But I knew that if something is in my memory, then it can be used, and I started digging. Nevertheless, gulp uses native nodejs streams, which means that you can do something about it.
I came to a home-made solution:
Please note : judging by the new event-stream documentation, the concat method was renamed to merge. I did this the last time six months ago, so now the method may have new subtleties of use - the code is taken from a real relatively old project that works with an old version of EventStream.
When you have 10-20 plugins, it becomes somewhat tedious to write them manually.
There is another plugin for this that creates a plugins object with plugin methods, but the same thing can be done much more clearly and simply:
If someone does not understand what exactly this code does - it opens the contents of devDependencies in package.json and connects all elements that start with gulp- in it as plugins [pluginName]. If the plugin is called something like gulp-css-base64, it will be available at plugins.css_base64.
Sometimes it is necessary to create something in memory and send it to the stream (at least with the same gluing). Again, there is a plugin for this, but why? If you can write everything yourself in three lines.
It all works on top of the Vynil FS from gulp-util, but what's the difference?
Why browserify in a post about gulp? Yes, because it can be called a meta-assembly system, which is used in other systems. Its capabilities have long gone beyond the simple gluing of js-modules, and in the next part of the post everything will come together in general.
If you use browserify and commonJS modules - tell me honestly, have you ever wanted to write like this?
This is the real code from the same project, for the post about which a UFO flew after me, by the way.
As it turned out, riveting your plugins for browserify is elementary.
The real task for building JS in the end looks like this:
What is this ... and how does it work? Yes, very simple.
The simplest wrapper looks something like this:
In the future, I will quote only the compile function - to save space.
If there is a browserify-ninja here who already knows all the plugins by heart, they will ask "so what?"
Yes, nothing.
In this form, plugins already exist.
But the trick is that we can change the syntax.
For instance:
And now in the jade template we can write
As a result, we can include templates on jade in js, and styles in templates on jade.
We can connect several collectors at once, for example:
This is what we do the DoT template JS function (handlebars-like template engine over HTML) wrapped in Jade.
And we can even ...
... drum roll ...
... use gulp plugins to create browserify plugins, which we can connect as gulp task
and, finally, decoupling the entire post. We can turn this data string into a stream (which I just talked about in the middle of the post), which can be used with gulp. We take the function that I showed above and get ...
Once again, very carefully:
We just passed through a bunch of gulp plugins the data that went to browserify.
Yes, it’s a bit hemorrhoid. But the result is worth it.
What for?To the glory of Satan, of course, because you couldn’t just take and configure in browserify the assembly of Stylus-styles, which would also suck out base64-pictures, and pass through the auto-prefixer and minification.
gulp is an amazingly elegant system that can be customized to suit most situations. And the fact that its plugins can be used in browserify (and, therefore, other projects) is generally brilliant. Yes, a little hemorrhoid, but it's something.
I hope you learned something new. More precisely, I am sure of this, but I wanted to say beautifully.
And I hope that the UFO will return me to Habr and give me a talk about neural networks inside Web Workers and algorithms that can give accurate recommendations on the user's musical preferences based on an extremely small amount of data.
However, everything that was in the article was at least little known, and half of it was generally unique, as I am sure, and each of these tips can significantly facilitate the work with gulp, so I really would be sorry if this material was irretrievably gone would.
Therefore, I tried to remove all references to the project and re-publish (with revisions and corrections) an article that, in fact, no one had yet seen. If you are a grunt fan, at least read the second part: the fact that you do not like gulp does not mean that you do not like browserify.
Summary:
- A simple way to handle errors;
- Universal structure for storing source files;
- Combining several streams (for example, compiled coffee and js) into one;
- Creating a stream of text;
- creating your own plugins for Browserify;
- creating plugins from Gulp plugins for Browserify.
I switched to gulp while working on one video portal. Now they are doing well, they are developing, and it seems that my build system still stands with them.
Then I switched to stylus .
Why? Extremely fast.
I bring many numbers from memory, so I can lie a little, but the proportions are the same, I remember them exactly.
Introductory:
- grunt;
- there are a large number (73, if not confusing) of style files that are almost unrelated to each other, assembly - by gluing. Basically, styles are components and page hooks and layouts, so that they are independent of each other, in almost all only a set of variables is connected;
- There are a large number of scripts that stick together into one. CoffeeScript compiles, regular js does not change. Each file is wrapped in a closure;
- there is an application file that is assembled from a large number of precompiled jade templates through some specialized plugin that makes an object with all the templates from the folder in the global space, plus it sticks together with the main application engine. In fact, the output is also a js file;
- the assembly time for each item is about 10 seconds on a Macbook Air 2013, i5, in total - about 30-40 seconds. I remind you that there is an SSD in the MBA, which is connected via PCIe and gives out the speed of working with a hard drive, which seems to be unattainable in principle on a regular sata connection. On computers with an HDD, assembly time may take more than a minute.
Everything indicates that there is a stop in the work with the hard drive, and in only one task out of three, everything can be solved with some kind of cache.
A study of the problem also indicated that styles compile the longest.
The solution to the problem was as follows:
- replace grunt with gulp. The problem with compilation, and then gluing styles and scripts is solved - the step of writing to the disk of each individual file is removed;
- replace sass with stylus, translate everything to inclusions in a global file. Style compilation is accelerated to less than a second. Apparently, transferring each file from a node to a ruby consumed a lot of resources. And ruby-sass is not very fast. The transfer, by the way, happened without any problems at all - sass used the basic one, without mixins and functions, and from a certain point of view, the sass format can be called a subset of the stylus format;
- Translating all coffeescript to JS to speed up compilation is good, these were mostly old widgets;
- Translate the js assembly for the application to browserify for caching.
As a result, the assembly of the entire project began to take 3-4 seconds, of individual file types - about a second.
watchdog worked out generally generally instantly, of course.
Since then I have been using the gulp + browserify stack.
Of the advantages of a browser browser - wrapping widgets in a closure, plus essentially the simplest code validation - it will not miss code that cannot parse.
However, the failed code forced the collector to crash. This was not the case, of course.
Error processing
At first the solution was something like:
gulp.task('build-html', function () {
...
.pipe(plugins.jade()).on('error', console.log.bind(console))
...
But, as it turned out, this is not very good: the stream "freezes" and does not restart on updating.
Why is this happening? The fact is that the task does not end, but remains hanging in this state, and in gulp, it seems that restarting is impossible without the end of the previous execution.
I have been looking for a good solution for this issue for a long time, but over time I wrote my own function to solve this problem:
function log(error) {
console.log([
'',
"----------ERROR MESSAGE START----------".bold.red.underline,
("[" + error.name + " in " + error.plugin + "]").red.bold.inverse,
error.message,
"----------ERROR MESSAGE END----------".bold.red.underline,
''
].join('\n'));
this.end();
}
gulp.task('build-html', function () {
...
.pipe(plugins.jade()).on('error', log)
...
It also closes the stream (this.end ()), causing the completion of the task.
If desired, you can add here, for example, growl alerts, but I personally have enough.
The function requires a pre-installed npm-package color and gives a very beautiful output. If you do not want to put extra packages, you can remove the methods from the flowers.
The most important thing here is in the last line.
When we execute this.end (), the specific gulp task terminates. Yes, this is a little crap in memory, but the watchdog task will be able to restart your style assembly when you update them.
It looks like this:
Folders and files
If everything is neatly arranged in folders like:
- assets
- styles
- scripts
- templates
I congratulate you.
But I personally have everything lying around like this:
And it's convenient, much more convenient than before. Why? Yes, because I had the opportunity to arbitrarily structure the files that are included in each other without changing anything in the collector.
You write @require in styles, layouts and inclusions in templates and browserify for scripts, everything just works.
In the end, it's all going to index.html, app.js and style.css - the very base for any project.
How did I get this?
In all projects, I try to stick to a similar scheme:
gulp.task('build-js', function () {
return gulp.src('src/**/[^_]*.js')
...
gulp.task('build-html', function () {
return gulp.src('src/**/[^_]*.jade')
...
gulp.task('build-css', function () {
return gulp.src('src/**/[^_]*.styl')
...
What kind of glob path is this?
This is a selection of all files that do not begin with underscores. At any depth. Accordingly, if you name the file src / lib / _some_lib.js, it will not be compiled on its own. But require it with pleasure picks up.
Gluing the results of different tasks
Now I do not use this technique, because I switched to a circuit with inclusions of everything and everything in the code, I write mostly from memory, so I can lie a little.
But it is very interesting, and in due time I did not find it anywhere.
When I needed to solve a problem such as “glue all CoffeeScript files and js files from the vendor folder, and then from the main folder”, at first I was upset because I did not know what to do. Why such a sequence - I think it is clear - vendor scripts must be loaded first, and if you do it somehow else, everything will mix up.
But I knew that if something is in my memory, then it can be used, and I started digging. Nevertheless, gulp uses native nodejs streams, which means that you can do something about it.
I came to a home-made solution:
var es = require('event-stream');
gulp.task('build', function(){
return es.concat(
gulp.src('scripts/vendor/*.coffee').pipe(coffee()),
gulp.src('scripts/vendor/*.js'),
gulp.src('scripts/*.coffee').pipe(coffee()),
gulp.src('scripts/*.js')
)
.pipe(concat())
.pipe(dest(...));
})
Please note : judging by the new event-stream documentation, the concat method was renamed to merge. I did this the last time six months ago, so now the method may have new subtleties of use - the code is taken from a real relatively old project that works with an old version of EventStream.
Plugin connection
When you have 10-20 plugins, it becomes somewhat tedious to write them manually.
There is another plugin for this that creates a plugins object with plugin methods, but the same thing can be done much more clearly and simply:
var gulp = require('gulp'),
plugins = {};
Object.keys(require('./package.json')['devDependencies'])
.filter(function (pkg) { return pkg.indexOf('gulp-') === 0; })
.forEach(function (pkg) {
plugins[pkg.replace('gulp-', '').replace(/-/g, '_')] = require(pkg);
});
If someone does not understand what exactly this code does - it opens the contents of devDependencies in package.json and connects all elements that start with gulp- in it as plugins [pluginName]. If the plugin is called something like gulp-css-base64, it will be available at plugins.css_base64.
How to create a stream from text
Sometimes it is necessary to create something in memory and send it to the stream (at least with the same gluing). Again, there is a plugin for this, but why? If you can write everything yourself in three lines.
var gutil = require('gulp-util');
function string_from_src(filename, string) {
var src = require('stream').Readable({objectMode: true});
src._read = function () {
this.push(new gutil.File({cwd: "", base: "", path: filename, contents: new Buffer(string)}));
this.push(null);
};
return src;
}
It all works on top of the Vynil FS from gulp-util, but what's the difference?
Plugins for browserify
Why browserify in a post about gulp? Yes, because it can be called a meta-assembly system, which is used in other systems. Its capabilities have long gone beyond the simple gluing of js-modules, and in the next part of the post everything will come together in general.
If you use browserify and commonJS modules - tell me honestly, have you ever wanted to write like this?
var vm = new Vue({
template: require('./templates/_app.html.jade'),
...
This is the real code from the same project, for the post about which a UFO flew after me, by the way.
As it turned out, riveting your plugins for browserify is elementary.
The real task for building JS in the end looks like this:
gulp.task('build-js', function () {
return gulp.src('src/**/[^_]*.js')
.pipe(plugins.browserify(
{
transform: [require('./lib/html-jadeify'), 'es6ify'],
debug : true
}
)).on("error", log)
.pipe(gulp.dest("build"));
});
What is this ... and how does it work? Yes, very simple.
The simplest wrapper looks something like this:
var through = require('through'),
jade = require('jade');
function Jadify(file) {
var data = '';
if (/\.html\.jade$/.test(file) === false)
return through();
else
return through(write, end);
function write(buf) { data += buf; }
function end() {
compile(file, data, function (error, result) {
if (error) stream.emit('error', error);
else stream.queue(result);
stream.queue(null);
});
}
}
function compile(file, data, callback) {
callback(null,
'module.exports = "' + jade.render(data, {filename: file})+ '"\n';
);
}
Jadify.compile = compile;
Jadify.sourceMap = true; // use source maps by default
module.exports = Jadify;
In the future, I will quote only the compile function - to save space.
If there is a browserify-ninja here who already knows all the plugins by heart, they will ask "so what?"
Yes, nothing.
In this form, plugins already exist.
But the trick is that we can change the syntax.
For instance:
callback(null,
'module.exports = "' + jade.render(data, {filename: file})
.replace(/"/mg, '\\"')
.replace(/\n/mg, '\\n')
.replace(/@inject '([^']*)'/mg, '"+require("$1")+"')
+ '"\n'
);
And now in the jade template we can write
style @inject './_font_styles.styl'
As a result, we can include templates on jade in js, and styles in templates on jade.
We can connect several collectors at once, for example:
callback(null, 'module.exports = ' + dot.template(jade.render(data, {
filename: file
})) + '\n');
This is what we do the DoT template JS function (handlebars-like template engine over HTML) wrapped in Jade.
And we can even ...
... drum roll ...
... use gulp plugins to create browserify plugins, which we can connect as gulp task
and, finally, decoupling the entire post. We can turn this data string into a stream (which I just talked about in the middle of the post), which can be used with gulp. We take the function that I showed above and get ...
function string_src(filename, string) {
var src = require('stream').Readable({ objectMode: true });
src._read = function () {
this.push(new gutil.File({ cwd: "", base: "", path: filename, contents: new Buffer(string) }));
this.push(null)
};
return src;
}
function compile(path, data, cb) {
string_src(path, data)
.pipe(gulp_stylus())
.pipe(gulp_css_base64({maxWeightResource: 32 * 1024}))
.pipe(gulp_autoprefixer())
.pipe(gulp_cssmin())
.on('data', function(file){
cb(null, "module.exports = \""+
file.contents.toString()
.replace(/"/mg, '\\"')
.replace(/\n/mg, '\\n')
+ '"');
})
.on('error', cb);
}
Once again, very carefully:
string_src(path, data)
.pipe(gulp_stylus())
.pipe(gulp_css_base64({maxWeightResource: 32 * 1024}))
.pipe(gulp_autoprefixer())
.pipe(gulp_cssmin())
.on('data', function(file){
cb(null, "module.exports = \""+
file.contents.toString()
.replace(/"/mg, '\\"')
.replace(/\n/mg, '\\n')
+ '"');
})
We just passed through a bunch of gulp plugins the data that went to browserify.
Yes, it’s a bit hemorrhoid. But the result is worth it.
What for?
Conclusion
gulp is an amazingly elegant system that can be customized to suit most situations. And the fact that its plugins can be used in browserify (and, therefore, other projects) is generally brilliant. Yes, a little hemorrhoid, but it's something.
I hope you learned something new. More precisely, I am sure of this, but I wanted to say beautifully.
And I hope that the UFO will return me to Habr and give me a talk about neural networks inside Web Workers and algorithms that can give accurate recommendations on the user's musical preferences based on an extremely small amount of data.