Building projects using Gulp.js. Yandex Workshop

    Hi, my name is Boris. I work in Yandex in the testing department and create tools that make our testers' lives easier and happier. Our team is partly research, so we can afford to use quite unusual tools and experiments. Recently, I told my colleagues about one of such experiments: Gulp.js . Today I would like to share this experience with you.



    First, a little background on how web technologies evolved. In the beginning, there was no frontend as a separate concept, most of the logic was executed on the server. Therefore, a variety of tasks for assembling scripts and styles, as well as preparing pictures, fonts and other resources, were performed by the backend and their assemblers, for example, Apache Ant or Maven. The frontend was at a disadvantage, the tools provided by these assemblers were not very suitable for him. This problem began to be solved only recently when Grunt appeared. This is the first compiler written in JS. Each front-end vendor knows JavaScript, so it can easily write tasks under Grunt and understand what has already been written. This led to the success of this collector. Grunt has a bunch of advantages, but there are also disadvantages.

    For example, this is how a simple Grunt file looks like.

    Gruntfile.js
    module.exports = function (grunt) {
      "use strict";
      // Project configuration.
      grunt.initConfig({
          pkg: grunt.file.readJSON('package.json'),
          jshint: {
              files: ['<%= pkg.name %>.js']
          },
          concat: {
              options: {
                  banner: '/*! <%= pkg.name %> <%= grunt.template.today("yyyy-mm-dd") %> */\n'
              },
              main: {
                  src: ['<%= pkg.name %>.js'],
                  dest: 'build/<%= pkg.name %>.js'
              }
          },
          uglify: {
              main: {
                  src: 'build/<%= pkg.name %>.js',
                  dest: 'build/<%= pkg.name %>.min.js'
              }
          }
      });
      grunt.loadTasks('tasks/');
      grunt.registerTask('default', ['jshint', 'concat', 'uglify']);
      return grunt;
    };
    


    We have a task, plugins are used to complete it. If we need more action, we plug in more plugins. As a result, we get a huge sheet of code in which nothing can be found. And since the Grunt-file is large, the assembly becomes prohibitively long. And how to speed it up is completely incomprehensible, because Grunt architecture does not have any means for this.

    Gruntfile.js
    module.exports = function (grunt) {
        "use strict";
        // Project configuration.
        grunt.initConfig({
            pkg: grunt.file.readJSON('package.json'),
            karma: {
                options: {
                    configFile: 'karma.conf.js'
                },
                unit: {},
                travis: {
                    browsers: ['Firefox']
                }
            },
            jshint: {
                files: ['<%= pkg.name %>.js']
            },
            concat: {
                options: {
                    banner: '/*! <%= pkg.name %> <%= grunt.template.today("yyyy-mm-dd") %> */\n'
                },
                main: {
                    src: ['<%= pkg.name %>.js'],
                    dest: 'build/<%= pkg.name %>.js'
                }
            },
            uglify: {
                main: {
                    src: 'build/<%= pkg.name %>.js',
                    dest: 'build/<%= pkg.name %>.min.js'
                }
            },        
            copy: {
                main: {
                    expand: true,
                    cwd: 'docs/',
                    src: ['**', '!**/*.tpl.html'],
                    dest: 'build/'
                }
            },
            buildcontrol: {
                options: {
                    dir: 'build',
                    connectCommits: false,
                    commit: true,
                    push: true,
                    message: 'Built %sourceName% from commit %sourceCommit% on branch %sourceBranch%'
                },
                pages: {
                    options: {
                        remote: 'git@github.com:just-boris/angular-ymaps.git',
                        branch: 'gh-pages'
                    }
                }
            }
        });
        grunt.loadNpmTasks('grunt-contrib-uglify');
        grunt.loadNpmTasks('grunt-contrib-concat');
        grunt.loadNpmTasks('grunt-contrib-jshint');
        grunt.loadNpmTasks('grunt-karma');
        grunt.loadNpmTasks('grunt-contrib-copy');
        grunt.loadNpmTasks('grunt-build-control');
        grunt.registerTask('test', 'Run tests on singleRun karma server', function() {
            if (process.env.TRAVIS) {
                //this task can be executed in Travis-CI
                grunt.task.run('karma:travis');
            } else {
                grunt.task.run('karma:unit');
            }
        });
        grunt.registerTask('build', ['jshint', 'test', 'concat', 'uglify']);
        grunt.registerTask('default', ['build', 'demo']);
        grunt.registerTask('build-gh', ['default', 'buildcontrol:pages']);
        return grunt;
    };
    


    Therefore, the only way out is to try to start all over from the beginning and enter from the other side. Well, for starters, you can see what is already useful in the world. For example, there is a UNIX shell. It has a useful concept - pipeline - the direction of the exhaust of one process to the input of another process, and it can send it to the next and so on in the chain.

    $ cat *.coffee \
        | coffee \
        | concat \
        | uglify \
        > build/app.min.js
    

    Thus, we can build a real conveyor that will perform our assembly. It’s damn logical to assemble on a conveyor belt. This also applies to frontend tasks. However, if you do this on a clean shell, some problems may arise. Firstly, not every operating system has a shell, and secondly, we do not have commands that, for example, will convert coffee to JS.

    But Gulp can do this. This utility is written in JavaScript. It uses the same principle as the Shell script, but here the function is used for piping instead of the vertical bar pipe().

    gulp.src('*.coffee')
        .pipe(coffee())
        .pipe(concat())
        .pipe(uglify())
        .pipe(gulp.dest('build/'))
    

    Those. we can do exactly the same thing, it’s clear that what he is doing, if necessary, you can swap blocks, delete and generally configure your conveyor as you like.

    Gulp is already quite stable, developed to the third version and found its fans. It is installed in our favorite way:

    npm install -g gulp
    

    I decided to test it on one of my projects and was surprised to find that the assembly with it helps a little faster than with Grunt. And now I’ll try to explain why.



    The thing is that the most expensive operation during assembly is accessing the file system: the assembly takes place on the processor, the file system is somewhere far away, you need to go to it, and this takes some time. In the diagram, the red arrows just show these operations. It can be seen that in Gulp there are only two of them (read at the input, written down at the output), and in Grunt - four: each plugin reads and writes. Well, since everything works faster, why not switch to Gulp. But first, I decided to check everything carefully. I prepared a test case, which collects and packs coffee files and styles, described this task for Grunt and Gulp, ran them one by one and saw that there really is a gain, gulp is about a quarter faster: 640 ms versus 850. I also prepared another test, a little more complicated. In it we still need to slightly preprocess styles. Most styles, of course, are in bootstrap. Let's try to collect it from the source less-files, and then, to reduce its size, let's go through CSSO. In Gulp, this is done quite easily: there is a plug-in for lessboth and csso.

    var gulp = require('gulp');
    var csso = require('gulp-csso');
    var less = require('gulp-less');
    gulp.task('default', function() {
      return gulp.src('bower_components/bootstrap/less/bootstrap.less')
        .pipe(less())
        .pipe(csso())
        .pipe(gulp.dest('dest/'));
    });
    

    A larger grunt file.

    module.exports = function(grunt) {
      require('time-grunt')(grunt);
      grunt.loadNpmTasks('grunt-contrib-less');
      grunt.loadNpmTasks('grunt-csso');
      grunt.initConfig({
        less: {
          compile: {
            src: 'bower_components/bootstrap/less/bootstrap.less',
            dest: 'dest/bootstrap.css'
          }
        },
        csso: {
          compile: {
            src: 'dest/bootstrap.css',
            dest: 'dest/bootstrap.min.css'
          }
        }
      });
      grunt.registerTask('default', ['less', 'csso']);
    };
    

    As a result, Gulp won again: 2 seconds against 2.3. Grunt spent 300 milliseconds reading and writing junk files.

    There aren’t as many plugins for Gulp as there are for Grunt, but the 400 that are are enough for typical tasks. Well, if you still lack something, you can always write your own. The main idea of ​​Gulp is streams. They are already in the kernel node.js, for this you do not need to connect anything. Consider a small example: a plugin that will welcome everyone. We are his word, and he is greeting us:



    This is how it will look in JavaScript:

    var stream = require('stream'),
      greeterStream = new stream.Transform({objectMode: true});
    greeterStream._transform = function(str) {
      this.push('Hello, '+str+'!');
    };
    greeterStream.pipe(process.stdout)
    greeterStream.write('world'); // Hello, world!
    greeterStream.write('uncle Ben'); // Hello, uncle Ben!
    

    We have a ready-made native object in which we must define a method _transform. A string is supplied to his input so that we process it and return it. We write in it, and it converts. You don’t need to connect anything, this is the native API node.js. To see how it all fits into Gulp, remove the cover from it and take a look inside. There we will see two modules: Orchestrator and Vinyl fs. Orchestrator conducts flows, lines them up, tries to execute them with maximum parallelism and generally takes care that everything works like an orchestra. Everything is a little more interesting with Vinyl. A stream is a data set, and we collect files. This is more than just data, it is also a name, extension and other attributes. I would like to somehow separate the continuous stream into separate files. All this is done by Vinyl. In fact, this is a wrapper over files: we get not just data, but objects. Vinyl puts down all the necessary fields there.

    var coffeeFile = new File({
      cwd: "/",
      base: "/test/",
      path: "/test/file.coffee"
      contents: new Buffer("test = 123")
    });
    

    Each plug-in does this, for example, gulp-freeze , which I wrote specifically to show how simple it is. It is designed to freeze static. In Gulp, this is all done very simply: we have content, we calculate the md5 hash from it and say that this is the file name. Then we write the file further to the stream. Everything else Gulp will do for us: read the files, give them to our plugin, then transfer it to the rest of the plugins and finally write to the file system. And we write only the most interesting, our plugin.

    var through = require('through2');
    module.exports = function() {
        return through.obj(function(/**Vinyl*/file, enc, callback) {
            var content = file.contents.toString('utf-8'),
                checksum = createMD5(content),
                file.path = checksum;
            this.push(file);
            callback();
       });
    };
    

    And since we have nothing superfluous, the test is quite simple. We will create a test stream in which we put fake data, and we may not even use the file system. If we write a large plug-in, and CI will be configured for it, for example, Travis, then we will be pleasantly surprised by the speed of the build. For all test cases, you can generate virtual files, write them to the stream and listen to the output. If the output is correct data, everything is fine, the test is passed, if not, we have a mistake, we’ll go to fix it.

    var freeze = require('./index.js')
    var testStream = freeze()
    testStream.on('data', function(file) {
        //assert here
    });
    testStream.write(fakeFile);
    

    Sometimes it’s not even necessary to write a plugin. Some functions can be inserted directly into the stream. For example, no one has written a Gulp plugin for the Yate template engine yet. But we can call it directly:

    var map = require('vinyl-map');
    var yate = require('yate');
    gulp.src('*.yate')
        .pipe(map(function(code, filename) {
            // здесь может быть любое ваше что угодно
            return yate.compile(code.toString('utf-8')).js;
        }))
        .pipe(gulp.dest('dist/'))
    


    There are more exotic applications of such a system. For example, this collector can replace Jekyll. Let's say we have articles in markdown, and from them we collect web pages in HTML. This scheme fits perfectly into the ideology of Gulp, with its help you can collect Jekyll-templates. To do this, you just need to read our posts, process them, you may have to write a couple of small plug-ins, and as a result get a full Jekyll port on node.js. Very comfortably. It seems to me that in Grunt it is impossible to do this in principle.

    gulp.task('jekyll', function() {
        return gulp.src('_posts/**')
        .pipe(liquid.collectMeta()) //собираем метаданные из постов
        .pipe(post())               //генерируем ссылку на пост
        .pipe(gulp.src('!_*/?*'))   //добавляем остальные файлы
        .pipe(markdown())           //конвертируем в html, если нужно
        .pipe(liquid.template())    //шаблонизируем
        .pipe(gulp.dest('_site/')); //записываем результат
    });
    


    PS The report was told in the spring of 2014, over the past six months the tools have evolved, something could change, but the basic idea remains the same.

    Also popular now: