How we collect + a couple of tricks in Gulp in js django-project

Hello!

This is not a guide, I am sharing the experience of how, in a large Django project, from the ugly junk of jQuery scripts, we gradually came to the assembly and minification of complex frontend applications on AngularJS using gulp and browserify.

Background


There is a large long-term Django project with a bunch of legacy code, a billion dependencies and a team without an official frontend developer. Somehow it happened that I gradually became more engaged in js, got involved in the frontend and now it already takes more than half of my work time.

In the history of the front-end of our project (and, accordingly, my development as a js-developer), there are three large stages:

jQuery is our everything

This was the period when, having mastered a couple of jQuery methods, mastered selectors and learned how to show / hide elements on the page with animation, you consider yourself an accomplished frontend developer. All newcomers went through this and everyone knows how it looks: each piece of functionality is a separate file, there are a dozen script connections on large pages, no system - each script is for itself, with all the consequences, as they say. There was no specific place to store vendor libraries, each next developer threw a new lib where he wanted. In addition to everything that I wrote myself, there was a huge bunch of old scripts written before me.

Knockout + RequireJS

There was a need to write more complex interfaces, wizards and other things for the admin panel. By this time, it was understood that jQuery is not a panacea, and that you need to somehow organize your code. Knockout and RequireJS came to the rescue. RequireJS allowed breaking code into modules, specifying dependencies, reusing modules on different pages, building a normal file structure for each application. At least some kind of system appeared: a config file was created for RequireJS with paths to all libraries, it was used on all knockout pages, all vendor libraries settled in one place. There was only one problem: even now only one script was connected in the template, the rest of the bunch of dependencies was already pulled by RequireJS itself, and often the module files were so small, that ping to the server was longer than the download time - meaningless brakes. I often pointed out this problem and offered different solutions, but the response of the authorities was always the same: “This is the admin panel. This is not critical. We will not waste time on this. ”

AngularJS + Gulp + Browserify + Uglify

Finally, hands reached the Customer Area: tricky interfaces, plus UX requirements. It was already impossible to ignore the problem of loading scripts. At that time, I already gained experience in developing on NodeJS using the build scripts for the frontend. Now I couldn’t look at the config file for RequireJS and the systematic dump of the vendor libraries without tears.

A little about how the project generally works. Each django application has its own static folder; during development, the django dev server looks for scripts connected to pages in these folders. During the deployment, collectstatic is made on the production, which collects all the files in one folder so that they can be returned by the web server. Nothing unusual.

I wanted to get the following:
  • normal package manager for the frontend;
  • normal code format in the form of reusable-modules;
  • assembly of js-application into one file and its minification.


The question arose - from which side should this be screwed to the project so as not to disturb the usual workflow and frighten the authorities with new dependencies in the form of NodeJS (read as “a new language in the team of pythonists”) and its utilities?

It was decided that all manipulations with the js code (assembly, minification) will be done before the commit, the finished package will be copied to the folder with the statics of the corresponding django application and connected from there. Thus, the deployment process will remain unchanged, plus no new dependencies in production.

We embark on the true path


Environment

So, the first thing we need:

  • nodejs ;
  • gulp - to describe assembly tasks;
  • npm - to install packages required for assembly;
  • bower - to install packages needed in the frontend.


They should be installed globally in the system, as we need their console utilities. Fortunately, we are developing in Vagrant, so I just added the appropriate chef recipes to his config. After installation in the root of the project, you need to run npm init and bower init and set the minimum necessary parameters, the output will be package.json and bower.json . The final step in preparing the environment will be the introduction of node_modules / and bower_components / in .gitignore, since the entire assembly will be done directly during development.

When using bower and npmto install packages, do not forget to use the --save-dev argument, so that the package information is saved in bower.json and package.json respectively, and other developers can easily raise the environment by simply running npm install and bower install in the project root.

Directory structure

I decided to store the source code for js applications in a separate directory in the root of the project. At first I wanted to analyze the directory structure on the fly during assembly, but I thought that sooner or later, for every smart analyzer, there would be a task that would have to be crutched, so I decided to just create a config in which I will describe all these applications. So the config-spa.js file appeared in the root of the project :

module.exports = {
    apps: {
        'appname': {        // имя js-приложения
            main: 'app.js',  // имя главного файла
            path: './spa/dj-app/appname/',  // путь до приложения
            bundle: 'appname.min.js',         // имя скомпилированного пакета
            dest: './dj-app/static/dj-app/js/',  // путь до каталога со статикой соответствующего django-приложения
            watch: ['./spa/dj-app/appname/**/*.js']  // список glob-путей для слежения за изменениями (для автоматической перекомпиляции при разработке)
        },
        ...
    }
}


  • spa / - the directory in which all js applications will be located
  • dj-app - names of the django application in which the assembled package will be used


Thus, it is easy to understand which application the scripts belong to. General modules are placed in directories named common.

gulpfile.js


It remains the case for small - a description of the tasks for the assembly. In general, the result was a standard gulpfile, but there are a couple of tricks that can be useful to someone.

Parsing command line arguments and the first trick

Since we have several applications, we had to somehow indicate which application to build, or indicate that we need to rebuild all of them.
Another argument is a flag that cancels the minification of the application so that you can see normal stack traces during debugging.

What is the trick? Firstly, in the fact that I parsed the arguments as a separate task, so that it can be indicated in the dependencies of other tasks, and secondly, once parsed arguments are stored in a global variable so that when calling some tasks from others they will work with the same settings.

// подключения библиотек опущены, см. полный файл в конце
var config = require('./config-spa'),
    argv = {parsed: false}
gulp.task('parseArgs', function() {
    // prevent multiple parsing when watching
    if (argv.parsed) return true
    // check the process arguments
    var options = minimist(process.argv)
    if (_.size(options) === 1) {
        printArgumentsErrorAndExit()
    }
    // готовим список приложений, сверяя его с конфигом
    var apps = []
    if (options.app && config.apps[options.app]) {
        apps.push(options.app)
    } else if (options.all) {
        apps = _.keys(config.apps)
    }
    if (!apps.length) printArgumentsErrorAndExit()
    argv.apps = apps
    // dev - флаг, отменяющий минификацию
    if (options.dev) argv.dev = true
    argv.parsed = true
})
function printArgumentsErrorAndExit() {
    gutil.log(gutil.colors.red('You must specify the app or'), gutil.colors.yellow('--all'))
    gutil.log(gutil.colors.red('Available apps:'))
    _.each(config.apps, function(item, i) {
        gutil.log(gutil.colors.yellow('  --app ' + i))
    })
    // break the task on error
    process.exit()
}


Build application

function bundle() {
    return through.obj(function(file, enc, cb) {
        var b = browserify({entries: file.path})
        file.contents = b.bundle()
        this.push(file)
        cb()
    })
}
gulp.task('build', ['parseArgs'], function(cb) {
    var prefix = gutil.colors.yellow('  ->')
    async.each(argv.apps,
        function(app, cb) {
            gutil.log(prefix, 'Building', gutil.colors.cyan(app), '...')
            var conf = config.apps[app]
            if (!conf) return cb(new Error('No conf for app ' + app))
            gulp.src(path.join(conf.path, conf.main))
                .pipe(bundle())
                .pipe(gulpif(!argv.dev, streamify(uglify())))
                .pipe(rename(conf.bundle))
                .pipe(gulp.dest(conf.dest))
                .on('end', function() { cb() })
        },
        function(err) {
            cb(err)
        }
    )
})


  • function bundle () {...} - a self-written wrapper for browserify. Whoever uses it knows for a long time that browserify itself can work with streams, therefore the gulp-browserify package has not been used for a long time;
  • [parseArgs] - specify in the dependencies of tasks for parsing command line arguments. Thus, we are sure that the argv variable already contains valid settings;
  • async.each, cb () - enumeration of the applications specified in the arguments. Why are there asinks and troubles with callbacks? The fact is that the assembly procedure itself (gulp.src (). Pipe () ...) is asynchronous, and the task can complete before the entire chain is executed, and this, in turn, leads to the fact that dependent tasks start their execution earlier. There are three possible solutions - callback for task, return from stream task - return gulp.src () ... and return promise. We cannot return the stream here, because there are several of them, so I settled on the callback;
  • .pipe (gulp.dest (conf.dest)) - the collected package is copied to the static folder specified in the js application config, so that when deployed, collectstatic will do its job without additional gestures.


Recompilation at changes in files

Task for monitoring changes in js application files:

gulp.task('watch', ['build'], function() {
    var targets = []
    _.each(argv.apps, function(app) {
        var conf = config.apps[app]
        if (!conf) return
        if (conf.watch) {
            if (_.isArray(conf.watch)) {
                targets = _.union(targets, conf.watch)
            } else {
                targets.push(conf.watch)
            }
        }
    })
    targets = _.uniq(targets)
    // start watching files
    gulp.watch(targets, ['build'])
})


  • ['build'] - specify the build task in the dependencies. Firstly, he will rebuild the application before the start of the observation, secondly, we know that before the build build the command line arguments are parsed;
  • _.each (argv.apps, ...) - iterate over the applications specified in the arguments, look at their settings in the config, collect the targets to observe the changes;
  • gulp.watch (targets, ['build']) - start the observation, the build taskis executed upon changes. There is one drawback - if we run watch for several applications, then with any changes they will be rebuilt everything, but in reality it is unlikely that ever (never) it will be necessary to monitor several applications at the same time, so we don’t bother.


Reassemble with minification after the watch is completed - the second trick

The development process looks like this: launch the django dev server , launch the gulp watch and write / debug the front-end application. Thus, the development process itself ensures that the actual compiled application will immediately appear in the static folder with any changes, and we no longer need additional steps during the deployment. But the problem is that development is usually carried out with the --dev parameter (without minification), and now, a couple of times in the park, committing to production a non-minified package of 2 megabytes in size, I thought that I would have to come up with some kind of reminder, and better automation.

So in the watch task, the following code appeared:

    // handle Ctrl+C and build a minified version on exit
    process.on('SIGINT', function() {
        if (!argv.dev) process.exit()
        argv.dev = false
        console.log()
        gutil.log(gutil.colors.yellow('Building a minified version...'))
        gulp.stop()
        gulp.start('build', function() {
            process.exit()
        })
    })


  • catch CTRL + C;
  • if watch was launched with minification, then we simply end the process;
  • argv.dev = false - cancel the minification ban so that the next build will collect the package for production for us;
  • gulp.stop () - complete all current tasks;
  • gulp.start ('build', function () {...}) - call the build taskand exit after it is complete. It is very important here that the callback after the build was called correctlyin the build task, which I mentioned earlier, otherwise the task will end before the package is copied to the static folder and the process exits. The start method isnot in the documentation for gulp because it is actually not its method: it was inherited from Orchestrator.


The result is: run gulp watch --app appname --dev , debug the application, press CTRL + C to stop watch and gulp immediately collects the minified version of the package for us. We quietly commit and enjoy the result of our labors in production.

Total


We got a system for building js applications without any changes during the deployment process and without new dependencies on production. It allowed us to divide the code into modules and get one compact file on the output. Here you can add js-linter, tests, and much more.

In the same way, you can easily transfer, for example, styles to some Stylus and also minify them, but due to some human reasons, we have not yet done so.

To everyone who read, thanks for your attention.

Gulpfile is completely with an example application .

Also popular now: