We develop the front-end Diary.ru. Part one. Build and validate JavaScript code

Introduction

During the existence of Diary.ru (and this is more than 4 years), a huge amount of JavaScript code has accumulated: some were in a separate project in the form of connected files, some were determined directly on the control markup, and some were collected directly in code-behind with the help of StringBuilder. To this were added:

• a growing number of HTTP requests to receive static content - for example, 11 JavaScript files were loaded on all pages in a tag only ;
• global variables that sometimes overlap each other;

Having decided that it was time to do something with this, we set ourselves the first priority task: to remove all individually connected files from the tag into one minified package. At the same time, the code was divided into third-party and “ours”, which was planned to be checked by some sort of parser.

In this article, we will tell you how you solved this problem.

What to use?

First of all, we had to decide by what means we would organize the automatic assembly of this package. Of course, you could use any build system, from Ant to MSBuild; you could write your own simple script - for example, in Ruby or Python. As a result, we decided not to write our bikes and not to hammer nails with a tractor, but to use Grunt . For those who do not know: Grunt is a JavaScript task runner, it runs on node.js , and is distributed under the free MIT license. Despite the relative “youth” of this solution, it has already established itself as an excellent tool - it is used to build jQuery and QUnit, Tweetdeck on Twitter and Brackets in Adobe. In addition to these recommendations, we had our own reasons why we chose Grunt:

• Ease of use - in order to start working with it, you just need to install node.js.
• All tasks can be solved using JavaScript on node.js, use JSHint for parsing, UglifyJS to minify the code, and if you look into the future, node.js will be indispensable for unit testing, checking and building styles.
• A large selection of plugins for launching various tools, as well as a simple API for writing your own plugins.

By the way, it is no secret to anyone that our project works on ASP.NET, so we considered the possibility of using the Web Optimization Framework derived from it Bundle Transformer . However, we rejected these decisions for the following reasons:

• using these tools it is not possible to parse the code;
• the content delivered to the client is generated dynamically upon request, and this operation is in any case harder than the web server uploading a static file. Someone may say that this is a saving on matches, but:
  firstly , we do not agree with this - there are quite heavy operations in our project that already load the server;
  secondly , it was technically impossible to do this purely, since the project in which the JavaScript files are stored is not a web application for us,
  in addition, we needed static files in connection with the transition to CDN in the near future.

However, if in the future these tools will rise to the level of sprockets from Ruby on Rails, then I do not exclude that we will return to their consideration.

Go!

So, the system for the assembly is selected and it is time to act, but before further narration it is worth making a reservation. Since our application is written in ASP.NET, most developers work on Windows (which is not surprising), and the continuous integration process that we built using TeamCity (we wrote about this in a previous article ) also Windows Therefore, the author asks Unix-way fans to forgive him for the fact that the following will be described within the framework of the Windows ecosystem, and to perceive the entire experience below as a challenge.

Installing node.js on Windows has not been a problem for a long time. All you need to do is download the binary file from the official website, run it and poke into the "Next" button. Along with node.js, npm will also be installed - a package manager with which we will install both Grunt and everything necessary for its operation. First, create a file in the project package.json, in which we write the name of our project, its version, dependencies and version of node.js. It will look something like this:

{
    "name": "Dnevnik",
    "version": "0.1.0",
    "private": true,
    "dependencies": {
        "grunt": "0.4.0",
        "grunt-cli": "0.1.6", 
        "grunt-contrib-concat": "0.1.3",
        "grunt-contrib-jshint": "0.2.0",
        "grunt-contrib-uglify": "0.1.1",
        "grunt-hash": "0.2.2",
        "grunt-contrib-clean": "0.4.0"
    },
    "engines": {
        "node": "0.10.0"
    }
}

In the dependencies, we indicate Grunt and its version, as well as the necessary plugins. At the initial stage, we used only six plugins:

• grunt-cli - plugin to run Grunt from the command line
• grunt-contrib-concat - plugin for concatenating a list of files into one
• grunt-contrib-jshint- plugin for checking JavaScript code using the JSHint utility
• grunt-contrib-uglify- plugin for minifying JavaScript code using UglifyJS2
• grunt-hash - a plugin for adding a hash sum to file names (in order to flush the cache when the contents of the file change)
• grunt-contrib-clean - plugin for cleaning the directory from temporary files and artifacts

To install all packages with their dependencies, you need to run only one command in the console relative to the directory in which it is located package.json:

> npm install

After successful completion, a folder will appear in it .\node_modules, in which all the necessary modules will be contained (this is the standard name of the folder for modules installed via npm).
Next, you need to create Gruntfile.jsin the root directory of the application, it will contain all the logic of Grunt. Its structure is very simple:

module.exports = function (grunt) {
    'use strict';
    grunt.initConfig({});
    grunt.loadNpmTasks('grunt-contrib-jshint');
    grunt.loadNpmTasks('grunt-contrib-concat');
    grunt.loadNpmTasks('grunt-contrib-uglify');
    grunt.loadNpmTasks('grunt-hash');
    grunt.loadNpmTasks('grunt-contrib-clean');
    grunt.registerTask('default', ['jshint', 'concat', 'uglify', 'hash', 'clean']);
}; 

Essentially, this is a JavaScript script for node.js consisting of:

• a wrapper function that takes the grunt parameter,
• the function grunt.initConfig()to which the JavaScript object with the configuration of all tasks is passed,
• a function grunt.loadNpmTasks()that loads tasks from npm packages,
• function grunt.registerTask()that registers own tasks.

When starting a task, it tries to find an attribute with its name in the object that was passed to the function grunt.initConfig(), and from it receives all the settings through the attribute optionand goals through the other attributes. There can be an unlimited number of goals in a task, and each goal can override some settings for itself. You can read more about the configuration of tasks in the official documentation .

In order to start Grunt, you need to run the following command in the console relative to the application root directory:

> .\node_modules\.bin\grunt.cmd

Optionally, you can pass a parameter with the name of the task and the goal to be completed. If it starts without parameters, the task with the name will be executed default.
Next, we had to split those notorious 11 files, which contained various libraries and jQuery plugins, into separate atomic files. Some of them were compressed, and for the convenience of development, I wanted to have all the code in a normal, readable form. And if it’s easy to find a non-minified version of jQuery, then finding the right version of some ancient plug-in was no longer so trivial, I had to tinker with it. However, the result was worth the effort: now there was no minified code in the project, and you could easily get into the jQuery source code with a debugger.

In order for each of the developers not to need to install node.js and build the package, we made a simple mapping file in JSON format (of course, not the most beautiful solution, but we decided to make everything as simple as possible at first), in which the package name corresponded to the set from several files:

{"package.js": ["jquery.js", "foo.js", … "bar.js"]}

When the application started, this file was read, deserialized, and files were added to the page. And during the assembly, Grunt received information about which packages from which files should be built, and at the final stage converted it. So instead of a set of files in the list was one name of the assembled package.

After the code for running Grunt was written and the tasks were prepared, it was necessary to install node.js on all build agents in TeamCity and try it in action by running a script through PowerShell. In order not to download the necessary dependencies from the network each time (don’t think it’s not about our stinginess on traffic - we just don’t want to depend on the stability of the Internet or the npm repository), we decided to save them in a separate folder on each build agent and copy to the right place before use. “Cheap and cheerful,” we thought (about what this led to, read below). However, in this situation, it is worth considering that the paths in the folder.\node_modulescan be much longer than the maximum characters allowed in Windows 260 characters (Hi, MS-DOS), so the copy and xcopy commands will fly out with an error, unless robocopy with the / E flag can come to the rescue .

What problems we encountered and how to solve them.

Grunt slipped the first pig to us immediately after launching on TeamCity - we could not get a log of his work. Poking around in our PowerShell scripts and realizing that the problem is not on our side, we began to look at the issue tracker in the Grunt repository and found a noteworthy message there . It turns out that this problem appeared not only with us and it is associated with a bug in node.js, in which the stdin / stdout / stderr streams are not blocked on Windows. They promise to fix it in version 0.12.0, but for now, in order for Grunt to work with us, we had to resort to a not very beautiful hack: we started Grunt two times - the first time we got the correct exit code, and the second we redirected the stream output to a file, after which the contents of this file were displayed.

Not so long ago, a patch for Grunt appeared that fixes this error, but it is not in the main repository yet. So we had to download the fork directly from Github, and here we were faced with yet another trouble. The fact is that when we first started working with Grunt, there were only three cars in our build agents park. Now there are eight of them, and copying a new package on each of them is a tedious task. Without thinking twice, we decided to stir up a local npm repository, where we could always quickly pick up packages and where we could put our own, regardless of the connection and the availability of the official repository.

The official npm repository works with CouchDB, and to create a local repository we just needed to create its replication. We quickly picked up the virtual machine (again, running Windows) and installed CouchDB on it - since this is no more complicated than installing node.js. Further, in order to be able to access the repository from the local network, \etc\couchdb\local.initwo values ​​must be changed in the CouchDB configuration file :

secure_rewrites = false
bind_adress = 0.0.0.0

You can verify the correctness of the settings by sending a regular GET request to the 5984 port of the virtual machine and receiving something like this JSON response:

{"couchdb":"Welcome","version":"1.2.1"}

After that, it remains only to obtain information about all the modules used in the project and replicate them. To do this, in the root directory of the project, you can run the following command:

> npm shrinkwrap

She will create a file npm-shrinkwrap.jsonthat will contain all the information about the project, including all the dependencies. But, since we will only need their names, we will have to work a little more by writing a small recursive script that will get them from the resulting file (I will not give its code, since it is incredibly banal). Having received the list with the names of the packages, it remains for us to perform a normal HTTP request to CouchDB for their replication. We will use the utility for this curl(although you can use any other) and for convenience we will create a JSON file with the name deps.jsonwith the following contents:

{
    "source": "http://isaacs.iriscouch.com/registry/",
    "target": "registry", 
    "create_target": true,
    "doc_ids": ["_design/app", "_design/ghost"]
}

where the attribute value "doc_ids"needs to be supplemented with a list of necessary dependencies (packages "_design/app", "_design/ghost"organize the work of the database as a repository). And now just execute the following command in the console:

> .\curl.exe -X POST http://user:password@npm:5984/_replicate -d@deps.json -H "Content-Type: application/json"

The response from the server will again be in JSON format, and it is worth paying attention to two attributes: "ok"and "doc_write_failures". If the first value true, and the second 0, then packet replication was successful.

All that was left for us to do was publish the fork received from Github with a patch for Grunt. To do this, change the name of the version in package.jsonthe fork file , register it in the local user repository using the command:

> npm adduser --registry="http://npm:5984/registry/_design/app/_rewrite/"

And publish the package:

> npm publish --registry="http://npm:5984/registry/_design/app/_rewrite/"

That's it, the job is done, the package is published in our local repository, you just need to remember to change its version in the package.jsonproject.
Now, to install all packages, you can (and should) use the following command:

> npm install --registry="http://npm:5984/registry/_design/app/_rewrite/"

By the way, not so long ago we refused to use a PowerShell script to run Grunt on TeamCity and switched to using a plugin for it. The plugin is called TeamCity.Node and allows you to run node.js scripts, npm, Grunt and PhantomJS on TeamCity, while it checks whether node.js and npm are installed on the build agent. While we are absolutely happy with his work, if only because it was with his help that we learned that we forgot to put node.js on one of the agents.

What's next?

We look forward to the release of node.js 0.12 and Grunt 0.5, in which the errors described above should be fixed. And our plan for the future is something like this: firstly, we need to abandon the use of file mapping, and secondly, we need to move all JavaScript code from controls to separate files in order to reduce the amount of code and improve its support.

But we will talk about this in our next articles.