October 30, 2008 at

We roll out into battle

    To the battle!The implementation of the final web product is not the most pleasant procedure for the creator and is often accompanied by terrible stress. The developer’s dislike for releases is associated not only with feelings of responsibility and fear of the exploitation of the new version, but also with feelings of uncertainty: what will happen after we introduce it?

    Applications can be developed by a large team of programmers, quality engineers, graphic interfaces, but at the end of the project path, the last of the Mohicans takes responsibility. The lack of theoretical knowledge makes our hero nervous, because the experience gained as a result of trial and error is not enough for an hour for a systematically successful implementation. To figure out how to properly roll web projects into battle, let's start with the basics.

    General Staff Plan

    development workflow

    Having received the requirements, specifications, all kinds of recommendations and instructions, the developer allocates space for his maneuvers in the source code repository. Such an area can be a personal SVN branch, a new module in CVS, or a folder on the file system.

    It makes no sense to talk in detail about the subsequent cycles of "Develop - Test - Deploy": there are so many useful things written about them. I note only important observations:
    • The application of each developer should use personal databases and file storages. To use the general “garbage can” is an evil that generates serious errors.
    • The development of autotests is not always justified : their implementation should not take more than 20% of the working time. To develop through testing is an extreme degree of waste.
    • The more participants in the process, the more often it is necessary to merge and check the collective result. Continuous integration can help, but maintaining it can be overwhelming - you need to balance between automation and cost.
    The final development results are recorded in the release candidate (branches merge, a tag is created, etc.) and transferred to acceptance testing. A candidate release is the final monolithic product that cannot be modified. If it is necessary to correct errors, modify the functionality, it is necessary to start from the beginning and make a new candidate.

    Hard to learn

    It is worth noting that many managers conduct acceptance testing in the development environment, which is fundamentally wrong. It is clear that making a separate candidate for nanobags is very lazy (and wasteful). But once worked out the procedure of implementation on a test platform will significantly save the developer time and nerves. Moreover, no one bothers to group minor corrections into groups and fix them in one version of the product.

    The main hints of acceptance testing:
    • The candidate implementation scenario should be as close as possible to the release implementation scenario. If you plan to update the database structure in the final release, migrate the data, add third-party libraries, you need to do all this exactly in the order in which the release is supposed to be introduced.
    • The conditions of the test platform should be as close as possible to the conditions of the combat. If the application uses a cluster of 5 memcached servers, prepare> 1 server on the test platform. If the application uses 1 DBMS master server in RW mode and 2 slave servers in RO mode, recreate the full replication scheme. Save on equipment for testing distributed systems will help free VMWare Server (or any other analogue, except Virtuozzo). The versions of system libraries, OS kernels, etc., must also coincide with the versions of the corresponding components of the combat platform.
    • Load testing is a floating acceptance testing problem. It is impossible to completely recreate the release conditions: we do not know what exactly the load will be, we can only speculate. Yes, and to create a full copy of the combat platform is unprofitable. For adequate testing, bottlenecks are usually identified in the application, the maximum stability values ​​are measured for them and the results are approximated to the expected loads.
    • Testing should be regression , but you need to balance between quality and cost. If only cosmetic changes are made to the interface, it makes no sense to run the test plan completely.
    • Do not forget to roll back the application , the return to the previous version should be carefully worked out.
    The transition from a candidate to release may take a certain number of iterations: “Candidate - Testing - Change”. During the cycle, new bugs will be found, requirements will be changed and changes will be made. Ultimately, you have to break the vicious circle and make a release. And it’s extremely rare that a prerelease test report is 100% complete (developers in such cases are excused with the beta prefix ).
    I once taxied the development of a portal about advertising. The team then had to roll out ~ 20 candidates before introducing the release. The release was introduced from the 10th time. The final test report was successful by ~ 70%. Not the most successful project in my life.

    To weapons

    The release implementation order should be as follows:
    1. We are introducing new versions of the platform components . We update libraries, services, etc., while retaining a tool to roll back to older versions.
    2. We introduce the database and file storage . If the previous version of the application works in battle, you must prepare the old release in advance for supporting the new storage structure. The story is this:
      • add support for the new storage scheme to the previous release
      • introducing a modified previous release
      • we introduce a new storage structure and migrate data (the previous release should work in the old logic with the new storage)
      • introducing a new release
    3. We are implementing the application .
    4. Testing the application .
    5. We roll back (if necessary).
    In the case of a fakap, we roll back to the previous version in the reverse order:
    1. Roll back the application .
    2. Roll back the database and file storage (if necessary).
    3. Rollback platform components (if necessary).

    Application to battle

    Firstly, the application must be designed so that its folder structure is separated from the file storage.
    Right
    / var / www / myproject /
	/ etc /
	/ lib /
	/ classes /
	/ images /
	/ templates /
	index.php
/ var / www / userdata /
	/ images /
	/ video /
    		Not properly
    / var / www / myproject /
	/ etc /
	/ lib /
	/ classes /
	/ images /
	/ templates /
	/ userdata /
		/ images /
		/ video /
	index.php
    Secondly, it is necessary to control third-party library versions. To be sure that the supported version of the Zend Framework library is used in battle, you must either include the library in the folder structure of the application (and keep it in svn ), or present the release in the form of a package (for example, deb ) with which to check dependencies.

    Thirdly, many developers introduce the application into battle by updating from the source code repository ( svn up ). You can only compare this approach with rm –rf / . The release must be fully extracted from the repository and placed in a separate place from the previous release.
    Releases are placed in separate folders on the combat platform.
    user @ stable: ~ / apps / myproject $ ls -la
drwxr-xr-x 12 user www-data 4096 Oct 29 11:38.
drwxr-xr-x 5 user www-data 4096 May 13 23:22 ..
drwxr-xr-x 8 user www-data 4096 Oct 22 17:11 rel_0.5.1
drwxr-xr-x 8 user www-data 4096 Oct 27 13:17 rel_0.5.1.1
drwxr-xr-x 8 user www-data 4096 Oct 28 02:07 rel_0.5.1.2
drwxr-xr-x 8 user www-data 4096 Oct 29 11:38 rel_0.5.2
user @ stable: ~ / apps / myproject $ svn co svn + ssh: //user@svn.myproject.ru/myproject/tags/rel_0.5.2.1
...
user @ stable: ~ / apps / myproject $ ls -la
drwxr-xr-x 12 user www-data 4096 Oct 29 11:38.
drwxr-xr-x 5 user www-data 4096 May 13 23:22 ..
drwxr-xr-x 8 user www-data 4096 Oct 22 17:11 rel_0.5.1
drwxr-xr-x 8 user www-data 4096 Oct 27 13:17 rel_0.5.1.1
drwxr-xr-x 8 user www-data 4096 Oct 28 02:07 rel_0.5.1.2
drwxr-xr-x 8 user www-data 4096 Oct 29 11:38 rel_0.5.2
drwxr-xr-x 8 user www-data 4096 Oct 29 12:31 rel_0.5.2.1
    After extracting the source code, the project is installed (configs, rights, start data are configured) and tested next to the working release, for which a separate virtual host of the web server can be used (for example, www-test.myproject.ru ). After running the test plan, the previous release is replaced by a new one. The following options for replacing releases are known: changing the symbolic link, mount_null , changing the web server config , followed by graceful restart .
    With a symbolic link, we switch the project from one release to another.
    In the same way, you can roll back to the previous version.
    user @ stable: ~ / httpdocs / $ ls –la
drwxr-xr-x 3 user www-data 4096 Oct 29 11:40.
drwxr-xr-x 9 user www-data 4096 Oct 28 21:25 ..
lrwxrwxrwx 1 user www-data Oct 27 29 11:40 pro -> ../apps/myproject/rel_0.5.2
user @ stable: ~ / httpdocs / $ rm pro && ln –s ../apps/myproject/rel_0.5.2.1 pro
user @ stable: ~ / httpdocs / $ ls –la
drwxr-xr-x 3 user www-data 4096 Oct 29 11:40.
drwxr-xr-x 9 user www-data 4096 Oct 28 21:25 ..
lrwxrwxrwx 1 user www-data Oct 27 30 17:12 pro -> ../apps/myproject/rel_0.5.2.1
    If the application is distributed:
    • disable part of the backends by redirecting all traffic from the fronts to the remaining workers
    • On disabled backends, we conduct local implementations and tests
    • we switch traffic to fresh backends
    • repeat the procedure for backends
    • As a result of the block implementation of the front-ends, we switch to the normal state and distribute the load among all the back-ends.
    The more servers are involved and the more complex the application structure, the more automation and monitoring tools will be required. However, do not forget the favorite rule of futurologists: do not trust robots - they were created by people who tend to make mistakes.
    PS If you have enough strength to finish the story, wait for the second part about the delights of LiquiBase , the pitfalls of rsync and the migration of file storage.
    PPS During the correspondence with one of the habrayuzers it was formulated: the main thing in this informational appeal is not to impose my recommendations, but that I make a person think about problems that may arise in their specific situation. General recommendations on * ^ & no one needs. In the end, everyone does it his own way. And it is right. Because every project is unique.
    Tags:

    Also popular now: