About structuring and automation

    I would like to share a little experience on automation and standardization of the development process. The article does not claim to be true, but I hope it will be interesting reading for the end of the work week.

    In the beginning was the word


    To start, I’ll tell you a little how it all began:
    • A repository in which there was only one main branch.
    • 5 projects that should interact in a single ecosystem. And a few more are coming.
    • Constantly supplemented requirements leading to irreversible changes in the communication protocol between all components (not the best situation, but it already was).
    • Each of the projects had its own version format. Some were based on revisions, others on a date, third on both.
    • The lack of any consistency between all components (was only in words).
    • Lack of any automation of processes (something still was, but not a lot).
    • The lack of a single repository of all distributions (some were stored in svn, others in Track Studio, the third were rebuilt each time from the sources (it was funny to watch how testers unloaded application sources from svn and did make clean all to get a new binary for testing)).


    In the beginning there was a structuring


    The biggest problem was the lack of any mechanisms to control the consistency in functionality between the components of the system. There was not a single criterion, based on which, it could be safely said that the two components of the system taken at some point in time are consistent with each other.

    Consistent in functionality, we call components that communicate with each other in the same language of one version of the protocol. In our case, the protocol version is also closely intertwined with business logic. More precisely, no new business logic can be introduced without changing or expanding the protocol.

    During the long debate about “how to properly conduct svn”, the option of fixing tags was chosen and a number of requirements / rules were formulated which should be followed:

    Trunk
    • may not be consistent in functionality
    • not subject to manual comprehensive testing
    • not subject to roll out to production or merge to brunch

    Tag n
    • all components are consistent in functionality
    • functionality does not change
    • participates in the main testing process
    • May be marked as Production after testing.
    • It can be frozen and forgotten (deleted) after functional testing.
    • Bug Fix Merry in Trunk
    • From Trunk only critical bugs identified in earlier versions

    Disclaimer: in the case of a new system component that can already be started to be tested, but which obviously does not reach the functionality of the last tag, in order to get into the general testing process, such a component can be taken from the trunk on the responsibility of the developer and the general understanding that this absolutely exceptional and temporary situation until a new tag appears.

    Our tags, not such tags. But they are closer in essence to the concept of “tag” rather than to the concept of “brunch”.
    There was another SVN reference format for discussion - to conduct the main development in brunches. if we have a major change, then we create a separate brunch, implement this change there and merge it into the trunk. In this case, the trunk should always be stable, so that at any time it would be possible to take all the components of the system and they would be at least coordinated in terms of functionality, and at maximum ready for production. In our case, it does not look viable. Since sometimes one major change can come out at the time of the implementation of another. Then I would have to create a separate brunch on it and switch to it. Problems would begin at the moment when it would be necessary to keep two branches in the trunk.
    We have a very long development cycle, and the transition process from version to version has not yet been run. Therefore, giving our product to public testing, we would like to be able to make hot fixes. But without a tag (branch) for each final version, this would not be possible. We also do not have the opportunity to say that “they say it will be fixed in the next version”.
    Further in the article I will use the term “tag” in the meaning that I described above.

    We cut the scalps


    There would be no benefit from tags if all the components were consistent in functionality in it. The bottleneck in this case is the point in time when we cut the scalp and create a tag. As a rule, at one of the meetings we make the decision that in the next week we should prepare a trunk for creating a tag. That is, each of the projects should come into a state when it is coordinated in functionality with all the components with which it communicates directly. After that we create a tag. Which goes into testing. We name the tag in a special format MAJOR_NUM.MINOR_NUM.

    Single version format


    The next important step was the introduction of a criterion based on which it would be safe to say that the two components are consistent. We decided that the criterion would be the version. Moreover, the version itself should carry information based on which it can be understood that the two components are consistent.
    Version format: MAJOR_NUM.MINOR_NUM.REVISION. Those. The first part of the version is also the tag number. And we know that we have all the components in the tag consistent. Indication of two numbers at once in the tag name allows you to make a normal transition between major versions, i.e. instead 1.1.x, 1.2.x, 1.3.x, 2.4.x , we will have a 1.1.x, 1.2.x, 1.3.x, 2.1.x . The revision number remains and is used as usual. This is the revision number of each of the individual components, and not just the last revision of the tag.

    Then came the automation


    The next step was the implementation of CI (Continuous Integration). The choice fell on, I think many famous CI system, Jenkins. The choice was between Cruise control, with which I have not the most pleasant experience, Team City, which is paid for its normal use, and directly to Jenkins, which is free, widely distributed and well-documented. The free factor was important, since it was scary to ask the authorities for money for something that wasn’t a fact that would take root.

    I am your master you are my slave

    The components of our system are very diverse and currently use 3 build platforms: windows, linux and mac os. It turns out one master and 2 slave (I do not want to repeat the word “slave” anymore). The question of where to do the wizard did not arise, of course on Linux. The initial configuration was deployed from one master (Linux) and one slave (Windows). The process of automating the assembly of all our components has begun. In the course of work, the tag 1.1 already appeared and the number of tasks in Jenkins exceeded 10. And then the problems started. One day, Jenkins fell. We lifted him, he fell at the again. Once again. And it continued to fall, no matter how we run it through Tomcat or as a service. Regardless of the JAVA machine, open JDK or SUN java. Regardless of the current version. Rather, it was not Jenkins himself who fell, but the JAVA machine that he was crashing.JENKINS-16199 , though it all died out). Having spent a week and a half on all sorts of attempts to understand what is happening or at least why, it was decided to transfer the wizard to the Windows system. And lo and behold, in the same configuration everything began to work stably. Since time, the number of tasks has doubled, but everything continues to work. As a result, the final structure looks like this:
    Master on Windows, two Slaves, on Linux and MacOS, interaction on SSH through public keys.

    About implants

    We expanded the standard Jenkins delivery with the following plugins:
    Copy Artifact Plugin - We use it to use the artifact of one assembly in another. There is one simple point where you can stumble.
    If you save the artifact in one of the tasks like this

    And in the other, using this plugin, download this art,

    you should at least indicate the path to it as in the first case, and it doesn’t hurt to set the “Flatten directories” option to discard extra paths and copy only the art itself to the place where you need it.
    Jenkins description setter plugin - Allows you to specify some information about each assembly in the Build Description field. Those tasks that have a version output it there.
    Extra Columns Plugin- allows you to add the Build Description field to the View.

    Publish Over SSH - A great plugin for uploading files and executing commands on a remote machine. True, we are not using it yet due to the fact that auto-testing has not yet been established. The disadvantage of this plugin is that a preliminary configuration of all SSH connections is required and each task clearly indicates which connection to use at the stage of its creation.
    Python Plugin - part of the build steps is written in Python, because it is easier. Including there is a separate task that does almost everything the same as Publish Over SSH but only allows you to configure access settings to the remote machine on the fly.
    Xcode integration - Used to build a project on MacOS. We had to sweat in order to be able to make assemblies for different versions of iOS. As a result, on the build machine, it was allowed to use the sudo xcode-select -switch command without entering a password.

    Special, black, magic

    I already mentioned the need for remote fill and command execution, which the Publish Over SSH plugin almost perfectly solves . But which does not allow you to specify the desired target directly for each assembly process.
    We have our own installer, several testers and several machines. Many are too lazy to drag this file to themselves and do something with it to install the product. For these purposes, the “Remote deploy” link task was created.
    And so that she can:
    Pick up the installer from the tag that the user will indicate.
    Install it on the machine that the user will indicate.
    This is how it looks.

    This task is parameterized.
    Parameters
    TAG, HOST, USER, DEPLOY_DIR, OPENSSL_DIR, there’s no magic here, ordinary lines.
    JOBS_NUMBER - “Build selector for Copy Atrifact”, allows you to specify which version of the assembly to take, the last successful or some specific number or something else.
    PASSWORD is a parameter of type “Password Parameter” - closes the password with asterisks.
    This is how the desired assembly is taken from the desired task


    View Setup

    Views in Jenkins is a means of customizing the display of projects. We removed one column and added a “Build Description” for our Trunk views, Tag_X. We would like it to be the same in the default view of Jenkins. But Jenkins does not allow such manipulations with his main appearance. Instead, you can create your own view (Default), add all projects by mask. * And make this view the default view.

    We shoot at idle

    We have a special script for creating the version and all its possible variations. At the beginning, he appeared in each of the projects in which similar functionality was needed. After a while, we put it in one place and wherever it is needed, we pull it as an external dependence. And then Epic fail happened when he was accidentally corrected in one of the projects where he was addicted. Jenkins has lined up 15 projects for rebuilding, idle rebuilding.
    The first thing that came to mind was to lock the file for changes. They hung the needs-lock property on the file and blocked it (svn lock) from a special build of the user.
    But there is a better solution. Much better. Jenkins allows you to enter filtering before starting a build initiated by a repository change. This is located under “Source Control -> Advanced”. We are interested in the “Excluded Regions” parameter in which you can specify changes in which files and directories should be ignored.

    To the place of detention


    This concludes my story. At the moment, I have already left the place of work where this experience was gained and this article was written, so I can hardly answer any questions that arose about the details and settings.

    Thanks for attention.

    Acknowledgments


    FrimInc , zzapp for proofreading an article

    Also popular now: