How did we build

Introduction

When a project grows, acquires a lot of modules and dependencies, setting up a project to build manually can be prohibitive. In addition, all technical participants in the project, from developers, testers, to administrators, are involved in configuration, and therefore a waste of time.

In addition, it is necessary to constantly update development and test systems, and even confuse nothing. This is not without the practice of continuous integration .


This article describes the experience of implementing assemblies using maven, so basic maven knowledge is required. For a quick start, it will be enough to study the site http://www.apache-maven.ru , for a more detailed development of the topic - you should refer to the original source http://maven.apache.org

The purpose of the article is not to teach how to use maven, but to show the experience of implementation with positive and negative solutions. With this knowledge, it’s much easier and faster to deploy maven assemblies in your team.

Why not ant?

Anticipating such questions, I’ll say that we use ant , but not for the assembly itself, but for the technical functions, more on that below.

Another important advantage of maven is the ability to collect only the code that you work with, everything else is collected at night on a separate server.

Environment

First, it’s worth talking about the structure of our projects and development approaches. This is very important, due to the fact that the project has been developing for many years and has a structure that does not correspond to maven-webapp-artifact.

And so, we have many modules stored in one SVN repository. Each module has a main development branch (trunk), as well as tags (tags) and branches (branches) as necessary. Most modules contain both java classes and jsp pages, scripts, graphics, etc. Most modules are stand-alone web applications and can work independently. Examples of modules: core, soap, workflow, addons, common ... I counted about 100 of them.

The project most often consists of a separate module, which hosts specific resources and logic in java classes. Sometimes, there is no such module. The final assembly of the project consists of a set of modules in the form of a single web application.

The libraries required by the module are stored in the lib folder directly in the repository, for web resources there is its own sx-war folder. As well as the standard folders src and test.



Naturally, almost all modules depend on core, and from each other, as necessary.

There is also a database with meta information. Based on it, the modules interact, all work with the user is built. For example, a publication block is also a unit of meta information and is stored in the database. That is, in addition to the data itself, the database stores settings for working with this data. They are configured in the console, which is also built on publishing units. It is possible to transfer meta information using service packs.

Ideally, each module has its own common package with meta information. The database for core is the lightest database and can be expanded by any module. The kernel is almost an application-server, only without the ability to dynamically load / unload multifunction modules. Not all modules have a separate meta, but we strive for this :)

BC

We have such a phrase. It means that “this” was a long time ago, even before the start of our work in the company. I’ll tell you how the assembly setup went before me. I also had to spend a lot of time setting up for the first time.

And so you got a job with us. There is a second day. The first went to view subsystem screencasts and work with meta-information. It is necessary to configure the assembly. Well, if the knowledge base for the project contains information on the necessary modules for the project. Often this information is not relevant (forget to update or update from case to case). Information about the dependencies of the modules themselves cannot be found, and the one that is is also becoming outdated.

The trunk versions of all the necessary modules are extracted from SVN, a project is created in the IDE. And then the hardest part: configure the import of all the necessary libraries and communication modules. It is still complicated when it is necessary to exclude some versions of libraries from some modules and use others. Configure the assembly of all modules in one folder. Next, configure the web application server (using tomcat).

And now the project is configured, then you need to configure the second, third, version of the first (use branches to work on errors). The developer becomes uneasy.

Dependencies have changed, everyone’s assemblies have broken. And administrators use ant to create production assemblies - everything breaks down too. Often there are problems due to different assemblies between the administrator and the developer.

First try

We did not have special people in charge of the assembly, nor was there any good maven knowledge. Therefore, the first attempt to implement maven on projects failed.

The parent pom.xml was placed in a separate repository folder. After that, in a separate project folder, svn-externals were configured and all the modules necessary for the project were included, plus the parent pom.xml in a separate folder: The



classical approach using the hierarchy of modules in a single SVN structure did not suit us, because each module is used in dozens of projects. And not only in one single. And any manipulations with parent pom on projects are undesirable. In addition, fixing the version of any module will automatically affect all projects.

All required libraries were not searched at http://search.maven.org, because there are a lot of them and they are all impersonal in the repository (libraries without versions are stored in the lib folder. For example axis.jar, jtds.jar, velocity.jar). They did just that: all the libraries were loaded into the Nexus using a script. GroupId and ArtifactId were set using the mask sx. <Library name> (sx.axis).

Here's what a typical pom.xml module looked like:


It can be seen that it is necessary to change the list of modules in it for each project.

During assembly, each module was assembled in a separate war, and then unpacked into a common deploy using maven-overlay .

Reasons for failure

Completely crossed out the benefits of the maven approach using svn-externals. This property does not allow to unambiguously fix the code of all modules without manually editing the properties of the svn folder. This becomes almost impossible with the presence of 10-15 modules and the release of a patch for a certain version. And I really wanted to use the maven release plugin .

pros

• Everything is going "here and now." The developer does not depend on the update period of SNAPSHOT dependencies (builds from the main development branch);
• The developer can make changes to several modules at once (a questionable point, because the encapsulation of the modules is violated);
• The entire set of necessary modules is visible;

Minuses

• Complete assembly of all modules is a lengthy procedure;
• You cannot properly lock the code version;
• All libraries are not taken from the public repository, adding a new one is a separate procedure;
• Jar-hell is possible when someone decides to use the library from the public repository and it is already in the corporate one (both versions, or even three, will get into the deployment!). This happens because GroupId: ArtifactId: Version changes;
• The code seems to be stored in one SVN folder, and you need to remove something completely different (with svn-externals). Not transparent - for aesthetes;

Second attempt

The second attempt to implement assemblies developed in several iterations.

Iteration 1

First of all, it was necessary to obtain a mechanism for fixing the code (release of the version) of both individual modules and the entire product as a whole. To do this, we got rid of the use of svn-externals. Each module turned out to be autonomous and was assembled separately. All dependencies that the module has got into the final assembly.

The parent pom.xml has also been modified in accordance with the tasks. All modules use it as a base:

Iteration 2

A version release plugin has been implemented in all project modules . To do this, blocks were added to each module:


There were thoughts to take this configuration to the base pom.xml, and use the ArtifactId-Version property, but not in all projects, this value matches the name of the folder in the repository. Therefore, the idea was postponed until better times. The first version received just the basic for all pom.xml modules.

A new profile has been added to the base pom.xml: production. It uploads the assembly to corporate FTP and can later be extended to projects.

Iteration 3

After an unpleasant collision with jar-hell , it was decided to put the libraries in order in all assemblies. And it was not easy, because there were more than 100 libraries in the kernel and almost all without versions. Each had to be found in the public repository, to “calculate” the version. Libraries of “surprises” were opened with internal assemblies and patches, modified versions. We carefully placed them in the maven repository, preserving the coordinates of the artifact, but with the postscript _custom in Version.

We use Nexus as a repository. A gateway to public repositories with caching was configured on it. There is a single entry point: corporate maven repository. Before that, there were more than three: libraries, releases, daily builds, and other public repositories. All required authorization. Three similar repositories in each pom.xml, and even the access settings in settings.xml caused constant confusion.

They tried to leave a link to the repository only in the base pom.xml. This is quite realistic, but during the first assembly of any module, the maven should load the base module from the repository, and its coordinates are unknown. You can do it manually, after that everything works fine. Refused due to the opacity of this action.

Iteration 4

We set up nightly builds of all the basic and design modules in the Hudson continuous integration system . There were no problems with them, because the modules are now completely independent. It makes no difference whether it is a public library or a project module. Maven-based builds run after each code change in SVN, and successful artifacts fall into the Nexus repository.

Over time, we configured the cascading assembly of modules in tasks, as well as deployment to development assemblies and assemblies for testing. Deployment is performed using fairly simple ant scripts. They extract the latest module assembly ( dependency: get ) from Nexus , unpack it on the server and restart the application server (the scripts themselves can be found at the end of the article).

pros

• You can work with each module individually;
• By fixing the versions of all add-on modules, you can commit the entire project code;
• Continuous integration eliminates routine work and allows you to quickly transfer functionality to a productive environment and to the customer;
• The release of the module version is fully controlled and quite simple;
• Now you can develop with only one specific module. Indeed, the general deployment of several modules can affect the final behavior of the code;
• Thanks to Hudson, you can always see who broke the assembly;
• You don’t need to set up projects at all, the IDE does everything itself. It is only necessary to start the assembly (and the mvn package is enough);
• It is always clear which code the module depends on. And before, the trunk was always going with its errors, broken commits, etc .;

Minuses

• The IDE does not have a common project with all modules, so making changes to several modules at once requires separate work with each module. This is certainly not very convenient, but conceptually correct, because any module should be self-sufficient;
• You cannot release all versions of components in cascade when you release the main product
• ...

We develop a solution

In the process, edits are made to the assembly. For those who are still assembling "the old fashioned way", a page in the knowledge base is opened, which contains the dependencies of the modules. This information is updated based on the pom.xml module every night as a separate task in Hudson (do not forget about DRY ).

To speed up the assembly of modules, unnecessary work during assembly was excluded. For example, we build a soap module, which depends on core. The core code ends up in the final soap assembly. Next is going to project, which depends on core and soap. soap is not used separately, therefore core is excluded from its assembly, and it is included in project. Thus, we speed up the assembly. To enable soap assembly with the inclusion of the dependent core, a context profile has been added, which starts if necessary.

You can notice that all modules are found in pom.xml twice: one record is used during compilation, the second means that the module is in deploy. In order not to replace the versions of modules throughout pom.xml, all versions of corporate modules were moved to separate properties and are located at the very beginning of the file. This reduces the likelihood of errors when switching to another version:



ant-scripts for working with maven artifacts

Conclusion

So continuous integration was introduced in our project. The need for manual code updates on servers has disappeared. You can always see the results of the assembly, which are sent via RSS. The big screen of assembly monitoring has not yet been set.

New developers do not spend much time setting up assemblies. And this is only pleasing.

The release of code versions has been adjusted, now this process is almost completely automated and does not take much time. It has become much easier to maintain previously released versions.

New developments fall into the development contexts during the day, which means they reach the customer faster. Difficult integration errors also began to appear, which means that they can be corrected faster.

UPDATE: Useful article on implementation experience from commentators:http://bazhenov.me/blog/2011/04/09/build.html