Maven We collect only the changed
When working in a multi-module maven project, you often have to make changes to several related modules at the same time. And if you want to build only affected modules, then unfortunately maven does not provide anything automatic. If you google a little, then on stackoverflow you can find a simple one-line solution:
mvn install -amd -pl $(svn st | colrm 1 8 | sed 's /.* ' | xargs echo | sed 's- -,:-g' | sed 's ^ : ')This could be the end. But I wanted more - more specifically and how I achieved this under the cut.
For those new to bash, mvn or svn, a quick explanation of the script:
- The main part is mvn install -amd -pl project_list . Team for maven, collect projects from project_list and their dependent
- All that is inside
$(...)is getting local changes in svn and pulling project names from these changes
Why is it necessary
Consider the following simple scenario:
- Add a new method to the Parentable interface in module A
- We implement this method in the Child (implements Parentable) class in module B
To check the assembly, you need to separately assemble A, then B. If only the implementation in Child changes, then it is enough to rebuild only B. However, if the signature of the method changes, then you need to collect both modules again.
With this kind of work, one has to wonder "is the assembly of project B relevant?" It’s good when there are 2-3 projects, but if there are more of them, and refactoring through the IDE is also used, then it is very easy to miss the moment when the assembly ceases to be relevant.
What's wrong with the script above
The script presented above solves this problem clumsily: it always collects A and B. This is not very effective in terms of time, but it is relatively safe, because you will not forget to collect anything. I used this approach for about a month, but then I decided that it was better to forget to collect than to spend this extra time.
Besides this problem, there are a number of others related to maven features when this script just doesn't work. They will be discussed below.
What was done
I took the idea of this script as a basis and modified it with a file (in this case, a rasp). The goal was to get a script that collects everything that we changed, but have not yet collected. And nothing more.
Modifying a script on bash is a thankless task for a number of reasons. Therefore, development was conducted in python. From the pros:
- Cross-platform
- Modularity
- Almost all linux users and many window users have it.
- Simplicity
Of course, I want to just take it and use it, so only standard libraries are of the dependencies.
So, let's start the main part of the article - a description of what problems had to be faced and what solutions were used.
What can maven
To do as little as possible yourself, it would be nice to figure out what maximum you can squeeze out of maven. The maximum is not very large. The following areas can be collected in maven:
- One project
- One project and all its modules (recursively)
- List of modules within one project.
Option 3 looks the most suitable and it is used in the introductory bash script. Of the advantages, when working with this option, you can specify the parameters:
- -amd (also make dependents) - maven will also collect all dependent modules
- -am (also make) - also the modules on which our list depends
For local assembly, these options are not very useful, because dependency assembly (amd) can be assigned to the CI and in addition, our IDE can indicate compilation errors without maven building.
And building parent projects (-am) is redundant, because we can pull unchanged projects from the maven repository.
Unfortunately, this is the end of what maven can help us with. In fairness, it is worth saying that there are special third-party plugins, incl. incremental, but in this case you have to work with what is already there and you can’t change anything in the project.
Problem 1: we collect only actual changes
To understand what has changed, it is most convenient to use the status of files in the version control system. However, VCS will not tell us anything about when these changes were made and whether they were included in the assembly. Nevertheless, the list of all changes in VCS is our active projects and the starting point for further analysis.
To find out how our changes coincide with the assembly, we compare the date the files were modified with the date of the last build (target / artifact_id-version.jar).
For a first approximation, it’s good, but there are nuances that must be taken into account:
- If other changes come to us from VCS, then our project needs to be reassembled again
- target / artifact_id-version.jar is the default value. May be completely different
- Modified files may not be related to the assembly (for example, the project file for your IDE).
Point 1 can be solved by analyzing not only locally modified files, but also all files in active projects.
Two other problems are solved simply by obtaining the necessary information from the pom file and taking it into account during assembly.
Problem 2: Rollback Changes
If there were changes and we collected an artifact, then when you roll back these changes, the artifact needs to be rebuilt. But the status in VCS does not provide any information about such files. Therefore, it is worth storing this information yourself. To do this, after analyzing the current changed projects, we save this information somewhere and at the next build we combine the current changes and the previous ones from the saved file.
Problem 3: modules not included in the main project
The correct structure of a maven project is one in which each project is described as a module in the parent project. Those. having executed mvn install in the root, we must use all internal projects.
Everything would be simpler if everything was correct. But it's not always the case. In this case, there are untied projects. This does not suit the correct maven, and if you slip him such a project in -pl, then he will spit out an error in you, because -pl only works with projects that are listed as submodules in your parent project.
To solve the problem, you need to collect such projects separately. Those. Maven is launched as many times as we have unrelated root projects.
To complicate things, a situation may still arise when these projects are interconnected, so you need to collect them in a certain order.
Problem 4: local maven repository synchronization
Maven always makes builds based on artifacts in the local repository. Periodically, it can synchronize the local repository and download updates from the global repository.
So a situation may arise when you have collected your changes and no longer change the files. But sooner or later your local assembly will be superseded by external changes. One solution is to check the date of changes, not in the target file, but in the local repository. And if they are different, then you need to make a new build.
However, already collected files in target remain correct and nobody will drive them out of there. And in order not to do double work, you can simply drop them into the local repository, instead of doing the full assembly.
Problem 5: verbose maven
This is not quite a problem, but for me personally it is always a question why maven issues so many logs. Unless to sit and meditate, looking at the running lines. But if everything goes well, these logs are not at all interesting to me. If everything went wrong, then the logs are really useful. It is by this principle that I derive the logs.
As a nice bonus to pure output, a significant increase in build speed, as IO is far from the cheapest operation.
Problem 6: slow maven vs fast workarounds
Many of the problems above could be solved more easily if additional information was received from the maven. But he is very slow and performs any commands for a very long time. Using them in a loop is a surefire way to add a strong performance overhead.
Therefore, all problems are solved to the best of their abilities and capabilities on their own, and the maven is used only once - to start the assembly itself.
However, all attempts to bypass the maven are not complete and may not take into account a number of additional factors. They are enough for me on the current configuration of the project, but any more sophisticated configurations are not taken into account.
Bonus 1: script for the build server
Everything described above was written and used for local assembly when we make the changes ourselves.
But after a successful experience of use, I decided that this approach can be used on our continuous integration server and collect only what has changed between revisions. Of course, it is necessary to take into account most of the problems described for local assembly. So both versions of the script go hand in hand.
It is important that such an incremental assembly gives exactly the same result as the full one. This can be easily achieved using the same maven -amd parameter described above. Those. what is changed and / or offended is collected.
Regarding the gain received - in our case, the average time of incremental assembly is 10 times less.
At TeamCity, such an opportunity is out of the box. No on bamboo. I don’t know about other CIs
Bonus 2: python project mini-minifier
This section has nothing to do with maven or java, and may be of interest only to python developers.
The script that I am describing here is written as a project with several modules, because it's easier for me to work with code.
But to share a script that contains several files is not very convenient.
Alternatively, you can build a single binary from scripts and issue it. But in this case, the possibility of fixing on the fly and understanding what is happening there is lost.
Therefore, another script was written in haste, which takes a python file as input, collapses it and all the dependent modules. The output is a single file. At the same time, unnecessary empty lines are deleted.
Epilogue
The script that I am describing here has been used by me and several colleagues for a year now. The script for CI is currently undergoing test operation.
The performance of the script is quite tolerable: I don’t notice its analysis and optimization (i.e. they are less than a second) within our projects.
So far, this only works with svn, as there is no need to collect in other VCS. But optionally, you can easily add others. The benefit of VCS is not required there so much.
I will be extremely happy if my approach and script will allow someone else to save their time and effort in building with maven
Questions, corrections and additions are highly appreciated.
