December 3, 2010 at 14:03Practical aspects of using svn: externals
Despite the fact that distributed version control systems (Git, Mercurial, Bazaar) are gaining popularity, the good old Subversion is still widely used. In this article, I will consider the pros and cons of using external dependencies (svn: externals) in SVN repositories in practice.
Let's look at an example of what svn: externals are. Suppose we have a project that uses third-party open-source code, for example, the well-known PDF library iText. Usually we completely copy all the necessary code of this library to our repository. Subsequently, when iText updates are released, we manually replace the old files with new ones.
svn: externals provide a more convenient way. We select some folder of our repository and set the svn: externals property for it in this form (how to do it using TortoiseSVN ):
The first parameter determines where to download the changes, the second - where. Now, every time we take the latest updates from the repository, the current version of iText is automatically downloaded. Conveniently.
Consider the benefits of using svn: externals.
The principle of DRY (Don't Repeat Yourself) - the cornerstone in the development of quality software - extends to source code management. If the same files are duplicated in several repositories - this is not very good.
For example, in established teams, as a rule, a base of common code is formed, which is constantly used in various projects. All projects will not necessarily be in one large repository. Most likely - in different. In this case, without svn: externals, we are doomed to copy the same files between different repositories, as well as to manually update them everywhere in case of changes.
A direct consequence of the previous paragraph is that the source code is stored only in the original repository, we only download it from there to our working copy. The size of the repository can be very critical if you use paid SVN hosting with a limit on the total size of the repository (s).
It seems reasonable to split projects according to the rule “1 project <-> 1 repository”. With this approach, using svn: externals helps to simplify the separation of dependent projects.
Of course, we can work on all projects / modules within the same repository, but this approach is not without drawbacks. For example, all developers can upload many unnecessary changes to their working copies. It will not work to restrict developers access to those projects to which they should not have access. Nearby are unrelated projects. Anyway, keeping everything on the heap does not look like the right approach.
However, if we still work in one large repository, svn: externals can also be useful. Example: suppose in our repository (in trunk) the following folder structure: / Common / C # and / OurCustomer / Windows / HelloCSharp /. If in the HelloCSharp project we want to use code from Common / C #, then we have three ways:
The second is normal, but with minor flaws. For example, when transferring the Common / C # folder, we will be forced to change all relative paths in HelloCSharp. Or - to look at shared files you will have to go around the folders in the repository.
The third way through svn: externals looks the most ideologically correct. We simply load the common code to the specified location, in fact - just refer to it. A kind of symbolic link within SVN.
What was described in the very first example. If we want to upload the whole boost to our repository - not worth it. Using svn: externals for this purpose would be more appropriate.
The pros look very convincing, but do not rush with the widespread introduction of svn: externals in your repositories. Let's evaluate the problems that svn: externals brings:
If we use external dependencies, then we can forget about previous fast updates. Each dependency is at least one additional HTTP connection during the upgrade. Even if nothing has changed in the external repository, we will still wait. During real work, one gets the feeling that the matter is not limited to only additional connections, since the update time is growing very noticeably.
svn: externals is really slow. At least in the current implementation of Subversion.
If you use svn: externals in this form:
you can never guarantee that our previously run code is still working. Any change in the external repository could break it. The code may stop compiling, or latent errors will appear, and without any of our involvement.
This approach can hardly be recommended for use at all. Even if externals within the same repository. Even if we have full control over the external repository. With this approach, the history of the repository ceases to be unchanged. Having rolled back to some revision, we get an unpredictable result, since the code downloaded through svn: externals may change in the time elapsed since the revision was created. It is very likely that previously hassle-free code simply does not compile.
However, do not put an end to svn: externals right now, there is a solution to this problem - you need to reference a specific revision in an external repository, like this:
Now we will always receive a specific version of the code for this revision. When we want to receive updates, we update the revision number to a suitable one and download the updates. We check that our code compiles and works as before (we run the tests). If everything is fine, commit the changes to svn: externals.
If an external server (for example, itext.svn.sourceforge.net) is unavailable, then we will not be able to upgrade to either fresh or previous revisions (if externals are also used there).
If, of course, we want all the old revisions to remain operational, otherwise we can restrict ourselves to only changes for the current revision. But this, of course, is a disastrous path.
In this case, it is not very difficult to completely modify the repository, you must:
Both cases entail extremely negative consequences - our repository will become completely inoperative.
Example: in our repository lies the iText folder completely. Then, after reading one clever article, we decide to implement svn: externals instead. We successfully implement it, it works, everyone is happy. However, if now we want to roll back from the current revision to some earlier one, it will not work.
This is due to the fact that when you roll back the svn: externals installation, the target folder (iText) is not physically deleted. And when you need to replace it with a regular tracked folder in the repository, you get a conflict that the current version of SVN cannot resolve.
In this case, there is only one solution - to do a full checkout from zero revision to the right one.
If we decide to transfer the code from the SVN repository to the repository, for example, to Git or Mercurial, then svn: externals is a serious hindrance. There are three solutions: either do not go over at all, or ignore this problem and transfer it as it happens (in this case, we get old broken revisions), or completely get rid of svn: externals by hand.
The last path is thorny, but the result is worth it - we get a completely independent repository that we can import anywhere. How to get rid of svn: externals specifically is a topic for a future article.
To use svn: externals or not - everyone decides for himself. We initially worked without external dependencies, then when decomposing projects into several repositories, we began to actively use svn: externals, but over time, we realized that the disadvantages outweighed the advantages. In the end, we generally decided to migrate to Mercurial, and for this we had to completely get rid of svn: externals.
And finally, some tips for using svn: externals:
Let's look at an example of what svn: externals are. Suppose we have a project that uses third-party open-source code, for example, the well-known PDF library iText. Usually we completely copy all the necessary code of this library to our repository. Subsequently, when iText updates are released, we manually replace the old files with new ones.
svn: externals provide a more convenient way. We select some folder of our repository and set the svn: externals property for it in this form (how to do it using TortoiseSVN ):
iText https://itext.svn.sourceforge.net/svnroot/itext/trunk
The first parameter determines where to download the changes, the second - where. Now, every time we take the latest updates from the repository, the current version of iText is automatically downloaded. Conveniently.
About the pros
Consider the benefits of using svn: externals.
Avoid duplication
The principle of DRY (Don't Repeat Yourself) - the cornerstone in the development of quality software - extends to source code management. If the same files are duplicated in several repositories - this is not very good.
For example, in established teams, as a rule, a base of common code is formed, which is constantly used in various projects. All projects will not necessarily be in one large repository. Most likely - in different. In this case, without svn: externals, we are doomed to copy the same files between different repositories, as well as to manually update them everywhere in case of changes.
Do not take up space in the repository
A direct consequence of the previous paragraph is that the source code is stored only in the original repository, we only download it from there to our working copy. The size of the repository can be very critical if you use paid SVN hosting with a limit on the total size of the repository (s).
Easier to structure projects and modules
It seems reasonable to split projects according to the rule “1 project <-> 1 repository”. With this approach, using svn: externals helps to simplify the separation of dependent projects.
Of course, we can work on all projects / modules within the same repository, but this approach is not without drawbacks. For example, all developers can upload many unnecessary changes to their working copies. It will not work to restrict developers access to those projects to which they should not have access. Nearby are unrelated projects. Anyway, keeping everything on the heap does not look like the right approach.
However, if we still work in one large repository, svn: externals can also be useful. Example: suppose in our repository (in trunk) the following folder structure: / Common / C # and / OurCustomer / Windows / HelloCSharp /. If in the HelloCSharp project we want to use code from Common / C #, then we have three ways:
- copy the desired code to HelloCSharp
- use relative paths
- use svn: externals to load code from our_repository / trunk / Common / C # into HelloCSharp / Common, for example.
The second is normal, but with minor flaws. For example, when transferring the Common / C # folder, we will be forced to change all relative paths in HelloCSharp. Or - to look at shared files you will have to go around the folders in the repository.
The third way through svn: externals looks the most ideologically correct. We simply load the common code to the specified location, in fact - just refer to it. A kind of symbolic link within SVN.
Easier to use open-source code in our repository.
What was described in the very first example. If we want to upload the whole boost to our repository - not worth it. Using svn: externals for this purpose would be more appropriate.
About cons
The pros look very convincing, but do not rush with the widespread introduction of svn: externals in your repositories. Let's evaluate the problems that svn: externals brings:
Slow updates (svn update)
If we use external dependencies, then we can forget about previous fast updates. Each dependency is at least one additional HTTP connection during the upgrade. Even if nothing has changed in the external repository, we will still wait. During real work, one gets the feeling that the matter is not limited to only additional connections, since the update time is growing very noticeably.
svn: externals is really slow. At least in the current implementation of Subversion.
The code may stop working by itself
If you use svn: externals in this form:
iText https://itext.svn.sourceforge.net/svnroot/itext/trunk
you can never guarantee that our previously run code is still working. Any change in the external repository could break it. The code may stop compiling, or latent errors will appear, and without any of our involvement.
This approach can hardly be recommended for use at all. Even if externals within the same repository. Even if we have full control over the external repository. With this approach, the history of the repository ceases to be unchanged. Having rolled back to some revision, we get an unpredictable result, since the code downloaded through svn: externals may change in the time elapsed since the revision was created. It is very likely that previously hassle-free code simply does not compile.
However, do not put an end to svn: externals right now, there is a solution to this problem - you need to reference a specific revision in an external repository, like this:
iText –r 247 https://itext.svn.sourceforge.net/svnroot/itext/trunk
Now we will always receive a specific version of the code for this revision. When we want to receive updates, we update the revision number to a suitable one and download the updates. We check that our code compiles and works as before (we run the tests). If everything is fine, commit the changes to svn: externals.
If the server we are referring to is unavailable, we will not be able to upgrade
If an external server (for example, itext.svn.sourceforge.net) is unavailable, then we will not be able to upgrade to either fresh or previous revisions (if externals are also used there).
If the server we are referring to has moved - you need to completely modify the repository
If, of course, we want all the old revisions to remain operational, otherwise we can restrict ourselves to only changes for the current revision. But this, of course, is a disastrous path.
In this case, it is not very difficult to completely modify the repository, you must:
- Dump the repository using “svnadmin dump"
- In the resulting dump, find all lines of a similar type:
V 66 iText –r 247 https://itext.svn.sourceforge.net/svnroot/itext/trunk PROPS-END
- Replace the path to the repository in them and update the number in the first line - this is the number of bytes in the description of the property, for example:
V 123 iText -r 155 http://new_server.com/svnroot/itext/trunk/ iTextSharp -r 155 http://new_server.com/svnroot/itextsharp/trunk/ PROPS-END
Here, for clarity, I added a second line to make it clearer how the number V is calculated. - Delete our old repository and roll up a new dump using “svnadmin load”
If the server we are linking to has changed Subversion for something else or just stopped working - everything is gone
Both cases entail extremely negative consequences - our repository will become completely inoperative.
Update to revision may stop working
Example: in our repository lies the iText folder completely. Then, after reading one clever article, we decide to implement svn: externals instead. We successfully implement it, it works, everyone is happy. However, if now we want to roll back from the current revision to some earlier one, it will not work.
This is due to the fact that when you roll back the svn: externals installation, the target folder (iText) is not physically deleted. And when you need to replace it with a regular tracked folder in the repository, you get a conflict that the current version of SVN cannot resolve.
In this case, there is only one solution - to do a full checkout from zero revision to the right one.
Inability to correctly migrate to another version control system
If we decide to transfer the code from the SVN repository to the repository, for example, to Git or Mercurial, then svn: externals is a serious hindrance. There are three solutions: either do not go over at all, or ignore this problem and transfer it as it happens (in this case, we get old broken revisions), or completely get rid of svn: externals by hand.
The last path is thorny, but the result is worth it - we get a completely independent repository that we can import anywhere. How to get rid of svn: externals specifically is a topic for a future article.
Conclusion
To use svn: externals or not - everyone decides for himself. We initially worked without external dependencies, then when decomposing projects into several repositories, we began to actively use svn: externals, but over time, we realized that the disadvantages outweighed the advantages. In the end, we generally decided to migrate to Mercurial, and for this we had to completely get rid of svn: externals.
And finally, some tips for using svn: externals:
- Use only links to specific revisions in an external repository;
- Try not to use too many external dependencies, otherwise the updates will last a very long time;
- Use svn: externals for only one root folder (for example, trunk), and do not set this property for several folders scattered throughout the repository. Since the target folder can be set using relative paths (for example, Common / Java / iText), it is much more convenient to determine all the dependencies in one place than in several;
- If you install / update svn: externals for a folder - do not mix these changes with code changes, file transfers and other actions. It is better to make a separate commit for only changing the svn: externals property. It will greatly simplify life if someday you decide to completely get rid of svn: externals;
- Try not to change svn: externals too often, it will also simplify life when getting rid of external dependencies.
