Git subrepo

The git-subrepo project has been around for a long time, but there is an unfair mention of it. The author of git-subrepo is Ingy döt Net .

If you look at the history of the komits of the master branch of the project, it may seem that the project stopped developing 2 years ago. However, work on the project is underway and I would like to hope that version 0.4.0 will be released soon .

An important feature of this tool is that on the user side there is no need to install git-subrepo until the user decides to do commit in the upstream-repository of subprojects. In addition, the user receives a fully prepared and customized source tree at the time of copying the main repository using the standard git-clone (1) command .

When choosing the means of supporting the submodules / subtrees / subprojects of the main container repository, the developer first of all determines the range of possibilities that this or that mechanism provides and answers the following questions:

• whether it is necessary to store the full history of the subproject in the main repository or enough squashed commits;
• whether the ability to deliver changes from the subproject to the upstream repository of the subtree is needed;
• is there a need to connect fixed tags of the upstream repository of the subproject or is it enough to connect the branches;
• whether it will be necessary to delete further both the subprojects themselves and, which have become unnecessary, a part of the history of these subprojects;
• whether the user will have to take any actions to manually configure the subprojects after cloning the repository of the main project;
• how time-consuming will be the question of analyzing the history of connecting subprojects and specific audits, from which the subproject originates;
• how a particular tool will affect the configuration management policy (Source Configuration Management) and how much this tool will complicate the daily work of engineers.

Of course, this list of questions cannot reflect all the completeness of the input parameters necessary for the correct choice, but for a preliminary review of existing tools, it is quite sufficient and, speaking of the git-subrepo project , we urge the reader to consider this project from these positions.

Git-subrepo installation

The git-subrepo package , on the developer’s side, can be installed both locally, in its home directory, and at the system level.

In the first case, it is enough to clone the git-subrepo repository into the desired directory, for example, ~ / bin :

bash-4.4$ cd ~/bin
bash-4.4$ git clone https://github.com/ingydotnet/git-subrepo.git

and set up environment variables

bash-4.4$ vim subrepo-env.sh
#!/bin/sh
export GIT_SUBREPO_ROOT="/home/username/bin/git-subrepo"export PATH="/home/username/bin/git-subrepo/lib:$PATH"export MANPATH="/home/username/bin/git-subrepo/man:$MANPATH"
:wq
bash-4.4$ source ./subrepo-env.sh

If you look at the variables defined in the git-subrepo Makefile :

# Install variables:
PREFIX ?= /usr/local
INSTALL_LIB  ?= $(DESTDIR)$(shell git --exec-path)
INSTALL_EXT  ?= $(INSTALL_LIB)/$(NAME).d
INSTALL_MAN1 ?= $(DESTDIR)$(PREFIX)/share/man/man1

it is easy to find out that at the system level, git-subrepo is installed in the directory where Git is located :

bash-4.4$
bash-4.4$ git --exec-path
/usr/libexec/git-core
bash-4.4$

Thus, the command for installing git-subrepo may look, for example, as follows:

bash-4.4$ make PREFIX=/usr install

The presence of variable DESTDIR allows without any additional effort (of course, if we are not in a cross-environment) to make a distribution package for all the Linux , which can be useful for DevOps engineers.

Install git-subrepo on behalf of the superuser:

bash-4.4$
bash-4.4$ cd git-subrepo/
bash-4.4$ make PREFIX=/usr install
install -C -d -m 0755 /usr/libexec/git-core/
install -C -m 0755 lib/git-subrepo /usr/libexec/git-core/
install -C -d -m 0755 /usr/libexec/git-core/git-subrepo.d/
install -C -m 0755 lib/git-subrepo.d/help-functions.bash lib/git-subrepo.d/bash+.bash /usr/libexec/git-core/git-subrepo.d/
install -C -d -m 0755 /usr/share/man/man1/
install -C -m 0644 man/man1/git-subrepo.1 /usr/share/man/man1/
bash-4.4$

To analyze the capabilities of git-subrepo, we need a simple test environment, where we can reproduce standard scripts of work.

Test environment

We will create three directories owner , remote , user , in which we will place the upstream and local repositories of the developer and the user.

bash-4.4$ vim _init.sh
#!/bin/sh

CWD=`pwd`
mkdir remote owner user
cd remote
git init --bare build-system.git
git init --bare platform.git
cd ../owner
git clone$CWD/remote/build-system.git
git clone$CWD/remote/platform.git
cd build-system
echo -e "\n[master] build-system 1.0.0\n" >README
git add README
git commit -m "init build-system master 1.0.0"
git push
cd ../platform
echo -e "\n[master] platform 1.0.0\n" >README
git add README
git commit -m "init platform master 1.0.0"
git push
cd ../../user
git clone$CWD/remote/build-system.git
git clone$CWD/remote/platform.git
cd$CWD
:wq
bash-4.4$
bash-4.4$ ./_init.sh
bash-4.4$

Here,

The author of the project and users have their own copies of upstream-repositories on their machines, presented in our example in the owner and user directories, respectively.

The author’s task is to include the builld-system subproject in the main platform tree to provide the following possibilities to users and project participants:

• clone the main repository with the build-system subproject included in it and at the same time do not worry about setting up versions or revisions. That is, each branch of the platform repository must correspond to a specific revision of a specific build-system repository branch and the user must receive the configured source tree in one git-clone (1) operation , without any additional actions.
• deliver your changes to the upstream project repositories, both to the main and auxiliary ones.
• receive changes made by other project participants or users, of course, if they have the appropriate rights.

Consider the actions of the author, which he must implement to implement these requirements.

Subproject connection

To connect a new subtree, use the git subrepo clone command , which is similar in its purpose to the git-clone (1) command . The required parameter of the command is the URL of the remote repository. You can also specify the directory in which the subproject will be located and the remote repository branch. We will work with master branches, therefore, in our example, we omit unnecessary command parameters.

So, the author of the project, on his working machine, can connect the build-system subproject using the git subrepo clone ../../remote/build-system.git/ build-system command :

bash-4.4$
bash-4.4$ cd owner/platform/
bash-4.4$ git subrepo clone ../../remote/build-system.git/ build-system
Subrepo '../../remote/build-system.git' (master) cloned into 'build-system'.
bash-4.4$

Consider the changes that occurred in the local platform repository :

bash-4.4$
bash-4.4$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
nothing to commit, working tree clean
bash-4.4$
bash-4.4$
bash-4.4$ git subrepo status
1 subrepo:
Git subrepo 'build-system':
  Remote URL:      ../../remote/build-system.git
  Upstream Ref:    b2f5079
  Tracking Branch: master
  Pulled Commit:   b2f5079
  Pull Parent:     b5e76a7
bash-4.4$ 

The history of the build-system subproject is not supplied to the main tree, we have only one squashed committee, which is accompanied by reference information. This information comes under version control in the ./build-system/.gitrepo/config file :

[subrepo]
	remote = ../../remote/build-system.git
	branch = master
	commit = b2f507918f2821cb7dd90c33223ed5cc3c9922a2
	parent = b5e76a713f895565b06ee3ccfa29f19131ba06dd
	method = merge
	cmdver = 0.4.1

Information about subprojects can be obtained using the git subrepo config command , for example, you can learn the revision of the remote / build-system.git upstream project that just came to the main repository using the command:

bash-4.4$
bash-4.4$ git subrepo config build-system commit
Subrepo 'build-system' option 'commit' has value 'b2f507918f2821cb7dd90c33223ed5cc3c9922a2'.
bash-4.4$

It should be mentioned that the original git-subrepo package stores information about the subprojects not in the .gitrepo / config file, but in the .gitrepo file . If you prefer to store information in a hidden subdirectory, you can select the git-subrepo-0.3.2 branch or the git-subrepo-0.4.1 branch of the git-subrepo child repository .

So, we got the latest version of the remote / build-system.git upstream-repository master branch and placed it in the build-system subdirectory of the main project platform .

To deliver these changes to the remote / platform.git upstream repository , the author needs to run the git push command :

bash-4.4$
bash-4.4$ git push
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 4 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 849 bytes | 849.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To /home/prog/0.4.1/remote/platform.git
   <font color="#8b0000">b5e76a7..6b831e4</font>  master -> master
bash-4.4$

More information on git subrepo commands can be obtained from the ReadMe.pod file or on the command line.

bash-4.4$ git subrepo help <command>

eg:

bash-4.4$ git subrepo helpclone

Now consider everything that happens on the part of the user.

Getting the code by users

At the moment, when the user has not yet received updates to the platform.git upstream repository , his copy contains one README file

bash-4.4$
bash-4.4$ cd user/platform/
bash-4.4$ ls
README
bash-4.4$

containing one line:

bash-4.4$
bash-4.4$ cat README 
[master] platform 1.0.0
bash-4.4$

After removing changes to the upstream repository

bash-4.4$
bash-4.4$ git pull
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From /home/prog/0.4.1/remote/platform
   b5e76a7..6b831e4  master     -> origin/master
Updating <font color="#8b0000">b5e76a7..6b831e4</font>
Fast-forward
 build-system/.gitrepo/config | 12 ++++++++++++
 build-system/README          |  3 +++
 2 files changed, 15 insertions(+)
 create mode 100644 build-system/.gitrepo/config
 create mode 100644 build-system/README
bash-4.4$

the user will have at his disposal the code of the build-system subproject of exactly that revision, which was defined by the author of the project. The user can at any time refine the current revision using the config command :

bash-4.4$
bash-4.4$ git subrepo config build-system/ commit
Subrepo 'build-system' option 'commit' has value 'b2f507918f2821cb7dd90c33223ed5cc3c9922a2'.
bash-4.4$

It is noteworthy that the user does not need to make additional settings and he can rely on the fact that the author of the project delivered to him exactly that build-system revision , which is necessary for the current version of the platform to work .

This is exactly what the author of the project wanted.

Delivery of changes to the upstream project

Suppose now that our user is a member of the project and is allowed to deliver changes not only to the remote / platform.git upstream repository , but also to the upstream repository of the remote / build-system.git subproject .

Then, if the user makes a change:

bash-4.4$
bash-4.4$ cd build-system/
bash-4.4$ vim README
bash-4.4$ cat README
[master] build-system 1.0.1
bash-4.4$
bash-4.4$ git commit -a -m "update BS version to 1.0.1"
[master d30b001] update BS version to 1.0.1
 1 file changed, 1 insertion(+), 1 deletion(-)
bash-4.4$
bash-4.4$ cd ..
bash-4.4$ git log
commit d30b001286b08708f5c30c1f5346a90e4339f969 (HEAD -> master)
Author: user <___@_____>
Date:   Tue Oct 30 10:49:32 2018 +0300
    update BS version to 1.0.1
 . . .
bash-4.4$ 

he can put them in the upstream repository as follows:

bash-4.4$
bash-4.4$ git subrepo push build-system/
Subrepo 'build-system' pushed to '../../remote/build-system.git' (master).
bash-4.4$

bash-4.4$
bash-4.4$ git status
On branch master
Your branch is ahead of 'origin/master' by 2 commits.
  (use "git push" to publish your local commits)
nothing to commit, working tree clean
bash-4.4$
bash-4.4$ git push
Enumerating objects: 14, done.
Counting objects: 100% (14/14), done.
Delta compression using up to 4 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), 992 bytes | 992.00 KiB/s, done.
Total 9 (delta 1), reused 0 (delta 0)
To /home/prog/0.4.1/remote/platform.git
   d00be9f..deccb66  master -> master
bash-4.4$

Direct work with upstream repository

Now consider what happened in the remote / build-system.git upstream repository .

bash-4.4$
bash-4.4$ cd owner/build-system/
bash-4.4$
bash-4.4$ git pull
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /home/prog/0.4.1/remote/build-system
   b2f5079..d229920  master     -> origin/master
Updating b2f5079..d229920
Fast-forward
 README | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bash-4.4$
bash-4.4$ git log
commit d229920c7de34405bc7b8d47f36d420987687908 (HEAD -> master, origin/master)
Author: user <___@_____>
Date:   Tue Oct 30 10:49:32 2018 +0300
    update BS version to 1.0.1
commit b2f507918f2821cb7dd90c33223ed5cc3c9922a2
Author: user <___@_____>
Date:   Tue Oct 30 10:05:30 2018 +0300
    init build-system master 1.0.0
bash-4.4$

That is, the author of the project received the changes made by the project participant.

Of course, the author can make changes directly to the upstream-repository of the build-system project :

bash-4.4$
bash-4.4$ cd owner/build-system/
bash-4.4$
bash-4.4$ vim README
bash-4.4$ cat README
[master] build-system 1.0.2
bash-4.4$ git commit -a -m "update build-system version to 1.0.2"
[master 8255f59] update build-system version to 1.0.2
 1 file changed, 1 insertion(+), 1 deletion(-)
bash-4.4$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 281 bytes | 281.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To /home/prog/0.4.1/remote/build-system.git
   d229920..8255f59  master -> master
bash-4.4$

And all participants, as well as users of the main project, will be able to receive these changes using the git subrepo pull command.

bash-4.4$
bash-4.4$ cd owner/platform/
bash-4.4$
bash-4.4$ git subrepo pull build-system/
Subrepo 'build-system' pulled from '../../remote/build-system.git' (master).
bash-4.4$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
nothing to commit, working tree clean
bash-4.4$ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 4 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 670 bytes | 670.00 KiB/s, done.
Total 6 (delta 1), reused 0 (delta 0)
To /home/prog/0.4.1/remote/platform.git
   6b831e4..b6f4a7b  master -> master
bash-4.4$

findings

If the developer does not need to keep the history of the subprojects in the main repository and, when delivering the code, he operates with branches rather than fixed tags, then git-subrepo is quite suitable for the organization of daily work.

Conventionally, among the disadvantages of git-subrepo is the fact that the operation git subrepo clone is possible only with respect to the subproject branches. In other words, the user cannot connect a subproject referring to his fixed tag or a specific revision, that is, commands like

bash-4.4$ git subrepo clone ../../remote/build-system.git build-system -t 1.0.1
bash-4.4$ git subrepo clone ../../remote/build-system.git build-system 7f5d1113eb0bc6

not valid.

LITERATURE:

• Git - Subtree Merging
• Mastering Git subtrees
• Mastering Git submodules
• Git subrepo
• Branch model and git module management for a large project