Mercurial + Nginx configuration for managing a large number of repositories

    Under the cat, an example configuration of the mercurial + nginx bundle is described and an automation script for all of the above is given.

    Tasks:

    Creation of repository repository based on Mercurial version control system with the possibility of secure data transfer and separation of access levels.
    Automation of the above actions.

    Decision:

    Proxying the embedded http server (hg serve) with proxy-level sharing.
    To prevent data interception, the transmission is carried out using the HTTPS protocol.
    To minimize resource consumption, Nginx acts as a proxy.
    Access control is done at the location directive level.

    Agreement:

    Linux distribution where everything will happen - Gentoo
    https://hg.expample.com/reponame - link to access the repository
    hg.example.com - domain name of the repository repository server
    reponame - name of the required repository
    / home / repos - root folder for of repositories
    / etc / hg - root folder for configuration files
    This reader has minimal Linux administration skills

    Installing the necessary packages:

    hg ~ # emerge mercurial
    hg ~ # emerge nginx
    We will launch the built-in http server with the command
    hg ~ # /usr/bin/hg serve -d -A /var/log/hg_access.log -p 8080 -a 127.0.0.1 --pid-file /var/run/hgserver.pid --encoding utf8 --webdir-conf /etc/hg/web.config
    Startup directives:
    -d - start the server as a daemon
    -A /var/log/hg_access.log - server access log
    -p 8080 - port number on which the server listens for requests
    -a 127.0.0.1 - ip address on which the server starts
    - pid-file /var/run/hgserver.pid - file with server process identifier
    --encoding utf8 - encoding
    --webdir-conf /etc/hg/web.config - server configuration file
    hg ~ # cat /etc/hg/web.config
    [web] //секция параметров веб сервера
    allow_push = * //разрешаем всем “поднимать” изменения (контроль доступа идет на уровне прокси)
    push_ssl = false //не используем ssl (шифрование идет на уровне прокси)
    [paths] //секция “путей”
    rep1=/home/repos/rep1 //выставляем соответствие: название - расположение для репозитория rep1
    rep2=/home/repos/rep2 //выставляем соответствие: название - расположение для репозитория rep2
    We write the Include directive in the main Nginx configuration file
    hg ~ # cat /etc/nginx/nginx.conf |grep -i include
    include "/etc/hg/nginx.conf";
    Example configuration file /etc/hg/nginx.conf
    hg ~ # cat /etc/hg/nginx.conf
    server
    {
    listen 443;
    server_name hg.example.com;
    client_max_body_size 128M;
    ssl on;
    ssl_certificate /etc/ssl/nginx/nginx.pem;
    ssl_certificate_key /etc/ssl/nginx/nginx.key;
    location /repo1
    {
    proxy_pass http://127.0.0.1:8080;
    auth_basic "Restricted";
    auth_basic_user_file /etc/hg/nginx/repo1.pass;
    access_log /var/log/nginx/repo1.hg.example.com.ssl_access_log main;
    error_log /var/log/nginx/repo1.hg.example.com.ssl_error_log info;
    }
    location /repo2
    {
    proxy_pass http://127.0.0.1:8080;
    auth_basic "Restricted";
    auth_basic_user_file /etc/hg/nginx/repo2.pass;
    access_log /var/log/nginx/repo2.hg.example.com.ssl_access_log main;
    error_log /var/log/nginx/repo2.hg.example.com.ssl_error_log info;
    }
    }
    Consider the section: location The / repo1
    proxy_pass - specify where to send the request in a successful authorization
    auth_basic - using the HTTP Basic the Authentication
    auth_basic_user_file - specify which file is password database
    access_log and the error_log - Directive login to access the repository
    create authorization files for each of the repositories team the htpasswd
    (program htpasswd is included in the apache package , so you have to install it with emerge apache )
    hg ~ # htpasswd -bc /etc/hg/nginx/repo2.pass test2 testpass2
    hg ~ # htpasswd -bc /etc/hg/nginx/repo1.pass test1 testpass1
    -b - use the password specified on the command line
    -c - create a new password database file
    /etc/hg/nginx/repo1.pass - name of the password database file
    test1 - login of the added user
    testpass1 - password of the added user It
    remains to initialize the repositories with the hg init command
    hg ~ # hg init /home/repos/repo1
    hg ~ # hg init /home/repos/repo2
    We start Nginx team
    hg ~ # /etc/init.d/nginx start
    Repositories repo1 and repo2
    are now available at
    https://hg.example.com/repo1 and https://hg.example.com/repo2 respectively.
    With configured separation of access levels.
    This could have ended the article, because for a simple configuration, which is updated every 3-5 months, this is enough. But what to do when the configuration has to be changed often? The right system administrator will immediately begin to look for a way to make his life easier.

    There are at least three options for solving the problem:

    1. Use a ready-made repository management product.
    2. Delegate the honorable duty to his subordinate.
    3. Create a control system yourself.
    Unfortunately, neither the first nor the second option suited me , so I had to overpower my laziness and do it myself.
    The result was a script that completely automates the process of managing the repository server.

    In which it was implemented:

    1. Create (if necessary) and provide access to repositories.
    2. Denying access to inactive repositories. (When deleted from the configuration file, the repository is not physically deleted).
    3. Regenerating user passwords for Nginx authorization.
    4. Distribution of access to repositories at the user level.

    The configuration file syntax allows:

    1. use of empty lines for visual separation of sections
    2. use of the “#” symbol to highlight comments

    The configuration file consists of three sections.

    [users] - секция описывающая пары: логин-пароль.разделителем служит занк “=”
    user1=pass1
    user2=pass2
    user3=pass3
    [repos] - секция описывающая доступные репозитории
    repo1
    repo2
    repo3
    [access] - секция описывающая права доступа к репозиториям, разделителем служит символ “,”
    repo1 = user1 , user2,user3 - к репозиторию имеют доступ user1,user2,user3
    repo2 = user1,user2 - к репозитроию имеют доступ только user1 и user2
    repo3 = user3 - доступ к репозиторию только для пользователя user3
    Script Listing /usr/local/sbin/hgmkrep.sh
    1. #! / bin / bash
    2. tmphtpass = "/ var / tmp / htpass" # define a temporary password database file
    3. repohome = "/ home / repos /" # define the root folder for the repositories
    4. hgservepid = "/ var / run / hgserver.pid" #pid hg serve 
    5. hgaccesslog = "/ var / log / hg_access.log" #access log file for hg server
    6. domain = "exapmple.com" #tld server name 
    7. confdir = "/ etc / hg /" # define the root folder for the configs
    8. confile = $ {confdir} "repo.cfg" # main config file /etc/hg/repo.cfg
    9. webconfig = $ {confdir} "web.config" # config for hg server /etc/hg/web.config
    10. nginxconfig = $ {confdir} "nginx.conf" # config for nginx /etc/hg/nginx.conf
    11. nginxauthdir = $ {confdir} "nginx /" # password database folder for accessing repositories
    12. [-s $ {confile}] || echo "where is config file?" # check the presence of the main config
    13. [-s $ {confile}] || exit 0 # upset if there is no main config
    14. # parse the [repos] section for repositories
    15. repos = `cat $ {confile} | sed '/ ^ $ / d' | sed '/ ^ # / d' | sed 's / \ // g' | awk '/ \ [repos \] / {
    16. is_repos = 1;
    17. while (is_repos == 1)
    18.     {if (getline <= 0 || index ($ 0, "[") == 1)
    19.         {is_repos = 0;}
    20.     else
    21.         {print $ 0;}}} ''
    22. # Check the presence of folders with repositories and, if necessary, create
    23. for i in $ {repos}
    24. do; [-d $ {repohome} $ {i}] || / usr / bin / hg init $ {repohome} $ {i}; done
    25. # generate a config for hg server
    26. echo "[web]
    27. allow_push = *
    28. push_ssl = false
    29. [paths] "> $ {webconfig}
    30. # allow access only to active repositories
    31. for i in $ {repos}
    32. do; echo $ {i} = $ {repohome} $ {i} >> $ {webconfig}; done
    33. # reboot hg serve
    34. [-a $ {hgservepid}] && / bin / kill `/ bin / cat $ {hgservepid}` && rm $ {hgservepid}
    35. / usr / bin / hg serve -d -A $ {hgaccesslog} -p 8080 -a 127.0.0.1 --pid-file $ {hgservepid} --encoding utf8 --webdir-conf $ {webconfig}
    36. # create a config for nginx
    37. echo "server 
    38.     {
    39.     listen 443;
    40.     server_name hg. "$ {domain}";
    41.     client_max_body_size 128M;
    42.     ssl on;
    43.     ssl_certificate /etc/ssl/nginx/nginx.pem;
    44.     ssl_certificate_key /etc/ssl/nginx/nginx.key; "> $ {nginxconfig}
    45. # create lacation for active repositories
    46.     for i in $ {repos}
    47.         do
    48.     echo "location /" $ {i} "
    49.         {
    50.         proxy_pass http://127.0.0.1:8080;
    51.         auth_basic \ "Restricted \";
    52.         auth_basic_user_file "$ {nginxauthdir} $ {i}". pass;
    53.         access_log /var/log/nginx/"${i►".hg."${domainasket".ssl_access_log main;
    54.         error_log /var/log/nginx/"${iasket".hg."${domainasket".ssl_error_log info;
    55.         } ">> $ {nginxconfig}
    56.         done
    57. echo "}" >> $ {nginxconfig}
    58. # create (reset just in case) a temporary password database
    59. cat / dev / null> $ {tmphtpass}
    60. # parse the main config section [users]
    61. # generate passwords for all active users
    62. cat $ {confile} | sed '/ ^ $ / d' | sed '/ ^ # / d' | sed 's / \ // g' | awk -v passfile = $ tmphtpass' / \ [users \] / {
    63. is_users = 1;
    64. while (is_users == 1)
    65.     {if (getline <= 0 || index ($ 0, "[") == 1)
    66.         {is_users = 0;}
    67.     else
    68.         {split ($ 0, userpass, "="); system ("htpasswd -b" passfile "" userpass [1] "" userpass [2]);}}} ''
    69. # parse the [access] section of the main config
    70. # and get a list of privileges of the form repo = user1, user2
    71. access = `cat $ {confile} | sed '/ ^ $ / d' | sed '/ ^ # / d' | sed 's / \ // g' | awk '/ \ [access \] / {
    72. is_access = 1; while (is_access == 1)
    73.     {if (getline <= 0 || index ($ 0, "[") == 1)
    74.         {is_access = 0;}
    75.     else
    76.         {print $ 0;}}} ''
    77. # check if there is a folder for storing password databases
    78. [-d $ {nginxauthdir}] || mkdir -p $ {nginxauthdir}
    79. # delete old password database files
    80. find $ {nginxauthdir} -type f -name * .pass -delete
    81. # for each repository from the [access] section, we generate a personal password database
    82. for i in $ {access}
    83. do; echo $ {i} | sed 's /, / \ | / g' | awk -v tmphtpass = $ {tmphtpass} -v nginxauthdir = $ {nginxauthdir} \
    84. 'BEGIN {FS = "="} {system ("cat" tmphtpass "| egrep \" "$ 2" \ ">" nginxauthdir "" $ 1 ".pass")}' done
    85. # restart nginx
    86. /etc/init.d/nginx restart
    87. # delete the temporary password database file
    88. rm $ {tmphtpass}
    I also post the link to a working script and an example config : hgmkrep.tar.gz

    Existing disadvantages:

    1. Lack of differentiation for reading and writing, each user who is allowed to the emperor’s body has repository rights rw by default
    2. The groups are not implemented to restrict access
    3. The configuration is not rolled back if the config syntax is incorrect (the system will fall into collapse)
    4. When increasing the number of users and repositories the config abundantly loses its visibility

    Also popular now: