Mercurial + Nginx configuration for managing a large number of repositories
Under the cat, an example configuration of the mercurial + nginx bundle is described and an automation script for all of the above is given.
Automation of the above actions.
To prevent data interception, the transmission is carried out using the HTTPS protocol.
To minimize resource consumption, Nginx acts as a proxy.
Access control is done at the location directive level.
https://hg.expample.com/reponame - link to access the repository
hg.example.com - domain name of the repository repository server
reponame - name of the required repository
/ home / repos - root folder for of repositories
/ etc / hg - root folder for configuration files
This reader has minimal Linux administration skills
-d - start the server as a daemon
-A /var/log/hg_access.log - server access log
-p 8080 - port number on which the server listens for requests
-a 127.0.0.1 - ip address on which the server starts
- pid-file /var/run/hgserver.pid - file with server process identifier
--encoding utf8 - encoding
--webdir-conf /etc/hg/web.config - server configuration file
proxy_pass - specify where to send the request in a successful authorization
auth_basic - using the HTTP Basic the Authentication
auth_basic_user_file - specify which file is password database
access_log and the error_log - Directive login to access the repository
create authorization files for each of the repositories team the htpasswd
(program htpasswd is included in the apache package , so you have to install it with emerge apache )
-c - create a new password database file
/etc/hg/nginx/repo1.pass - name of the password database file
test1 - login of the added user
testpass1 - password of the added user It
remains to initialize the repositories with the hg init command
are now available at
https://hg.example.com/repo1 and https://hg.example.com/repo2 respectively.
With configured separation of access levels.
This could have ended the article, because for a simple configuration, which is updated every 3-5 months, this is enough. But what to do when the configuration has to be changed often? The right system administrator will immediately begin to look for a way to make his life easier.
2. Delegate the honorable duty to his subordinate.
3. Create a control system yourself.
Unfortunately, neither the first nor the second option suited me, so I had to overpower my laziness and do it myself.
The result was a script that completely automates the process of managing the repository server.
2. Denying access to inactive repositories. (When deleted from the configuration file, the repository is not physically deleted).
3. Regenerating user passwords for Nginx authorization.
4. Distribution of access to repositories at the user level.
2. use of the “#” symbol to highlight comments
the emperor’s body has repository rights rw by default
2. The groups are not implemented to restrict access
3. The configuration is not rolled back if the config syntax is incorrect (the system will fall into collapse)
4. When increasing the number of users and repositories the config abundantly loses its visibility
Tasks:
Creation of repository repository based on Mercurial version control system with the possibility of secure data transfer and separation of access levels.Automation of the above actions.
Decision:
Proxying the embedded http server (hg serve) with proxy-level sharing.To prevent data interception, the transmission is carried out using the HTTPS protocol.
To minimize resource consumption, Nginx acts as a proxy.
Access control is done at the location directive level.
Agreement:
Linux distribution where everything will happen - Gentoohttps://hg.expample.com/reponame - link to access the repository
hg.example.com - domain name of the repository repository server
reponame - name of the required repository
/ home / repos - root folder for of repositories
/ etc / hg - root folder for configuration files
This reader has minimal Linux administration skills
Installing the necessary packages:
hg ~ # emerge mercurial
hg ~ # emerge nginx
We will launch the built-in http server with the commandhg ~ # /usr/bin/hg serve -d -A /var/log/hg_access.log -p 8080 -a 127.0.0.1 --pid-file /var/run/hgserver.pid --encoding utf8 --webdir-conf /etc/hg/web.config
Startup directives: -d - start the server as a daemon
-A /var/log/hg_access.log - server access log
-p 8080 - port number on which the server listens for requests
-a 127.0.0.1 - ip address on which the server starts
- pid-file /var/run/hgserver.pid - file with server process identifier
--encoding utf8 - encoding
--webdir-conf /etc/hg/web.config - server configuration file
hg ~ # cat /etc/hg/web.config
[web] //секция параметров веб сервера
allow_push = * //разрешаем всем “поднимать” изменения (контроль доступа идет на уровне прокси)
push_ssl = false //не используем ssl (шифрование идет на уровне прокси)
[paths] //секция “путей”
rep1=/home/repos/rep1 //выставляем соответствие: название - расположение для репозитория rep1
rep2=/home/repos/rep2 //выставляем соответствие: название - расположение для репозитория rep2
We write the Include directive in the main Nginx configuration filehg ~ # cat /etc/nginx/nginx.conf |grep -i include
include "/etc/hg/nginx.conf";
Example configuration file /etc/hg/nginx.confhg ~ # cat /etc/hg/nginx.conf
server
{
listen 443;
server_name hg.example.com;
client_max_body_size 128M;
ssl on;
ssl_certificate /etc/ssl/nginx/nginx.pem;
ssl_certificate_key /etc/ssl/nginx/nginx.key;
location /repo1
{
proxy_pass http://127.0.0.1:8080;
auth_basic "Restricted";
auth_basic_user_file /etc/hg/nginx/repo1.pass;
access_log /var/log/nginx/repo1.hg.example.com.ssl_access_log main;
error_log /var/log/nginx/repo1.hg.example.com.ssl_error_log info;
}
location /repo2
{
proxy_pass http://127.0.0.1:8080;
auth_basic "Restricted";
auth_basic_user_file /etc/hg/nginx/repo2.pass;
access_log /var/log/nginx/repo2.hg.example.com.ssl_access_log main;
error_log /var/log/nginx/repo2.hg.example.com.ssl_error_log info;
}
}
Consider the section: location The / repo1 proxy_pass - specify where to send the request in a successful authorization
auth_basic - using the HTTP Basic the Authentication
auth_basic_user_file - specify which file is password database
access_log and the error_log - Directive login to access the repository
create authorization files for each of the repositories team the htpasswd
(program htpasswd is included in the apache package , so you have to install it with emerge apache )
hg ~ # htpasswd -bc /etc/hg/nginx/repo2.pass test2 testpass2
hg ~ # htpasswd -bc /etc/hg/nginx/repo1.pass test1 testpass1
-b - use the password specified on the command line -c - create a new password database file
/etc/hg/nginx/repo1.pass - name of the password database file
test1 - login of the added user
testpass1 - password of the added user It
remains to initialize the repositories with the hg init command
hg ~ # hg init /home/repos/repo1
hg ~ # hg init /home/repos/repo2
We start Nginx teamhg ~ # /etc/init.d/nginx start
Repositories repo1 and repo2are now available at
https://hg.example.com/repo1 and https://hg.example.com/repo2 respectively.
With configured separation of access levels.
This could have ended the article, because for a simple configuration, which is updated every 3-5 months, this is enough. But what to do when the configuration has to be changed often? The right system administrator will immediately begin to look for a way to make his life easier.
There are at least three options for solving the problem:
1. Use a ready-made repository management product.2. Delegate the honorable duty to his subordinate.
3. Create a control system yourself.
Unfortunately, neither the first nor the second option suited me
The result was a script that completely automates the process of managing the repository server.
In which it was implemented:
1. Create (if necessary) and provide access to repositories.2. Denying access to inactive repositories. (When deleted from the configuration file, the repository is not physically deleted).
3. Regenerating user passwords for Nginx authorization.
4. Distribution of access to repositories at the user level.
The configuration file syntax allows:
1. use of empty lines for visual separation of sections2. use of the “#” symbol to highlight comments
The configuration file consists of three sections.
[users] - секция описывающая пары: логин-пароль.разделителем служит занк “=”
user1=pass1
user2=pass2
user3=pass3
[repos] - секция описывающая доступные репозитории
repo1
repo2
repo3
[access] - секция описывающая права доступа к репозиториям, разделителем служит символ “,”
repo1 = user1 , user2,user3 - к репозиторию имеют доступ user1,user2,user3
repo2 = user1,user2 - к репозитроию имеют доступ только user1 и user2
repo3 = user3 - доступ к репозиторию только для пользователя user3
Script Listing /usr/local/sbin/hgmkrep.shI also post the link to a working script and an example config : hgmkrep.tar.gz
- #! / bin / bash
- tmphtpass = "/ var / tmp / htpass" # define a temporary password database file
- repohome = "/ home / repos /" # define the root folder for the repositories
- hgservepid = "/ var / run / hgserver.pid" #pid hg serve
- hgaccesslog = "/ var / log / hg_access.log" #access log file for hg server
- domain = "exapmple.com" #tld server name
- confdir = "/ etc / hg /" # define the root folder for the configs
- confile = $ {confdir} "repo.cfg" # main config file /etc/hg/repo.cfg
- webconfig = $ {confdir} "web.config" # config for hg server /etc/hg/web.config
- nginxconfig = $ {confdir} "nginx.conf" # config for nginx /etc/hg/nginx.conf
- nginxauthdir = $ {confdir} "nginx /" # password database folder for accessing repositories
- [-s $ {confile}] || echo "where is config file?" # check the presence of the main config
- [-s $ {confile}] || exit 0 # upset if there is no main config
- # parse the [repos] section for repositories
- repos = `cat $ {confile} | sed '/ ^ $ / d' | sed '/ ^ # / d' | sed 's / \ // g' | awk '/ \ [repos \] / {
- is_repos = 1;
- while (is_repos == 1)
- {if (getline <= 0 || index ($ 0, "[") == 1)
- {is_repos = 0;}
- else
- {print $ 0;}}} ''
- # Check the presence of folders with repositories and, if necessary, create
- for i in $ {repos}
- do; [-d $ {repohome} $ {i}] || / usr / bin / hg init $ {repohome} $ {i}; done
- # generate a config for hg server
- echo "[web]
- allow_push = *
- push_ssl = false
- [paths] "> $ {webconfig}
- # allow access only to active repositories
- for i in $ {repos}
- do; echo $ {i} = $ {repohome} $ {i} >> $ {webconfig}; done
- # reboot hg serve
- [-a $ {hgservepid}] && / bin / kill `/ bin / cat $ {hgservepid}` && rm $ {hgservepid}
- / usr / bin / hg serve -d -A $ {hgaccesslog} -p 8080 -a 127.0.0.1 --pid-file $ {hgservepid} --encoding utf8 --webdir-conf $ {webconfig}
- # create a config for nginx
- echo "server
- {
- listen 443;
- server_name hg. "$ {domain}";
- client_max_body_size 128M;
- ssl on;
- ssl_certificate /etc/ssl/nginx/nginx.pem;
- ssl_certificate_key /etc/ssl/nginx/nginx.key; "> $ {nginxconfig}
- # create lacation for active repositories
- for i in $ {repos}
- do
- echo "location /" $ {i} "
- {
- proxy_pass http://127.0.0.1:8080;
- auth_basic \ "Restricted \";
- auth_basic_user_file "$ {nginxauthdir} $ {i}". pass;
- access_log /var/log/nginx/"${i►".hg."${domainasket".ssl_access_log main;
- error_log /var/log/nginx/"${iasket".hg."${domainasket".ssl_error_log info;
- } ">> $ {nginxconfig}
- done
- echo "}" >> $ {nginxconfig}
- # create (reset just in case) a temporary password database
- cat / dev / null> $ {tmphtpass}
- # parse the main config section [users]
- # generate passwords for all active users
- cat $ {confile} | sed '/ ^ $ / d' | sed '/ ^ # / d' | sed 's / \ // g' | awk -v passfile = $ tmphtpass' / \ [users \] / {
- is_users = 1;
- while (is_users == 1)
- {if (getline <= 0 || index ($ 0, "[") == 1)
- {is_users = 0;}
- else
- {split ($ 0, userpass, "="); system ("htpasswd -b" passfile "" userpass [1] "" userpass [2]);}}} ''
- # parse the [access] section of the main config
- # and get a list of privileges of the form repo = user1, user2
- access = `cat $ {confile} | sed '/ ^ $ / d' | sed '/ ^ # / d' | sed 's / \ // g' | awk '/ \ [access \] / {
- is_access = 1; while (is_access == 1)
- {if (getline <= 0 || index ($ 0, "[") == 1)
- {is_access = 0;}
- else
- {print $ 0;}}} ''
- # check if there is a folder for storing password databases
- [-d $ {nginxauthdir}] || mkdir -p $ {nginxauthdir}
- # delete old password database files
- find $ {nginxauthdir} -type f -name * .pass -delete
- # for each repository from the [access] section, we generate a personal password database
- for i in $ {access}
- do; echo $ {i} | sed 's /, / \ | / g' | awk -v tmphtpass = $ {tmphtpass} -v nginxauthdir = $ {nginxauthdir} \
- 'BEGIN {FS = "="} {system ("cat" tmphtpass "| egrep \" "$ 2" \ ">" nginxauthdir "" $ 1 ".pass")}' done
- # restart nginx
- /etc/init.d/nginx restart
- # delete the temporary password database file
- rm $ {tmphtpass}
Existing disadvantages:
1. Lack of differentiation for reading and writing, each user who is allowed to2. The groups are not implemented to restrict access
3. The configuration is not rolled back if the config syntax is incorrect (the system will fall into collapse)
4. When increasing the number of users and repositories the config abundantly loses its visibility