April 6, 2010 at 04:08Deploy Mercurial Repositories via FastCGI Using Nginx on FreeBSD
I have succumbed to the fashion and exciting perspectives of DVCS recently. This pushed me off the beaten track of Subversion + Trac and forced me to look for new schemes for storing source codes in different companies. And provide them convenient access to developers, customers and other interested individuals.
It so happened that I specialize in FreeBSD and am not so good at Linux. And I prefer where you can use Nginx instead of Apache httpd. Therefore, I decided to create for myself a unified architecture that will allow you to store an unlimited number of repositories and to differentiate access for them to different groups of people on this platform.
Of course, Bitbucket is our everything. But any developer has closed projects that I would not want to upload to public. You can, of course, pay $ 50 per month for the opportunity to host 25 projects on bitbucket. I personally think that it is better to spend this money on a dedicated server and raise as many projects as you like. It will not be so convenient, but its own and with the possibility of tuning, backup and other goodies.
You must have a repository of Mercurial repositories with the ability to access them via the web. Access rights must be specified read / write for specific users.
Details on how to deploy multiple repositories can be found on the official page . To do this, use the cgi hgwebdir.cgi script. We, of course, will use its FastCGI version of hgwebdir.fcgi.
Mercurial does not include a self-contained FastCGI server that can be run on the desired port. It is assumed that hgwebdir.fcgi will be started by apache httpd or lighttpd. To solve this problem, we will use the fastcgi wrapper spawn-fcgi.
So, first of all, install the following ports
Copy the default script to our settings directory. Edit our hgwebdir.fcgi. It looks like this for me. Next, in the configuration file /usr/home/www/cgi/hgweb.config, we specify the paths to the collections and the address setting. The baseurl parameter says that all links that hgwebdir will generate will contain / hg at the beginning of the path ( remember, we have the address where the repositories dev.example.com/hg/ should be available)
In the FreeBSD configuration file /etc/rc.conf, add the following lines. You can specify the script itself as spawn_fcgi_app, and not the path to the python interpreter. But then the service will not stop correctly, because system scripts will look for a process called hgwebdir.fcgi to stop, but in reality the process is called python, because hgwebdir.fcgi is a program that runs using the interpreter. Now you can run the daemon
In the configuration file /usr/local/etc/nginx/nginx.conf, create a new server section with the following contents
At this point, be extremely careful. Nginx by default does not set the variables that are necessary for correct operation with mercurial. Therefore, I will try to explain the meaning of each line. They were given to me by studying the sources of mercurial :)
set_real_ip_from , real_ip_header - as I mentioned above, external https access is served by a separate nginx, standing on a server that looks to the world. So that on our local server the real addresses of the clients get to the logs, and not the address of this external server that proxies requests, and these directives were added.
if block and PATH_INFO variable- to determine the repository that is being accessed, hgwebdir uses the PATH_INFO variable. We need to cut off the beginning of the path / hg, because otherwise, hgwebdir will think that you are trying to access the repository under the name "hg". Here you need to be careful and use nginx's $ uri variable rather than $ request_uri to calculate PATH_INFO, as some manuals on the Internet mention. Because $ request_uri also contains $ args in addition to the path. And if you pass it to PATH_INFO, then you will receive 404 Not Found when trying to work with the repository.
client_max_body_size- by default, the maximum length of a request to nginx is 1M. Naturally, if you try to commit changes by more than 1M, or make the initial push of a large repository, the request will fail. Therefore, this size is increased to 100M. I think in most cases this is enough.
include fastcgi_params - we connect other standard variables of type QUERY_STRING, etc.
REMOTE_USER - in this variable hgwebdir expects to get the username.
Well, the last two lines - setting http basic auth authorization.
That's all, now you can fix other parameters not related to hg in nginx.conf (for example, where to put the logs) and start the web server.
Add
Now we are ready to create the first repository. Everything is done from the user www, because it is under him that nginx and hgwebdir work, and he needs write permissions to this directory. Let's make a test repository. We give read rights to user user1 and read / write permissions to user2. We edit test / .hg / hgrc so that it looks like this. Done, you can check access.
In addition to general migration from subversion to mercurial, I now look intently at redmine instead of trac. I liked the idea that users can register and create projects themselves. Unfortunately, I did not find a ready-made solution on how to automatically create an hg repository for the created project. Moreover, with the cloning of access rights as described in this article. If someone knows - I will be grateful for the reference. In any case, I'm going to solve this problem in the near future either by writing a plugin for redmine, or in a more different / correct way.
If you liked this article and would like to see a sequel, leave feedback. I will then share my new knowledge of integrating our new environment with redmine.
It so happened that I specialize in FreeBSD and am not so good at Linux. And I prefer where you can use Nginx instead of Apache httpd. Therefore, I decided to create for myself a unified architecture that will allow you to store an unlimited number of repositories and to differentiate access for them to different groups of people on this platform.
Of course, Bitbucket is our everything. But any developer has closed projects that I would not want to upload to public. You can, of course, pay $ 50 per month for the opportunity to host 25 projects on bitbucket. I personally think that it is better to spend this money on a dedicated server and raise as many projects as you like. It will not be so convenient, but its own and with the possibility of tuning, backup and other goodies.
Formulation of the problem
You must have a repository of Mercurial repositories with the ability to access them via the web. Access rights must be specified read / write for specific users.
Specific points
- Repositories should be stored in the directory / usr / home / www / repos /
- Mercurial settings are stored in / usr / home / www / repos / cgi /
- Hosting takes place on a local server. Therefore, access is only via http. Outside, access via https will be served by another server, also based on nginx, and transferred to the local port on port 80.
- Let's say the server is called dev.example.com. The list of repositories should be available at
http://dev.example.com/hg/
Installing prerequisites
Details on how to deploy multiple repositories can be found on the official page . To do this, use the cgi hgwebdir.cgi script. We, of course, will use its FastCGI version of hgwebdir.fcgi.
Mercurial does not include a self-contained FastCGI server that can be run on the desired port. It is assumed that hgwebdir.fcgi will be started by apache httpd or lighttpd. To solve this problem, we will use the fastcgi wrapper spawn-fcgi.
So, first of all, install the following ports
- www / spawn-fcgi
- devel / mercurial
- www / nginx
- www / py-flup
Customize hgwebdir
Copy the default script to our settings directory. Edit our hgwebdir.fcgi. It looks like this for me. Next, in the configuration file /usr/home/www/cgi/hgweb.config, we specify the paths to the collections and the address setting. The baseurl parameter says that all links that hgwebdir will generate will contain / hg at the beginning of the path ( remember, we have the address where the repositories dev.example.com/hg/ should be available)
sudo -u www mkdir /usr/home/www/cgi/
cp /usr/local/share/mercurial/www/hgwebdir.fcgi /usr/home/www/cgi/
#!/usr/bin/env python
from mercurial import demandimport; demandimport.enable()
from mercurial.hgweb.hgwebdir_mod import hgwebdir
from flup.server.fcgi import WSGIServer
WSGIServer(hgwebdir('/usr/home/www/cgi/hgweb.config')).run()
[collections]
/usr/home/www/repos = /usr/home/www/repos
[web]
baseurl = /hg
Configure spawn-fcgi
In the FreeBSD configuration file /etc/rc.conf, add the following lines. You can specify the script itself as spawn_fcgi_app, and not the path to the python interpreter. But then the service will not stop correctly, because system scripts will look for a process called hgwebdir.fcgi to stop, but in reality the process is called python, because hgwebdir.fcgi is a program that runs using the interpreter. Now you can run the daemon
# Mercurial
spawn_fcgi_enable="YES"
spawn_fcgi_app="/usr/local/bin/python"
spawn_fcgi_app_args="/usr/home/www/cgi/hgwebdir.fcgi"
spawn_fcgi_pidfile="/var/run/hg.pid"
spawn_fcgi_bindsocket="/var/run/hg.socket"
sudo /usr/local/etc/rc.d/spawn-fcgi start
Configure nginx
In the configuration file /usr/local/etc/nginx/nginx.conf, create a new server section with the following contents
server {
listen 80;
server_name dev.example.com;
set_real_ip_from 172.16.224.1;
real_ip_header X-Forwarded-For;
location / {
root / usr / local / www / nginx;
index index.html index.htm;
}
location / hg {
if ($ uri ~ /hg(/.*)$) {
set $ path $ 1;
}
client_max_body_size 100M;
fastcgi_pass unix: /var/run/hg.socket;
include fastcgi_params;
fastcgi_param PATH_INFO $ path;
fastcgi_param REMOTE_USER $ remote_user;
auth_basic "Mercurial repositories";
auth_basic_user_file hg_htpasswd;
}
}
At this point, be extremely careful. Nginx by default does not set the variables that are necessary for correct operation with mercurial. Therefore, I will try to explain the meaning of each line. They were given to me by studying the sources of mercurial :)
set_real_ip_from , real_ip_header - as I mentioned above, external https access is served by a separate nginx, standing on a server that looks to the world. So that on our local server the real addresses of the clients get to the logs, and not the address of this external server that proxies requests, and these directives were added.
if block and PATH_INFO variable- to determine the repository that is being accessed, hgwebdir uses the PATH_INFO variable. We need to cut off the beginning of the path / hg, because otherwise, hgwebdir will think that you are trying to access the repository under the name "hg". Here you need to be careful and use nginx's $ uri variable rather than $ request_uri to calculate PATH_INFO, as some manuals on the Internet mention. Because $ request_uri also contains $ args in addition to the path. And if you pass it to PATH_INFO, then you will receive 404 Not Found when trying to work with the repository.
client_max_body_size- by default, the maximum length of a request to nginx is 1M. Naturally, if you try to commit changes by more than 1M, or make the initial push of a large repository, the request will fail. Therefore, this size is increased to 100M. I think in most cases this is enough.
include fastcgi_params - we connect other standard variables of type QUERY_STRING, etc.
REMOTE_USER - in this variable hgwebdir expects to get the username.
Well, the last two lines - setting http basic auth authorization.
That's all, now you can fix other parameters not related to hg in nginx.conf (for example, where to put the logs) and start the web server.
Add
nginx_enable="YES"to /etc/rc.conf and dosudo /usr/local/etc/rc.d/nginx startRepository Creation Example
Now we are ready to create the first repository. Everything is done from the user www, because it is under him that nginx and hgwebdir work, and he needs write permissions to this directory. Let's make a test repository. We give read rights to user user1 and read / write permissions to user2. We edit test / .hg / hgrc so that it looks like this. Done, you can check access.
# Идем в директорию с репозиториями
cd /usr/home/www/repos
# Создаем новый репозиторий
sudo -u www hg init test
# Добавляем пользователей в nginx
# -c - создать файл
# htpasswd - из пакета www/apache22
sudo htpasswd -b -c /usr/local/etc/nginx/hg_htpasswd user1 pass1
sudo htpasswd -b /usr/local/etc/nginx/hg_htpasswd user2 pass2
[web]
# Нам не нужен ssl, поскольку о нем заботится nginx
push_ssl = False
allow_push = user2
allow_read = user1,user2
Instead of a conclusion
In addition to general migration from subversion to mercurial, I now look intently at redmine instead of trac. I liked the idea that users can register and create projects themselves. Unfortunately, I did not find a ready-made solution on how to automatically create an hg repository for the created project. Moreover, with the cloning of access rights as described in this article. If someone knows - I will be grateful for the reference. In any case, I'm going to solve this problem in the near future either by writing a plugin for redmine, or in a more different / correct way.
If you liked this article and would like to see a sequel, leave feedback. I will then share my new knowledge of integrating our new environment with redmine.