Squid-based failover proxy in Windows domain

UPD: the differences in memory consumption between my configuration and the one used by the authors of negative comments are caused by the fact that with Squid, in addition to the standard script wbinfo_group.pl, which determines the membership of a domain user in a domain group, a script is used that determines user login in order for squid to be able to provide the user with rights different from those granted to his group.

Once, on a rainy gray evening, I had a need to implement a proxy server, but not a simple one, but one that would have the following functionality:

• provision / restriction of access depending on the membership of the account in a specific Active Directory group;
• granting / restricting access depending on the rights granted to the account (for example, so that the whole group can be allowed to use search engines, and one very cunning user of this group should be disconnected from a strictly defined search engine);
• providing access without any restrictions for the "VIP" MAC addresses;
• providing access to a minimum set of resources to all users (including non-domain ones);
• the number of movements that a user must perform in order to work with a proxy server should be minimized;
• proxy server administration is performed through the web interface;
• The total cost of ownership of this proxy server should be minimal.

Naturally, my eyes fell on Squid with ntlmssp authentication. There was a utility for configuring Squid through the web interface - SAMS. But somehow I did not grow together with him: either the php version had to be rolled back from 5.3 to 5.2, then it did not cling to the domain, then it did not see users. By no means do I want to say that this is a bad product, my friend deployed and tuned it (so the source is reliable), just the location of the stars did not allow me to become a SAMS adherent.
Pretty tormented with Google, various products and solutions, I thought: “the essence of the configuration is to generate the squid.conf file, I somehow wrote to php, and why not implement the web interface myself?” No sooner said than done. Soon, bags ran in, users scolded, brought me service notes on opening access to Internet resources.
It would seem, what else is needed from life? But no, after some time, several shortcomings in the solution infrastructure crawled to the surface.

1. In the event of a domain controller’s malfunction, loss of connection with it, the backup communication channel with the outside world, which is the Internet (and the main one is corporate mail), ceased to function, because users could not log in.
2. The user logs in to the proxy using the ntlmssp protocol using ntlm_auth helper (oil), one instance of which occupies about 60 megabytes in RAM.
   
   As you can see, the winbindd process, which eats up the same amount of memory, is also involved in authorization. Total, 120 megabytes per request. It was experimentally found that the auth_param ntlm children parameter , indicating the number of threads for authorization, should exceed the number of users by about 10%. With auth_param ntlm children = 200, 600 megabytes of memory are used on the proxy server.
   
   It was experimentally found that if at the same time 150 users with mail agents and other naughty people work on the proxy, then the server can behave unpredictably, for example, freeze, lose connection with a domain controller, etc. Again, I note that perhaps this is my curvature, but it was she who prompted me to comprehend the topic of this opus.
3. CPU resources are practically not involved, while there is a reservation of a huge amount of RAM.

Accordingly, new Wishlist appeared.

1. Reduce the load on the proxy server and get the opportunity to expand the composition of proxy servers with a flick of the wrist.
2. If there are problems with authorization, switch to the non-domain configuration, and if problems disappear, switch back to the domain configuration. Along the way, you can notify the administrator of emerging and disappearing problems.

Reduced proxy server load + scalability

The starting point to the solution is the ability to load balance in iptables.

# iptables -t nat -A PREROUTING --dst $GW_IP -p tcp --dport 3128 -m state --state NEW -m statistic --mode nth --every 3 --packet 0 -j DNAT --to-destination $PROXY1
#iptables -t nat -A PREROUTING --dst $GW_IP -p tcp --dport 3128 -m state --state NEW -m statistic --mode nth --every 3 --packet 1 -j DNAT --to-destination $PROXY2
#iptables -t nat -A PREROUTING --dst $GW_IP -p tcp --dport 3128 -m state --state NEW -m statistic --mode nth --every 3 --packet 2 -j DNAT --to-destination $PROXY3

The code for this listing allows you to evenly distribute the load between the three proxy servers. Now we need three proxies that will be hiding behind one IP address. My solution is presented in the figure below.

Designations:

• LAN 192.168.0.0/24 - local area network;
• PROXY - the proxy server that the user sees;
• LAN 192.168.1.0/24 - local area network for proxy servers;
• PROXY1..PROXYN - Linux-based virtual machines on which Squid is deployed, and Samba, integrated with Active Directory;
• PROXY_CONFIG - a virtual machine on which the proxy server management interface is deployed.

In this case, the “proxy server” that the user sees is actually a hypervisor of virtual machines on which the required number of proxy servers and the interface for managing them are deployed.

With this structure, you can easily provide:

• packet passing from VIP VIP MAC addresses bypassing proxies;
• scalability of proxies;
• fast recovery after a failure;
• field for experimenting with Squid’s configuration (for this it’s enough to deploy a new virtual machine and test everything on it);
• quick removal of the problem virtual machine from the proxy pool;
• more or less optimal use of hardware resources.

Squid's configuration is the same for all proxies, so the squid.conf file is stored on the PROXY_CONF computer in an nfs ball.

To implement this scheme, the following hardware and software components were used:

• Computer CPU Core i3 / RAM 4 Gb / HDD 500 Gb;
• OS Ubuntu-Server 11.04;
• VirtualBox + phpvirtualbox.

To automatically start and stop virtual machines when turning on and off the computer, vmstart and vmstop scripts were written that turn all virtual machines on and off. I will not give their contents, since this is simply using the VBoxHeadless and VBoxManage commands in conjunction with ping to check the status of the machine.
Thus, several problems have been solved:

• reducing the load on the proxy server - packets with a request for a new connection are scattered to N virtual proxy servers in order of priority, one proxy server with 650 MB RAM is quite comfortable with auth_param ntlm children = 200, with three proxies the number of authorization flows already equal to 600;
• scalability - if necessary, unload the server, you can simply copy the hard drive of an existing proxy and in 5-10 minutes create a new machine in full combat readiness;
• rational use of hardware resources - two previous tasks have been solved due to the increased load on the processor;
• fault tolerance - it is enough to have copies of the PROXY_CONFIG and PROXY1 machines and after the fall you can restore the whole scheme by slowly drinking coffee, and for complete peace of mind you can periodically backup the hypervisor’s hard drive, since Linux will not drop into the blue screens when you try to insert the hard drive into another system unit.

Automatic switching between domain and non-domain configurations

Time has shown that in case of failures, it would be nice if the proxy server could determine whether everything is normal with the connection to the domain controller, and otherwise apply a controller-independent configuration, and when restoring the controller, return to the functionality for which all this is conceived.
Talk less, work more!

#!/bin/bash
# Проверяем, локальный или сетевой конфиг сейчас в работе
/bin/ls -la /etc/squid/squid.conf | /bin/grep 'local'
LOCAL=$?
echo Local configuration - $LOCAL
# Проверяем, авторизует ли контроллер домена
/usr/bin/ntlm_auth --username=ad_user --password=ad_pass 2>/dev/null 1>/dev/null
if [ $? != 0 ]; then
       # Если контроллер не авторизует
       echo No connection with domain
       if [ $LOCAL -eq 1 ]; then
		# Если при этом включена доменная конфигурация
		# то меняем ее на локальную
		echo Reconfigure to local
        	/bin/rm /etc/squid/squid.conf
        	/bin/ln -s /etc/squid/squid.conf.local /etc/squid/squid.conf
        	/usr/sbin/squid -k reconfigure
		# уведомляем админа почтой о том, что случилось
                /usr/bin/php5 /home/squiduser/mail/mail.php 'from' 'Proxy XX - Gone to non-domain mode' '=('
       fi
       # Пробуем автризоваться в домене 
       /etc/init.d/joindomain
else
       # Если связь с доменом в порядке
       echo OK connection to domain
       if [ $LOCAL -eq 0 ]; then
	       # Если при этом включена локальная конфигурация
	       # то меняем ее на сетевую
               echo Reconfigure to domain
               /bin/rm /etc/squid/squid.conf
               /bin/ln -s /var/www/squid/squid.conf /etc/squid/squid.conf
               /usr/sbin/squid -k reconfigure
	       # уведомляем админа о том, что все вернулось на круги своя
               /usr/bin/php5 /home/squiduser/mail/mail.php 'from' 'Gone to domain mode' '=)'
       fi
fi

Notes:

• the file /etc/squid/squid.conf is a symbolic link files with local or domain configuration (squid.conf and squid.conf.local);
• for notification, the mail.php script is used, which sends a message to corporate mail;
• /etc/init.d/joindomain - script to enter the computer into the domain.

The logic of the script is shown in the figure below.

In order not to synchronize the script in the future on all proxy servers, drop it into the network share on the PROXY_CONFIG machine, mount it on all proxies and add the script to cron every minute. Thus, if the connection with the domain is lost, the downtime will be reduced to one minute, that is, the angry accountants will not have time to call or have time, but they will rejoice almost immediately.
Well, that's all. Everything works, there are no problems. I would like to immediately answer frequently asked questions about the configuration described.

Question 1

Isn’t it easier to deploy one real proxy server on a good hardware platform than bother with virtual machines?

Answer

expand - yes, easier to manage - no. Of course, if the organization is small and you are responsible for both proxies and controllers, then there is no point in fencing such a garden. In the case of the presence of a distributed network of branches, branches, when the duties and responsibilities of administrators are divided, then such a system, "waiting" for the best infrastructure is the best that has come to mind. If earlier, when a sharp connection of two new branches, I had to go into the configuration and increase the number of authorization flows, wading through the terrible brakes of the OS deployed on the hardware, then if the CD fails, disconnect the network cable from the proxy, because without the CD everything hangs even locally, spend a few dozens of minutes to wait for sagging, now I have already forgotten what the system unit looks like on which all this is deployed, I only remember that it was black.

Question 2

Why check the status of the connection to the domain, isn’t it easier to troubleshoot the controller?

Answer

see answer to question 1 regarding separation of powers.