September 28, 2015 at 13:32Transparent Squid with filtering HTTPS resources without certificate spoofing (x86)
- From the sandbox
- Tutorial
It's no secret that in large offices the topic of Internet filtering is quite relevant. A lot of software and hardware solutions cope with this task. But at present, all those sites that we cut earlier work on the HTTPS protocol, i.e. port 443. As you know, this protocol cannot be traced, listened to, etc. And any caching filtering proxy server, redirector, etc. only filters HTTP, i.e. port 80. How to cut Vkontakte, Odnoklassniki, iphide.info and many other similar sites? How to block access to personal mail in the organization, if its use is prohibited by orders in the organization? Yes, you can filter by IP addresses, but they often change, and many resources have several IP addresses. Blocking them at the firewall level is somehow not an Orthodox decision at all, and not entirely convenient.
And recently, a friend told me that he was raising a caching proxy with HTTPS filtering in his office, I was interested in this.
And he raised Squid 3.5.8 . As it turned out, it reworked the organization of intercepting encrypted HTTPS sessions (ssl_bump), instead of intercepting modes ("modes"), actions ("actions") were introduced, which can also be used in ACLs. The server-first mode, in which a connection is first made to the target server, and then a secure connection is created between the client and the proxy, is now available as a bump action. The "none" mode, in which a TCP tunnel is created without decrypting traffic, is now available as the "splice" action.
To ensure backward compatibility, the “peek-and-splice” action has been added, in which the decision on the type of connection created at the beginning (client proxy or proxy server) is made based on SSL hello messages. Added actions "peek" and "stare" to obtain a client or server certificate while maintaining the possibility of further use of the "splice" and "bump" modes for connections. The terminate action has been added to close connections to a client or server. That's just the SSL BUMP, PEEK-and-SPLICE and Terminate we need. In general, the working scheme is actually quite simple. Squid connects to an HTTPS resource, receives its certificate, and can “see” some data about the resource, in particular the server name, which we just need to block! All manuals that are on the Internet, every now and then describe the Man in the middle (MITM) attack with the substitution of certificates, in which, in principle, some sites and client banks do not work, and users clearly see that they are being watched. Together with a friend, we worked together to achieve the filtering and tracking of HTTPS without changing certificates, without MITM and other things, and all this in a transparent mode without setting up browsers!
Subsequently, I encountered some difficulties, in particular, Squid began to segoltitsya under heavy load. But the problem has been resolved. The fact is that in Openssl there are some bugs that are fixed in the Libressl library. Therefore, you must first integrate into the Libressl system , then after making the patch in the bio.cc file in the Squid sources, proceed to compile the latter. Go! I will make a reservation that I use the Debian Jessie x86 distribution, and I finally built Squid version 3.5.9 (the latest version at the moment), and the article is designed for a more or less experienced Linux user, as some points are omitted, but only the most is said the main thing, since I'm too lazy to chew everything.
To get started, prepare for building packages:
Do not forget to go to the folder where you will collect the source so as not to litter your home. Next, download, compile and install Libressl:
We collect and install, after which we re-read the hashes of the libraries:
Well, you need to configure the use of LibreSSL by default:
Let's check whether it was possible to install Libressl:
Happened!
After completing these steps, you need to edit sources.list to include the sources from the testing branch there (this is necessary, since we need to compile a new libecap, which in turn is necessary for building Squid):
Update package cache:
And now download the necessary sources from Testing:
Next, build libecap:
Let's remove the junk and install the novice:
Download thelatest and most recent Squid snapshot:
We will update the Squid sources received earlier to new ones, and further we will work already in the directory with the newly-made sources:
In order for all the buns we need to work, we need to compile Squid with the necessary options, so we will add the following compilation options to debian / rules:
Download the patch for bio.cc, which is needed to work correctly with Libressl, from here bugs.squid-cache.org/attachment.cgi?id=3216 and apply it
Now you can start compiling and building Squid. But not so fast! Everything compiles without problems, at least on the x86 architecture, but at the very end, at the stage of building deb packages, you will be kindly told in the console: “ah ah ah, I can’t understand what dependencies are needed for libssl.so.32” (this version of the library from Libressl). It is understandable how Debian knows about it. We trick the system by specifying the option “do not check dependencies”, like this:
And now let's start compilation and assembly:
After the build, install the language pack for Squid:
Next, install the newly made packages:
If the system has matched up for unsatisfied dependencies, we will do:
Almost ready. Let's go to the / etc / squid directory and change something there. Create the pem file required for SSL-bumping:
And bring squid.conf to the following form:
Next, wrap the required ports on Squid with a firewall:
All is ready! You can solemnly run Squid:
Let's see if everything is normal with Squid:
Если ошибок нет, можно работать. К сожалению, при блокировке HTTPS ресурсов, не появляется сообщение Squid'a «Доступ запрещен», а вместо этого браузер выдает ошибку о невозможности создания соединения. Если кто-то подскажет, как это сделать, буду очень рад.
UPD: в версии Кальмара, которую компилировал я изначально, т.е. 3.5.9, найден досадный баг (или фича), из-за которого спустя время перестают открываться некоторые HTTPS сайты. Решение: компилировать версию 3.5.8.
UPD2: создал очередной багрепорт по проблеме в 3.5.9, тему буду обновлять, если что-то прояснится.
UPD3: вышла версия 3.5.10 с исправленными багами, по крайней мере, патч на файл bio.cc там уже применен. Не тестировал пока-что
UPD4: подредактировал статью немного.
UPD5: прямая ссылкаTo download the archive with all the necessary packages (SQUID 3.5.8) , so far this is the only working version! The rest, which are versions higher, have a bug due to which Squid stops working when transparently proxing HTTPS. VERSION FOR X86 !!!
ATTENTION!!! To avoid confusion, install version 3.5.8! There are no problems on it, and if everything is done exactly according to the article, then everything will work! If you put the version above, then I can not guarantee the correct operation of transparent filtering!
Another Update !!! Debian Testing src repository is now 3.5.10 !!! add squeeze src repository to sources.list. There is version 3.1, but it's okay, everything will be updated to version 3.5.8 as it should, only change the paths to directories in accordance with the version of the downloaded sources!
UPD 12/02/15: I apologize if I forced someone to crutch. I reloaded the squid 3.5.8 archive, one deb package was missing there! squid 3.5.8_all.
UPD 12/02/15: Attention! If you have a problem with installing libressl, then you need to do this: first install openssl, and then install libressl (if you are installing the version from the packages archive, do not forget to replace openssl with libressl, as in the article!). This will solve the “invisibility” problem of the libssl.so.32 UPD library 12/10/15
: The next version of the article has appeared, with a slightly different approach to compiling Squid and building packages. Right here
UPD 12/14/15: I hasten to share great news with my colleagues! I found a way to get new versions of Squid to work without much dancing with a tambourine! It is necessary that the clients and the Squid settings have the same DNS! In my case, Bind is spinning on the gateway with Squid. He assigned clients to him, and Kalmar directive:
UPD 12/14/15: I don’t know why, but at the moment on Debian 8.2, when building Squid, as described above, a message is displayed stating that the mime.conf file was not found. And really, because in the sources of Kalmar there is mime.conf.default. I have found a solution. Two files must be edited:
1) debian / squid-common.install, and give the line
in this form:
2) It is also necessary to make sure that after installing the package, the file is automatically renamed into a “readable view”, i.e. in mime.conf. To do this, edit the debian / squid-common.postinst file, bringing it into this form:
That is, add the file rename line, and leave the rest as is, do not touch.
Everything, after that we form packages, they will be formed and delivered.
UPD 06/19/17: since the linux-admin.tk domain was stolen from me, I had to create a new, already “normal” domain, so I changed the link to download the archive.
UPD 05/05/18: wrote a new article , which solves the problem of glitches of new versions of Squid + in logs beautiful domain names instead of ip addresses + bypass ILV locks.
And recently, a friend told me that he was raising a caching proxy with HTTPS filtering in his office, I was interested in this.
And he raised Squid 3.5.8 . As it turned out, it reworked the organization of intercepting encrypted HTTPS sessions (ssl_bump), instead of intercepting modes ("modes"), actions ("actions") were introduced, which can also be used in ACLs. The server-first mode, in which a connection is first made to the target server, and then a secure connection is created between the client and the proxy, is now available as a bump action. The "none" mode, in which a TCP tunnel is created without decrypting traffic, is now available as the "splice" action.
To ensure backward compatibility, the “peek-and-splice” action has been added, in which the decision on the type of connection created at the beginning (client proxy or proxy server) is made based on SSL hello messages. Added actions "peek" and "stare" to obtain a client or server certificate while maintaining the possibility of further use of the "splice" and "bump" modes for connections. The terminate action has been added to close connections to a client or server. That's just the SSL BUMP, PEEK-and-SPLICE and Terminate we need. In general, the working scheme is actually quite simple. Squid connects to an HTTPS resource, receives its certificate, and can “see” some data about the resource, in particular the server name, which we just need to block! All manuals that are on the Internet, every now and then describe the Man in the middle (MITM) attack with the substitution of certificates, in which, in principle, some sites and client banks do not work, and users clearly see that they are being watched. Together with a friend, we worked together to achieve the filtering and tracking of HTTPS without changing certificates, without MITM and other things, and all this in a transparent mode without setting up browsers!
Subsequently, I encountered some difficulties, in particular, Squid began to segoltitsya under heavy load. But the problem has been resolved. The fact is that in Openssl there are some bugs that are fixed in the Libressl library. Therefore, you must first integrate into the Libressl system , then after making the patch in the bio.cc file in the Squid sources, proceed to compile the latter. Go! I will make a reservation that I use the Debian Jessie x86 distribution, and I finally built Squid version 3.5.9 (the latest version at the moment), and the article is designed for a more or less experienced Linux user, as some points are omitted, but only the most is said the main thing, since I'm too lazy to chew everything.
To get started, prepare for building packages:
apt-get install git fakeroot build-essential devscripts
apt-cache policy squid3
apt-get build-dep squid3
apt-get build-dep libecap2
apt-get install libssl-dev libgnutls28-dev
Do not forget to go to the folder where you will collect the source so as not to litter your home. Next, download, compile and install Libressl:
wget http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.1.6.tar.gz
tar -xzvf libressl-2.1.6.tar.gz
cd libressl-2.1.6
We collect and install, after which we re-read the hashes of the libraries:
./configure
make
checkinstall --pkgname libressl --pkgversion 2.1.6
dpkg -i libressl_2.1.6-1_i386.deb
ldconfig
Well, you need to configure the use of LibreSSL by default:
mv /usr/bin/openssl /usr/bin/openssl-1
update-alternatives --install /usr/bin/openssl openssl /usr/bin/openssl-1 10
update-alternatives --install /usr/bin/openssl openssl /usr/local/bin/openssl 50
update-alternatives --config openssl
Let's check whether it was possible to install Libressl:
openssl version
LibreSSL 2.1.6
Happened!
After completing these steps, you need to edit sources.list to include the sources from the testing branch there (this is necessary, since we need to compile a new libecap, which in turn is necessary for building Squid):
deb-src http://ftp.de.debian.org/debian/ testing main contrib non-free
Update package cache:
apt-get update
And now download the necessary sources from Testing:
apt-get source squid3/testing
apt-get source libecap3/testing
Next, build libecap:
cd libecap-1.0.1/
dpkg-buildpackage -us -uc -nc -d
Let's remove the junk and install the novice:
apt-get purge libecap2
dpkg -i libecap3_1.0.1-2_i386.deb
dpkg -i libecap3-dev_1.0.1-2_i386.deb
Download the
wget http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.8.tar.gz
We will update the Squid sources received earlier to new ones, and further we will work already in the directory with the newly-made sources:
cd squid3-3.5.7/
uupdate -v 3.5.8 ../squid-3.5.8.tar.gz
cd ../squid3-3.5.8/
In order for all the buns we need to work, we need to compile Squid with the necessary options, so we will add the following compilation options to debian / rules:
--enable-ssl
--enable-ssl-crtd
--with-openssl
Download the patch for bio.cc, which is needed to work correctly with Libressl, from here bugs.squid-cache.org/attachment.cgi?id=3216 and apply it
patch -p0 -i bug-4330-put_cipher_by_char-t1.patch
Now you can start compiling and building Squid. But not so fast! Everything compiles without problems, at least on the x86 architecture, but at the very end, at the stage of building deb packages, you will be kindly told in the console: “ah ah ah, I can’t understand what dependencies are needed for libssl.so.32” (this version of the library from Libressl). It is understandable how Debian knows about it. We trick the system by specifying the option “do not check dependencies”, like this:
export DEB_DH_SHLIBDEPS_ARGS_ALL=--dpkg-shlibdeps-params=--ignore-missing-info
And now let's start compilation and assembly:
dpkg-buildpackage -us -uc -nc
After the build, install the language pack for Squid:
apt-get install squid-langpack
Next, install the newly made packages:
dpkg -i squid-common_3.5.8-1_all.deb
dpkg -i squid_3.5.8-1_i386.deb
dpkg -i squid3_3.5.8-1_all.deb
dpkg -i squidclient_3.5.8-1_i386.deb
If the system has matched up for unsatisfied dependencies, we will do:
apt-get -f install
Almost ready. Let's go to the / etc / squid directory and change something there. Create the pem file required for SSL-bumping:
openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout squidCA.pem -out squidCA.pem
And bring squid.conf to the following form:
acl localnet src 192.168.1.0/24 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
dns_nameservers 8.8.8.8
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
#прозрачный порт указывается опцией intercept
http_port 192.168.1.254:3128 intercept options=NO_SSLv3:NO_SSLv2
#также нужно указать непрозрачный порт, ибо если захотите вручную указать адрес
#прокси в браузере, указав прозрачный порт, вы получите ошибку доступа, поэтому нужно
#указывать непрозрачный порт в браузере, если конечно такое желание будет, к тому же в логах #сыпятся ошибки о том, что непрохрачный порт не указан=)
http_port 192.168.1.254:3130 options=NO_SSLv3:NO_SSLv2
#и наконец, указываем HTTPS порт с нужными опциями
https_port 192.168.1.254:3129 intercept ssl-bump options=ALL:NO_SSLv3:NO_SSLv2 connection-auth=off cert=/etc/squid/squidCA.pem
always_direct allow all
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER
#укажем правило со списком блокируемых ресурсов (в файле домены вида .domain.com)
acl blocked ssl::server_name "/etc/squid/blocked_https.txt"
acl step1 at_step SslBump1
ssl_bump peek step1
#терминируем соединение, если клиент заходит на запрещенный ресурс
ssl_bump terminate blocked
ssl_bump splice all
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
coredump_dir /var/spool/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
cache_dir aufs /var/spool/squid 20000 49 256
maximum_object_size 61440 KB
minimum_object_size 3 KB
cache_swap_low 90
cache_swap_high 95
maximum_object_size_in_memory 512 KB
memory_replacement_policy lru
logfile_rotate 4
I’ll talk a little about the config. The documentation for ssl_bump, peek-n-splice is quite extensive, but for our task you need to know the following. There are several stages of “handshaking” (ie shaking hands, interacting with the server). They are described in the official documentation . We are interested in the example of Peek at SNI and Bump . That is, as the name implies, we look at the SNI information and bump the connection. Before this, we indicate with the DONT_VERIFY_PEER option that it is necessary to accept certificates even if they have not passed the verification and with the sslproxy_cert_error option we indicate that we need to disable certificate verification on the server. The specified rule is "acl step1 at_step SslBump1"- this is the first of three possible steps ssl_bump'a. When performing this step, we get only the SNI, and nothing more. This is enough for us. Next, we use this ACL with the string ssl_bump peek step1 , that is, we directly look at the SNI, and after that we block the connection if server_name from the blocked list is found in the SNI.
Next, wrap the required ports on Squid with a firewall:
iptables -t nat -A PREROUTING -p tcp -m tcp -s 192.168.1.0/24 --dport 443 -j REDIRECT --to-ports 3129
iptables -t nat -A PREROUTING -p tcp -m tcp -s 192.168.1.0/24 --dport 80 -j REDIRECT --to-ports 3128
All is ready! You can solemnly run Squid:
systemctl start squid
Let's see if everything is normal with Squid:
systemctl status squid
● squid.service - LSB: Squid HTTP Proxy version 3.x
Loaded: loaded (/etc/init.d/squid)
Active: active (running) since Сб 2015-09-26 21:09:44 YEKT; 2h 6min ago
Process: 1798 ExecStop=/etc/init.d/squid stop (code=exited, status=0/SUCCESS)
Process: 1818 ExecStart=/etc/init.d/squid start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/squid.service
├─1850 /usr/sbin/squid -YC -f /etc/squid/squid.conf
├─1852 (squid-1) -YC -f /etc/squid/squid.conf
├─1853 (logfile-daemon) /var/log/squid/access.log
└─1854 (pinger)
сен 26 21:09:44 squid squid[1850]: Squid Parent: will start 1 kids
сен 26 21:09:44 squid squid[1850]: Squid Parent: (squid-1) process 1852 started
сен 26 21:09:44 squid squid[1818]: Starting Squid HTTP Proxy: squid.
Если ошибок нет, можно работать. К сожалению, при блокировке HTTPS ресурсов, не появляется сообщение Squid'a «Доступ запрещен», а вместо этого браузер выдает ошибку о невозможности создания соединения. Если кто-то подскажет, как это сделать, буду очень рад.
UPD: в версии Кальмара, которую компилировал я изначально, т.е. 3.5.9, найден досадный баг (или фича), из-за которого спустя время перестают открываться некоторые HTTPS сайты. Решение: компилировать версию 3.5.8.
UPD2: создал очередной багрепорт по проблеме в 3.5.9, тему буду обновлять, если что-то прояснится.
UPD3: вышла версия 3.5.10 с исправленными багами, по крайней мере, патч на файл bio.cc там уже применен. Не тестировал пока-что
UPD4: подредактировал статью немного.
UPD5: прямая ссылкаTo download the archive with all the necessary packages (SQUID 3.5.8) , so far this is the only working version! The rest, which are versions higher, have a bug due to which Squid stops working when transparently proxing HTTPS. VERSION FOR X86 !!!
ATTENTION!!! To avoid confusion, install version 3.5.8! There are no problems on it, and if everything is done exactly according to the article, then everything will work! If you put the version above, then I can not guarantee the correct operation of transparent filtering!
Another Update !!! Debian Testing src repository is now 3.5.10 !!! add squeeze src repository to sources.list. There is version 3.1, but it's okay, everything will be updated to version 3.5.8 as it should, only change the paths to directories in accordance with the version of the downloaded sources!
UPD 12/02/15: I apologize if I forced someone to crutch. I reloaded the squid 3.5.8 archive, one deb package was missing there! squid 3.5.8_all.
UPD 12/02/15: Attention! If you have a problem with installing libressl, then you need to do this: first install openssl, and then install libressl (if you are installing the version from the packages archive, do not forget to replace openssl with libressl, as in the article!). This will solve the “invisibility” problem of the libssl.so.32 UPD library 12/10/15
: The next version of the article has appeared, with a slightly different approach to compiling Squid and building packages. Right here
UPD 12/14/15: I hasten to share great news with my colleagues! I found a way to get new versions of Squid to work without much dancing with a tambourine! It is necessary that the clients and the Squid settings have the same DNS! In my case, Bind is spinning on the gateway with Squid. He assigned clients to him, and Kalmar directive:
dns_nameservers 127.0.0.1
UPD 12/14/15: I don’t know why, but at the moment on Debian 8.2, when building Squid, as described above, a message is displayed stating that the mime.conf file was not found. And really, because in the sources of Kalmar there is mime.conf.default. I have found a solution. Two files must be edited:
1) debian / squid-common.install, and give the line
etc/squid/mime.conf
in this form:
etc/squid/mime.conf.default
2) It is also necessary to make sure that after installing the package, the file is automatically renamed into a “readable view”, i.e. in mime.conf. To do this, edit the debian / squid-common.postinst file, bringing it into this form:
#! /bin/sh
set -e
case "$1" in
configure)
mv /etc/squid/mime.conf.default /etc/squid/mime.conf
...........
That is, add the file rename line, and leave the rest as is, do not touch.
Everything, after that we form packages, they will be formed and delivered.
UPD 06/19/17: since the linux-admin.tk domain was stolen from me, I had to create a new, already “normal” domain, so I changed the link to download the archive.
UPD 05/05/18: wrote a new article , which solves the problem of glitches of new versions of Squid + in logs beautiful domain names instead of ip addresses + bypass ILV locks.
I want to say thanks to comrade Dmitry Rakhmatullin, without him I would not have been able to do what is written above. Also, special thanks to the Squid developers who promptly responded to my bug report about the error in libssl. And thanks to the guys Nadz Goldman and gmax007 from Toaster, who sent me on the right track my idea of porting Squid to a server physically separate from the main gateway.
