December 9, 2015 at 15:13"Transparent" Squid with HTTPS resource filtering without certificate spoofing (x86, x64 - universal instruction)
- Tutorial
Hello! Past articleAbout transparent proxying HTTPS with Squid, it was quite successful. A lot of reviews came by mail about the successful installation of this system. But there were also letters asking for help. The problems were completely solvable. But not so long ago, a colleague approached me asking for help in installing this system on x64 architecture (Debian). Here we are puzzled. Firstly, it turned out that the previous article was unsuitable for this because of the lack of the necessary sources in the Debian repository (there is now 3.5.10). It was not possible to find the Debian sources needed in the first article, and checkinstall generated strange errors. Secondly, I wanted a more universal solution that would work without problems on x64, x86, and (if possible) on other distributions. A solution has been found. It turned out a small addition to the previous article + some clarifications. This instruction allows you to compile both x86 and x64 versions of Squid and create the appropriate packages. The instruction will be divided into several paragraphs and subparagraphs. If interested, go to cat:
Let's go in order.
1)
a) To get started, prepare for the assembly of packages:
Do not forget to go to the folder where you will collect the sources, so as not to spoil your home.
b) Next, download Libressl:
c) And now we collect:
2) Now you can install Libressl:
After installation, you need to configure the use of LibreSSL by default:
Let's check whether it was possible to install Libressl:
If the console exhaust is similar, then everything worked out. Move on.
3) Next in line is Libecap.
a) You need to edit sources.list to include the sources from the testing branch there (this is necessary, since we need to compile a new libecap, which in turn is necessary for building Squid):
Update package cache:
And now download the necessary sources from Testing:
Next, build libecap:
b) Remove the junk and install the novice:
4) We got to the compilation of Squid itself.
a) Download thelatest most working Squid snapshot:
Unpack:
b) Download the patch for bio.cc, and patch:
5) And this stage is one of the most responsible. You must configure Squid with the options you need. The debian / rules file was used in the previous article, but we will compile Squid in this instruction using make, and we will create packages using checkinstall. Therefore, there will be more options. And here are some:
Be extremely careful. We are more interested in, as in the previous article, three options: --enable-ssl, --enable-ssl-crtd, --with-openssl. You can change the remaining options according to your preferences (if you want to change them, be sure to read the configuration documentation).
6) Now we get to compiling.
a) Compile.
b) An ambiguous stage. It is necessary to create the directories / usr / share / squid / and / usr / share / squid / icons, otherwise the next step will fail due to the lack of these folders (why checkinstall does not create them myself, I did not understand, unfortunately):
c) And now we create installation packages:
7) We are approaching the finals. Install Squid:
8) Try to run squid:
And see the big FIG! Wow ... Trying the old fashioned way:
And also see the big FIG. Why? Because checkinstall did not include Squid service files in the package. No problem. Create the desired systemd service yourself:
With the following contents:
Turn on the created service
9) Yes, you're right, that's not all. Since we compiled completely original sources (with the exception of the patch on bio.cc), the configuration files were installed like squid.conf.default, mime.conf.default, etc. Of course, Squid did not hear about them. Rename them to Squid's readable form:
10) And this is not all =) You must manually create a folder for Squid's logs and assign appropriate rights to it:
11) And here it is - the final stage. Launch Squid'a and check the status of the service!
If the console exhaust looks similar, or rather there are no errors and the line “Active: active (running)” is present, then you have successfully installed Squid with support for transparent HTTPS proxying! Congratulations!
If you do not want to compile anything, then you can download the archive with ready deb packages (x64 version!). If you install from ready-made packages, then you will need steps: 2, 3 (b), 7, 8, 9, 10, 11 .
I also want to note that checkinstall allows you to create rpm packages, and you can take advantage of this. The only thing is that you need to collect all the packages using checkinstall, but I think there will be no problems with this, since the main and most complex ones are already collected by checkinstall.
Squid'a configuration file with the necessary directives, job description, etc. read in the previous article !
UPD 12/12/15: At the request of the user, nikitasius checked Cloudflare. Comrade nikitasius wrote in the comments:
UPD 12/14/15: I’m in a hurry to share great news with my colleagues! I found a way to get new versions of Squid to work without much dancing with a tambourine! It is necessary that the clients and the Squid settings have the same DNS! In my case, Bind is spinning on the gateway with Squid. He assigned clients to him, and Kalmar directive:
UPD 12/16/15: Packages for Centos!
UPD 04/25/16: Request for those who tested the x64 assembly to write whether everything works. There are reports of glitches, in particular, a heavy load on the CPU after some time and the subsequent segfault. I myself can’t check for now
UPD 05/05/18: I wrote a new article where the problem of glitches of new versions of Squid + in the logs beautiful domain names instead of ip addresses + bypass of ILV locks is solved.
Let's go in order.
1)
a) To get started, prepare for the assembly of packages:
apt-get install git fakeroot checkinstall build-essential devscripts patch
apt-cache policy squid3
apt-get build-dep squid3
apt-get build-dep libecap2
apt-get install libssl-dev libgnutls28-dev
Do not forget to go to the folder where you will collect the sources, so as not to spoil your home.
b) Next, download Libressl:
wget http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.1.6.tar.gz
tar -xzvf libressl-2.1.6.tar.gz
cd libressl-2.1.6
c) And now we collect:
./configure
make
checkinstall --pkgname libressl --pkgversion 2.1.6
2) Now you can install Libressl:
dpkg -i libressl_2.1.6-1_amd64.deb
ldconfig
After installation, you need to configure the use of LibreSSL by default:
mv /usr/bin/openssl /usr/bin/openssl-1
update-alternatives --install /usr/bin/openssl openssl /usr/bin/openssl-1 10
update-alternatives --install /usr/bin/openssl openssl /usr/local/bin/openssl 50
update-alternatives --config openssl
Let's check whether it was possible to install Libressl:
openssl version
LibreSSL 2.1.6
If the console exhaust is similar, then everything worked out. Move on.
3) Next in line is Libecap.
a) You need to edit sources.list to include the sources from the testing branch there (this is necessary, since we need to compile a new libecap, which in turn is necessary for building Squid):
deb-src http://ftp.de.debian.org/debian/ testing main contrib non-free
Update package cache:
apt-get update
And now download the necessary sources from Testing:
apt-get source libecap3/testing
Next, build libecap:
cd libecap-1.0.1/
dpkg-buildpackage -us -uc -nc -d
b) Remove the junk and install the novice:
apt-get purge libecap2
libecap3_1.0.1-2_amd64.deb
libecap3-dev_1.0.1-2_amd64.deb
4) We got to the compilation of Squid itself.
a) Download the
wget http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.8.tar.gz
Unpack:
tar -xf squid-3.5.8.tar.gz
cd squid-3.5.8
b) Download the patch for bio.cc, and patch:
wget -O bug-4330-put_cipher_by_char-t1.patch http://bugs.squid-cache.org/attachment.cgi?id=3216
patch -p0 -i bug-4330-put_cipher_by_char-t1.patch
» patching file src/ssl/bio.cc
5) And this stage is one of the most responsible. You must configure Squid with the options you need. The debian / rules file was used in the previous article, but we will compile Squid in this instruction using make, and we will create packages using checkinstall. Therefore, there will be more options. And here are some:
./configure --build=x86_64-linux-gnu \
--prefix=/usr \
--includedir=${prefix}/include \
--mandir=${prefix}/share/man \
--infodir=${prefix}/share/info \
--sysconfdir=/etc \
--localstatedir=/var \
--libexecdir=${prefix}/lib/squid \
--srcdir=. \
--disable-maintainer-mode \
--disable-dependency-tracking \
--disable-silent-rules \
--datadir=/usr/share/squid \
--sysconfdir=/etc/squid \
--mandir=/usr/share/man \
--enable-inline \
--disable-arch-native \
--enable-async-io=8 \
--enable-storeio=ufs,aufs,diskd,rock \
--enable-removal-policies=lru,heap \
--enable-delay-pools \
--enable-cache-digests \
--enable-icap-client \
--enable-follow-x-forwarded-for \
--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB \
--enable-auth-digest=file,LDAP \
--enable-auth-negotiate=kerberos,wrapper \
--enable-auth-ntlm=fake,smb_lm \
--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group \
--enable-url-rewrite-helpers=fake \
--enable-eui \
--enable-esi \
--enable-icmp \
--enable-zph-qos \
--enable-ecap \
--disable-translation \
--with-swapdir=/var/spool/squid \
--with-logdir=/var/log/squid \
--with-pidfile=/var/run/squid.pid \
--with-filedescriptors=65536 \
--with-large-files \
--with-default-user=proxy \
--enable-ssl \
--enable-ssl-crtd \
--with-openssl \
--enable-linux-netfilter \
'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' \
'LDFLAGS=-fPIE -pie -Wl,-z,relro -Wl,-z,now' \
'CPPFLAGS=-D_FORTIFY_SOURCE=2' \
'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'
Be extremely careful. We are more interested in, as in the previous article, three options: --enable-ssl, --enable-ssl-crtd, --with-openssl. You can change the remaining options according to your preferences (if you want to change them, be sure to read the configuration documentation).
6) Now we get to compiling.
a) Compile.
make
b) An ambiguous stage. It is necessary to create the directories / usr / share / squid / and / usr / share / squid / icons, otherwise the next step will fail due to the lack of these folders (why checkinstall does not create them myself, I did not understand, unfortunately):
mkdir -p /usr/share/squid/icons
c) And now we create installation packages:
checkinstall --pkgname squid --pkgversion 3.5.8
7) We are approaching the finals. Install Squid:
dpkg -i squid_3.5.8-1_amd64.deb
Spoiler heading
Yes, that's right, it turned out only one file, while in the previous article there were several, as is usual in Debian.
8) Try to run squid:
systemctl start squid
And see the big FIG! Wow ... Trying the old fashioned way:
service squid start
And also see the big FIG. Why? Because checkinstall did not include Squid service files in the package. No problem. Create the desired systemd service yourself:
touch /etc/systemd/system/squid.service
nano /etc/systemd/system/squid.service
With the following contents:
## Copyright (C) 1996-2015 The Squid Software Foundation and contributors
##
## Squid software is distributed under GPLv2+ license and includes
## contributions from numerous individuals and organizations.
## Please see the COPYING and CONTRIBUTORS files for details.
##
[Unit]
Description=Squid Web Proxy Server
After=network.target
[Service]
Type=simple
ExecStart=/usr/sbin/squid -sYC -N
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
[Install]
WantedBy=multi-user.target
Spoiler heading
In fact, this same service is in the archive with the Squid source code. For what reasons Checkinstall did not include it in the package, it is not known.
Turn on the created service
systemctl enable squid
9) Yes, you're right, that's not all. Since we compiled completely original sources (with the exception of the patch on bio.cc), the configuration files were installed like squid.conf.default, mime.conf.default, etc. Of course, Squid did not hear about them. Rename them to Squid's readable form:
cp /etc/squid/squid.conf.default /etc/squid/squid.conf
cp /etc/squid/mime.conf.default /etc/squid/mime.conf
cp /etc/squid/cachemgr.conf.default /etc/squid/cachemgr.conf
cp /etc/squid/errorpage.css.default /etc/squid/errorpage.css
10) And this is not all =) You must manually create a folder for Squid's logs and assign appropriate rights to it:
mkdir /var/log/squid
chown proxy /var/log/squid
11) And here it is - the final stage. Launch Squid'a and check the status of the service!
systemctl start squid
systemctl status -l squid
● squid.service - Squid Web Proxy Server
Loaded: loaded (/etc/systemd/system/squid.service; enabled)
Active: active (running) since Пт 2015-12-04 23:32:04 YEKT; 2min 41s ago
Main PID: 590 (squid)
CGroup: /system.slice/squid.service
├─590 /usr/sbin/squid -sYC -N
└─591 (logfile-daemon) /var/log/squid/access.log
дек 04 23:32:04 squidX64 squid[590]: Max Swap size: 0 KB
дек 04 23:32:04 squidX64 squid[590]: Using Least Load store dir selection
дек 04 23:32:04 squidX64 squid[590]: Current Directory is /
дек 04 23:32:04 squidX64 squid[590]: Finished loading MIME types and icons.
дек 04 23:32:04 squidX64 squid[590]: HTCP Disabled.
дек 04 23:32:04 squidX64 squid[590]: Pinger socket opened on FD 16
дек 04 23:32:04 squidX64 squid[590]: Squid plugin modules loaded: 0
дек 04 23:32:04 squidX64 squid[590]: Adaptation support is off.
дек 04 23:32:04 squidX64 squid[590]: Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 14 flags=9
дек 04 23:32:05 squidX64 squid[590]: storeLateRelease: released 0 objects
If the console exhaust looks similar, or rather there are no errors and the line “Active: active (running)” is present, then you have successfully installed Squid with support for transparent HTTPS proxying! Congratulations!
If you do not want to compile anything, then you can download the archive with ready deb packages (x64 version!). If you install from ready-made packages, then you will need steps: 2, 3 (b), 7, 8, 9, 10, 11 .
I also want to note that checkinstall allows you to create rpm packages, and you can take advantage of this. The only thing is that you need to collect all the packages using checkinstall, but I think there will be no problems with this, since the main and most complex ones are already collected by checkinstall.
Squid'a configuration file with the necessary directives, job description, etc. read in the previous article !
Thanks to Tatyana Illarionova and the Squid developers for their help in creating this recipe!
UPD 12/12/15: At the request of the user, nikitasius checked Cloudflare. Comrade nikitasius wrote in the comments:
Apparently for sites that for Cloudflare such a system will not work correctly ...So, I checked it. I added one of its domains from Cloudflare to the HTTPS blocking blacklist, the browser doesn’t go to it, but the browser calmly goes to other domains that are listed in the certificate. So, the Cloudflare test passed
They usually have a bunch of domains in certificates.
In case example.com is in the same certificate with freepron.cum, the squid will work on freepron, if the latter is in the local bath, right?
UPD 12/14/15: I’m in a hurry to share great news with my colleagues! I found a way to get new versions of Squid to work without much dancing with a tambourine! It is necessary that the clients and the Squid settings have the same DNS! In my case, Bind is spinning on the gateway with Squid. He assigned clients to him, and Kalmar directive:
dns_nameservers 127.0.0.1
UPD 12/16/15: Packages for Centos!
UPD 04/25/16: Request for those who tested the x64 assembly to write whether everything works. There are reports of glitches, in particular, a heavy load on the CPU after some time and the subsequent segfault. I myself can’t check for now
UPD 05/05/18: I wrote a new article where the problem of glitches of new versions of Squid + in the logs beautiful domain names instead of ip addresses + bypass of ILV locks is solved.
For those who think this is MITM =)
In the survey, the last point is of course a joke =) If you CAREFULLY read the previous article, it will immediately become clear that there is no attack on the user, since there is no substitution of certificates !!!
Only registered users can participate in the survey. Please come in.
Have you already installed Squid with transparent proxying HTTPS?
- 20.4% Yes 57
- 51.6% Not yet, I’ll do 144
- 27.9% Why! ???? This is MITM !!! 78
