May 3, 2018 at 19:47Transparent HTTPS proxy to bypass Roskomnadzor locks
- Tutorial
Hello, Habrovsk citizens! I think many people have recently encountered problems with access to the necessary resources due to Roskom’s attempts to disgrace the shame of surveillance to block Telegram. And I think the comments here are superfluous. Fact - these resources are not to blame for anything, but they are blocked. Problems arose with Viber, ReCaptcha, GoogleFonts, Youtube, etc. (except for the telegram itself). This happened in my organization, and some of the innocent services we need as air. At some time, everything was solved using proxy servers, but they were unstable or completely disconnected (they were also blocked by our great and powerful ILV).
After reading a bunch of articles, the idea came up to teach Squid to let individual URLs through Tor. Whether to use such a method is up to you. But I will say that after the implementation all the problems that were before that disappeared. Who cares, we go under the cat.
So, for starters, a little theory. As we all know, Tor is not an HTTP proxy; it cannot be made a direct peer for the underlying Squid. It provides SOCKS proxying (of course, not only, but this is what we need). In order for us to marry Tor with Squid, we need something that could play the role of a conductor from Tor to Squid and vice versa. And of course, ladies and gentlemen, this is Privoxy. Just the same, he is able to be a direct peer, and send everything further to Tor.
It was, as I said, a bunch of articles were read, but not one suited me. This article came across here , but it didn’t quite suit me, since I do not need a bump. In general, all available articles, almost everything, imply either bump or only http, and in my case I need HTTPS, splice, and transparency. Also saw this andit , but there are completely different approaches. Its pros and cons. I chose for myself a bunch of Squid + Tor.
I've previously written about how to make a transparent Squid with HTTPS proxies without replacing the certificates. And of course, I tried to implement the idea on it. But I was disappointed. HTTP requests went perfectly to TOR, but HTTPS didn’t. The problem is not very well known, and I learned from one of the developers that this is a drawback of older versions of Squid. But during the experiments, a solution was found - Squid 3.5.27, in which this bug was fixed + beautiful domain names in the logs (https), instead of ip addresses. But even here, several disappointments awaited me, which will be discussed below. But everything, as they say, is doped with a file.
So, the source data:
Collect Squid pens or put ready-made packages (links below), you decide. If you think that I haveplayed blackjack and w ... viruses in them , then compile yourself. You also need to compile if you have a different distribution kit (if Ubunt, then install). We will collect the version of Squid 3.5.23, which at the time of writing, is lying in the Stretch repository , upgrading it to the fresh 3.5.27 sources from the website. Unlike my first article about HTTPS + Squid, we will build without Libressl.
So, prepare for the assembly:
Next, we get the sources of Squid 3.5.23
Download this particular archive with Squid sources:
We go to the Squid source directory and update the sources to the newly downloaded sources:
Go to the newly-made catalog with updated sources:
Add compilation options to debian / rules:
Next, you need to patch the source with this patch:
What is it for? I will explain. When I wrote the first article about peek and splice, I said that newer versions do not work, and it was, and just this patch fixes the very problem, which was that Squid selectively breaks HTTPS connections, with an interesting message in cache.log:
The fact is that on one host something resolves to one IP, sometimes to another on the next, on Squid itself to the third, because There is a DNS cache and it is not updated synchronously. Squid does not match the ip domain in its cache (because it updated its cache a little earlier or later) and terminates the connection. It seems like protection, but nowadays it is considered normal (round-robin DNS). Developers are reinsured. And we don’t need it at all! To those who say that this patch may pose a security risk, I will answer that I consulted with Yuri Voinov, who is directly related to the Squid development team, about this patch. There is no threat here!
So, we created the file for the patch, we threw the code, we need to patch it:
Next, you must cancel the application of one patch during compilation (otherwise you will get an error that this patch cannot be applied, since it has already been applied). We go to debian / patches / series and comment out there 0003-SQUID-2018_1.patch , putting a # in front of it :
Well, then - compilation and assembly of packages!
Install squid-langpack
and install fresh packages
If apt swears on addiction, do
Next, you need to turn off Squid from startup (by default, the init file is used, Squid complains about the inaccessibility of the PID file)
and create a systemd service in the directory / etc / systemd / system (the service file is in the sources, and is fully copied here)
Turn it on
Install tor, privoxy
I personally did not touch the Tor config at all, but the Privoxy config can be reduced to this view:
Almost ready. Let's go to the / etc / squid directory and change something there. Create the pem file needed for splice:
And bring squid.conf to the following form:
The url_regex list looks like this (the list is given as an example!):
Read more about this list format in the documentation office. The result did not take long to wait - everything worked, as planned. And it works to this day. Perhaps the article was late, but it will probably come in handy in the future).
I will replenish the topic if possible.
Ready-made Squid packages (.deb x86)
UPD 05/05/18:you can add a line to torcc
and basically abandon Privoxy. Thanks dartraiden for the comment.
Found a bug. Not recommended HTTPTunnelPort, Privoxy is necessary until things until you exit this is . Thanks to Yuri Voinov!
Also fixed an article on libecap3. You do not need to compile anything, but simply install it from the Stretch repository. Thanks to AlucoST for the UPD comment
05/05/18: comrade dartraiden suggests that you can add Tor to the config
This excludes the use of output nodes in these countries.
After reading a bunch of articles, the idea came up to teach Squid to let individual URLs through Tor. Whether to use such a method is up to you. But I will say that after the implementation all the problems that were before that disappeared. Who cares, we go under the cat.
Why is this?
The article is written solely for the purpose of helping those who unlawfully suffer from the total nonsense that is going on in our country. It is also aimed at those who need exactly “transparent” Squid with HTTPS without certificate substitution and the ability to track visits both via HTTP and HTTPS, since the article is essentially a continuation of my other article, since here I propose fixing a long-standing bug that did not allow to see https domain names in the logs and did not allow to use newer versions of Squid. Well, also just for those who are interested in using Squid.
What are the benefits?
- Unlimited scalability
- Relative ease of support and administration
- If this is important, then you can provide anonymous access to the resources indicated in the list (although this is not the topic of the article, anonymity is provided for
- Stability. When using several Tor services (different config files), you can connect them to Squid and get round robin.
- Absolute free. Forever and ever.
I don’t know anything, Squid \ Tor is slow, I’ll go and get VPS with VPN over the hill
Congratulations! You really decided that Roskomnadzor caused you inconvenience, and you still have to pay for it to get out of the situation? OK. Feel free to skip the article and raise the VPN tunnels. By the way, the VPN is successfully blocked. Easy. And in the light of recent events, I will tell you that in the near future no one will be able to use VPS to bypass locks at the law level. And plus to everything, your VPS can also get into the lock "simply because telegram is sitting next to it." Tor will not block, no way. If you configure it with obfs, then never anywhere (the topic is, perhaps, for a separate article, since this obfs is not considered). And how many will need to raise such VPS with VPN? How to serve it? Here, the solution is much easier, reliable, and if desired, quite fast, and also free. So before you mislead other readers,
Tor will block! In China, over there, is already blocked
Not. Not. And again no. In China, Tor works with obfs configured. It works great. There is no way in the World to block Tor. Even China with its capacities, minds and finances failed to do this.
Tor is slow! And if you work through obfs, then generally horror!
Refer to the official documentation and a bunch of articles on the Internet that describe how to make sure that the speed is at a decent level. And again, having set up several copies of Tor in this way with different configs, you can attach them to Squid and get round robin.
So, for starters, a little theory. As we all know, Tor is not an HTTP proxy; it cannot be made a direct peer for the underlying Squid. It provides SOCKS proxying (of course, not only, but this is what we need). In order for us to marry Tor with Squid, we need something that could play the role of a conductor from Tor to Squid and vice versa. And of course, ladies and gentlemen, this is Privoxy. Just the same, he is able to be a direct peer, and send everything further to Tor.
It was, as I said, a bunch of articles were read, but not one suited me. This article came across here , but it didn’t quite suit me, since I do not need a bump. In general, all available articles, almost everything, imply either bump or only http, and in my case I need HTTPS, splice, and transparency. Also saw this andit , but there are completely different approaches. Its pros and cons. I chose for myself a bunch of Squid + Tor.
I've previously written about how to make a transparent Squid with HTTPS proxies without replacing the certificates. And of course, I tried to implement the idea on it. But I was disappointed. HTTP requests went perfectly to TOR, but HTTPS didn’t. The problem is not very well known, and I learned from one of the developers that this is a drawback of older versions of Squid. But during the experiments, a solution was found - Squid 3.5.27, in which this bug was fixed + beautiful domain names in the logs (https), instead of ip addresses. But even here, several disappointments awaited me, which will be discussed below. But everything, as they say, is doped with a file.
So, the source data:
- Debian Stretch (9) x86 (did not try in x64)
- Squid Squid 3.5.23 from the repository
- Fresh Squid sorts from offsite
- Openssl
- Libecap3
- Tor
- Privoxy
- Straight arms and lots of coffee with cookies
Collect Squid pens or put ready-made packages (links below), you decide. If you think that I have
So, prepare for the assembly:
apt-get install fakeroot build-essential devscripts
apt-get build-dep squid3
apt-get install libecap3
apt-get install libecap3-dev
apt-get install libssl1.0-dev
apt-get install libgnutls28-dev
IMPORTANT!
It is very important to install libssl1.0-dev, and not another version, otherwise Squid will either lag, or it will not be collected at all due to incomprehensible errors
Next, we get the sources of Squid 3.5.23
apt-get source squid3
Download this particular archive with Squid sources:
wget -O squid-3.5.27-2018.tar.gz http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.27-20180318-r1330042.tar.gz
We go to the Squid source directory and update the sources to the newly downloaded sources:
cd squid3-3.5.23/
uupdate -v 3.5.27-2018 ../squid-3.5.27-2018.tar.gz
Go to the newly-made catalog with updated sources:
cd ../squid3-3.5.27-2018
Add compilation options to debian / rules:
--enable-ssl \
--enable-ssl-crtd \
--with-opensslTip
By the way, you can cut down options you don’t need, this will speed up compilation
Next, you need to patch the source with this patch:
client_side_request.patch
--- src/client_side_request.cc Thu Aug 18 00:36:42 2016
+++ src/client_side_request.cc Mon Sep 19 04:41:45 2016
@@ -519,20 +519,10 @@
// note the DNS details for the transaction stats.
http->request->recordLookup(dns);
- if (ia != NULL && ia->count > 0) {
- // Is the NAT destination IP in DNS?
- for (int i = 0; i < ia->count; ++i) {
- if (clientConn->local.matchIPAddr(ia->in_addrs[i]) == 0) {
- debugs(85, 3, HERE << "validate IP " << clientConn->local << " possible from Host:");
- http->request->flags.hostVerified = true;
- http->doCallouts();
- return;
- }
- debugs(85, 3, HERE << "validate IP " << clientConn->local << " non-match from Host: IP " << ia->in_addrs[i]);
- }
- }
- debugs(85, 3, HERE << "FAIL: validate IP " << clientConn->local << " possible from Host:");
- hostHeaderVerifyFailed("local IP", "any domain IP");
+ debugs(85, 3, HERE << "validate IP " << clientConn->local << " possible from Host:");
+ http->request->flags.hostVerified = true;
+ http->doCallouts();
+ return;
}
void
What is it for? I will explain. When I wrote the first article about peek and splice, I said that newer versions do not work, and it was, and just this patch fixes the very problem, which was that Squid selectively breaks HTTPS connections, with an interesting message in cache.log:
SECURITY ALERT: Host header forgery detected on ... (local IP does not match any domain IP)The fact is that on one host something resolves to one IP, sometimes to another on the next, on Squid itself to the third, because There is a DNS cache and it is not updated synchronously. Squid does not match the ip domain in its cache (because it updated its cache a little earlier or later) and terminates the connection. It seems like protection, but nowadays it is considered normal (round-robin DNS). Developers are reinsured. And we don’t need it at all! To those who say that this patch may pose a security risk, I will answer that I consulted with Yuri Voinov, who is directly related to the Squid development team, about this patch. There is no threat here!
So, we created the file for the patch, we threw the code, we need to patch it:
patch -p0 -i client_side_request.patch
Next, you must cancel the application of one patch during compilation (otherwise you will get an error that this patch cannot be applied, since it has already been applied). We go to debian / patches / series and comment out there 0003-SQUID-2018_1.patch , putting a # in front of it :
#0003-SQUID-2018_1.patch
Well, then - compilation and assembly of packages!
dpkg-buildpackage -us -uc -ncInstall squid-langpack
apt-get install squid-langpackand install fresh packages
dpkg -i squid-common_3.5.27-2018-1_all.deb
dpkg -i squid_3.5.27-2018-1_i386.deb
dpkg -i squid3_3.5.27-2018-1_all.debIf apt swears on addiction, do
apt-get -f install
Next, you need to turn off Squid from startup (by default, the init file is used, Squid complains about the inaccessibility of the PID file)
systemctl disable squidand create a systemd service in the directory / etc / systemd / system (the service file is in the sources, and is fully copied here)
cat /etc/systemd/system/squid3.service
## Copyright (C) 1996-2018 The Squid Software Foundation and contributors
##
## Squid software is distributed under GPLv2+ license and includes
## contributions from numerous individuals and organizations.
## Please see the COPYING and CONTRIBUTORS files for details.
##
[Unit]
Description=Squid Web Proxy Server
After=network.target
[Service]
Type=simple
ExecStart=/usr/sbin/squid -sYC -N
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
[Install]
WantedBy=multi-user.target
Turn it on
systemctl enable squid3.serviceInstall tor, privoxy
apt-get install tor privoxyI personally did not touch the Tor config at all, but the Privoxy config can be reduced to this view:
listen-address 127.0.0.1:8118
toggle 0
enable-remote-toggle 0
enable-remote-http-toggle 0
enable-edit-actions 0
forward-socks5t / 127.0.0.1:9050 .
max-client-connections 500Almost ready. Let's go to the / etc / squid directory and change something there. Create the pem file needed for splice:
openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout squidCA.pem -out squidCA.pemAnd bring squid.conf to the following form:
acl localnet src 192.168.0.0/24 # Ваша локалка
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl SSL method CONNECT
#Укажем DNS для Squid. Крайне рекомендую использовать одинаковые DNS тут и у клиентов
dns_nameservers 77.88.8.8
# Список доменов, которые нужно пустить через Tor
acl rkn url_regex "/etc/squid/tor_url"
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
icp_access deny all
htcp_access deny all
#прозрачный порт указывается опцией intercept
http_port 192.168.0.1:3128 intercept options=NO_SSLv3:NO_SSLv2
#также нужно указать непрозрачный порт, ибо если захотите вручную указать адрес
#прокси в браузере, указав прозрачный порт, вы получите ошибку доступа, поэтому нужно
#указывать непрозрачный порт в браузере, если конечно такое желание будет, к тому же в логах #сыпятся ошибки о том, что непрохрачный порт не указан=)
http_port 192.168.0.1:3130 options=NO_SSLv3:NO_SSLv2
#и наконец, указываем HTTPS порт с нужными опциями
https_port 192.168.0.1:3129 intercept ssl-bump options=ALL:NO_SSLv3:NO_SSLv2 connection-auth=off cert=/etc/squid/squidCA.pem
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER
#укажем правило со списком блокируемых ресурсов (в файле домены вида .domain.com)
acl blocked ssl::server_name "/etc/squid/blocked_https.txt"
acl step1 at_step SslBump1
ssl_bump peek step1
#терминируем соединение, если клиент заходит на запрещенный ресурс
ssl_bump terminate blocked
ssl_bump splice all
# Никогда не пускать напрямую домены, указанные в списке РКН
never_direct allow rkn
# Указываем прокси, куда отправлять домены из списка, в нашем случае - Privoxy
cache_peer 127.0.0.1 parent 8118 0 no-query no-digest default
cache_peer_access 127.0.0.1 allow rkn
cache_peer_access 127.0.0.1 deny all
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
coredump_dir /var/spool/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
logfile_rotate 4
pid_filename /var/run/squid.pid
The url_regex list looks like this (the list is given as an example!):
zenway\.ru
\.*google\.com
\.*viber\.*
\.amazon\.com
\.fbcdn\.net
\.slack\.*
media\.api\.viber\.com*
static\.viber\.com*
secure\.viber.*
\*.cloudfront\.net
fonts\.gstatic\.com
med-edu\.ru
Comment
You can use non-regex, you can use dstdomain, you can also “screw” lists with illegally blocked subnets (dst)
Read more about this list format in the documentation office. The result did not take long to wait - everything worked, as planned. And it works to this day. Perhaps the article was late, but it will probably come in handy in the future).
I will replenish the topic if possible.
Ready-made Squid packages (.deb x86)
UPD 05/05/18:
HTTPTunnelPort 8118and basically abandon Privoxy. Thanks dartraiden for the comment.
Found a bug. Not recommended HTTPTunnelPort, Privoxy is necessary until things until you exit this is . Thanks to Yuri Voinov!
Also fixed an article on libecap3. You do not need to compile anything, but simply install it from the Stretch repository. Thanks to AlucoST for the UPD comment
05/05/18: comrade dartraiden suggests that you can add Tor to the config
ExcludeExitNodes {ru}, {ua}, {by}This excludes the use of output nodes in these countries.
Many thanks to Yuri Voinov , who helped in solving problems with the work of this bunch!