Cascading Squids

At the time of too expensive anlim (64kbit - 1000rur), we created a proxy cluster with friends in order to increase the total throughput. Time passed, prices changed. Now they are already more friendly - Mbit anlim (with nightly doubling the speed) costs all the same 1000 rubles. But, despite this, we still use the cluster sometimes. I decided to share with the public the method of creating such goodness, if anyone would be interested.
For the experiments we need:

• 1 server with squid installed on it + some other proxy (if you want this server to be not only central, but also share the Internet).
  I’ll talk about raising a cluster based on Debian, 2 Internet channels and 3 squids (squids for parent proxies were chosen for 2 reasons: because of the possibility of providing statistics on current connections; because of laziness to look for an alternative)
• any number of computers, with any OS and any http proxies
• optionally - apache, php and SqStat script - for viewing proxy activity

We will assume that we have 2 cars - 192.168.1.1 (it also has a 2nd ip, with a 2nd provider - 192.168.2.1) and 192.168.1.2
A little google - I found a note without much 5 years ago - Squid 2 chans . 2 points were drawn from it - how, in fact, to organize a cluster and how to proxy on different interfaces.

To begin with - we will create additional squid services.
First, we ’ll fix the server config a bit ( /etc/squid/squid.conf ). Because I don’t need a cache - I chopped it off. This is done by setting the cache_dir parameter as follows:

cache_dir null / var / spool / squid

Because disable the cache - chop off the log and write to the cache

cache_store_log none

Let's open access to the proxy for everyone (you need to insert before the entry 'http_access deny purge'):

acl all src 0.0.0.0/0.0.0.0
http_access allow all

Let's open access to the proxy via the cache_mgr protocol from the localhost (I do not remember point-blank which parameters are specified in the default config - so it won’t hurt to check what is already there before insertion):

acl manager proto cache_object # this line is most likely already in the config
http_access allow manager localhost
http_access deny manager

Listen to port 8080 (by default - 3128):

http_port 8080

To limit access to the proxy - I prefer to use iptables (by default - all packets are discarded) rather than authorization, so the proxy from the config is open to everyone:

iptables -N proxy
iptables -A proxy -j REJECT
iptables -I proxy -s 192.168.1.2 -j ACCEPT
iptables -I INPUT -p tcp --dport 8080 -j proxy

Now add the parent proxies (I say right away - with the configuration below - nothing will work. But then, as the story goes, the error will be found and fixed :)):

cache_peer 127.0.0.1 parent 8081 0 no-query no-digest round-robin weight = 4
cache_peer 127.0.0.1 parent 8082 0 no-query no-digest round-robin weight = 1
cache_peer 192.168.1.2 parent 8080 0 no-query no- digest round-robin weight = 4

Total - 3 parent proxies, from which cache (no-query) and cache-digest (no-digest) will not be requested, parents will cycle through (round-robin). Well, a different weight - because the 2nd channel is weak.
We prohibit a squid to go directly (not through parents):

never_direct allow all

We create 2 copies of the config - /etc/squid/squid_2.conf and /etc/squid/squid_3.conf . We delete from them the lines beginning with cache_peer, never_direct. We change the value of the http_port parameter to 8081 and 8082, respectively. We change the paths to the logs (hereinafter in the text - the changes will be indicated only for squid_2 , for squid_3 - the changes are similar):

access_log /var/log/squid_2/access.log
cache_log /var/log/squid_2/cache.log

It also does not hurt to create this path and change the owner, so that the squid can write logs:

mkdir / var / log / squid_2
chown proxy: proxy / var / log / squid_2

Specify the location of the pid file:

pid_filename /var/run/squid_2.pid

I did not change the cache location - all 3 proxies work quietly with 1 directory. But squid -z (initialize the cache) before the first run does not hurt.

Now let's start creating init scripts. In the /etc/init.d/squid file, we comment on the code responsible for initializing the cache (since the script is not designed for null cache):

# if [-d "$ cdr" -a! -d "$ cdr / 00"]
# then
# log_warning_msg "Creating squid spool directory structure"
# $ DAEMON -z
# fi

Copy /etc/init.d/squid => /etc/init.d/squid_2 .
Change squid_2 :

NAME = squid_2
...
SQUID_ARGS = "- D -sYC -f /etc/squid/squid_2.conf"

Also, for peace of mind, you can change the messages that the script produces:

log_daemon_msg "Starting Squid HTTP proxy" "squid_2"
...
log_daemon_msg "Stopping Squid HTTP proxy" "squid_2"
...
log_daemon_msg "Restarting Squid HTTP proxy" squid_2 "

For the next focus - at the beginning of the script should be approximately the following lines:

# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6

We configure the launch of the newly created service with the system:

update-rc.d squid_2 defaults

Launch child proxies:

invoke-rc.d squid_2 start
invoke-rc.d squid_3 start

Here it is, the moment of truth - we launch the central proxy

invoke-rc.d squid start

and…

FATAL: ERROR: cache_peer 127.0.0.1 specified twice

... and we are brutally cracked.
Having thought a little (hands do not reach google) - we replace in one record 127.0.0.1 => localhost.
We launch, and here it is - happiness.
The squid on the 2nd machine can be configured by copying /etc/squid/squid.conf and deleting references to cache_peer and never_direct from there + to correct access rights via the cache_mgr protocol.
For testing, configure your favorite browser to use proxies and open any page. We look /var/log/squid/access.log , there should be approximately the following lines:

1214499645.364 15335 89.189.176.111 TCP_MISS / 206 214755 GET en.download.nvidia.com/Windows/177.41/177.41_geforce_winxp_64bit_international_whql.exe - ROUNDROBIN_PARENT / 192.168.1.2 application / octet-stream
1214476116.1.189.138.148.1481 891 891 894.18 898.18.1 898.18 898.18 898.18.1 898.18.1 898.1 898.1 8 thosegy its 891.1481 489.68.148.148.18.138 then 481 then then. .nvidia.com / Windows / 177.41 / 177.41_geforce_winvista_64bit_international_whql.exe - ROUNDROBIN_PARENT / 127.0.0.1 application / octet-stream
1214499650.695 3564 89.189.176.111 TCP_MISS / 206 370947 GET ru.download.nvidge_ 1 / 41_ln_v1_1_4141_win_sht - ROUNDROBIN_PARENT / localhost application / octet-stream
1214499658.899 52092 89.189.176.111 TCP_MISS / 206 1115575 GETen.download.nvidia.com/Windows/177.41/177.41_geforce_winvista_64bit_international_whql.exe - ROUNDROBIN_PARENT / 192.168.1.2 application / octet-stream

Next up is SqStat. Download the archive , unpack it into any www-folder, rename config.inc.php.defaults => config.inc.php and edit:

$ squidhost [0] = "localhost";
$ squidport [0] = 8080;
$ cachemgr_passwd [0] = "";
$ resolveip [0] = false; # for beauty - you can also set true :)
$ hosts_file [0] = "hosts.txt";
$ group_by [0] = "host";

$ squidhost [] = "localhost";
$ squidport [] = 8081;
$ cachemgr_passwd [] = "";
$ resolveip [] = false;
$ hosts_file [] = "hosts.txt";
$ group_by [] = "host";

$ squidhost [] = "localhost";
$ squidport [] = 8082;
$ cachemgr_passwd [] = "";
$ resolveip [] = false;
$ hosts_file [] = "hosts.txt";
$ group_by [] = "host";

$ squidhost [] = "localhost";
$ squidport [] = 8081;
$ cachemgr_passwd [] = "";
$ resolveip [] = false;
$ hosts_file [] = "hosts.txt";
$ group_by [] = "host";

$ squidhost [] = "192.168.1.2";
$ squidport [] = 8080;
$ cachemgr_passwd [] = "";
$ resolveip [] = true;
$ hosts_file [] = "hosts.txt";
$ group_by [] = "host";

Open browser sqstat.php and observe the activity.
If anyone is interested - I can lay out I post a slightly modified version - additionally displays the total speed \ volume by the grouping parameter - http://narod.ru/disk/26137171001/sqstat.class.php.bz2.html , there is only 1 file - sqstat.class.php , so you need to unpack the archive into the folder with the original sqstat.

In passive mode, 3 squids consume 2.3% of the memory, which in terms of MB is 14.72.
PS # squid -v
Squid Cache: Version 2.6.STABLE5

UPD: nevertheless, I hope they understand that when downloading to 1 stream through such a proxy, there will be no increase in speed? to increase the speed it is necessary to start the download with n threads, where n is the number of proxies in the cluster (if the weights are the same).