Combined load balancing of Internet channels

Prehistory

Sooner or later, the system administrator is faced with the need to distribute traffic across several channels, while naturally wanting each channel to be used to the maximum. Faced with a similar need, and deciding not to reinvent the wheel, he turned to the help of search engines. Since I have a server on Ubuntu, I turned my attention to the article http://help.ubuntu.ru/wiki/ip_balancing. Implemented “Method 1”, but during the test the following critical problems were noticed: when using links on some sites they did not open (for example, when trying to turn on music on the VKontakte resource). The reason is obvious - the request went through another channel. After thinking about the situation, I decided to combine an approach to balancing. The logic is simple - most of all the traffic eats torrents and similar programs, so we share the traffic. As a result, we distribute traffic with ports up to 11000 approximately evenly in terms of the number of subscribers — subnets, and traffic with ports 11000-60000 equalizes the load on the channels.

Settings

It is assumed that routing tables have been created for each of the channels, let's call them chan1 , chan2 , chan3 - three channels, respectively.
Somewhere, for example, in /etc/rc.local we add something like:

ip rule add prio 101 fwmark 1 table chan1
ip rule add prio 102 fwmark 2 table chan2
ip rule add prio 103 fwmark 4 table chan3

Create the script /etc/rc.balance:

#!/bin/bash
lst='/etc/rc.balance.lst'
########### Flushing ##################
/sbin/iptables -t mangle -F PREROUTING
/sbin/iptables -t mangle -F POSTROUTING
/sbin/iptables -t mangle -F OUTPUT
#######################################
/etc/rc.baltor
/sbin/iptables -t mangle -A PREROUTING -d 172.16.0.0/16 -j RETURN
/sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -m state --state INVALID -j DROP
while read net mark
do
    /sbin/iptables -t mangle -A PREROUTING -s $net -m state --state new,related -j CONNMARK --set-mark $mark
done<$lst
/sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -p udp --sport 11000:60000 --dport 11000:60000 -m state --state new,related -j BALANCE
/sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -p tcp --sport 11000:60000 --dport 11000:60000 -m state --state new,related -j BALANCE
/sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -j CONNMARK --restore-mark
exit 0

We create a list of distribution of grids by channels (in the second column - the mark):

172.16.0.0/22		1
172.16.4.0/22		2
172.16.8.0/22		4

Script /etc/rc.baltor - balancing rules:

#!/bin/bash
/sbin/iptables -t mangle -F BALANCE
lst='/etc/rc.cnload.lst'
mrk=1
while read kld
do
    /sbin/iptables -t mangle -A BALANCE -j CONNMARK --set-mark $mrk
    /sbin/iptables -t mangle -A BALANCE -m statistic --mode random --probability 0.$kld -j RETURN
    mrk=`expr $mrk \* 2`
done < $lst
/sbin/iptables -t mangle -A BALANCE -j CONNMARK --set-mark $mrk
exit 0

Script /etc/rc.cnload - calculation of probability depending on the channel load:

#!/bin/bash
cn1=800000 # ширина первого канала в килобитах в секунду
cn2=600000 # второго ..
cn3=400000 # и третьего
if1='eth1' # интерфейс первого канала
if2='eth2' # второго
if3='eth3' # третьего
lst='/etc/rc.cnload.lst'
a1=`ifconfig $if1 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'`
a2=`ifconfig $if2 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'`
a3=`ifconfig $if3 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'`
sleep 20
b1=`ifconfig $if1 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'`
b2=`ifconfig $if2 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'`
b3=`ifconfig $if3 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'`
c1=`expr \( $b1 - $a1 \) \* 8 / 20000`
c2=`expr \( $b2 - $a2 \) \* 8 / 20000`
c3=`expr \( $b3 - $a3 \) \* 8 / 20000`
d1=`expr \( $cn1 - $c1 \) \* 100 / $cn1`
d2=`expr \( $cn2 - $c2 \) \* 100 / $cn2`
d3=`expr \( $cn3 - $c3 \) \* 100 / $cn3`
# выполняем корректировку при загрузке одного из каналов более 60%
if [ $d1 -lt "40" -o $d2 -lt "40" -o $d3 -lt "40" ]
then
    e1=`expr 100 \* $d1 / \( $d1 + $d2 + $d3 \)`
    e2=`expr 100 \* $d2 / \( $d2 + $d3 \)`
    f1=`head -n 1 $lst | tail -n 1`
    f2=`head -n 2 $lst | tail -n 1`
    # корректировка если коэффициенты изменились 
    if [ $e1 -ne $f1 -a $e2 -ne $f2 ]
    then
        echo $e1 > $lst
        echo $e2 >> $lst
        /etc/rc.baltor
    fi
fi
exit 0

Add to /etc/rc.local

/etc/rc.balance

and in / etc / crontab

*/1 *	* * *	root    /etc/rc.cnload

It is important that at the time of start a file with the coefficients /etc/rc.cnload.lst already exists, it can be compiled by running the script /etc/rc.cnload.

Conclusion

This method has been successfully implemented in a network with 8000 subscribers. In addition to balancing, dynamic shaping is used, but this is a topic for another article.

All balance in everything.