Gentle balancing between multiple providers on an office gateway

  • Tutorial
This article describes the configuration of a Linux-based gateway to balance traffic between channels of different providers.


The result achieved in this manual differs from the result of similar manuals : the same external IP address is used for each client, which eliminates problems with Internet services that are not ready to change the client's IP address in one session.

Problems


For an ordinary consumer who does not have any own block of addresses and does not participate in the exchange of routes at the level of telecom operators, access to the Internet is possible only from those addresses that each Internet provider provides him with for each channel. This means that any connection of several channels from "home" Internet providers to one node requires not only a choice between different upstream gateways, but also the choice of the correct outgoing address to communicate with them.

This state of affairs runs counter to the way the standard routing mechanism works. In general, the route in the routing table depends only on the destination address and does not change the sender address. Therefore, the first task that needs to be solved is to arrange the correspondence of the sending interface, gateway and outgoing address when forwarding packets from clients.

The second task that needs to be solved is how to reasonably distribute the load from customers.

Balancing using multipath routes (as in the link at the beginning of the article) is carried out at the destination address and does not always look like an attractive solution. Where it would be worth having the same external address, a client in one session can get a different one and vice versa - when accessing popular resources, all clients will use one common channel.

Balancing at the sender address can compromise between compatibility and granularity of distribution. Each computer in the local network will always have the same external address, and with a considerable number of computers in the network, we will get a decent load distribution over the channels. This division can be achieved if hashes are formed from local addresses, the entire hash space is divided in proportion to the channel weights, and the value of this hash is used to select a channel.

Instruments


The Linux kernel has the ability to use several different tables simultaneously, depending on what criteria the package meets. Often this mechanism is called policy based routing (PBR). This mechanism is controlled through a set of iproute2 utilities.

Simplifying the statement of the problem somewhat, we can say: all that is required of us to distinguish routes, interfaces and outgoing addresses is to create an additional routing table for each provider, which looks as if this channel of this provider were the only one. Then you need to add the rules (the same routing policies) that float the appropriate routing table for the corresponding traffic.

As for sender balancing, you can use the netfilter (iptables) network filter for this. Using the HMARK action, we mark the packet with a hash from the address. By the fwmark criterion in the routing rules (ip rule command), we will forward the packet to the desired routing table. iproute2 and iptables play well in pairs here.

Example


As an example, I’ll take a Debian-based gateway that provides Internet access through three channels from two different providers. Such an example was specially chosen, since it considers the delicate situation when the same superior gateway of one provider is accessible through 2 different wires. The configuration described will be similar in all other debian-like operating systems, including Ubuntu.

The gateway has 4 working interfaces:
Int.DescriptionCapacityAddressGateway
eth0the local network10.0.0.1/16-
eth1first channel of the first provider100 Mbps100.1.1.92/24100.1.1.1
eth2the only channel from the second provider80 Mbps200.2.2.22/24200.2.2.1
eth3second channel of the first provider100 Mbps100.1.1.93/24100.1.1.1
Each channel is connected via regular ethernet with a static address.

Initial state

Suppose we already have access configured through one channel. The file / etc / network / interfaces has the form:
/etc/network.interfaces, 3 interfaces, 1 main
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.0.0

auto eth1
iface eth1 inet static
address 100.1.1.92
gateway 100.1.1.1
netmask 255.255.255.0

auto eth2
iface eth2 inet static
address 200.2.2.22
# gateway 200.2.2.1 #correct gateway value, but commented out to avoid routing conflicts
netmask 255.255.255.0

auto eth3
iface eth3 inet dhcp
address 100.1.1.93
# gateway 100.1.1.1 # correct gateway value, but commented out to avoid routing conflicts
netmask 255.255.255.0

As you can see, only one of the interfaces has a default route.

In /etc/sysctl.conf, the value of net.ipv4.ip_forward is set to 1. The iptables-persistent package is installed and the contents of the /etc/iptables/rules.v4 file are as follows:
/etc/iptables/rules.v4, NAT rules and nothing more
# Generated by iptables-save v1.4.21 on Tue Feb 30 13:14:06 2016
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Tue Feb 30 13:14:06 2016
# Generated by iptables-save v1.4.21 on Tue Feb 30 13:14:06 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -s 10.0.0.0/16 -i eth0 -j ACCEPT # Proposed by ValdikSS due to security clues
-A FORWARD -d 10.0.0.0/16 -o eth0 -j ACCEPT # --//--
-A FORWARD -j DROP # --//--
COMMIT
# Completed on Tue Feb 30 13:14:06 2016
# Generated by iptables-save v1.4.21 on Tue Feb 30 13:14:06 2016
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE
-A POSTROUTING -s 10.0.0.0/8 -o eth2 -j MASQUERADE
-A POSTROUTING -s 10.0.0.0/8 -o eth3 -j MASQUERADE
COMMIT
# Completed on Tue Feb 30 13:14:06 2016

There are only three rules for translating addresses of packets coming out through three external interfaces. It’s not important to use a masquerade, but I settled on it. Since the gateway is currently configured on only one interface, in fact, only the first rule works.

Routing rules

Now the most difficult: we will solve the first problem of matching addresses, interfaces and gateways. We will prepare three additional routing tables, which we will use for different providers.
Let's start with the optional step - in order not to mention routing tables everywhere by their number, we will put them in the file of correspondence of numbers and table names / etc / iproute2 / rt_tables:
/ etc / iproute2 / rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep

10 Provider1_Cable1
20 Provider2
30 Provider1_Cable2

Pretty simple for now. Now fill these tables with routes. Let each table for each external channel contain all routes of all interfaces, except routes through other external interfaces. When raising each external interface, add a rule that instructs to use the agreed additional routing table if the sender address is equal to the address on this interface.
/ etc / network / interfaces: routes in additional tables and routing rules for the source address
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback
up ip route add 127.0.0.0/8 dev lo table Provider1_Cable1
up ip route add 127.0.0.0/8 dev lo table Provider2
up ip route add 127.0.0.0/8 dev lo table Provider1_Cable2

auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.0.0
up ip route add 10.0.0.0/16 dev eth0 table Provider1_Cable1
up ip route add 10.0.0.0/16 dev eth0 table Provider2
up ip route add 10.0.0.0/16 dev eth0 table Provider1_Cable2

auto eth1
iface eth1 inet static
address 100.1.1.92
gateway 100.1.1.1
netmask 255.255.255.0
up ip route add 100.1.1.0/24 dev eth1 table Provider1_Cable1
up ip route add default dev eth1 via 100.1.1.1 table Provider1_Cable1
up ip rule add from 100.1.1.92 table Provider1_Cable1

auto eth2
iface eth2 inet static
address 200.2.2.22
# gateway 200.2.2.1 #correct gateway value, but commented out to avoid routing conflicts
netmask 255.255.255.0
up ip route add 200.2.2.22/24 dev eth2 table Provider2
up ip route add default dev eth2 via 200.2.2.1 table Provider2
up ip rule add from 200.2.2.22 table Provider2

auto eth3
iface eth3 inet dhcp
address 100.1.1.93
# gateway 100.1.1.1 # correct gateway value, but commented out to avoid routing conflicts
netmask 255.255.255.0
up ip route add 100.1.1.0/24 dev eth3 table Provider1_Cable2
up ip route add default dev eth3 via 100.1.1.1 table Provider1_Cable2
up ip rule add from 100.1.1.93 table Provider1_Cable2

Please note that only one gateway gets into the main routing table anyway - all but one of the gateway directives are commented out.
Here additional routes and rules are issued by up directives, which simply execute the command when raising the interface. Commands for adding routes and rules are grouped under the interfaces through which they can be implemented - this is most clearly seen in the example of the lo interface.

After rebooting the network, we can notice how the output of the list of rules has changed in the ip rule list command:
ip rule list
# ip ru li
0: from all lookup local
32763: from 100.1.1.93 lookup Provider1_Cable2
32764: from 200.2.2.22 lookup Provider2
32765: from 100.1.1.92 lookup Provider1_Cable1
32766: from all lookup main
32767: from all lookup default

The corresponding table can be viewed with the ip ro li table XXX command.

Already in this position, the server is ready to use all the interfaces at once. We will verify this by trying to use all the interfaces in turn and checking the address observed by the outside:
checking external addresses through different interfaces
# curl --interface 100.1.1.92 http://canihazip.com/s ; echo
100.1.1.92
# curl --interface 100.1.1.93 http://canihazip.com/s ; echo
100.1.1.93
# curl --interface 200.2.2.22 http://canihazip.com/s ; echo
200.2.2.22

If there are real “white” addresses on the external interfaces, then you can try to join the server using any address — the server will communicate through each of them through the correct channel and the correct provider. If so, then the hardest part is behind.

Balancing

In my example, there are two channels of 100 Mb / s and one 80 Mb / s. In order to divide the load equally between them, it is enough for me to divide it into 14 parts and send 5 parts to two 100-megabit channels and 4 parts to an 80-megabit channel. Add a rule to the firewall that marks the packet with numbers from 10,000 to 10013, depending on the outgoing IP address: Simple enough. Parameters and values ​​speak for themselves. Now it remains to direct traffic marked with different numbers through different interfaces. As a result, / etc / network / interfaces takes the form:
# iptables -t mangle -A PREROUTING -s 10.0.0.0/16 -j HMARK --hmark-tuple src --hmark-offset 10000 --hmark-mod 14 --hmark-rnd 0xfeedcafe
# /etc/init.d/netfilter-persistent save


/ etc / network / interfaces final
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback
up ip route add 127.0.0.0/8 dev lo table Provider1_Cable1
up ip route add 127.0.0.0/8 dev lo table Provider2
up ip route add 127.0.0.0/8 dev lo table Provider1_Cable2

auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.0.0
up ip route add 10.0.0.0/16 dev eth0 table Provider1_Cable1
up ip route add 10.0.0.0/16 dev eth0 table Provider2
up ip route add 10.0.0.0/16 dev eth0 table Provider1_Cable2

auto eth1
iface eth1 inet static
address 100.1.1.92
gateway 100.1.1.1
netmask 255.255.255.0
up ip route add 100.1.1.0/24 dev eth1 table Provider1_Cable1
up ip route add default dev eth1 via 100.1.1.1 table Provider1_Cable1
up ip rule add from 100.1.1.92 table Provider1_Cable1
up ip rule add from 10.0.0.0/8 fwmark 10000 table Provider1_Cable1
up ip rule add from 10.0.0.0/8 fwmark 10003 table Provider1_Cable1
up ip rule add from 10.0.0.0/8 fwmark 10006 table Provider1_Cable1
up ip rule add from 10.0.0.0/8 fwmark 10009 table Provider1_Cable1
up ip rule add from 10.0.0.0/8 fwmark 10012 table Provider1_Cable1

auto eth2
iface eth2 inet static
address 200.2.2.22
# gateway 200.2.2.1 #correct gateway value, but commented out to avoid routing conflicts
netmask 255.255.255.0
up ip route add 200.2.2.22/24 dev eth2 table Provider2
up ip route add default dev eth2 via 200.2.2.1 table Provider2
up ip rule add from 200.2.2.22 table Provider2
up ip rule add from 10.0.0.0/8 fwmark 10002 table Provider2
up ip rule add from 10.0.0.0/8 fwmark 10005 table Provider2
up ip rule add from 10.0.0.0/8 fwmark 10008 table Provider2
up ip rule add from 10.0.0.0/8 fwmark 10011 table Provider2

auto eth3
iface eth3 inet dhcp
address 100.1.1.93
# gateway 100.1.1.1 # correct gateway value, but commented out to avoid routing conflicts
netmask 255.255.255.0
up ip route add 100.1.1.0/24 dev eth3 table Provider1_Cable2
up ip route add default dev eth3 via 100.1.1.1 table Provider1_Cable2
up ip rule add from 100.1.1.93 table Provider1_Cable2
up ip rule add from 10.0.0.0/8 fwmark 10001 table Provider1_Cable2
up ip rule add from 10.0.0.0/8 fwmark 10004 table Provider1_Cable2
up ip rule add from 10.0.0.0/8 fwmark 10007 table Provider1_Cable2
up ip rule add from 10.0.0.0/8 fwmark 10010 table Provider1_Cable2
up ip rule add from 10.0.0.0/8 fwmark 10013 table Provider1_Cable2

For each interface, lines of the type “up ip rule add from 10.0.0.0/8 fwmark MARK table TABLE” were added. Each of them sends packets with the corresponding marking for routing to the specified table. In my example, numbers are mixed between interfaces for the sake of uniformity.

All!


Optionally, you can add scripts to such a routing scheme that detect a drop in one of the channels and rebalance the load with this in mind. However, implementation details depend on taste and it is better to leave this to the reader.

Also popular now: