August 4, 2010 at 04:29Routing setup for home multihomed server
- Tutorial
Now, the presence of several Internet connections on one, including the home server, is not uncommon. City lokalka, ADSL, 3G modems ... Add home local and external virtual (VPN) networks to this network, and get a vigorous mixture of interfaces between which you need to route traffic, balance traffic between different channels on the Internet (when they are), and switch from non-working channels for workers (when they fall off).
Judging by the posts on the internet, most people who come across this situation have a very poor idea of how this is set up. It should be noted that in linuh really managing routing is very complicated and confusing - a consequence of evolutionary development and support for (partial) compatibility. I want to describe the principles of configuring multihomed routingservers on a specific, rather complicated example: on the server there are three physical network interfaces (one to the home LAN and two to the ADSL modems), two ADSL connections (ADSL modems in bridge mode, so pppd lifts the same server) to different providers (one with static IP, the other with dynamic), plus a VPN to the company’s server - a total of six interfaces.
The topic is quite complicated, so understanding the material will require at least a minimal understanding of the routing (what is the default route and gateway), firewall (packet labeling, connection tracking, the connection between different tables and firewall chains and routing), pppd (ip-up / scripts ip-down) and IP and TCP protocols.
So, we have three network interfaces:
To configure routing earlier (and now - in simple cases), the route command was used. You can forget about it - such settings are made only through the ip command (from the iproute2 package), and the simultaneous use of both ip and route will bring nothing but trouble. At the time of the route command, the routing table was one. Now there are several routing tables (and one of them, main, is the same with which the route command works). These tables are listed in / etc / iproute2 / rt_tables:
You can view their contents with the ip route list table <table name> command . There is nothing interesting in the local table - there are link-local routing rules through the available interfaces; in the table main, the basic rules of routing; the default table is empty by default; the unspec table (which can also be accessed as all ) displays all routing rules from all existing tables.
To create new tables, you need to add their names to this file (more precisely, this is only necessary to access your tables by name - you can also refer to them by their numbers).
As soon as we had more than one routing table, we also needed a mechanism by which it is possible to select the routing table to be used - it is called routing rules (RPDB, routing policy database) . The main advantage of this mechanism is the ability to select routing not only based on the destination address (as the route command did and does ip route ... ), but also by other criteria (source address, interface, tos, fwmark, ... fields). It works like this: you specify (via the ip rule command ...) any number of rules in the format “criteria-> routing table”. If for a given packet the criteria in the routing rules match, and in the specified routing table there is a route for this packet (based on the destination address), then it will be executed; and if not, then we return to the routing rules and check other options. Here's what the default routing rules look like:
The first column is priority; rules are scanned in order of increasing priority. These rules mean that you first look at the local table (where link-local are the routing rules), then the main table (which you usually controlled through the route command ) and where the default route is usually specified - i.e. on this, the execution of routing rules stops, then (if the default route was not specified in main ) the default table (empty by default).
Here is a simple example of how all this economy can be used so that packets from the home LAN address 192.168.2.100 are processed using separate routing rules that are different from the default routing (we will give it access only to the first ADSL modem and to the Internet through the first provider ):
Now a little about the firewall. With it, it is also possible to influence routing.
As you can see in the diagram, packets from, for example, local applications go through the “Routing Decision” stage twice. As far as I understand, the first time this happens at the time the packet is created, in order to determine the outgoing interface and the outgoing IP for it (which can check the firewall rules, so that they should already be known when the packet gets into the firewall), plus rerouting for of this packet is calculated after executing the MANGLE chain of the firewall, which could change any fields of the packet, which could affect the routing of this packet.
In our case, the easiest way to influence routing through a firewall is to change the packet fwmark field in the MANGLE chain, and then use the routing rules to select a routing table based on the value of the packet fwmark field (for this you need to create several routing tables with different rules - for example, in register default route in one table through one provider, and in another - through another provider).
The path to this end result is thorny in places (bugs of the pppd and openvpn features do not let you relax) and therefore will be described below. In the meantime, let's see what we need to get.
To begin with, we will configure the simultaneous use of both providers, with load balancing in the ratio of 8 to 2 (respectively, the channel width). This method selects a routing for each individual destination address - i.e. if the connection initially went through the first provider, then all of its packets will only go through the first provider, and all subsequent connections to this address for some time will only go through this provider. Balancing not by connections but by a package is done differently, and will work only if we have several connections to one provider.
Because Since the kernel caches information about the relationship between destination addresses and the default routing selected for them, you may need to reset this cache (so that the kernel again, with a probability of 8 to 2, selects the default routing for this destination address):
Because in some cases, we need to control through which of the external channels to send the packet, we will create a separate routing table for each channel with its default route:
Now we will add routing rules so that using the firewall, setting the desired fwmark values, we can select the desired routing table:
A value of 0x4 / 0x4 (value / mask) means that bits from the 0w mask (that is, only the third bit) will be taken from the fwmark field, which must match the value 0x4. Those. in fwmark, the third bit must be set so that the rule for the vpn table works, and the remaining bits do not matter. This approach allows you to set several bits in the fwmark in the firewall, thus indicating several “suitable” channels for this package. The priority of these channels (if they are all available) is determined by the priority of the routing rules. If a channel is unavailable, the corresponding routing table will still be viewed, but it will not have a default route, so you will return to the following routing rules and the next suitable channel will be found.
All of these commands have not affected anything yet - the fwmark firewall does not expose, so the routing rules for the tables isp1 , isp2 and vpn do not work, and by default, routing is determined by the main table as before :
Now we can indicate that access to the website with statistics and the control panel of the second provider should go through the second ADSL:
If it is necessary that these rules work not only for outgoing packets from our server, but also for packets from home LAN, then you need to add all these rules to the firewall not only in the OUTPUT chain, but also in the FORWARD chain.
All this will work fine as long as we are talking only about outgoing connections. But we have a server ... and there are incoming connections to the server. Let's say we have a website raised on static IP ssss (channel of the first provider). Now, if the admin of the second provider from the serverhe wants to look (through lynx) on our website, nothing will come of it! The fact is that he will send a request to our IP ssss (on which the website) through the channel of our first provider, and our server will send a response to this request through the channel of the second provider (and, accordingly, from the dddd address - we have, of course, the SNAT / MASQUERADE of packets coming out through the ppp0 , ppp1 and tun0 interfaces ) is configured - according to the firewall settings that we just made: packets to the addresssend through the channel of the second provider. The admin web browser that sent the request to the ssss address does not expect to receive a response from the dddd address, so this will not work.
To solve this problem, it is necessary to guarantee responses to incoming connections from the interface from which packets were originally received. To do this, unfortunately, you will have to use connection tracking with a firewall (conntrack) to set fwmark (connmark). Unfortunately - because conntrack is a big complicated and buggy garbage, but there is no choice. This is done as follows (these rules must be added to the firewall first, BEFORE the rules setting fwmark like described above):
Here we mark incoming packets (more precisely, the connections to which they belong) according to the interfaces from which they came. Moreover, the fwmark value (more precisely, connmark) is chosen so that it corresponds to the routing table through which you need to send responses to these packets (1,2,4 - for isp1, isp2 and vpn; 8 - this is not used in fwmark routing rules, t .e. packets with this fwmark will be sent according to the usual routing in the main table - this is quite suitable for packets from local networks).
This labeling does not affect the routing of incoming packets in any way, but this allows us to actually bind this labeling to all packets of this connection, including outgoing packets. Further, for outgoing packets on the same connection (responses to incoming packets), we set the fwmark value to the same value that the incoming packets had and which is tied to this connection (more precisely, copy the connmark value to fwmark). And if, as a result, the outgoing packets are set to fwmark (not 0), then all the following fwmark installation rules cannot be used to select the outgoing channel - this packet must be sent through the same interface through which the incoming packet arrived, so stop (-j ACCEPT )
In total, the correct firewall configuration option according to our problem statement looks like this:
Together with the rules and routing table settings, we get exactly what we needed. It remains to figure out how to set these settings in dynamics, taking into account the fact that channels can fall and rise.
Honestly, the article turned out to be rather big, so I will not clutter it with my ip-up / down scripts as I was going to. If someone asks in the comments - add to the article later. And now I will briefly describe the main tasks and problems that arose during the implementation of the above settings.
Judging by the posts on the internet, most people who come across this situation have a very poor idea of how this is set up. It should be noted that in linuh really managing routing is very complicated and confusing - a consequence of evolutionary development and support for (partial) compatibility. I want to describe the principles of configuring multihomed routingservers on a specific, rather complicated example: on the server there are three physical network interfaces (one to the home LAN and two to the ADSL modems), two ADSL connections (ADSL modems in bridge mode, so pppd lifts the same server) to different providers (one with static IP, the other with dynamic), plus a VPN to the company’s server - a total of six interfaces.
The topic is quite complicated, so understanding the material will require at least a minimal understanding of the routing (what is the default route and gateway), firewall (packet labeling, connection tracking, the connection between different tables and firewall chains and routing), pppd (ip-up / scripts ip-down) and IP and TCP protocols.
General configuration and problem statement
So, we have three network interfaces:
- eth0 (192.168.0.2), connected to the first ADSL modem (192.168.0.1)
- eth1 (192.168.1.2), connected to the second ADSL modem (192.168.1.1)
- eth2 (192.168.2.1), is connected to the home LAN (192.168.2.x) and is an Internet gateway for this LAN
- ppp0 (ssss), static IP through the first ADSL modem to the first provider (8Mbps)
- ppp1 (dddd), dynamic IP through the second ADSL modem to the second provider (2Mbps)
- tun0 (vvvv), static IP via VPN to the company server
- доступ с сервера и домашней локалки в инет при наличии любого из ADSL соединений
- балансирование трафика между ADSL соединениями если доступны оба, в соответствии с шириной каналов
- VPN должен быть поднят через любое ADSL соединение, а если доступны оба, то через первого провайдера (канал шире)
- почта должна отправляться через первого провайдера (там статический IP привязанный к моему домену)
- доступ к веб-сайту со статистикой и контрольной панелью второго провайдера должен идти через второй ADSL (провайдер закрыл доступ к своему сайту снаружи)
Немного теории
To configure routing earlier (and now - in simple cases), the route command was used. You can forget about it - such settings are made only through the ip command (from the iproute2 package), and the simultaneous use of both ip and route will bring nothing but trouble. At the time of the route command, the routing table was one. Now there are several routing tables (and one of them, main, is the same with which the route command works). These tables are listed in / etc / iproute2 / rt_tables:
# cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
You can view their contents with the ip route list table <table name> command . There is nothing interesting in the local table - there are link-local routing rules through the available interfaces; in the table main, the basic rules of routing; the default table is empty by default; the unspec table (which can also be accessed as all ) displays all routing rules from all existing tables.
To create new tables, you need to add their names to this file (more precisely, this is only necessary to access your tables by name - you can also refer to them by their numbers).
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
127.0.0.0 127.0.0.1 255.0.0.0 UG 0 0 0 lo
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
# ip route list table main
127.0.0.0/8 via 127.0.0.1 dev lo scope link
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.1
As soon as we had more than one routing table, we also needed a mechanism by which it is possible to select the routing table to be used - it is called routing rules (RPDB, routing policy database) . The main advantage of this mechanism is the ability to select routing not only based on the destination address (as the route command did and does ip route ... ), but also by other criteria (source address, interface, tos, fwmark, ... fields). It works like this: you specify (via the ip rule command ...) any number of rules in the format “criteria-> routing table”. If for a given packet the criteria in the routing rules match, and in the specified routing table there is a route for this packet (based on the destination address), then it will be executed; and if not, then we return to the routing rules and check other options. Here's what the default routing rules look like:
# ip rule list
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
The first column is priority; rules are scanned in order of increasing priority. These rules mean that you first look at the local table (where link-local are the routing rules), then the main table (which you usually controlled through the route command ) and where the default route is usually specified - i.e. on this, the execution of routing rules stops, then (if the default route was not specified in main ) the default table (empty by default).
Here is a simple example of how all this economy can be used so that packets from the home LAN address 192.168.2.100 are processed using separate routing rules that are different from the default routing (we will give it access only to the first ADSL modem and to the Internet through the first provider ):
# echo '100 local100' >> /etc/iproute2/rt_tables
# ip route add 192.168.0.0/24 dev eth0 table local100
# ip route add default via <шлюз_первого_провайдера> dev ppp0 table local100
# ip rule add from 192.168.2.100 lookup local100 priority 5
# ip route list table local100
192.168.0.0/24 dev eth0 scope link
default via <шлюз_первого_провайдера> dev ppp0
# ip rule list
0: from all lookup local
5: from 192.168.2.100 lookup local100
32766: from all lookup main
32767: from all lookup default
Now a little about the firewall. With it, it is also possible to influence routing.
As you can see in the diagram, packets from, for example, local applications go through the “Routing Decision” stage twice. As far as I understand, the first time this happens at the time the packet is created, in order to determine the outgoing interface and the outgoing IP for it (which can check the firewall rules, so that they should already be known when the packet gets into the firewall), plus rerouting for of this packet is calculated after executing the MANGLE chain of the firewall, which could change any fields of the packet, which could affect the routing of this packet.
In our case, the easiest way to influence routing through a firewall is to change the packet fwmark field in the MANGLE chain, and then use the routing rules to select a routing table based on the value of the packet fwmark field (for this you need to create several routing tables with different rules - for example, in register default route in one table through one provider, and in another - through another provider).
Routing setup: final result
The path to this end result is thorny in places (
Balancing between two providers
To begin with, we will configure the simultaneous use of both providers, with load balancing in the ratio of 8 to 2 (respectively, the channel width). This method selects a routing for each individual destination address - i.e. if the connection initially went through the first provider, then all of its packets will only go through the first provider, and all subsequent connections to this address for some time will only go through this provider. Balancing not by connections but by a package is done differently, and will work only if we have several connections to one provider.
# ip route replace default scope global \
nexthop via <шлюз_isp1> dev ppp0 weight 8 \
nexthop via <шлюз_isp2> dev ppp1 weight 2
Because Since the kernel caches information about the relationship between destination addresses and the default routing selected for them, you may need to reset this cache (so that the kernel again, with a probability of 8 to 2, selects the default routing for this destination address):
# ip route flush cache
Manual channel selection for specific connections
Because in some cases, we need to control through which of the external channels to send the packet, we will create a separate routing table for each channel with its default route:
# echo '1 isp1' >> /etc/iproute2/rt_tables
# echo '2 isp2' >> /etc/iproute2/rt_tables
# echo '3 vpn' >> /etc/iproute2/rt_tables
# ip route add default via <шлюз_isp1> dev ppp0 table isp1
# ip route add default via <шлюз_isp2> dev ppp1 table isp2
# ip route add default via <шлюз_vpn> dev tun0 table vpn
Now we will add routing rules so that using the firewall, setting the desired fwmark values, we can select the desired routing table:
# ip rule add priority 100 fwmark 0x4/0x4 lookup vpn
# ip rule add priority 101 fwmark 0x1/0x1 lookup isp1
# ip rule add priority 102 fwmark 0x2/0x2 lookup isp2
A value of 0x4 / 0x4 (value / mask) means that bits from the 0w mask (that is, only the third bit) will be taken from the fwmark field, which must match the value 0x4. Those. in fwmark, the third bit must be set so that the rule for the vpn table works, and the remaining bits do not matter. This approach allows you to set several bits in the fwmark in the firewall, thus indicating several “suitable” channels for this package. The priority of these channels (if they are all available) is determined by the priority of the routing rules. If a channel is unavailable, the corresponding routing table will still be viewed, but it will not have a default route, so you will return to the following routing rules and the next suitable channel will be found.
All of these commands have not affected anything yet - the fwmark firewall does not expose, so the routing rules for the tables isp1 , isp2 and vpn do not work, and by default, routing is determined by the main table as before :
# ip route list ### "table main" можно не указывать
<шлюз_vpn> dev tun0 proto kernel scope link src
<шлюз_isp1> dev ppp0 proto kernel scope link src
<шлюз_isp2> dev ppp1 proto kernel scope link src
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.1
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.2
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2
127.0.0.0/8 via 127.0.0.1 dev lo scope link
default
nexthop via <шлюз_isp1> dev ppp0 weight 8
nexthop via <шлюз_isp2> dev ppp1 weight 2
# ip route list table isp1
default via <шлюз_isp1> dev ppp0
# ip route list table isp2
default via <шлюз_isp2> dev ppp1
# ip route list table vpn
default via <шлюз_vpn> dev tun0
# ip rule list
0: from all lookup local
100: from all fwmark 0x4/0x4 lookup vpn
101: from all fwmark 0x1/0x1 lookup isp1
102: from all fwmark 0x2/0x2 lookup isp2
32766: from all lookup main
32767: from all lookup default
Now we can indicate that access to the website with statistics and the control panel of the second provider should go through the second ADSL:
# iptables -t mangle -A OUTPUT -d /32 -j MARK --set-mark 0x2
If it is necessary that these rules work not only for outgoing packets from our server, but also for packets from home LAN, then you need to add all these rules to the firewall not only in the OUTPUT chain, but also in the FORWARD chain.
Reply from the same interface
All this will work fine as long as we are talking only about outgoing connections. But we have a server ... and there are incoming connections to the server. Let's say we have a website raised on static IP ssss (channel of the first provider). Now, if the admin of the second provider from the server
To solve this problem, it is necessary to guarantee responses to incoming connections from the interface from which packets were originally received. To do this, unfortunately, you will have to use connection tracking with a firewall (conntrack) to set fwmark (connmark). Unfortunately - because conntrack is a big complicated and buggy garbage, but there is no choice. This is done as follows (these rules must be added to the firewall first, BEFORE the rules setting fwmark like described above):
# iptables -t mangle -A INPUT -i ppp0 -j CONNMARK --set-mark 0x1
# iptables -t mangle -A INPUT -i ppp1 -j CONNMARK --set-mark 0x2
# iptables -t mangle -A INPUT -i tun0 -j CONNMARK --set-mark 0x4
# iptables -t mangle -A INPUT -i eth+ -j CONNMARK --set-mark 0x8
# iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
# iptables -t mangle -A OUTPUT -m mark ! --mark 0x0 -j ACCEPT
Here we mark incoming packets (more precisely, the connections to which they belong) according to the interfaces from which they came. Moreover, the fwmark value (more precisely, connmark) is chosen so that it corresponds to the routing table through which you need to send responses to these packets (1,2,4 - for isp1, isp2 and vpn; 8 - this is not used in fwmark routing rules, t .e. packets with this fwmark will be sent according to the usual routing in the main table - this is quite suitable for packets from local networks).
This labeling does not affect the routing of incoming packets in any way, but this allows us to actually bind this labeling to all packets of this connection, including outgoing packets. Further, for outgoing packets on the same connection (responses to incoming packets), we set the fwmark value to the same value that the incoming packets had and which is tied to this connection (more precisely, copy the connmark value to fwmark). And if, as a result, the outgoing packets are set to fwmark (not 0), then all the following fwmark installation rules cannot be used to select the outgoing channel - this packet must be sent through the same interface through which the incoming packet arrived, so stop (-j ACCEPT )
In total, the correct firewall configuration option according to our problem statement looks like this:
# iptables -t mangle -A INPUT -i ppp0 -j CONNMARK --set-mark 0x1
# iptables -t mangle -A INPUT -i ppp1 -j CONNMARK --set-mark 0x2
# iptables -t mangle -A INPUT -i tun0 -j CONNMARK --set-mark 0x4
# iptables -t mangle -A INPUT -i eth+ -j CONNMARK --set-mark 0x8
# iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
# iptables -t mangle -A OUTPUT -m mark ! --mark 0x0 -j ACCEPT
# iptables -t mangle -A OUTPUT -p tcp -m tcp --dport 25 -j MARK --set-mark 0x1
# iptables -t mangle -A OUTPUT -d /32 -p udp -m udp --dport 1130 -j MARK --set-mark 0x1
# iptables -t mangle -A OUTPUT -d /32 -j MARK --set-mark 0x2
Together with the rules and routing table settings, we get exactly what we needed. It remains to figure out how to set these settings in dynamics, taking into account the fact that channels can fall and rise.
The thorny path to the end result
Honestly, the article turned out to be rather big, so I will not clutter it with my ip-up / down scripts as I was going to. If someone asks in the comments - add to the article later. And now I will briefly describe the main tasks and problems that arose during the implementation of the above settings.
- pppd itself does not know how to raise the multiple default route, so you need to run it with the nodefaultroute option and set the default route with handles in / etc / ppp / ip- {up, down} scripts (or user scripts that are called automatically from these scripts - in different distributions it is configured differently).
- Since the channels go up one at a time, at the time the ip-up script is called, there may not be any default route (this channel went up first) and then it should set the usual single default route through this channel; or a default route may already exist through another channel (this channel went up the second), and then it needs to replace a single default route with a double one (with nexthops and weight) - this is done in the main routing table .
- It is also necessary to add a single default route through this channel to the corresponding routing table ( isp1 or isp2 ).
- In the ip-down script, logically, you need to do the opposite operation (if one of the two channels is disconnected, then you need to replace the double default route with the usual via the remaining channel, if the last channel is disconnected, you need to delete the default route altogether) ... But in practice, pppd itself removes the default route , even if it was double and even if pppd was started with the nodefaultroute option (this is probably a bug in pppd). Therefore, the task in ip-down is the opposite: if it was not the only channel, then raise the usual default route through the remaining channel. Unfortunately, this is not easy either: pppd removes the default route in parallel with the ip-down script, i.e. race condition occurs ! So in ip-down you must first wait until pppd removes the default route, and only then raise it again.
- After making changes to the routing (both in ip-up and ip-down) it will not hurt to make ip route flush cache .
- One more thing: both channels can rise / fall at the same time, and their ip-up / down scripts can also work simultaneously. Most often this will lead to a race condition and damage to the default route. :) Therefore, it is necessary to provide a lock at the start of these scripts , ensuring that only one of them will work at a time - the easiest way to do this is through the chpst utility from the runit package or the setlock utility from the daemontools package.
- openvpn can call up / down user scripts too ...
- In up, you need to add a single default route through this channel to the corresponding routing table ( vpn ).
- After making changes to the routing (both in up and down) it will not hurt to make ip route flush cache .
- In down, nothing is required in this configuration (the default route from the vpn table will be deleted automatically by the kernel when this channel falls), but if something needs to be done, then keep in mind that the up script is run as root, and down from the user openvpn ( or whatever you specified in the openvpn settings) (this is probably a bug too), and the openvpn user usually does not have enough rights to manage the routing (you have to use sudo).