September 24, 2009 at 13:22Ubuntu, KVM and proxy_arp - how to trick an evil provider
One company located colocation servers for internal needs and immediately bought / 30 addresses for their needs. It was configured as aliases (eth0: 0, eth0: 1, etc.). Everything worked perfectly, until after some time a sensible idea appeared to distribute different services to different virtual machines. Since Ubuntu Server was used as the host, the choice of KVM as a virtualizer occurred by itself. Both here and in the rest of the net, a lot of clever words have already been written about installing and configuring KVM and the network environment, I will not stop there, I will only tell about small children's rakes conveniently enclosed by the provider.
First of all, it should be noted that everything happened on a living actively used technological machine, where service interruptions were undesirable, therefore all reconfigurations were performed at night with the corresponding state of the brain;)
So, leaving eth0 intact (so that important production services are not interrupted), we extinguish all aliases, create the br0 bridge, stick eth0 into it and start the virtual machines, just as it is written in all KVM primers, sticking tapX into the same bridge.
Voila - IPs are visible, cars ping each other, time to open champagne when it is discovered that only the host is available from the Internet. Omitting the intricacies of searching for a problem, I immediately turn to the point - at the provider, where the server was on the colocation, the switch is configured with port security, i.e. let out only nailed when installing the MAC. The provider refused to allow poppies of virtual machines, for which I do not blame him, and it is his right to establish an internal technical policy. In response to the confused question: “What can we do?” The answer was again from the KVM primer: leave aliases on the interface, indicate the tenth grid on the virtual machines and then “iptables -j DNAT bla-bla-bla”
Having grieved slightly over some clumsiness of such a solution and thoughtfully smoked Google, an alternative was found with the keyword proxy_arp.
First things first we do
In
By default, a regular bridge simply transfers packets from one interface to another unchanged. It only considers the hardware address of the packet to determine in which direction the packet should be transmitted. This means that Linux can forward any kind of traffic, even one that it does not know if the packets have a hardware address.
The pseudo-bridge works somewhat differently and looks more like a hidden router than a bridge, but like a bridge, it has some effect on the network architecture. True, this is not quite a bridge, since packets actually pass through the core and can be filtered, changed, redirected or sent along a different route.
Another advantage of the pseudo-bridge is that it cannot transmit protocol packets that it “does not understand” - which prevents the network from being filled with all kinds of “garbage”.
In this scheme, when a provider switch wants to establish a connection with a virtual machine located behind the host, it sends the host an ARP request packet, which, translated into human language, may sound something like this: “Who has the address 1.2.3.2? Report to 1.2.3.254. ” In response to the request, 1.2.3.1 will respond with a short packet “I'm here! My hardware address is xx: xx: xx: xx: xx: xx. "
When 1.2.3.254 receives a response, it will remember the hardware address 1.2.3.2 and will store it in the cache for some time.
When configuring proxy_arp, we instructed the eth0 interface to respond to ARP requests. This will force the router to send packets to the bridge, which will then process them and forward them to the appropriate virtual machine.
Thus, whenever a provider router on one side of the bridge asks for the hardware address of a computer located on the other side, the bridge responds to the request, as if to say “pass everything through me”, as a result of which all traffic will go to eth0 and will be transmitted to desired virtualka.
In preparing the article, the materials of the Linux Advanced Routing & Traffic Control HOWTO were used.
First of all, it should be noted that everything happened on a living actively used technological machine, where service interruptions were undesirable, therefore all reconfigurations were performed at night with the corresponding state of the brain;)
So, leaving eth0 intact (so that important production services are not interrupted), we extinguish all aliases, create the br0 bridge, stick eth0 into it and start the virtual machines, just as it is written in all KVM primers, sticking tapX into the same bridge.
Voila - IPs are visible, cars ping each other, time to open champagne when it is discovered that only the host is available from the Internet. Omitting the intricacies of searching for a problem, I immediately turn to the point - at the provider, where the server was on the colocation, the switch is configured with port security, i.e. let out only nailed when installing the MAC. The provider refused to allow poppies of virtual machines, for which I do not blame him, and it is his right to establish an internal technical policy. In response to the confused question: “What can we do?” The answer was again from the KVM primer: leave aliases on the interface, indicate the tenth grid on the virtual machines and then “iptables -j DNAT bla-bla-bla”
Having grieved slightly over some clumsiness of such a solution and thoughtfully smoked Google, an alternative was found with the keyword proxy_arp.
First things first we do
apt-get install uml-utilitiesIn
/etc/network/interfacesprescribe:
We start virtual machines: etc.
Next, in virtual machines, we specify the host address 1.2.3.1 as gw and get a working bunch of virtual machines that stumbled behind the MAC of the host machine from the provider.
For those who do not yet consider themselves network guru, a few words about proxy_arp and how it differs from bridge.auto eth0
iface eth0 inet static
address 1.2.3.1
netmask 255.255.255.0
network 1.2.3.0
broadcast 1.2.3.255
gateway 1.2.3.254
post-up sysctl -w net.ipv4.ip_forward=1
post-up sysctl -w net.ipv4.conf.eth0.proxy_arp=1
pre-down sysctl -w net.ipv4.ip_forward=0
auto qtap1
iface qtap1 inet manual
tunctl_user root
uml_proxy_arp 1.2.3.2
uml_proxy_ether eth0
up ip link set qtap1 up
down ip link set qtap1 down
auto qtap2
iface qtap2 inet manual
tunctl_user root
uml_proxy_arp 1.2.3.3
uml_proxy_ether eth0
up ip link set qtap2 up
down ip link set qtap2 down
kvm -m 512 -hda image.img -net nic,macaddr=00:01:02:03:04:05 -net tap,ifname=qtap1,macaddr=00:01:02:03:04:05,script=no -daemonize -vnc :1By default, a regular bridge simply transfers packets from one interface to another unchanged. It only considers the hardware address of the packet to determine in which direction the packet should be transmitted. This means that Linux can forward any kind of traffic, even one that it does not know if the packets have a hardware address.
The pseudo-bridge works somewhat differently and looks more like a hidden router than a bridge, but like a bridge, it has some effect on the network architecture. True, this is not quite a bridge, since packets actually pass through the core and can be filtered, changed, redirected or sent along a different route.
Another advantage of the pseudo-bridge is that it cannot transmit protocol packets that it “does not understand” - which prevents the network from being filled with all kinds of “garbage”.
In this scheme, when a provider switch wants to establish a connection with a virtual machine located behind the host, it sends the host an ARP request packet, which, translated into human language, may sound something like this: “Who has the address 1.2.3.2? Report to 1.2.3.254. ” In response to the request, 1.2.3.1 will respond with a short packet “I'm here! My hardware address is xx: xx: xx: xx: xx: xx. "
When 1.2.3.254 receives a response, it will remember the hardware address 1.2.3.2 and will store it in the cache for some time.
When configuring proxy_arp, we instructed the eth0 interface to respond to ARP requests. This will force the router to send packets to the bridge, which will then process them and forward them to the appropriate virtual machine.
Thus, whenever a provider router on one side of the bridge asks for the hardware address of a computer located on the other side, the bridge responds to the request, as if to say “pass everything through me”, as a result of which all traffic will go to eth0 and will be transmitted to desired virtualka.
In preparing the article, the materials of the Linux Advanced Routing & Traffic Control HOWTO were used.