Floating IP addresses for networking in public and private OpenStack clouds
Posted by Piotr Siwczak
Recently I described how VlanManager worksand how it provides network scalability and user isolation. But until now, I have only talked about networks with a fixed IP address belonging to different users. Although by default fixed IP addresses are allocated to instances, they do not guarantee the immediate availability of instances from outside the network (or from other data centers). Imagine the following scenario:
You are launching a small website with one www server, a database server, and a firewall that performs network address translation (NAT) and traffic filtering. Usually you apply the following settings:
-All servers communicate within the network within a private (non-routable) network range (for example, 192.168.0.0/24).
-There is one public routable IP range on which the www server is visible.
You do the following: -
Assign a public IP address to the firewall.
-Create a NAT rule on the firewall to route traffic from the public IP address to the private IP address of the www server.
The fixed IP addresses in OpenStack work the same way as the 192.168.0.0/24 network range in the example above. They guarantee only interaction between instances within the same OpenStack cluster. But OpenStack also introduces another pool of IP addresses called “floating IPs”. Floating IP addresses are publicly routable IP addresses that you buy from your Internet service provider (the one that fits in the firewall above). Users can assign IP addresses to their instances, making them accessible from an external network.
Floating IP addresses are not assigned to virtual machines by default. Cloud users must explicitly “take” them from the pool configured by the OpenStack administrator, and then assign them to their virtual machines. As soon as the user has taken the floating IP address from the pool, he becomes its “owner” (that is, at any time he can detach the IP address from the virtual machine and attach it to another). If the virtual machine for some reason ceases to exist, the user does not lose the floating IP address - he can then assign it to another instance. Unfortunately, it is now impossible to share a single floating IP address between multiple virtual machines for load balancing, such as with elastic load balancing on Amazon EC2.
On the other hand, fixed IP addresses are assigned dynamically by the nova-network component when the VM boots. There is no way to tell OpenStack to assign a specific fixed IP address to a virtual machine. Thus, you may find yourself in a situation where, upon the accidental shutdown of a virtual machine after its recovery from a snapshot, a new instance is loaded with a new fixed IP address.
System administrators can configure multiple pools of floating IP addresses. However, unlike fixed IP pools, floating IP pools cannot be associated with specific users. Each user can “take” a floating IP address from any pool of floating IP addresses. But the main motivation for creating multiple pools of floating IPs is for each pool to serve its Internet access provider. Thus, we can guarantee the ability to connect, even if one of the suppliers fails.
As a result, the basic functions of floating IP addresses are as follows:
-Floating IP addresses are not automatically assigned to virtual machines by default (they must be assigned to instances manually).
-If the virtual machine ceases to exist, the user can reuse the floating IP address by assigning it to another instance.
-Users can take floating IP addresses from various pools defined by the cloud administrator to provide the ability to connect to virtual machines by different Internet service providers or from external networks.
The “public availability” of a floating IP address is a relative concept. For public clouds, you might want to define a pool of floating IP addresses as a pool of IP addresses publicly accessible from the Internet. Then your clients can assign them to virtual machines to access them via SSH from their home or work computers:
If your corporate data center is running in your data center, then the pool of floating IP addresses can be any range of IP addresses that provides access to OpenStack instances from the rest of the data center.
For the traffic of your data center, you can define the following range: 10.0.0.0/16.
Inside OpenStack, you can create the following range of fixed IP addresses: 192.168.0.0/16, divided into user subnets.
To make OpenStack instances accessible from the entire data center, you can define the pool of floating IP addresses as a 10.0.0.0/8 subnet (for example, 10.0.0.0/16) and register it with OpenStack so that users can get IP addresses from there.
As I mentioned earlier, the system administrator first registers the pool of floating IP addresses in OpenStack:
Thus, the public pool becomes available to users.
Now users follow this procedure:
-Download instance:
+ -------------------------------------- + --------- + -------- + ------------------------------- - +
| ID | Name | Status | Networks |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
| 79935433-241a-4268-8aea-5570d74fcf42 | inst1 | ACTIVE | private = 10.0.0.4 |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
-List the available pools of floating IP addresses :
nova floating-ip-pool-list
+ ------ +
| name |
+ ------ +
| pub |
| test |
+ ------ +
-Take a floating IP address from the pool “pub” (or optionally from the pool “test”):
nova floating-ip-create pub
+ ------------ + ------------- + --- + ------ ---------- +
| Ip | Instance Id | Fixed Ip | Pool |
+ --------------- + ------------- + ---------- + ------ +
| 172.24.4.225 | None | None | pub |
+ --------------- + ------------- + ---------- + ------ +
- Assign a floating IP address to the instance:
(where the first argument is the uuid of the instance and the second is the floating IP address itself)
-Check all the settings are correct:
nova floating-ip-list
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| Ip | Instance Id | Fixed Ip | Pool |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| 172.24.4.225 | 79935433-241a-4268-8aea-5570d74fcf42 | 10.0.0.4 | pub |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
Now the instance should be visible from outside the OpenStack cluster by the floating IP address.
What happens inside an instance after adding a floating IP? The correct answer is nothing. If you connect to it via SSH and look at the network configuration, you will see that there is one network interface with a fixed fixed IP address.
All configuration is performed on the computing node. All work related to the floating IP address is performed by the nova-network service: a network address translator (NAT) is organized between the fixed and floating IP addresses of the instance. An explanation of how NAT works can be found here .
Take a look at the following chart:
The diagram displays one computing node configured in network mode with the distribution of nodes and VlanManager used to configure fixed IP networks. The computing node is equipped with two network interfaces: the eth0 interface is allocated for traffic with a fixed IP / VLAN, and eth1 is the interface through which the computing node is connected to an external network; it has floating IPs. (For information on how VlanManager configures fixed IP networks, see the previous article. )
Please note that while the eth0 interface (fixed / private) is not configured, the eth1 interface is assigned an IP address, which is the gateway to default for the compute node (91.207.15.105).
When a user assigns a floating IP address (91.207.16.144) to VM_1, two things happen:
-The floating IP address is configured as a secondary address on eth1: this is the output of the ip addr show eth1 command, which contains the following actions:
inet 91.207. 15.105 / 24 scope global eth1 # primary eth1 ip
inet 91.207.16.144/32 scope global eth1 # floating ip of VM_1
- The NAT rule set for the floating IP address is configured in the iptables tables. Below are all the relevant entries from the nat table of the computing node (except for the command “iptables –S -t nat". For details on how to configure NAT using iptables tables in Linux, see a third here ):
# the this rule Ensures That packets The Originating from node compute clause
# where the instance resides, will reach the instance via its floating IP:
-A nova-network-OUTPUT -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# ensures that all external traffic to the floating IP
# is directed to the fixed IP of the instance
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# all the traffic originating from the instance will be SNAT-ted to its floating IP
-A nova-network-float-snat -s 10.0.0.3/32 -j SNAT --to-source 91.207.16.144
In general, nova-network adds additional chains to those predefined in the NAT table. The order of the chains in relation to traffic by a floating IP address is shown below (using the rules shown above):
Chain OUTPUT - Chain nova-network-OUTPUT - Rule: -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Chain PREROUTING - Chain nova-network-PREROUTING - Rule: -d 91.207.16.144/32 - j DNAT --to-destination 10.0.0.3
Chain POSTROUTING - Chain nova-postrouting-bottom - Chain nova-network-snat - Chain nova-network-float-snat - Rule: -s 10.0.0.3/32 -j SNAT - to-source 91.207.16.144 -The
code responsible for setting the rules is located in nova / network / linux_net.py in the function:
def floating_forward_rules (floating_ip, fixed_ip):
return [('PREROUTING', '-d% s -j DNAT - -to% s '% (floating_ip, fixed_ip)),
(' OUTPUT ',' -d% s -j DNAT --to% s '% (floating_ip, fixed_ip)),
(' float-snat ',
' -s % s -j SNAT --to% s'% (fixed_ip,floating_ip))]
We return to the diagram. If the user wants to access the virtual machine by his IP address from the external network (for example, “ping 91.20.16.144 ″):
-Traffic reaches the public interface (eth1) of the computing node. In nova-network-PREROUTING, a DNAT is executed that changes the destination IP address of packets from 91.207.16.144 to 10.0.0.3.
-The computing node accesses its routing table and sees that the network 10.0.0.0 is available on the br100 interface (except for the “ip route show” of the computing node):
10.0.0.0/24 dev br100
This way it routes the packet to the br100 interface, then the packet reaches virtual machine.
If the virtual machine sends a packet to the outside world (for example, “ping 8.8.8.8):
- Since the destination address is not in the local network of the virtual machine, packets are sent to the default gateway of the virtual machine with the IP address 10.0.0.1 (the device address is “br100 ″ on the computing node).
-The computing node checks its routing tables and finds that there is no 8.8.8.8 address on directly connected networks, so it forwards the packet to the default gateway (which is the primary address of eth1 - 91.207.15.105 in this case).
-The packet falls into the POSTROUTING chain and is transmitted to the “nova-network-float-snat” chain, where its original IP address is rewritten to the floating IP address (91.207.16.144).
When using OpenStack, system administrators delegate full control of iptables tables to nova services. The set of custom rules is very complex and is easily broken by any outside interference. Moreover, each time the nova-network daemon is restarted, it applies all the rules in the iptables table chains associated with OpenStack. If there is a need to somehow change the behavior of iptables tables, this can be done by changing the code in the appropriate places linux_net.py (for NAT rules, these will be floating_forward_rules).
It is also worth saying that nova-network does not track its tables in any way. Thus, if we manually delete any rules from the chains associated with OpenStack, they will not be restored the next time nova-network is launched.
Thus, a system administrator can easily accidentally open unwanted access to a computing node. Remember that nova-network placed the floating IP address as secondary on the eth1 interface and configured DNAT rules that direct traffic to the fixed IP address of the virtual machine:
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT - -to-destination 10.0.0.3
Thus, all traffic directed to 91.207.16.144 goes to address 10.0.0.3.
Now let's imagine that the system administrator solved the problems of connecting to the network at night and accidentally deleted all NAT rules by typing:
iptables –F –t nat
The above NAT rule has been removed, but the secondary IP address 91.207.16.144 is still assigned to eth1. Thus, we can still access the address 91.207.16.144 from the outside, but instead of getting to the virtual machine, we now have access to the compute node itself (the destination IP address is no longer translated according to the DNAT rule, since we deleted all NAT rules). This security hole will not be closed until the next restart of the nova-network process, which will re-create the rules.
There are flags in the nova.conf service that affect the behavior of floating IP addresses:
# the interface to which floating ips are attached
# as secondary addresses
public_interface = "eth1"
# the pool from which floating IPs are taken by default
default_floating_pool = "pub »
# <br> we CAN the add a floating the ip automatically to every instance That is spawned
auto_assign_floating_ip to false =
In addition to providing access to virtual machines directly from the Internet, the floating IP mechanism gives cloud users some flexibility. After “picking up” a floating IP address, they can change their ownership, that is, on the go assign them to various virtual machines and reassign them, which greatly facilitates the release of new code and system updates. For system administrators, this poses a potential security risk, since the underlying mechanism (iptables) is very complex and not monitored by OpenStack. Therefore, it is important that only OpenStack software can change the firewall policies and that they do not change manually.
Original article in English
Recently I described how VlanManager worksand how it provides network scalability and user isolation. But until now, I have only talked about networks with a fixed IP address belonging to different users. Although by default fixed IP addresses are allocated to instances, they do not guarantee the immediate availability of instances from outside the network (or from other data centers). Imagine the following scenario:
You are launching a small website with one www server, a database server, and a firewall that performs network address translation (NAT) and traffic filtering. Usually you apply the following settings:
-All servers communicate within the network within a private (non-routable) network range (for example, 192.168.0.0/24).
-There is one public routable IP range on which the www server is visible.
You do the following: -
Assign a public IP address to the firewall.
-Create a NAT rule on the firewall to route traffic from the public IP address to the private IP address of the www server.
The fixed IP addresses in OpenStack work the same way as the 192.168.0.0/24 network range in the example above. They guarantee only interaction between instances within the same OpenStack cluster. But OpenStack also introduces another pool of IP addresses called “floating IPs”. Floating IP addresses are publicly routable IP addresses that you buy from your Internet service provider (the one that fits in the firewall above). Users can assign IP addresses to their instances, making them accessible from an external network.
Differences between floating and fixed IPs
Floating IP addresses are not assigned to virtual machines by default. Cloud users must explicitly “take” them from the pool configured by the OpenStack administrator, and then assign them to their virtual machines. As soon as the user has taken the floating IP address from the pool, he becomes its “owner” (that is, at any time he can detach the IP address from the virtual machine and attach it to another). If the virtual machine for some reason ceases to exist, the user does not lose the floating IP address - he can then assign it to another instance. Unfortunately, it is now impossible to share a single floating IP address between multiple virtual machines for load balancing, such as with elastic load balancing on Amazon EC2.
On the other hand, fixed IP addresses are assigned dynamically by the nova-network component when the VM boots. There is no way to tell OpenStack to assign a specific fixed IP address to a virtual machine. Thus, you may find yourself in a situation where, upon the accidental shutdown of a virtual machine after its recovery from a snapshot, a new instance is loaded with a new fixed IP address.
System administrators can configure multiple pools of floating IP addresses. However, unlike fixed IP pools, floating IP pools cannot be associated with specific users. Each user can “take” a floating IP address from any pool of floating IP addresses. But the main motivation for creating multiple pools of floating IPs is for each pool to serve its Internet access provider. Thus, we can guarantee the ability to connect, even if one of the suppliers fails.
As a result, the basic functions of floating IP addresses are as follows:
-Floating IP addresses are not automatically assigned to virtual machines by default (they must be assigned to instances manually).
-If the virtual machine ceases to exist, the user can reuse the floating IP address by assigning it to another instance.
-Users can take floating IP addresses from various pools defined by the cloud administrator to provide the ability to connect to virtual machines by different Internet service providers or from external networks.
Floating IPs — internal or external clouds
The “public availability” of a floating IP address is a relative concept. For public clouds, you might want to define a pool of floating IP addresses as a pool of IP addresses publicly accessible from the Internet. Then your clients can assign them to virtual machines to access them via SSH from their home or work computers:
If your corporate data center is running in your data center, then the pool of floating IP addresses can be any range of IP addresses that provides access to OpenStack instances from the rest of the data center.
For the traffic of your data center, you can define the following range: 10.0.0.0/16.
Inside OpenStack, you can create the following range of fixed IP addresses: 192.168.0.0/16, divided into user subnets.
To make OpenStack instances accessible from the entire data center, you can define the pool of floating IP addresses as a 10.0.0.0/8 subnet (for example, 10.0.0.0/16) and register it with OpenStack so that users can get IP addresses from there.
Work with floating IPs
As I mentioned earlier, the system administrator first registers the pool of floating IP addresses in OpenStack:
nova-manage floating create --ip_range=PUBLICLY_ROUTABLE_IP_RANGE --pool POOL_NAMEThus, the public pool becomes available to users.
Now users follow this procedure:
-Download instance:
+ -------------------------------------- + --------- + -------- + ------------------------------- - +
| ID | Name | Status | Networks |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
| 79935433-241a-4268-8aea-5570d74fcf42 | inst1 | ACTIVE | private = 10.0.0.4 |
+ -------------------------------------- + --------- + -------- + -------------------------------- +
-List the available pools of floating IP addresses :
nova floating-ip-pool-list
+ ------ +
| name |
+ ------ +
| pub |
| test |
+ ------ +
-Take a floating IP address from the pool “pub” (or optionally from the pool “test”):
nova floating-ip-create pub
+ ------------ + ------------- + --- + ------ ---------- +
| Ip | Instance Id | Fixed Ip | Pool |
+ --------------- + ------------- + ---------- + ------ +
| 172.24.4.225 | None | None | pub |
+ --------------- + ------------- + ---------- + ------ +
- Assign a floating IP address to the instance:
nova add-floating-ip 79935433-241a-4268-8aea-5570d74fcf42 172.24.4.225(where the first argument is the uuid of the instance and the second is the floating IP address itself)
-Check all the settings are correct:
nova floating-ip-list
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| Ip | Instance Id | Fixed Ip | Pool |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
| 172.24.4.225 | 79935433-241a-4268-8aea-5570d74fcf42 | 10.0.0.4 | pub |
+ -------------- + ---------------------------------- ---- + ---------- + ------ +
Now the instance should be visible from outside the OpenStack cluster by the floating IP address.
How floating IPs work
What happens inside an instance after adding a floating IP? The correct answer is nothing. If you connect to it via SSH and look at the network configuration, you will see that there is one network interface with a fixed fixed IP address.
All configuration is performed on the computing node. All work related to the floating IP address is performed by the nova-network service: a network address translator (NAT) is organized between the fixed and floating IP addresses of the instance. An explanation of how NAT works can be found here .
Take a look at the following chart:
The diagram displays one computing node configured in network mode with the distribution of nodes and VlanManager used to configure fixed IP networks. The computing node is equipped with two network interfaces: the eth0 interface is allocated for traffic with a fixed IP / VLAN, and eth1 is the interface through which the computing node is connected to an external network; it has floating IPs. (For information on how VlanManager configures fixed IP networks, see the previous article. )
Please note that while the eth0 interface (fixed / private) is not configured, the eth1 interface is assigned an IP address, which is the gateway to default for the compute node (91.207.15.105).
When a user assigns a floating IP address (91.207.16.144) to VM_1, two things happen:
-The floating IP address is configured as a secondary address on eth1: this is the output of the ip addr show eth1 command, which contains the following actions:
inet 91.207. 15.105 / 24 scope global eth1 # primary eth1 ip
inet 91.207.16.144/32 scope global eth1 # floating ip of VM_1
- The NAT rule set for the floating IP address is configured in the iptables tables. Below are all the relevant entries from the nat table of the computing node (except for the command “iptables –S -t nat". For details on how to configure NAT using iptables tables in Linux, see a third here ):
# the this rule Ensures That packets The Originating from node compute clause
# where the instance resides, will reach the instance via its floating IP:
-A nova-network-OUTPUT -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# ensures that all external traffic to the floating IP
# is directed to the fixed IP of the instance
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
# all the traffic originating from the instance will be SNAT-ted to its floating IP
-A nova-network-float-snat -s 10.0.0.3/32 -j SNAT --to-source 91.207.16.144
In general, nova-network adds additional chains to those predefined in the NAT table. The order of the chains in relation to traffic by a floating IP address is shown below (using the rules shown above):
Chain OUTPUT - Chain nova-network-OUTPUT - Rule: -d 91.207.16.144/32 -j DNAT --to-destination 10.0.0.3
Chain PREROUTING - Chain nova-network-PREROUTING - Rule: -d 91.207.16.144/32 - j DNAT --to-destination 10.0.0.3
Chain POSTROUTING - Chain nova-postrouting-bottom - Chain nova-network-snat - Chain nova-network-float-snat - Rule: -s 10.0.0.3/32 -j SNAT - to-source 91.207.16.144 -The
code responsible for setting the rules is located in nova / network / linux_net.py in the function:
def floating_forward_rules (floating_ip, fixed_ip):
return [('PREROUTING', '-d% s -j DNAT - -to% s '% (floating_ip, fixed_ip)),
(' OUTPUT ',' -d% s -j DNAT --to% s '% (floating_ip, fixed_ip)),
(' float-snat ',
' -s % s -j SNAT --to% s'% (fixed_ip,floating_ip))]
We return to the diagram. If the user wants to access the virtual machine by his IP address from the external network (for example, “ping 91.20.16.144 ″):
-Traffic reaches the public interface (eth1) of the computing node. In nova-network-PREROUTING, a DNAT is executed that changes the destination IP address of packets from 91.207.16.144 to 10.0.0.3.
-The computing node accesses its routing table and sees that the network 10.0.0.0 is available on the br100 interface (except for the “ip route show” of the computing node):
10.0.0.0/24 dev br100
This way it routes the packet to the br100 interface, then the packet reaches virtual machine.
If the virtual machine sends a packet to the outside world (for example, “ping 8.8.8.8):
- Since the destination address is not in the local network of the virtual machine, packets are sent to the default gateway of the virtual machine with the IP address 10.0.0.1 (the device address is “br100 ″ on the computing node).
-The computing node checks its routing tables and finds that there is no 8.8.8.8 address on directly connected networks, so it forwards the packet to the default gateway (which is the primary address of eth1 - 91.207.15.105 in this case).
-The packet falls into the POSTROUTING chain and is transmitted to the “nova-network-float-snat” chain, where its original IP address is rewritten to the floating IP address (91.207.16.144).
Safety notes
When using OpenStack, system administrators delegate full control of iptables tables to nova services. The set of custom rules is very complex and is easily broken by any outside interference. Moreover, each time the nova-network daemon is restarted, it applies all the rules in the iptables table chains associated with OpenStack. If there is a need to somehow change the behavior of iptables tables, this can be done by changing the code in the appropriate places linux_net.py (for NAT rules, these will be floating_forward_rules).
It is also worth saying that nova-network does not track its tables in any way. Thus, if we manually delete any rules from the chains associated with OpenStack, they will not be restored the next time nova-network is launched.
Thus, a system administrator can easily accidentally open unwanted access to a computing node. Remember that nova-network placed the floating IP address as secondary on the eth1 interface and configured DNAT rules that direct traffic to the fixed IP address of the virtual machine:
-A nova-network-PREROUTING -d 91.207.16.144/32 -j DNAT - -to-destination 10.0.0.3
Thus, all traffic directed to 91.207.16.144 goes to address 10.0.0.3.
Now let's imagine that the system administrator solved the problems of connecting to the network at night and accidentally deleted all NAT rules by typing:
iptables –F –t nat
The above NAT rule has been removed, but the secondary IP address 91.207.16.144 is still assigned to eth1. Thus, we can still access the address 91.207.16.144 from the outside, but instead of getting to the virtual machine, we now have access to the compute node itself (the destination IP address is no longer translated according to the DNAT rule, since we deleted all NAT rules). This security hole will not be closed until the next restart of the nova-network process, which will re-create the rules.
Configure floating IPs
There are flags in the nova.conf service that affect the behavior of floating IP addresses:
# the interface to which floating ips are attached
# as secondary addresses
public_interface = "eth1"
# the pool from which floating IPs are taken by default
default_floating_pool = "pub »
# <br> we CAN the add a floating the ip automatically to every instance That is spawned
auto_assign_floating_ip to false =
Summary comments
In addition to providing access to virtual machines directly from the Internet, the floating IP mechanism gives cloud users some flexibility. After “picking up” a floating IP address, they can change their ownership, that is, on the go assign them to various virtual machines and reassign them, which greatly facilitates the release of new code and system updates. For system administrators, this poses a potential security risk, since the underlying mechanism (iptables) is very complex and not monitored by OpenStack. Therefore, it is important that only OpenStack software can change the firewall policies and that they do not change manually.
Original article in English