Scalable networks in OpenStack. Part 1: flat topology
Author: Piotr Siwczak
Recently, the networks in OpenStack have evolved from a simple, hardly usable model to a more advanced one with the ability to completely isolate owners. Support for various user requirements in OpenStack is implemented using the so-called. "Network managers." The network manager determines the network topology of a particular OpenStack deployment. Starting with the Essex version, the user can choose one of three options for network managers: FlatManager, FlatDHCPManager, VlanManager. In this part of the article we will consider the first two, for the latter the second part will be assigned.
The managers of FlatManager and FlatDHCPManager have a lot in common. Both are based on the concept of bridged networking using a single bridge. As an example, consider a network consisting of several nodes.
For each computing node, 1 virtual bridge is created, the name of which is specified in the Nova configuration using the option:
All virtual machines launched by OpenStack join this dedicated bridge.
Network Bridge on OpenStack Computing Node
The main drawback of the single-bridge approach to a compute node is the well-known limitation of network bridges in Linux - a bridge can only be connected to one physical interface (the limitation can be bypassed using VLANs, but they are not supported by FlatDHCPManager and FlatManager). For this reason, isolation at the second level is not possible, and all nodes operate in the same ARP domain.
At the core of FlatManager and FlatDHCPManager is the idea of having 1 “flat” pool of IP addresses defined for the entire cluster. This address space is shared between all user machines, regardless of which owner they belong to. Each owner can pick up any address available in the pool.
Network Manager FlatManager provides the most primitive set of operations. Its role is only to connect the user machine to the bridge on the computing node. By default, the manager does not configure IP on the user machine. This task rests with the system administrator and can be performed using an external DHCP server.
Network Topology Using FlatManager
Network Manager FlatDHCPManager connects the specified user machine to the bridge and provides a DHCP server.
Thus, on each computing node:
• the network bridge receives the address from the “flat” pool of IP addresses
• the dnsmasq process is started, which listens to the IP address of the bridge interface
• the bridge acts as a default gateway for all user machines running on the computing node
Network topology using FlatDHCPManager
For the dnsmaq service, the FlatDHCPManager network manager creates static entries on each compute node to ensure that the same IP address is received for a given user machine. The record file is created from the Nova database and consists of the MAC address, IP, and machine name. It is envisaged that the dnsmasq server distributes addresses only to machines running locally on a compute node. This is achieved by filtering records from the 'instances' table by the 'host' field. Also, the default gateway is set to the IP address of the bridge interface. In the following figure, you can see that the default gateway depends on which computing node the virtual machine is located on.
Network gateway for machines running on different computing nodes
The following is the routing table for the vm_1 and vm_3 machines, each of them has a different default gateway:
root @ vm_1: ~ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth0
root @ vm_3: ~ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.4 0.0.0.0 UG 0 0 0 eth0
By default, all virtual machines in a “flat” network can see each other regardless of which owner they belong to. Isolation can be achieved using the following flag from nova.conf:
This sets up iptables policies that block any traffic between machines (even within the same owner) until it is resolved using security groups.
From a practical point of view, “flat” network managers look suitable for homogeneous rather small internal corporate clouds, in which there are no owners as such, or their number is very limited. Typically, the use case is a dynamically scalable web server or HPC cluster. For such tasks, it is usually sufficient to have a single space of IP addresses managed either by an external DCHP server or by the dnsmasq process provided by OpenStack. On the other hand, this solution is limited in terms of scalability, since all virtual machines must be in the same L2 domain.
Limitations in scalability and separation of ownership are removed to one degree or another in the network manager VlanManager, which we will discuss in our next post.
Original article in English
Recently, the networks in OpenStack have evolved from a simple, hardly usable model to a more advanced one with the ability to completely isolate owners. Support for various user requirements in OpenStack is implemented using the so-called. "Network managers." The network manager determines the network topology of a particular OpenStack deployment. Starting with the Essex version, the user can choose one of three options for network managers: FlatManager, FlatDHCPManager, VlanManager. In this part of the article we will consider the first two, for the latter the second part will be assigned.
The managers of FlatManager and FlatDHCPManager have a lot in common. Both are based on the concept of bridged networking using a single bridge. As an example, consider a network consisting of several nodes.
For each computing node, 1 virtual bridge is created, the name of which is specified in the Nova configuration using the option:
flat_network_bridge=br100All virtual machines launched by OpenStack join this dedicated bridge.
Network Bridge on OpenStack Computing Node
The main drawback of the single-bridge approach to a compute node is the well-known limitation of network bridges in Linux - a bridge can only be connected to one physical interface (the limitation can be bypassed using VLANs, but they are not supported by FlatDHCPManager and FlatManager). For this reason, isolation at the second level is not possible, and all nodes operate in the same ARP domain.
At the core of FlatManager and FlatDHCPManager is the idea of having 1 “flat” pool of IP addresses defined for the entire cluster. This address space is shared between all user machines, regardless of which owner they belong to. Each owner can pick up any address available in the pool.
Flatmanager
Network Manager FlatManager provides the most primitive set of operations. Its role is only to connect the user machine to the bridge on the computing node. By default, the manager does not configure IP on the user machine. This task rests with the system administrator and can be performed using an external DHCP server.
Network Topology Using FlatManager
FlatDHCPManager
Network Manager FlatDHCPManager connects the specified user machine to the bridge and provides a DHCP server.
Thus, on each computing node:
• the network bridge receives the address from the “flat” pool of IP addresses
• the dnsmasq process is started, which listens to the IP address of the bridge interface
• the bridge acts as a default gateway for all user machines running on the computing node
Network topology using FlatDHCPManager
For the dnsmaq service, the FlatDHCPManager network manager creates static entries on each compute node to ensure that the same IP address is received for a given user machine. The record file is created from the Nova database and consists of the MAC address, IP, and machine name. It is envisaged that the dnsmasq server distributes addresses only to machines running locally on a compute node. This is achieved by filtering records from the 'instances' table by the 'host' field. Also, the default gateway is set to the IP address of the bridge interface. In the following figure, you can see that the default gateway depends on which computing node the virtual machine is located on.
Network gateway for machines running on different computing nodes
The following is the routing table for the vm_1 and vm_3 machines, each of them has a different default gateway:
root @ vm_1: ~ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth0
root @ vm_3: ~ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.4 0.0.0.0 UG 0 0 0 eth0
By default, all virtual machines in a “flat” network can see each other regardless of which owner they belong to. Isolation can be achieved using the following flag from nova.conf:
allow_same_net_traffic=FalseThis sets up iptables policies that block any traffic between machines (even within the same owner) until it is resolved using security groups.
From a practical point of view, “flat” network managers look suitable for homogeneous rather small internal corporate clouds, in which there are no owners as such, or their number is very limited. Typically, the use case is a dynamically scalable web server or HPC cluster. For such tasks, it is usually sufficient to have a single space of IP addresses managed either by an external DCHP server or by the dnsmasq process provided by OpenStack. On the other hand, this solution is limited in terms of scalability, since all virtual machines must be in the same L2 domain.
Limitations in scalability and separation of ownership are removed to one degree or another in the network manager VlanManager, which we will discuss in our next post.
Original article in English