Namespace Diagnostics in OpenStack Neutron
Posted by Damian Igby I
recently introduced you to my Neutron namespace talk, which was prepared for the Hong Kong OpenStack Summit. One of the commentators saw a video of my presentation and asked us to post a presentation here. In this post I will show you how to:
1. Define the namespace correctly.
2. Perform general diagnostics in the established namespace.
In addition, at the end of the post there is a presentation with my speech.
So, let's get right to the point. If you start two L3 + DHCP services on one node at once, in order to eliminate route conflict, you need to activate namespaces:
Use_namespaces = True
To disable namespaces, make sure that the following setting is set in the neutron.conf file used by the neutron server:
allow_overlapping_ips = False
. Also, the dhcp_agent.ini and l3_agent.ini files contain:
use_namespaces = False.
How namespaces are implemented in Neutron and how to recognize them? To begin with, it is necessary to state two very important facts:
1. Each l2-agent / private network has an associated DHCP namespace.
2. Each l3-agent / router has an associated router namespace.
OK, but how are namespaces named? At the beginning of the network namespace name is dhcp- followed by the network ID. By analogy, qrouter- comes at the beginning of the namespace of the router followed by the router ID. For example:
stack @ vmnet-mn: ~ $ sudo ip netns list
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
The above list of namespaces is typical for a single private network consisting of network and one router. For a better understanding of Fig. 1 illustrates such a simple network.
Figure 1. Illustration of a simple network with a router and DHCP namespaces
Fig. Figure 1 shows how a leased network consisting of a router and DHCP namespaces is built. For illustration purposes, a single virtual machine (VM1) is included in the network. As networks are built, similar network topologies will be created for each tenant. The following observations can be made from Fig. 1:
• The data center network is an external network through which packets arrive on the Internet.
• A router with an IP address of 10.0.0.1 connects the leased network to the data center network. It has an associated router namespace that allows you to isolate one tenant's routing tables from any other tenant when deploying OpenStack.
• A bridge denotes a leased network with an IP address of 10.0.0.0/24. A private network is attached to the DHCP server (in this case, the dnsmasq process) and it has an associated IP address (10 .0.0.2) just like a physical server connects to the network. Note that the router is usually assigned the first IP address, the DHCP server is assigned the next IP address, and the virtual machines are assigned the remaining IP addresses in the pool. Similarly, a DHCP server has an associated DHCP namespace that allows you to distribute IP addresses and transfer all traffic within a given leased network.
Without a proper understanding of network namespaces as a fundamental concept in building an OpenStack network, troubleshooting and network troubleshooting is akin to finding a needle in a haystack. Imagine a small OpenStack implementation with approximately 1,000 tenants, with each tenant of the Neutron network including at least one router and one private network. The total number of namespaces created will be at least 2000, since each leased router is a namespace, and each network will have an associated DHCP namespace (that is, two namespaces for each tenant, as shown in Figure 1). In this scenario, let’s say a tenant has problems distributing DHCP, allowing the instance to show the assigned IP addresses,
To solve this problem, you need to check the various components of the leased network topology shown in Fig. 1, mainly a DHCP agent. The first logical step is to define a tenant namespace. At the same time, we will see how this process differs from troubleshooting when deploying a traditional physical network. Here, not understanding how the namespace works in Neutron, the administrator will be completely confused and will not be able to continue troubleshooting using logical procedures.
So, the steps for finding and fixing errors are as follows:
1. Correctly determine the namespace.
2. Carry out general diagnostics in the established namespace.
Create two leased networks to illustrate the identification and elimination of errors in namespaces.
Figure 2. Network for tenant A
Figure 3. Network for tenant B
In the above diagrams, the network for tenant A has two private networks and one router; the network for tenant B is also two networks and one router. Consider these networks. Note that in a multimode configuration that includes the controller node, network node, and compute nodes, namespace commands will be executed from the network node.
@ vmnet-stack mn: the ip ~ $ sudo netns list
[sudo] password for stack:
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qdhcp-9f7ec6fd-ec9a-4b1b b854-a9832f190922-
qrouter-4587f8d0-08a7-434a-aaeb -74e68a50c48d
qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
qdhcp-e0fe9037-790a-4c6b-9ff4fc
We see a list of six namespaces of two leased networks (three namespaces for each tenant).
OK, but how do you know which namespace belongs to which tenant, because the names of tenants in namespaces are not designated? There are two ways to do this.
Firstly, it is worth noting that, as a rule, only the cloud administrator has access to the network node, so only the administrator can run namespace commands on the network node. One way to determine the namespace of a cloud administrator is to establish the source of the user credential file for the tenant. After that, all neutron commands will be executed under this user and only the components of the leased network to which this user belongs will be shown. We illustrate this as follows.
1. Identify the source of the credential file owned by tenant A.
2. Run the following commands:
stack @ vmnet-mn: ~ $ neutron router-list
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| id | name | external_gateway_info |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| 4adef4a5-1122-49b8-9da2-3cbf803c44dd | router_proj_one | {"Network_id": "ea00b8c3-7063-4780-ad73-ef0b47518f10"} |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
It can be seen from the list that tenant A has one router with the identifier 4adef4a5-1122-49b8-9da2-3cbf803c44dd
Having found this ID in the ip netns command above, we can easily determine that the qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd router namespace belongs to tenant A.
For the DHCP namespace:
stack @ vmnet-mn: ~ $ neutron net-list
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| id | name | subnets |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c | net_proj_one | f02b6df4-11af-411d-b41e-e4ce66d5510d 10.10.1.0/24 |
| ea00b8c3-7063-4780-ad73-ef0b47518f10 | ext_net | 4f7c6c9e-28d8-43b3-b7d4-276920216b31 |
| f5a69443-bf3f-4f9a-b3e2-5625e70514b4 | net2 | f6cd4df6-b37f-4615-94f6-8abcab38ef99 192.168.0.0/24 |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
In addition, from this list you can identify two private networks with identifiers 0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and f5a69443-bf3f-4f9a-b3e2-5625e70514b4.
After checking the command from the ip netns list above, we can also set up the following DHCP namespaces: qdhcp-e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
The procedures described above are effective and the simplest if the cloud administrator knows the tenant’s username / password, but this is not always the case. Therefore, another way that a cloud administrator can determine the tenant’s namespace is to ask the user for a network ID or network name. A rental network user can easily find out this information in the Horizon project's OpenStack Dashboard control panel. Knowing either the name or the ID, the cloud administrator can troubleshoot as follows:
1. Track the administrator credentials. Knowing them, the cloud administrator will have access to all the namespaces of the leased network.
2. Make a list of namespaces and select a tenant namespace.
stack @ vmnet-mn: ~ $ ip netns list | grep 9f7ec6fd-ec9a-4b1b-b854-a9832f190922
qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922
Now that we’ve figured out the tenant’s namespace, we can begin to troubleshoot it.
First, check the IP address.
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ifconfig
[sudo] password for stack:
lo Link encap: Local Loopback
inet addr: 127.0.0.1 Mask: 255.0.0.0
inet6 addr : :: 1/128 Scope: Host
UP LOOPBACK RUNNING MTU: 16436 Metric: 1
RX packets: 16 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 16 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 1344 (1.3 KB) TX bytes: 1344 (1.3 KB)
tap43cb2c73-40 Link encap: Ethernet HWaddr fa: 16: 3e: e6: 1b: 24
inet addr: 10.1.0.3 Bcast: 10.1.0.255 Mask: 255.255.255.0
inet6 addr: fe80 :: f816: 3eff: fee6: 1b24 / 64 Scope: Link
UP BROADCAST RUNNING MULTICAST MTU: 1500 Metric: 1
RX packets: 965 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 561 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 178396 (178.3 KB) TX bytes: 93900 (93.9 KB)
Then execute Interface ping:
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ping –c 3 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56 (84) bytes of data.
64 bytes from 10.1.0.3: icmp_req = 1 ttl = 64 time = 0.081 ms
64 bytes from 10.1.0.3: icmp_req = 2 ttl = 64 time = 0.035 ms
64 bytes from 10.1.0.3: icmp_req = 3 ttl = 64 time = 0.055 ms
The steps described above helped us establish the appropriate namespace and verify the correctness of the available (i.e., running and working) IP address. Further troubleshooting steps may include the following:
• Check if the dnsmasq process connected to the subnet is working for DHCP:
stack @ vmnet-mn: ~ $ ps –aux | grep dhcp
Based on the results of the above command, determine that the process in the selected network (for example, 10.0.0.0) is working, otherwise the dnsmasq process connected to this network may not start.
• Verify that the firewall is not blocking data transfers to / from the subnet.
Before summing up, let us look at the following:
• When creating a router or network, namespaces are not created immediately. For a network, DHCP namespaces are created only when the VM is connected, and for the router, the namespace is created when the gateway is installed. This means that there must be some activity before creating namespaces.
• When you delete a router or network, the associated namespaces are not deleted. They must be removed manually.
Just as network namespaces are critical to building an OpenStack network, understanding them by the OpenStack cloud administrator is important. We looked at the implementation of network namespaces in OpenStack, taking into account what data the OpenStack cloud administrator needs to have when solving problems with a rented OpenStack network.
Now, if you wish, you can familiarize yourself with the presentation on this topic. The Network and the Namespaces by Neutron IPtables - A Contents Technical Deep Dive from Mirantis Original article in English
recently introduced you to my Neutron namespace talk, which was prepared for the Hong Kong OpenStack Summit. One of the commentators saw a video of my presentation and asked us to post a presentation here. In this post I will show you how to:
1. Define the namespace correctly.
2. Perform general diagnostics in the established namespace.
In addition, at the end of the post there is a presentation with my speech.
So, let's get right to the point. If you start two L3 + DHCP services on one node at once, in order to eliminate route conflict, you need to activate namespaces:
Use_namespaces = True
To disable namespaces, make sure that the following setting is set in the neutron.conf file used by the neutron server:
allow_overlapping_ips = False
. Also, the dhcp_agent.ini and l3_agent.ini files contain:
use_namespaces = False.
How namespaces are implemented in Neutron and how to recognize them? To begin with, it is necessary to state two very important facts:
1. Each l2-agent / private network has an associated DHCP namespace.
2. Each l3-agent / router has an associated router namespace.
OK, but how are namespaces named? At the beginning of the network namespace name is dhcp- followed by the network ID. By analogy, qrouter- comes at the beginning of the namespace of the router followed by the router ID. For example:
stack @ vmnet-mn: ~ $ sudo ip netns list
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
The above list of namespaces is typical for a single private network consisting of network and one router. For a better understanding of Fig. 1 illustrates such a simple network.
Figure 1. Illustration of a simple network with a router and DHCP namespaces
Fig. Figure 1 shows how a leased network consisting of a router and DHCP namespaces is built. For illustration purposes, a single virtual machine (VM1) is included in the network. As networks are built, similar network topologies will be created for each tenant. The following observations can be made from Fig. 1:
• The data center network is an external network through which packets arrive on the Internet.
• A router with an IP address of 10.0.0.1 connects the leased network to the data center network. It has an associated router namespace that allows you to isolate one tenant's routing tables from any other tenant when deploying OpenStack.
• A bridge denotes a leased network with an IP address of 10.0.0.0/24. A private network is attached to the DHCP server (in this case, the dnsmasq process) and it has an associated IP address (10 .0.0.2) just like a physical server connects to the network. Note that the router is usually assigned the first IP address, the DHCP server is assigned the next IP address, and the virtual machines are assigned the remaining IP addresses in the pool. Similarly, a DHCP server has an associated DHCP namespace that allows you to distribute IP addresses and transfer all traffic within a given leased network.
Troubleshooting Neutron Namespace Errors
Without a proper understanding of network namespaces as a fundamental concept in building an OpenStack network, troubleshooting and network troubleshooting is akin to finding a needle in a haystack. Imagine a small OpenStack implementation with approximately 1,000 tenants, with each tenant of the Neutron network including at least one router and one private network. The total number of namespaces created will be at least 2000, since each leased router is a namespace, and each network will have an associated DHCP namespace (that is, two namespaces for each tenant, as shown in Figure 1). In this scenario, let’s say a tenant has problems distributing DHCP, allowing the instance to show the assigned IP addresses,
To solve this problem, you need to check the various components of the leased network topology shown in Fig. 1, mainly a DHCP agent. The first logical step is to define a tenant namespace. At the same time, we will see how this process differs from troubleshooting when deploying a traditional physical network. Here, not understanding how the namespace works in Neutron, the administrator will be completely confused and will not be able to continue troubleshooting using logical procedures.
So, the steps for finding and fixing errors are as follows:
1. Correctly determine the namespace.
2. Carry out general diagnostics in the established namespace.
Create two leased networks to illustrate the identification and elimination of errors in namespaces.
Figure 2. Network for tenant A
Figure 3. Network for tenant B
Step 1: Defining a Namespace
In the above diagrams, the network for tenant A has two private networks and one router; the network for tenant B is also two networks and one router. Consider these networks. Note that in a multimode configuration that includes the controller node, network node, and compute nodes, namespace commands will be executed from the network node.
@ vmnet-stack mn: the ip ~ $ sudo netns list
[sudo] password for stack:
qdhcp-eb2367bd-6e43-4de7-a0ab-d58ebf6e7dc0
qdhcp-9f7ec6fd-ec9a-4b1b b854-a9832f190922-
qrouter-4587f8d0-08a7-434a-aaeb -74e68a50c48d
qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd
qdhcp-e0fe9037-790a-4c6b-9ff4fc
We see a list of six namespaces of two leased networks (three namespaces for each tenant).
OK, but how do you know which namespace belongs to which tenant, because the names of tenants in namespaces are not designated? There are two ways to do this.
Firstly, it is worth noting that, as a rule, only the cloud administrator has access to the network node, so only the administrator can run namespace commands on the network node. One way to determine the namespace of a cloud administrator is to establish the source of the user credential file for the tenant. After that, all neutron commands will be executed under this user and only the components of the leased network to which this user belongs will be shown. We illustrate this as follows.
1. Identify the source of the credential file owned by tenant A.
2. Run the following commands:
stack @ vmnet-mn: ~ $ neutron router-list
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| id | name | external_gateway_info |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
| 4adef4a5-1122-49b8-9da2-3cbf803c44dd | router_proj_one | {"Network_id": "ea00b8c3-7063-4780-ad73-ef0b47518f10"} |
+ -------------------------------------- + ---------- ------- + ------------------------------------------ -------------- +
It can be seen from the list that tenant A has one router with the identifier 4adef4a5-1122-49b8-9da2-3cbf803c44dd
Having found this ID in the ip netns command above, we can easily determine that the qrouter-4adef4a5-1122-49b8-9da2-3cbf803c44dd router namespace belongs to tenant A.
For the DHCP namespace:
stack @ vmnet-mn: ~ $ neutron net-list
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| id | name | subnets |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
| e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c | net_proj_one | f02b6df4-11af-411d-b41e-e4ce66d5510d 10.10.1.0/24 |
| ea00b8c3-7063-4780-ad73-ef0b47518f10 | ext_net | 4f7c6c9e-28d8-43b3-b7d4-276920216b31 |
| f5a69443-bf3f-4f9a-b3e2-5625e70514b4 | net2 | f6cd4df6-b37f-4615-94f6-8abcab38ef99 192.168.0.0/24 |
+ -------------------------------------- + ---------- ---- + --------------------------------------------- -------- +
In addition, from this list you can identify two private networks with identifiers 0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and f5a69443-bf3f-4f9a-b3e2-5625e70514b4.
After checking the command from the ip netns list above, we can also set up the following DHCP namespaces: qdhcp-e0fe9037-790a-4c6b-9bf4-4b06f0cfcf5c and qdhcp-f5a69443-bf3f-4f9a-b3e2-5625e70514b4
The procedures described above are effective and the simplest if the cloud administrator knows the tenant’s username / password, but this is not always the case. Therefore, another way that a cloud administrator can determine the tenant’s namespace is to ask the user for a network ID or network name. A rental network user can easily find out this information in the Horizon project's OpenStack Dashboard control panel. Knowing either the name or the ID, the cloud administrator can troubleshoot as follows:
1. Track the administrator credentials. Knowing them, the cloud administrator will have access to all the namespaces of the leased network.
2. Make a list of namespaces and select a tenant namespace.
stack @ vmnet-mn: ~ $ ip netns list | grep 9f7ec6fd-ec9a-4b1b-b854-a9832f190922
qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922
Step 2: Find and fix namespace errors
Now that we’ve figured out the tenant’s namespace, we can begin to troubleshoot it.
First, check the IP address.
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ifconfig
[sudo] password for stack:
lo Link encap: Local Loopback
inet addr: 127.0.0.1 Mask: 255.0.0.0
inet6 addr : :: 1/128 Scope: Host
UP LOOPBACK RUNNING MTU: 16436 Metric: 1
RX packets: 16 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 16 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 1344 (1.3 KB) TX bytes: 1344 (1.3 KB)
tap43cb2c73-40 Link encap: Ethernet HWaddr fa: 16: 3e: e6: 1b: 24
inet addr: 10.1.0.3 Bcast: 10.1.0.255 Mask: 255.255.255.0
inet6 addr: fe80 :: f816: 3eff: fee6: 1b24 / 64 Scope: Link
UP BROADCAST RUNNING MULTICAST MTU: 1500 Metric: 1
RX packets: 965 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 561 errors: 0 dropped: 0 overruns: 0 carrier: 0
collisions: 0 txqueuelen: 0
RX bytes: 178396 (178.3 KB) TX bytes: 93900 (93.9 KB)
Then execute Interface ping:
stack @ vmnet-mn: ~ $ sudo ip netns exec qdhcp-9f7ec6fd-ec9a-4b1b-b854-a9832f190922 ping –c 3 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56 (84) bytes of data.
64 bytes from 10.1.0.3: icmp_req = 1 ttl = 64 time = 0.081 ms
64 bytes from 10.1.0.3: icmp_req = 2 ttl = 64 time = 0.035 ms
64 bytes from 10.1.0.3: icmp_req = 3 ttl = 64 time = 0.055 ms
The steps described above helped us establish the appropriate namespace and verify the correctness of the available (i.e., running and working) IP address. Further troubleshooting steps may include the following:
• Check if the dnsmasq process connected to the subnet is working for DHCP:
stack @ vmnet-mn: ~ $ ps –aux | grep dhcp
Based on the results of the above command, determine that the process in the selected network (for example, 10.0.0.0) is working, otherwise the dnsmasq process connected to this network may not start.
• Verify that the firewall is not blocking data transfers to / from the subnet.
conclusions
Before summing up, let us look at the following:
• When creating a router or network, namespaces are not created immediately. For a network, DHCP namespaces are created only when the VM is connected, and for the router, the namespace is created when the gateway is installed. This means that there must be some activity before creating namespaces.
• When you delete a router or network, the associated namespaces are not deleted. They must be removed manually.
Just as network namespaces are critical to building an OpenStack network, understanding them by the OpenStack cloud administrator is important. We looked at the implementation of network namespaces in OpenStack, taking into account what data the OpenStack cloud administrator needs to have when solving problems with a rented OpenStack network.
Now, if you wish, you can familiarize yourself with the presentation on this topic. The Network and the Namespaces by Neutron IPtables - A Contents Technical Deep Dive from Mirantis Original article in English