September 3, 2016 at 10:37LXC on QoS (replace ifb with veth)
Probably anyone who set up traffic prioritization in Linux was faced with the fact that the ifb capabilities for managing incoming traffic are rather scarce. Next, I will talk about a number of typical problems when building QoS, and how these problems were solved using containers.
It so happened that we can only control the speed of the network stream in the "exit" direction. What has already come to our interface, many bottlenecks have already passed, and it would seem that it makes no sense to discard or delay this traffic. But, since most protocols (TCP and those built on top of UDP, or have their own implementation) have mechanisms for controlling the outgoing stream, which can take into account the actual bandwidth of the client, and change the sending speed. In such a situation, it makes sense to control the flow that goes to the client on our side. There are many mechanisms that implement this; I will consider some of them.
The simplest and most trouble-free situation. Hereinafter, we will take it for granted (to simplify): the gateway itself does not generate traffic, it simply transmits it.
So, everything is simple here, we hang the queues on both interfaces, and control the speed to the clients (incoming), on the interface that looks to the local network (on its outgoing stream). Outgoing, respectively, on the external interface. Everything works, nothing interesting, move on.
Now, not everything is so good. How to divide the "incoming" stream between internal networks? Queues of neighboring interfaces do not know anything about the real load on the channel. Here you can divide the entire speed in half, and give half to each network. As a result, if the second network does not use the channel, and the second wants the maximum, then it will receive only half of it ... Sadness, but ifb (or its almost dead competitor imq) will help us.
The idea of ifb is that we place the pseudo-interface in front of our real one, and thus we begin to control the "incoming" stream of the real interface. Unfortunately, there is too much tar in this barrel. It turns out that ifb is so special that the labeling and filtering mechanisms of the iptables utility cannot be applied to it, it is only possible with tc. Why is tc bad? And the fact that in terms of filtering and labeling, he knows very little and the syntax is already completely different.
No matter how bad it is with tc, he can solve the task quite well, put it on the incoming interface, and we will remove the rules from the internal ones, they are no longer needed there.
Now everything is getting really bad. We have a new interface that works like a normal one, but, among other things, it “quietly” uses the bandwidth of another interface.
Now we can’t efficiently distribute traffic, ifb will not help us here either. The solution remains “forehead”: to allocate a fixed lane for the tunnel, and manage the contents of the tunnel as a normal interface (by hanging ifb and regular queues on it).
Everything is “not bad”, only in vain can we lose the speed given to the tunnel or to the regular network (how to look).
In general, the main idea in using netns is that it is easier to use a container (although it consumes more resources, mainly disk space).
So: we need an intermediate chain of interfaces between "external" and "internal" interfaces. It is on this chain that you can easily make QoS of all traffic, and the maximum that we lose from it is 4.5% (count the overhead yourself, here I took IPSec \ GRE) into tunneling and encryption (to ensure that everyone gets the set lane, we assume that all traffic that comes / leaves us comes from the tunnel).
I think that everyone can create a container on LXC, I will consider only some particulars that we may need.
So, in the container configuration we need:
Add our real external interface to the container (we will “push” it into the container, so it will “disappear” from the main namespace):
Or similarly for vlan:
Add the interface through which we will communicate with the main system (the script will raise the interface in the main system, the configuration must be set in advance):
Add the automatic start and stop of the interfaces of the main system, when starting / stopping the container:
Add the ability to use tun \ tap interfaces:
Let us control some routing parameters (who knows how to make more “narrow” permissions, I ask for a hint):
If necessary, make the container autostart.
In the main system, we need a profile for the route0 interface that will appear when the container starts, I assume that you are using NetworkManager.
What you need to configure yet:
Now we just set up the queues on the outgoing flows of interfaces connecting the main system with the container. Now, there is no problem with determining the destination of traffic, and we can use the entire available band from our provider (taking into account the% of overhead from the tunnels).
PS The era of "basins" that distribute the Internet under Linux is rapidly going away, the cost of hardware routers is becoming more attractive, and their power and flexibility are not much inferior to the "big" brothers, the same Mikrotik will solve this "problem" without any problems.
Why such problems?
It so happened that we can only control the speed of the network stream in the "exit" direction. What has already come to our interface, many bottlenecks have already passed, and it would seem that it makes no sense to discard or delay this traffic. But, since most protocols (TCP and those built on top of UDP, or have their own implementation) have mechanisms for controlling the outgoing stream, which can take into account the actual bandwidth of the client, and change the sending speed. In such a situation, it makes sense to control the flow that goes to the client on our side. There are many mechanisms that implement this; I will consider some of them.
Typical situations and problems
On one interface in the Internet and in a local area network, no tunnels
The simplest and most trouble-free situation. Hereinafter, we will take it for granted (to simplify): the gateway itself does not generate traffic, it simply transmits it.
So, everything is simple here, we hang the queues on both interfaces, and control the speed to the clients (incoming), on the interface that looks to the local network (on its outgoing stream). Outgoing, respectively, on the external interface. Everything works, nothing interesting, move on.
Add another internal interface
Now, not everything is so good. How to divide the "incoming" stream between internal networks? Queues of neighboring interfaces do not know anything about the real load on the channel. Here you can divide the entire speed in half, and give half to each network. As a result, if the second network does not use the channel, and the second wants the maximum, then it will receive only half of it ... Sadness, but ifb (or its almost dead competitor imq) will help us.
The idea of ifb is that we place the pseudo-interface in front of our real one, and thus we begin to control the "incoming" stream of the real interface. Unfortunately, there is too much tar in this barrel. It turns out that ifb is so special that the labeling and filtering mechanisms of the iptables utility cannot be applied to it, it is only possible with tc. Why is tc bad? And the fact that in terms of filtering and labeling, he knows very little and the syntax is already completely different.
No matter how bad it is with tc, he can solve the task quite well, put it on the incoming interface, and we will remove the rules from the internal ones, they are no longer needed there.
Now we will connect the two offices using the tunnel
Now everything is getting really bad. We have a new interface that works like a normal one, but, among other things, it “quietly” uses the bandwidth of another interface.
Now we can’t efficiently distribute traffic, ifb will not help us here either. The solution remains “forehead”: to allocate a fixed lane for the tunnel, and manage the contents of the tunnel as a normal interface (by hanging ifb and regular queues on it).
Everything is “not bad”, only in vain can we lose the speed given to the tunnel or to the regular network (how to look).
LXC will help us!
In general, the main idea in using netns is that it is easier to use a container (although it consumes more resources, mainly disk space).
So: we need an intermediate chain of interfaces between "external" and "internal" interfaces. It is on this chain that you can easily make QoS of all traffic, and the maximum that we lose from it is 4.5% (count the overhead yourself, here I took IPSec \ GRE) into tunneling and encryption (to ensure that everyone gets the set lane, we assume that all traffic that comes / leaves us comes from the tunnel).
I think that everyone can create a container on LXC, I will consider only some particulars that we may need.
So, in the container configuration we need:
Add our real external interface to the container (we will “push” it into the container, so it will “disappear” from the main namespace):
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = enp3s6
Or similarly for vlan:
lxc.network.type = vlan
lxc.network.flags = up
lxc.network.link = enp2s0
lxc.network.vlan.id = 603
Add the interface through which we will communicate with the main system (the script will raise the interface in the main system, the configuration must be set in advance):
lxc.network.type = veth
lxc.network.flags = up
lxc.network.veth.pair = route0
lxc.network.name = eth1
lxc.network.hwaddr = 02:b2:30:41:30:25
lxc.network.script.up = /usr/bin/nmcli connection up route0
Add the automatic start and stop of the interfaces of the main system, when starting / stopping the container:
lxc.hook.pre-start = /var/lib/lxc/route0/pre-start.sh
lxc.hook.post-stop = /var/lib/lxc/route0/post-stop.sh
/var/lib/lxc/route0/pre-start.sh
#!/bin/sh
/usr/bin/nmcli connection down vlan603 >/dev/null 2>&1
exit 0
/var/lib/lxc/route0/post-stop.sh
#!/bin/sh
/usr/bin/nmcli connection up vlan603 >/dev/null 2>&1
exit 0
Add the ability to use tun \ tap interfaces:
lxc.hook.autodev = /var/lib/lxc/route0/autodev.sh
/var/lib/lxc/route0/autodev.sh
#!/bin/bash
cd ${LXC_ROOTFS_MOUNT}/dev
mkdir net
mknod net/tun c 10 200
chmod 0666 net/tun
Let us control some routing parameters (who knows how to make more “narrow” permissions, I ask for a hint):
lxc.mount.auto = proc:rw sys:ro
If necessary, make the container autostart.
In the main system, we need a profile for the route0 interface that will appear when the container starts, I assume that you are using NetworkManager.
What you need to configure yet:
- Configure the route0 interface IP address on the main system, for example: 192.168.20.22/30 and the default route 192.168.20.21
- In the container, you need to configure the ip address of the eth1 interface, for example: 192.168.20.21/30 and the route to the internal network 192.168.21.0/24 via 192.168.20.22
- In the container, you need to configure the "external" interface, masking and forwarding.
- In the main system, we will most likely need only forwarding.
- Everything that should go in an external network (including tunnels), we place in the container.
Now we just set up the queues on the outgoing flows of interfaces connecting the main system with the container. Now, there is no problem with determining the destination of traffic, and we can use the entire available band from our provider (taking into account the% of overhead from the tunnels).
PS The era of "basins" that distribute the Internet under Linux is rapidly going away, the cost of hardware routers is becoming more attractive, and their power and flexibility are not much inferior to the "big" brothers, the same Mikrotik will solve this "problem" without any problems.
