Newbies about linux control in Linux

Published on November 21, 2011

Newbies about linux control in Linux

Some time ago I was asked to set up a simple traffic balancing in a remote branch. They work, poor fellows, through ADSL, and sending large volume emails (document scans) clogs them with the entire return channel, which leads to problems in working with office online programs via VPN.
They use Linux (Fedora) as a gateway. Prior to this, I saw a couple of times how this balancing is configured via ipfw on FreeBSD, and since I know the iptables mechanism quite well, I did not expect much problems. But searching the Internet, I was unpleasantly surprised that iptables was not my assistant at all. And the knowledge about the order of packets passing through its tables and rules will hardly be useful to me. Need to learn tc from iproute2 package.

Unexpectedly for myself, I spent two days in order to more or less understand traffic balancing using iproute2. At first, the article about HTB (not here ) was not the best for a beginner . Various examples from the Internet were also sometimes stupid, as they often did not contain a description of specific options or the meaning of their application. Therefore, I tried to collect the knowledge I received in one article, and most importantly to describe everything at a level accessible to beginners.

I’ll make a reservation right away, we’ll only cut traffic coming from the network interface. Incoming can also be adjusted, but this requires additional tricks.

Classless Disciplines

So, on Linux, a discipline (qdisc) is assigned to each network interface to control traffic . It is from disciplines that the entire traffic management system is built. But you should not be afraid, in fact, discipline is just an algorithm for processing a queue of network packets.
Several disciplines on one interface can be involved, and the so-called root discipline (root qdisc) is attached directly to the interface . In addition, each interface has its own root discipline.


By default, after booting the system, root qdisc sets the algorithm for processing packets of the pfifo_fast type .



# tc qdisc

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 1

pfifo_fast- This is the usual “First Input - First Output” algorithm, but with some prioritization of traffic. This type of discipline contains three FIFO queues with different packet processing priorities. Packets are laid out on them based on the ToS (Type of Service) flag in each IP packet. A packet falling into FIFO0 has the highest priority for processing, while in FIFO2 it has the smallest. ToS itself requires a separate discussion, so I propose to limit myself to the fact that the operating system itself knows which ToS to assign to the sent IP packet. For example, in telnet and ping packets, ToS will have different values.

0: - handle to the root discipline.
Descriptors should be of the form major_number: minor_number , but for disciplines the lowest number should always be 0, and therefore it can be omitted.

The priomap parameter 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 just sets the bit correspondence of the ToS field to each internal queue (band). For example, with ToS = 4, the packet is processed in queue 1, with ToS = 7 in queue 0.

Some sources indicate that the parameters of the pfifo_fast discipline cannot be changed, we believe.


Now we will consider how to limit the speed of the general outgoing traffic. To do this, we will assign a discipline of the TBF type (Token Bucket Filter) as the root discipline of the interface .

# tc qdisc add dev eth0 root tbf rate 180kbit latency 20ms buffer 1540

rate 180kbit - sets the transfer rate threshold on the interface.

latency 20ms - sets the maximum time a data packet spent waiting for a token.

buffer 1540 - set the size of the token buffer in bytes. In the examples, they write that for a limit of 10Mbit / s, a buffer of 10Kbytes is enough. The main thing is not to make it too small, more is possible. Approximate calculation formula: rate_in_Bytes / 100.


The discipline of TBF uses the token mechanism for its work. Tokens are generated by the system at a constant speed and placed in the buffer (bucket). For each token that leaves the buffer from the interface, an IP packet is sent.
If the packet transfer rates and the generation of tokens coincide, the data transfer process proceeds without delay.
If the packet transfer rate is less than the token rate, the latter begin to accumulate in the buffer and then can be used for short-term data transmission at a speed higher than the threshold.
If the packet transfer rate is higher, tokens begin to be missed. Data packets wait for new tokens for a while, and then begin to be discarded.

The described two disciplines belong to the so-called classless disciplines. They have a number of functional limitations: they are connected only to the interface (or the edge class), plus you cannot use packet filters for them. And accordingly, my task of balancing mail traffic with their help will not be possible.
By the way, the full set of classless disciplines is somewhat wider: pfifo, bfifo, sqf (provides the same speed of packets received from different streams), esqf, etc.

Class disciplines

If disciplines can be represented as segments of water pipes, then classes are connectors (fittings). It can be a simple fitting adapter, or it can be a tricky fitting splitter with a dozen taps.

The parent of the class can be either a discipline or another class.
Child classes can join a class (docking multiple fittings).

A class that does not have child classes is called a leaf class . Here, the data packets, having run through our “water supply system”, leave the traffic control system and are sent by the network interface. By default, any edge class has an attached discipline of type fifo, and it is it that determines the order in which packets are transmitted for this class. But the whole charm is that we can exchange this discipline for any other.
If you add a child class, this discipline is deleted.


Let's return to the task of balancing mail traffic and consider the class discipline prio .
It is very similar to pfifo_fast already described. But this discipline is special in that when it is assigned, three classes are automatically created (the number can be changed with the bands parameter).

Replace the root discipline of the interface with prio.

# tc qdisc add dev eth0 root handle 1: prio

handle 1: - set the handle to this root qdisc. In class disciplines, it is then indicated when classes are connected.

Check discipline:

# tc qdisc

qdisc prio 1: dev eth0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

Check classes:

# tc -d -s class show dev eth0

class prio 1: 1 parent one:
Sent 734914 bytes 7875 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
class prio 1: 2 parent 1:
Sent 1555058583 bytes 8280199 pkt (dropped 124919, overlimits 26443 requeues 0)
backlog 0b 0p requeues 0
class prio 1: 3 parent 1:
Sent 57934378 bytes 802213 pkt (dropped 70976, overlimits 284608 requeues 0)
backlog 0b 0p requeues 0

We see three classes with identifiers 1: 1, 1: 2 and 1: 3 connected to parent discipline 1: of prio type (classes must have a common identifier major_number with their parent).
That is, on the “pipe” root qdisc, which shares data streams in the same way as pfifo_fast does, we installed a tee. Guided by ToS, high-priority traffic gets to class 1: 1, normal traffic to class 1: 2, and “garbage” to class 1: 3.

Suppose the reverse channel ADSL produces a speed of 90KB / s. We divide it into 20KB / s for mail and 70KB / c for everything else.

Traffic from class 1: 1 will not be specifically limited. Its packets can always occupy at least the entire channel width due to the high priority of ToS, but the amount of traffic in this class will be negligible in comparison with the other two classes. Therefore, we do not reserve a separate strip for it.

Standard traffic, as a rule, falls into the class 1: 2. We connect to the second output of the tee class a discipline pipe at 70KB / s:

# tc qdisc add dev eth0 parent 1: 2 handle 10: tfb rate 70kbps buffer 1500 latency 50ms

To the third output of the tee, connect the discipline pipe at 20KB / c:

# tc qdisc add dev eth0 parent 1: 3 handle 20: tfb rate 20kbps buffer 1500 latency 50ms

All three of these classes are edge.

And now it remains only to direct the mail traffic not to class 1: 2, as happened before, but to class 1: 3. This is done using class discipline filters.

# tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 25 0xffff flowid 1: 3

parent 1: - the filter can only be attached to the discipline and is called from it. Based on the triggering of the filter, the discipline decides in which class the packet processing will continue.

protocol ip- we determine the type of network protocol

prio 1 - the parameter has been confusing for a long time, since it is used in classes and filters, plus this is the name of the discipline. Here prio sets the priority for triggering the filters, the filters with the lower prio are the first to be activated.

u32 - the so-called traffic classifier, which can perform packet selection by any of its criteria: by sender / receiver ip-address, source / receiver port, by protocol type. These conditions, in fact, are indicated below.

match ip dport 25 0xffff - sets the filter to trigger when packets are sent to port 25. 0xffff is a bitmask for the port number.

flowid 1: 3 - indicate in which class packets are sent when this filter is triggered.

Done rough, but complete the task.

We look at the packet statistics:

# tc -s -d qdisq show dev eth0
# tc -s -d class show dev eth0
# tc -s -d filter show dev eth0 You can

quickly remove all classes, filters and return the root qdisc of the interface to its original state command:

# tc qdisc del dev eth0 root


On the other hand, our return channel is already too thin to reserve 20KB / s only for sending email. Therefore, it is better to use the class discipline HTB (Hierarchical Token Bucket). It allows you to borrow bandwidth from the child classes of the parent.


# tc qdics add dev eth0 root handle 1: htb default 20

default 20 - set the class by default. It will process packets that do not fall into other classes of the htb discipline. If you do not specify it, then “default 0” will be assigned and all unclassified (non-filtering) traffic will be sent at the interface speed.

# tc class add dev eth0 parent 1: classid 1: 1 htb rate 90kbps ceil 90kbps

we attach to the root qdisc class with the identifier 1: 1. Thus, we limit the speed on the interface to 90KB / s.

classid 1: 1 - class identifier.

rate 90kbps - set the lower bandwidth threshold for the class.

ceil 90kbps - set the upper bandwidth threshold for the class.

# tc class add dev eth0 parent 1: 1 classid 1:10 htb rate 20kbps ceil 70kbps

create a class 1:10, a child of the class 1: 1. Then, outgoing mail traffic will be sent to it by a filter.

rate 20kbps - set the guaranteed lower bandwidth threshold for the class.

ceil 70kbps- set the upper bandwidth threshold for the class. If the parent class has free bandwidth (the presence of “extra” tokens), class 1:10 will be able to temporarily raise the data transfer rate, up to the specified limit of 70KB / s.

# tc class add dev eth0 parent 1: 1 classid 1:20 htb rate 70kbps ceil 90kbps

Create a default class. All other traffic will fall into it. In the same way, we set the rate and ceil parameters to expand the bandwidth if there is already no mail traffic.

# tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 25 0xffff flowid 1:10

a u32-based filter that forwards packets sent to port 25 to class 1:10.

By the way, the documentation states that in fact in HTB traffic shaping occurs only in the edge classes, in our case 1:10 and 1:20. Specification of bandwidth limitation parameters in other HTB classes is necessary only for the functioning of the borrowing system between classes.

When adding a class, it is also possible to specify the prio parameter . It sets the priority of the class (0 - max priority). Classes with lower priority are not processed as long as there is data in higher priority classes.

Linux Advanced Routing & Traffic Control HOWTO
Traffic Control HOWTO
A Tale of Linux and Traffic Management