November 21, 2016 at 12:43Monitoring and Configuring the Linux Network Stack: Retrieving Data
- Transfer
In this article, we will examine how packets are received on computers running the Linux kernel, and we will also analyze the monitoring and configuration of each component of the network stack as packets move from the network to user space applications. You will find a lot of source code here, because without a deep understanding of the processes, you cannot configure and monitor the Linux network stack.
We also recommend that you familiarize yourself with the illustrated manual on the same topic , there are explanatory diagrams and additional information.
Contents
1. General advice on monitoring and configuring the Linux network stack
2. Problem overview
3. Detailed analysis
3.1. Network device driver
3.2. SoftIRQ
3.3. Network device subsystem in Linux
3.4. Receive Packet Steering (RPS)
3.5. Receive Flow Steering (RFS) Mechanism
3.6. Hardware Accelerated Receive Flow Steering (aRFS)
3.7. Moving up the network stack using netif_receive_skb
3.8. netif_receive_skb
3.9. Registration of protocol level
3.10. Additional information
4. Conclusion
1. General advice on monitoring and configuring the network stack in Linux
The network stack is complex and there is no universal solution for all occasions. If performance and correctness when working with the network are critical to you or your business, then you will have to invest a lot of time, effort and money in understanding how the various parts of the system interact with each other.
Ideally, you should measure packet loss at each level of the network stack. In this case, you need to choose which components need to be configured. It is at this point, it seems to me, that many give up. This assumption is based on the fact that sysctl settings or / proc values can be used repeatedly and in bulk. In some cases, it is likely that the system is so permeated with interconnections and filled with nuances that if you want to implement useful monitoring or perform tuning, you will have to deal with the functioning of the system at a low level. Otherwise, just use the default settings. This may be enough until further optimization (and attachments to track these settings) is needed.
Many of the example settings given in this article are used solely as illustrations, and are not a recommendation for or against the use of a specific configuration or default settings. So before applying each setting, first think about what you need to monitor in order to detect a significant change.
It is dangerous to apply network settings by connecting to the machine remotely. You can easily block your access or even drop the network system. Do not apply the settings on working machines, first run them as far as possible on new ones, and then apply them to production.
2. Overview
You may want to have a copy of the device data sheet on hand. This article will discuss the Intel I350 controller, controlled by the igb driver. Download the specification from here .
The high-level path that the packet passes from arrival to the socket receive buffer looks like this:
- The driver loads and initializes.
- The packet arrives from the network to the network card.
- The packet is copied (via DMA) to the kernel's circular memory buffer.
- A hardware interrupt is generated to let the system know when a packet appears in memory.
- The driver calls NAPI to start the poll loop, if it has not already started.
- The ksoftirqd processes run on each CPU of the system. They are registered at boot time. These processes pull packets from the ring buffer by calling the poll NAPI function, registered by the device driver during initialization.
- Are unmapped those areas of memory in the ring buffer in which the network data was written.
- Data sent directly to memory (DMA) is transferred for further processing to the network layer in the form of 'skb'.
- If packet management is enabled, or if there are several receive queues in the network card, then incoming network data frames are distributed across several CPUs of the system.
- Network data frames are passed from the queue to the protocol layers.
- Protocol layers process data.
- Data is added to receive buffers attached to sockets by protocol layers.
Further we will consider in detail this entire stream. As a protocol layer, IP and UDP layers will be considered. Most of the information is also true for other protocol layers.
3. Detailed analysis
We will consider the Linux kernel version 3.13.0. Also, code samples and links to GitHub are used throughout the article.
It is very important to understand exactly how packets are received by the kernel. We will have to carefully read and understand the operation of the network driver, so that later it will be easier to delve into the description of the network stack.
As a network driver, igb will be considered. It is used in a fairly common server network card, Intel I350. So let's start by parsing this driver.
3.1. Network device driver
Initialization
The driver registers the initialization function called by the kernel when the driver loads. Registration is done using the module_init macro.
You can find the igb initialization function (igb_init_module) and register it with module_init in drivers / net / ethernet / intel / igb / igb_main.c . Everything is quite simple:
/**
* igb_init_module – подпрограмма (routine) регистрации драйвера
*
* igb_init_module — это первая подпрограмма, вызываемая при загрузке драйвера.
* Она выполняет регистрацию с помощью подсистемы PCI.
**/
static int __init igb_init_module(void)
{
int ret;
pr_info("%s - version %s\n", igb_driver_string, igb_driver_version);
pr_info("%s\n", igb_copyright);
/* ... */
ret = pci_register_driver(&igb_driver);
return ret;
}
module_init(igb_init_module);As we will see later, the bulk of the work of initializing the device occurs when pci_register_driver is called.
PCI initialization
The Intel I350 Network Card is a PCI express device .
PCI devices identify themselves through a series of registers in the PCI configuration space .
When the device driver is compiled, the macro MODULE_DEVICE_TABLE (from include / module.h ) is used to export the table of identifiers for PCI devices that the driver can control . Below we will see that the table is also registered as part of the structure.
This table is used by the kernel to determine which driver to load to control the device. Thus, the operating system understands which device is connected and which driver allows you to interact with it.
You can find the table and PCI device IDs for the igb driver, respectively, here drivers / net / ethernet / intel / igb / igb_main.c and here drivers / net / ethernet / intel / igb / e1000_hw.h :
static DEFINE_PCI_DEVICE_TABLE(igb_pci_tbl) = {
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I354_BACKPLANE_1GBPS) },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I354_SGMII) },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I354_BACKPLANE_2_5GBPS) },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I211_COPPER), board_82575 },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I210_COPPER), board_82575 },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I210_FIBER), board_82575 },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I210_SERDES), board_82575 },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I210_SGMII), board_82575 },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I210_COPPER_FLASHLESS), board_82575 },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_I210_SERDES_FLASHLESS), board_82575 },
/* ... */
};
MODULE_DEVICE_TABLE(pci, igb_pci_tbl);As we saw above, pci_register_driver is called by the driver initialization function.
This function registers the structure of pointers. Most of them are function pointers, but the PC device identifier table is also registered. The kernel uses the functions registered by the driver to start the PCI device.
From drivers / net / ethernet / intel / igb / igb_main.c :
static struct pci_driver igb_driver = {
.name = igb_driver_name,
.id_table = igb_pci_tbl,
.probe = igb_probe,
.remove = igb_remove,
/* ... */
};PCI Probe Function
When a device is identified by its PCI ID, the kernel can select the appropriate driver. Each driver registers a probe function in the kernel PCI system. The kernel calls this function for those devices for which drivers have not yet claimed. When one of the drivers claims to be a device, the others are no longer interrogated. Most drivers contain a lot of code that runs to prepare the device for use. The procedures performed vary greatly depending on the driver.
Here are some typical procedures:
- Enabling a PCI device.
- Query memory areas and I / O ports .
- Configure DMA mask .
- The ethtool functions supported by the driver are registered (will be described below).
- Watchdog timers are running (for example, the e1000e has a timer that checks if the hardware is hanging).
- Other procedures specific to this device. For example, bypassing or resolving hardware tricks, and the like.
- Create, initialize, and register a struct net_device_ops structure. It contains pointers to various functions needed to open the device, send data to the network, configure the MAC address, and so on.
- Create, initialize, and register a high-level struct net_device structure representing a network device.
Let's go through some of these procedures for the igb driver and the igb_probe function .
A quick look at PCI initialization
The code below from the igb_probe function performs basic PCI configuration. Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
err = pci_enable_device_mem(pdev);
/* ... */
err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
/* ... */
err = pci_request_selected_regions(pdev, pci_select_bars(pdev,
IORESOURCE_MEM),
igb_driver_name);
pci_enable_pcie_error_reporting(pdev);
pci_set_master(pdev);
pci_save_state(pdev);First, the device is initialized using pci_enable_device_mem. If the device is in sleep mode, it wakes up, memory sources are activated, and so on.
Then the DMA mask is configured. Our device can read and write to the addresses of 64-bit memory, therefore with the help of DMA_BIT_MASK (64) dma_set_mask_and_coherent is called.
By calling pci_request_selected_regions, memory areas are reserved. The PCI Express Advanced Error Reporting service starts if its driver is loaded. By calling pci_set_master, DMA is activated, and the PCI configuration space is saved by calling pci_save_state.
Fuh.
Additional PCI Driver Information for Linux
A full discussion of the operation of a PCI device is beyond the scope of this article, but you can read these materials:
- free-electrons.com/doc/pci-drivers.pdf
- wiki.osdev.org/PCI
- github.com/torvalds/linux/blob/v3.13/Documentation/PCI/pci.txt
Network device initialization
The igb_probe function does the important job of initializing a network device. In addition to the procedures specific to PCI, it performs more general operations for working with the network and functioning of the network device:
- Registers struct net_device_ops.
- Logs ethtool operations.
- Gets the default MAC address from the network card.
- Configures the net_device property flags.
- And does a lot more.
We will need all this later, so let's briefly go over it.
struct net_device_ops
struct net_device_ops contains function pointers to many important operations required by the network subsystem to control the device. We will mention this structure more than once in the article.
The net_device_ops structure is attached to the struct net_device in igb_probe. Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
{
/* ... */
netdev->netdev_ops = &igb_netdev_ops;In the same file, function pointers are stored in the net_device_ops structure. Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
static const struct net_device_ops igb_netdev_ops = {
.ndo_open = igb_open,
.ndo_stop = igb_close,
.ndo_start_xmit = igb_xmit_frame,
.ndo_get_stats64 = igb_get_stats64,
.ndo_set_rx_mode = igb_set_rx_mode,
.ndo_set_mac_address = igb_set_mac,
.ndo_change_mtu = igb_change_mtu,
.ndo_do_ioctl = igb_ioctl,
/* ... */As you can see, there are several interesting fields in the struct, for example, ndo_open, ndo_stop, ndo_start_xmit and ndo_get_stats64, which contain the addresses of functions implemented by the igb driver. We will consider some of them in more detail below.
Register ethtool
ethtool is a command line driven program. With it, you can receive and configure various drivers and hardware options. Under Ubuntu, this program can be installed like this: apt-get install ethtool.
Typically, ethtool is used to collect detailed statistics from network devices. Other applications will be described below.
The program communicates with drivers using the ioctl system call . The device driver registers a series of functions performed for ethtool operations, and the kernel provides glue.
When ethtool calls ioctl, the kernel finds the ethtool structure registered by the appropriate driver and performs registered functions. Implementation of the ethtool driver function can do anything - from changing a simple software flag in the driver to controlling how the physical NIC equipment works by writing register values to the device.
Using the igb_set_ethtool_ops call, the igb driver registers ethtool operations with igb_probe:
static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
{
/* ... */
igb_set_ethtool_ops(netdev);The entire igb driver ethtool code along with the igb_set_ethtool_ops function can be found in the drivers / net / ethernet / intel / igb / igb_ethtool.c file .
Adapted from drivers / net / ethernet / intel / igb / igb_ethtool.c :
void igb_set_ethtool_ops(struct net_device *netdev)
{
SET_ETHTOOL_OPS(netdev, &igb_ethtool_ops);
}In addition, you can find the igb_ethtool_ops structure with the igb driver ethtool functions configured in the corresponding fields.
Adapted from drivers / net / ethernet / intel / igb / igb_ethtool.c :
static const struct ethtool_ops igb_ethtool_ops = {
.get_settings = igb_get_settings,
.set_settings = igb_set_settings,
.get_drvinfo = igb_get_drvinfo,
.get_regs_len = igb_get_regs_len,
.get_regs = igb_get_regs,
/* ... */Each driver, at its discretion, decides which ethtool functions are relevant and which should be implemented. Unfortunately, not all drivers implement all the functions of ethtool.
The get_ethtool_stats function, which (if implemented) creates detailed statistical counters that are tracked either by the driver software or by the device itself, is quite interesting.
In the monitoring part, we will look at how to use ethtool to get these statistics.
IRQ
When a data frame is written to memory using DMA, how does the network card tell the system that the data is ready for processing?
Typically, a card generates an interrupt that signals the arrival of data. There are three common types of interrupts: MSI-X, MSI, and Legacy IRQ. Soon we will consider them. The interrupt generated when writing data to memory is quite simple, but if many frames arrive, then a large number of IRQs are also generated. The more interruptions, the less CPU time is available for servicing higher-level tasks, for example, user processes.
New Api (NAPI)was created as a mechanism to reduce the number of interrupts generated by network devices as packets arrive. But still, NAPI cannot completely save us from interruptions. Later we will find out why.
Napi
For a number of important features, NAPI differs from the legacy data collection method. It allows the device driver to register the poll function called by the NAPI subsystem to collect the data frame.
The algorithm for using NAPI drivers for network devices looks like this:
- The driver includes NAPI, but initially it is in an inactive state.
- A packet arrives and the network card directly sends it to memory.
- The network card generates IRQ by starting the interrupt handler in the driver.
- The driver wakes up the NAPI subsystem using SoftIRQ (more on this below). She begins to collect packets, calling the poll function registered by the driver in a separate thread of execution.
- The driver should disable subsequent interrupt generation by the network card. This is necessary in order to allow the NAPI subsystem to process packets without interference from the device.
- When all the work is done, the NAPI subsystem is turned off, and interrupt generation by the device is turned on again.
- The cycle repeats from point 2.
This method of collecting data frames has reduced the load compared to the legacy method, since many frames can be received at the same time without the need to simultaneously generate IRQ for each of them.
The device driver implements the poll function and registers it with NAPI, calling netif_napi_add. The driver also sets the weight. Most drivers hardcode value 64. Why exactly it, we will see further.
Typically, drivers register their poll NAPI functions during driver initialization.
Initializing NAPI in the igb driver
The igb driver does this with a long chain of calls:
- igb_probe calls igb_sw_init.
- igb_sw_init calls igb_init_interrupt_scheme.
- igb_init_interrupt_scheme calls igb_alloc_q_vectors.
- igb_alloc_q_vectors calls igb_alloc_q_vector.
- igb_alloc_q_vector calls netif_napi_add.
As a result, a number of high-level operations are performed:
- If MSI-X is supported , then it is enabled by calling pci_enable_msix.
- Various settings are calculated and initialized; for example, the number of transmit and receive queues that the device and driver will use to send and receive packets.
- igb_alloc_q_vector is called once for each created transmit and receive queue.
- Each call to igb_alloc_q_vector also calls netif_napi_add to register the poll function for a particular queue. When the poll function is called to collect packages, an instance of struct napi_struct will be passed to it.
Let's take a look at igb_alloc_q_vector to understand how callback poll and its private data are logged.
Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
static int igb_alloc_q_vector(struct igb_adapter *adapter,
int v_count, int v_idx,
int txr_count, int txr_idx,
int rxr_count, int rxr_idx)
{
/* ... */
/* размещает в памяти q_vector и кольца (rings) */
q_vector = kzalloc(size, GFP_KERNEL);
if (!q_vector)
return -ENOMEM;
/* инициализирует NAPI */
netif_napi_add(adapter->netdev, &q_vector->napi, igb_poll, 64);
/* ... */Above is the code for placing the igb_poll function in the memory of the queue for receiving and registering using the NAPI subsystem. We get a link to the struct napi_struct associated with this newly created receive queue (& q_vector-> napi). When the time comes to collect packets from the queue and igb_poll is called by the NAPI subsystem, this link will be passed to it.
We will understand the importance of the described algorithm when we examine the data stream from the driver to the network stack.
Bring up network device
Remember the net_device_ops structure that registered the set of functions for booting a network device, transmitting packets, setting a MAC address, and so on?
When the network device is loaded (for example, using ifconfig eth0 up), a function is called that is attached to the ndo_open field of the net_device_ops structure.
The ndo_open function usually does the following:
- Allocates memory for receive and transmit queues.
- Includes NAPI.
- Registers an interrupt handler.
- Enables hardware interrupts.
- And much more.
In the case of the igb driver, igb_open calls a function attached to the ndo_open field of the net_device_ops structure.
Preparing to receive data from the network
Most modern network cards use DMA to write data directly to memory, from where the operating system can retrieve it for further processing. Most often, the structure used for this is similar to a queue created on the basis of a ring buffer.
First, the device driver must, together with the OS, reserve in memory the area that will be used by the network card. Next, the card is informed about the allocation of memory, where later incoming data will be recorded, which can be taken and processed using the network subsystem.
It looks simple, but what if the frequency of the packets is so high that one CPU does not have time to process them? The data structure is based on a fixed-size memory area, so packets will be discarded.
In this case, the mechanism may help.Receive Side Scaling (RSS) , a multi-queue system.
Some devices can write incoming packets to several different areas of memory at the same time. Each area serves a separate queue. This allows the OS to use multiple CPUs for parallel processing of incoming data at the hardware level. But not all network cards can do this.
Intel I350 - can. We see evidence of this skill in the igb driver. One of the first things he does after loading is to call the igb_setup_all_rx_resources function . This function calls once for each receive queue another function - igb_setup_rx_resources, which organizes the DMA memory into which the network card will write incoming data.
If you are interested in the details, readgithub.com/torvalds/linux/blob/v3.13/Documentation/DMA-API-HOWTO.txt .
Using ethtool, you can configure the number and size of receive queues. Changing these parameters can significantly affect the ratio of processed and discarded frames.
To determine which queue to send data to, the network card uses a hash function in the header fields (source, destination, port, and so on).
Some network cards allow you to configure the weight of receive queues, so you can direct more traffic to specific queues.
Less common is the ability to configure the hash function itself. If you can configure it, you can direct a specific thread to a specific queue, or even drop packets at the hardware level.
Below we will look at how the hash function is configured.
Enabling NAPI
When a network device is loaded, the driver typically includes NAPI. We have already seen how drivers register poll functions using NAPI. Normally, NAPI does not power on until the device is booted.
Turning it on is pretty simple. A napi_enable call signals struct napi_struct that NAPI is enabled. As noted above, after turning on, NAPI is in an inactive state.
In the case of the igb driver, NAPI is enabled for each q_vector initialized after loading the driver, or when the counter or queue size is changed using ethtool.
Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
for (i = 0; i < adapter->num_q_vectors; i++)
napi_enable(&(adapter->q_vector[i]->napi));Register Interrupt Handler
After enabling NAPI, you need to register an interrupt handler. A device can generate interrupts in a variety of ways: MSI-X, MSI, and Legacy interrupts. Therefore, the code may be different, depending on the supported methods.
The driver must determine which method is supported by this device and register the corresponding handler function that executes when an interrupt is received.
Some drivers, including igb, try to register a handler for each method, in case of failure, move on to the next untried one.
It is preferable to use MSI-X interrupts, especially for network cards that support multiple receive queues. The reason is that each queue is assigned its own hardware interrupt, which can be processed by a specific CPU (using irqbalance or modifying / proc / irq / IRQ_NUMBER / smp_affinity). As we will soon see, the interrupt and the packet are processing the same CPU. Thus, incoming packets will be processed by different CPUs within the entire network stack, starting from the level of hardware interrupts.
If MSI-X is not available, then the driver uses MSI (if supported), which still has advantages over legacy interrupts. Read more about this on the English Wikipedia .
In the igb driver, the functions igb_msix_ring, igb_intr_msi, igb_intr act respectively as interrupt handlers for MSI-X, MSI, and Legacy.
Driver code trying each method can be found in drivers / net / ethernet / intel / igb / igb_main.c :
static int igb_request_irq(struct igb_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
struct pci_dev *pdev = adapter->pdev;
int err = 0;
if (adapter->msix_entries) {
err = igb_request_msix(adapter);
if (!err)
goto request_done;
/* переход к MSI */
/* ... */
}
/* ... */
if (adapter->flags & IGB_FLAG_HAS_MSI) {
err = request_irq(pdev->irq, igb_intr_msi, 0,
netdev->name, adapter);
if (!err)
goto request_done;
/* переход к легаси */
/* ... */
}
err = request_irq(pdev->irq, igb_intr, IRQF_SHARED,
netdev->name, adapter);
if (err)
dev_err(&pdev->dev, "Error %d getting interrupt\n", err);
request_done:
return err;
}As you can see, the driver first tries to use the igb_request_msix handler for MSI-X interrupts; if it fails, it switches to MSI. Request_irq is used to register the igb_intr_msi MSI handler. If this does not work, the driver switches to legacy interrupts. Request_irq is again used to register igb_intr.
Thus, the igb driver registers a function that will be executed when the network card generates an interrupt that signals the receipt of data and its readiness for processing.
Enabling Interrupts
At this point, almost everything is already configured. It remains only to turn on interrupts and wait for data to arrive. The activation procedure depends on the specific equipment, the igb driver does this in __igb_open by calling the igb_irq_enable helper function.
By writing registers, interrupts for this device are enabled:
static void igb_irq_enable(struct igb_adapter *adapter)
{
/* ... */
wr32(E1000_IMS, IMS_ENABLE_MASK | E1000_IMS_DRSTA);
wr32(E1000_IAM, IMS_ENABLE_MASK | E1000_IMS_DRSTA);
/* ... */
}The network device is now booted. Drivers can perform other procedures, such as starting timers, work queues, or other operations, depending on the specific device. Upon completion, the network card is loaded and ready for use.
Let's now look at monitoring and configuring network device driver options.
Network device monitoring
There are several different monitoring methods that differ in the detail of statistics and complexity. Let's start with the most detailed.
Using ethtool -S
You can install ethtool under Ubuntu like this: sudo apt-get install ethtool.
Now you can view the statistics by passing the -S flag with the name of the network device whose statistics you are interested in.
Monitor detailed statistics (e.g. packet drops) with `ethtool -S`.
$ sudo ethtool -S eth0
NIC statistics:
rx_packets: 597028087
tx_packets: 5924278060
rx_bytes: 112643393747
tx_bytes: 990080156714
rx_broadcast: 96
tx_broadcast: 116
rx_multicast: 20294528
....Monitoring can be a difficult task. Data is easy to obtain, but there is no standard representation of field values. Different drivers, even different versions of the same driver, can generate different field names that have the same value.
Look for values that have “drop,” “buffer,” “miss,” and so on. Next, you need to read data from the source. You can determine which values apply only to the software (for example, they are incremented in the absence of memory), and which come directly from the equipment by reading the registers. In the case of register values, refer to the network card specification to see what specific counters mean. Many ethtool names may be erroneous.
Using sysfs
sysfs also provide a lot of statistics, but it is slightly higher level than that provided directly by the network card.
You can find out the number of dropped incoming frames for, for example, eth0 by applying cat to the file.
Monitoring higher level network card statistics with sysfs:
$ cat /sys/class/net/eth0/statistics/rx_dropped
2The values of the counters will be scattered by the files: collisions, rx_dropped, rx_errors, rx_missed_errors and so on.
Unfortunately, this driver decides what these or other fields mean, which of them need to be incremented and where to get the values from. You may have noticed that some drivers regard a certain type of error situation as dropping a packet, while others see it as missing.
If these values are important to you, then study the source code of the driver to find out exactly what it thinks about each of the values.
Using / proc / net / dev
An even higher-level file / proc / net / dev, which provides a squeeze for each network adapter in the system.
We read / proc / net / dev to monitor high-level network card statistics:
$ cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
eth0: 110346752214 597737500 0 2 0 0 0 20963860 990024805984 6066582604 0 0 0 0 0 0
lo: 428349463836 1579868535 0 0 0 0 0 0 428349463836 1579868535 0 0 0 0 0 0This file contains a set of some values that you can find in the mentioned sysfs file files. It can be used as a general source of information.
As in the previous case, if these values are important for you, then refer to the source of the driver. Only in this case you can find out exactly when, where and why these values are incremented, that is, what is considered an error, discard or FIFO here.
Configure network devices
Checking the number of receive queues used
If the network card and device driver loaded on your system support RSS (multiple queues), then using ethtool you can usually configure the number of receive queues (receive channels, RX-channels).
Checking the number of queues for receiving a network card:
$ sudo ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 4The output reflects the pre-configured highs (imposed by the driver or hardware) and the current settings.
Note: not all device drivers support this operation.
The error that occurs if your network card does not support the operation:
$ sudo ethtool -l eth0
Channel parameters for eth0:
Cannot get device channel parameters
: Operation not supportedThis means that the driver did not implement the ethtool get_channels operation. The reason may be the lack of support for setting the number of queues from the network card, the lack of RSS support, or your driver version is too old.
Setting the number of receive queues
After you find the counters of the current and maximum number of queues, you can configure their values using sudo ethtool -L.
Note: some devices and their drivers support only combined queues, for receiving and transmitting, as in the example in the previous chapter.
Using ethtool -L, we assign 8 combined queues:
$ sudo ethtool -L eth0 combined 8If your device and driver allow you to separately configure the number of receive and transmit queues, then you can separately set 8 receive queues:
$ sudo ethtool -L eth0 rx 8Note: for most drivers, such changes will cause the interface to crash and reboot, because connections to it will be interrupted. Although a single change is not too important.
Setting the size of receive queues
Some network cards and their drivers support setting the size of the receive queue. The specific implementation is hardware dependent, but fortunately, ethtool provides a standard configuration method. An increase in size prevents network data from being dropped when there are a large number of incoming frames. True, the data can still be discarded at the software level, so to completely eliminate or reduce the drop, you will need to perform additional configuration.
Checking the current size of the network card queue using ethtool –g:
$ sudo ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512The output shows that the equipment supports 4096 transmit and receive descriptors, but currently uses 512.
Let's increase the size of each queue to 4096:
$ sudo ethtool -G eth0 rx 4096Note: for most drivers, such changes will cause the interface to crash and reboot, because connections to it will be interrupted. Although a single change is not too important.
Setting the weight of processing of receive queues
Some network cards allow you to configure the distribution of network data between receive queues by changing their weights.
This can be done if:
- The network card supports indirect flow addressing (flow indirection).
- Your driver implements the get_rxfh_indir_size and get_rxfh_indir functions from ethtool.
- You have a fairly new version of ethtool that supports the -x and -X command line options, which respectively display and configure the indirection table.
Checking the indirect addressing table of the receive stream:
$ sudo ethtool -x eth0
RX flow hash indirection table for eth3 with 2 RX ring(s):
0: 0 1 0 1 0 1 0 1
8: 0 1 0 1 0 1 0 1
16: 0 1 0 1 0 1 0 1
24: 0 1 0 1 0 1 0 1Here, on the left, the values of packet hashes and receive queues are displayed - 0 and 1. A packet with hash 2 will be addressed to queue 0, and a packet with hash 3 will be addressed to queue 1.
Example: evenly distribute processing between the first two receive queues:
$ sudo ethtool -X eth0 equal 2If you need to configure custom weights so that the number of packets addressed to specific queues (and consequently, the CPU), then you can do this on the command line using ethtool –X:
$ sudo ethtool -X eth0 weight 6 2Here, queue 0 is assigned weight 6, and queue 1 is weighted 2. Thus, most of the data will be processed by queue 0.
As we will now see, some network cards also allow you to configure the fields that are used in the hashing algorithm.
Configuring receive hash fields for network streams
Using ethtool, you can customize the fields that will be used to calculate the hashes used in RSS.
Using ethtool -n, check which fields are used for the hash of the UPD receive stream:
$ sudo ethtool -n eth0 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DAIn the case of eth0, the IPv4 source and destination address are used to calculate the hash of the UDP stream. Let's add inbound and outbound ports:
$ sudo ethtool -N eth0 rx-flow-hash udp4 sdfnThe sdfn value does not seem clear. An explanation of each letter can be found on the ethtool man page.
Configuring hash fields is quite useful, but ntuple filtering is even more useful and provides finer control over the distribution of streams in receive queues.
Ntuple filtering for network flow control
Some network cards support the ntuple filtering function. It allows you to specify (via ethtool) a set of parameters used to filter input data at the iron level and address to a specific receive queue. For example, you can specify that TCP packets arriving at a specific port be sent to queue 1.
In Intel network cards, this function is usually called Intel Ethernet Flow Director . Other manufacturers may give other names.
As we will see later, ntuple filtering is a critical component of another function, Accelerated Receive Flow Steering (aRFS). This greatly facilitates the use of ntuple if your network card supports it. We will consider aRFS later.
This function can be useful if the operational requirements of the system involve maximizing data locality in order to increase the frequency of hit hits to the CPU cache when processing network data. For example, the configuration of a web server running on port 80 looks like this:
- The server is assigned to CPU 2.
- IRQ processing for the receive queue is assigned to the same CPU.
- TCP traffic on port 80 is "filtered" using ntuple and sent to CPU 2.
- All incoming traffic to port 80 is processed by this CPU, starting from receiving data and up to user space programs.
- Efficiency monitoring requires careful monitoring of the system, including the frequency of successful cache accesses and network stack latency.
As mentioned above, ntuple filtering can be configured using ethtool, but first you need to make sure that this feature is enabled on your device. You can verify this by:
$ sudo ethtool -k eth0
Offload parameters for eth0:
...
ntuple-filters: off
receive-hashing: onAs you can see, ntuple-filters is off.
Turn on ntuple filters:
$ sudo ethtool -K eth0 ntuple onAfter you enable them or verify that the filters work, you can check the configured rules:
$ sudo ethtool -u eth0
40 RX rings available
Total 0 rulesFilters here do not have rules configured. You can add them via the command line using ethtool. Let's write that all TCP traffic coming to port 80 is forwarded to receive queue 2:
$ sudo ethtool -U eth0 flow-type tcp4 dst-port 80 action 2Ntuple filtering can also be used to drop packets of certain streams at the hardware level. This can be convenient for reducing the traffic load from some IP addresses. For more information on configuring filter rules, read man ethtool.
You can obtain statistics on the degree of success of your ntuple rules by checking the output of ethtool -S [device name]. For example, on Intel network cards fdir_match and fdir_miss give out the number of matches and misses of the filtering rules. For tracking the statistics of your device, refer to the source of the driver and the specification.
3.2. Softirq
Before proceeding to consider the network stack, we need to briefly walk through the SoftIRQ system from the Linux kernel.
What is SoftIRQ?
This is a mechanism for executing code outside the context of the interrupt handler implemented by the driver. This system is important because hardware interrupts can be disabled during all or part of the execution time of the handler. The longer the interrupts are disabled, the more likely it is to skip events. It is important to take any long-term actions beyond the limits of the interrupt handler in order to complete them as soon as possible and re-enable interrupts from the device.
There are mechanisms that can be used to defer work in the kernel for the needs of the network stack. We will consider SoftIRQ.
The SoftIRQ system can be represented as a series of kernel threads (one per CPU), in which the handler functions registered for different SoftIRQ events work. If you ever went upstairs and met ksoftirqd / 0 in the list of kernel threads, then this is just the SoftIRQ thread running on CPU 0.
The kernel subsystems (for example, for working with the network) can register the SoftIRQ handler by executing the open_softirq function. Next we will see how the network management system registers its SoftIRQ handlers. Now let's talk a little about how SoftIRQ works.
ksoftirqd
Since SoftIRQ is very important for deferring device drivers, the ksoftirqd process is created as part of the kernel life cycle quite early.
Take a look at the code from kernel / softirq.c showing how the ksoftirqd system is initialized:
static struct smp_hotplug_thread softirq_threads = {
.store = &ksoftirqd,
.thread_should_run = ksoftirqd_should_run,
.thread_fn = run_ksoftirqd,
.thread_comm = "ksoftirqd/%u",
};
static __init int spawn_ksoftirqd(void)
{
register_cpu_notifier(&cpu_nfb);
BUG_ON(smpboot_register_percpu_thread(&softirq_threads));
return 0;
}
early_initcall(spawn_ksoftirqd);As follows from the definition of struct smp_hotplug_thread, two function pointers are registered: ksoftirqd_should_run and run_ksoftirqd.
Both functions are called by kernel / smpboot.c as part of something resembling an event loop.
The code in kernel / smpboot.c first calls ksoftirqd_should_run, which determines if there are any pending SoftIRQs. If there is, then run_ksoftirqd is executed, which performs some minor calculations before calling __do_softirq.
__do_softirq
The __do_softirq function does some interesting things:
- Defines a pending SoftIRQ.
- It takes into account SoftIRQ time for statistics.
- Increments SoftIRQ execution statistics.
- Executes a handler for the pending SoftIRQ (which was registered using the open_softirq call).
So when you look at the graphs of CPU usage and see softirq or si, this measures the amount of CPU resources that are used by the deferred working context.
Monitoring
/ proc / softirqs
The softirq system increments statistical counters that can be read from / proc / softirqs. Monitoring these statistics will give you an idea of how often SoftIRQs are generated for different events.
Read / proc / softirqs to monitor SoftIRQ statistics:
$ cat /proc/softirqs
CPU0 CPU1 CPU2 CPU3
HI: 0 0 0 0
TIMER: 2831512516 1337085411 1103326083 1423923272
NET_TX: 15774435 779806 733217 749512
NET_RX: 1671622615 1257853535 2088429526 2674732223
BLOCK: 1800253852 1466177 1791366 634534
BLOCK_IOPOLL: 0 0 0 0
TASKLET: 25 0 0 0
SCHED: 2642378225 1711756029 629040543 682215771
HRTIMER: 2547911 2046898 1558136 1521176
RCU: 2056528783 4231862865 3545088730 844379888Thanks to this file, you will understand how the processing of incoming network data (NET_RX) is currently distributed between processes. If uneven, the counters of some CPUs will have higher values than others. This is an indication that you can benefit from the Receive Packet Steering / Receive Flow Steering described below. Be careful when relying on this file only when monitoring performance: during a period of high network activity, you can expect NET_RX to increase, but this may not happen. SoftIRQ NET_RX frequency may be affected by additional settings in the network stack, we will touch on this later.
So, be careful, but if you change the mentioned settings, then the changes can be seen in / proc / softirqs.
Now let's move on to the network stack and see how the received network data flows through it from top to bottom.
3.3. Linux Network Device Subsystem
We got acquainted with the operation of network drivers and SoftIRQ, now we are going to initialize the network device subsystem. Then we will trace the package following its arrival.
Initializing a Network Device Subsystem
The network device subsystem (netdev) is initialized in the net_dev_init function. In it, a lot of interesting things happen.
Initializing struct softnet_data structures
net_dev_init creates a set of struct softnet_data structures for each CPU in the system. These structures contain pointers to several things that are important for processing network data:
- List of NAPI structures to register for the CPU.
- Backlog for data processing.
- Processing weight
- A list of the receive offload structure .
- Receive packet steering settings .
- And much more.
Initializing SoftIRQ Handlers
net_dev_init registers the SoftIRQ receive and transmit handler that will be used to process incoming or outgoing network data. The code is pretty simple:
static int __init net_dev_init(void)
{
/* ... */
open_softirq(NET_TX_SOFTIRQ, net_tx_action);
open_softirq(NET_RX_SOFTIRQ, net_rx_action);
/* ... */
}Now let's see how the driver interrupt handler “raises” (or starts) the net_rx_action function registered on SoftIRQ NET_RX_SOFTIRQ.
Data arrival
Network data finally arrived!
We assume that the receive queue has sufficiently accessible descriptors, so that the packet is written to memory using DMA. Then the device generates an interrupt assigned to the packet (or, in the case of MSI-X, the interrupt is tied to the receive queue to which the arrived packet is addressed).
Interrupt handler
The handler executed after the interrupt is generated should generally try to push as much work as possible out of the context of the interrupt. This is critical because during the processing of one interrupt, others may be blocked.
Let's look at the source code for the MSI-X interrupt handler. This will help illustrate the idea that a handler does a minimal amount of work.
Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
static irqreturn_t igb_msix_ring(int irq, void *data)
{
struct igb_q_vector *q_vector = data;
/* Пишет значение ITR, вычисленное из предыдущего прерывания. */
igb_write_itr(q_vector);
napi_schedule(&q_vector->napi);
return IRQ_HANDLED;
}This is a very short handler, and before returning it performs two very fast operations.
- The igb_write_itr function simply updates a specific hardware register. It is used to track the frequency of occurrence of hardware interrupts. The register is used in conjunction with the hardware function “Interrupt Throttling” (also called “interrupt interconnection”, Interrupt Coalescing), which allows you to adjust the delivery of interrupt CPU. Soon we will see how ethtool provides a mechanism for adjusting the frequency of IRQ generation.
- Napi_schedule is called, waking up the NAPI processing cycle if it is not already active. Note that this loop executes in SoftIRQ, not an interrupt handler. The handler simply starts its execution if it is not already done.
It is important to see the real code demonstrating the operation of the above operations. This will help us understand how network data is processed in multiprocessor systems.
NAPI and napi_schedule
Let's see how the napi_schedule call from the hardware interrupt handler works.
As you remember, NAPI exists specifically for collecting network data without the use of interruptions from the network card, indicating that the data is ready for processing. It was mentioned above that the poll loop is bootstrapped when a hardware interrupt is received. In other words, NAPI is enabled but not active until the first packet arrives. At this point, the network card generates an interrupt and the NAPI starts. Below we will consider other cases when NAPI can be turned off, and before starting it, you need to generate an interrupt.
The poll loop starts when the driver interrupt handler calls napi_schedule. This is just a wrapper function defined in the header file, which in turn calls __napi_schedule.
Adapted from net / core / dev.c :
/**
* __napi_schedule – расписание получения
* @n: запись в расписании
*
* Будет задан старт функции получения записи
*/
void __napi_schedule(struct napi_struct *n)
{
unsigned long flags;
local_irq_save(flags);
____napi_schedule(&__get_cpu_var(softnet_data), n);
local_irq_restore(flags);
}
EXPORT_SYMBOL(__napi_schedule);In this code, __get_cpu_var is used to get the softnet_data structure registered to the current CPU. This structure is passed to ____napi_schedule together with the struct napi_struct structure taken from the driver. How many underscores.
Let's look at ____napi_schedule, taken from net / core / dev.c :
/* Вызывается с отключённым IRQ */
static inline void ____napi_schedule(struct softnet_data *sd,
struct napi_struct *napi)
{
list_add_tail(&napi->poll_list, &sd->poll_list);
__raise_softirq_irqoff(NET_RX_SOFTIRQ);
}This code does two important things:
- The struct napi_struct structure, taken from the driver interrupt handler code, is added to the poll_list attached to the softnet_data structure associated with the current CPU.
- __raise_softirq_irqoff is used to launch SoftIRQ NET_RX_SOFTIRQ. As a result, net_rx_action starts, registered during the initialization of the network device subsystem, if it is not already done.
As we will see shortly, the SoftIRQ net_rx_action handler function to collect the packets will call the NAPI poll function.
Note about CPU and network data processing
Note that all the code before that, which transferred the work from the hardware interrupt handler to SoftIRQ, used the structures associated with the current CPU.
Since the driver IRQ handler does little, the SoftIRQ handler will run on the same CPU as the IRQ handler. Therefore, it is important to configure how the specific IRQ will be processed by the CPU: after all, this CPU will be used not only to execute the driver interrupt handler, but also to collect packets in SoftIRQ via NAPI.
Later we will see that mechanisms like Receive Packet Steering can distribute part of the work to other CPUs within the network stack.
Network Data Monitoring
Hardware Interrupt Requests
Note: monitoring hardware interrupts does not give a complete picture of the correctness of packet processing. Many drivers disable hardware interrupts while NAPI is running. But still, this is an important part of the overall monitoring solution.
Read / proc / interrupts to monitor hardware interrupt statistics:
$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 46 0 0 0 IR-IO-APIC-edge timer
1: 3 0 0 0 IR-IO-APIC-edge i8042
30: 3361234770 0 0 0 IR-IO-APIC-fasteoi aacraid
64: 0 0 0 0 DMAR_MSI-edge dmar0
65: 1 0 0 0 IR-PCI-MSI-edge eth0
66: 863649703 0 0 0 IR-PCI-MSI-edge eth0-TxRx-0
67: 986285573 0 0 0 IR-PCI-MSI-edge eth0-TxRx-1
68: 45 0 0 0 IR-PCI-MSI-edge eth0-TxRx-2
69: 394 0 0 0 IR-PCI-MSI-edge eth0-TxRx-3
NMI: 9729927 4008190 3068645 3375402 Non-maskable interrupts
LOC: 2913290785 1585321306 1495872829 1803524526 Local timer interruptsBy monitoring statistics in / proc / interrupts, you can see the change in the number and frequency of hardware interrupts as packets arrive. You can also make sure that each receive queue of your network card is processed by the corresponding CPU. The quantity tells us only how many interrupts were, but this metric does not necessarily help to understand how much data was received or processed, because many drivers disable network card interrupts as part of the contract with the NAPI subsystem. Moreover, the use of interrupt coalescing also affects statistics obtained from this file. Its monitoring allows you to determine whether the interrupt aggregation settings you have selected really work.
To get a better idea of the correct processing of network data, you need to monitor / proc / softirqs and a number of files in / proc. We will talk about this below.
Configure network data arrival
Interrupt merge
This is the name of the method for preventing the transfer of interrupts from the device to the CPU until a certain number of events or the amount of specific work is typed.
This helps to avoid " interrupt storms " and increase throughput or delay, depending on the settings. Reducing the number of generated interrupts leads to an increase in throughput and delay, reducing the load on the CPU. With the increase in the number of interrupts, the opposite happens: the delay and throughput decrease, but the CPU load increases.
Historically, earlier versions of igb, e1000, and a number of other drivers only supported the InterruptThrottleRate parameter. In more recent versions, it has been replaced by a generic function from ethtool.
Retrieving the current IRQ merge settings:
$ sudo ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
...ethtool provides a generic interface for handling union settings. However, remember that not every device or driver supports all settings. Check the documentation or source code to find out what is supported in your case. As the documentation for ethtool says: "Everything that is not implemented by the driver will be silently ignored."
Some drivers support the curious “adaptive RX / TX IRQ coalescing” option. Usually it is sold in equipment. The driver needs to do some work to tell the network card to enable this option, as well as some bookkeeping (as we saw earlier in the igb driver code).
Enabling this option allows you to adapt the number of interrupts to improve latency with a small number of packets, and improve throughput with a large number of packets.
Enabling adaptive combining of receive interrupts:
$ sudo ethtool -C eth0 adaptive-rx onYou can use ethtool -C to configure several options. Some of the most commonly used are:
- rx-usecs: duration of the delay between the arrival of the packet and the generation of a receive interrupt, in milliseconds.
- rx-frames: the maximum number of frames received before generating a receive interrupt.
- rx-usecs-irq: The length of the receive interruption delay while it is being served by the host, in milliseconds.
- rx-frames-irq: the maximum number of frames received before generating a receive interrupt while the system is servicing the interrupt.
And there are many more.
I remind you that your hardware and driver can support only part of the options. Examine the driver source code and network card specification.
Unfortunately, customizable options are poorly documented everywhere except in the header file. Refer to the source code include / uapi / linux / ethtool.h and find an explanation for each option supported by ethtool (but not the fact that they are supported by your driver and network card).
Note: at first glance, combining interrupts looks like a very useful optimization. In some cases, this is true, but you need to make sure that the rest of the network stack is also configured correctly. Just changing the parameters of the union will probably bring little benefit.
Configure IRQ Bindings
If your map supports RSS, or if you are trying to optimize the locality of the data, you may need to use a specific set of CPUs to process the interrupts generated by the map.
This allows you to segment CPU usage. Such changes can affect the operation of the upper levels, as we saw in the example of the network stack.
If you decide to configure IRQ bindings, first check if the irqbalance daemon is working. It tries to automatically balance interrupts and CPU, and therefore can overwrite your settings. Disable irqbalance, or use --banirq along with IRQBALANCE_BANNED_CPUS so that irqbalance knows that it should not touch the interrupt set you configure and the CPU.
Then check the / proc / interrupts file for a list of interrupt numbers for each receive queue for your network card. Finally, make changes to / proc / irq / IRQ_NUMBER / smp_affinity, specifying for each number which CPU should handle this interrupt. Just specify a hexadecimal bitmask for this file so that the kernel knows which CPUs to use for handling interrupts.
Example: configure IRQ 8 binding to CPU 0:
$ sudo bash -c 'echo 1 > /proc/irq/8/smp_affinity'Start processing network data
When the SoftIRQ code determines which of the SoftIRQ are pending, net_rx_action is executed and the network data processing begins.
Let's take part of the net_rx_action processing loop to understand how it works, what you can configure and monitor.
Net_rx_action processing cycle
net_rx_action starts processing packets from memory that were sent there by the device via DMA.
The function iterates over the list of NAPI structures standing in the queue of the current CPU, in turn retrieves each structure that works with it.
The processing cycle limits the amount of work and execution time of the registered poll NAPI functions. He does this in two ways:
- tracking the work budget (which can be customized)
- and also checks the elapsed time.
Adapted from net / core / dev.c :
while (!list_empty(&sd->poll_list)) {
struct napi_struct *n;
int work, weight;
/* Если исчерпалось окно SoftIRQ - отфутболиваем
* Выполняйте это в течение двух тиков, это позволит сделать
* среднюю задержку на уровне 1.5/Гц.
*/
if (unlikely(budget <= 0 || time_after_eq(jiffies, time_limit)))
goto softnet_break;Thus, the kernel does not allow packet processing to occupy all CPU resources. budget is the entire available budget, which will be divided into all available NAPI structures registered on this CPU.
This is another reason why network cards with multiple queues require careful configuration of the IRQ binding. Recall that on a CPU that handles interrupts from the device, the SoftIRQ handler will also be executed. And as a result, the above cycle runs on the same CPU and budget calculations are performed.
Systems with multiple network cards, each of which supports multiple queues, may be in a situation where several NAPI structures are registered on one CPU. Processing data of all structures on one CPU is “paid” from the same budget.
If you do not have enough CPU to distribute interrupts, then you can increase net_rx_action budget so that each CPU processes more packets. An increase in the budget will entail an increase in the load on the CPU (especially sitime or si in top or other programs), but should reduce the delay, because the data will be processed faster.
Note: The CPU will be time limited by two jiffies , regardless of the assigned budget.
NAPI function poll and weight
Let me remind you that network device drivers use netif_napi_add to register the poll function. As we saw earlier, the igb driver contains this code:
/* инициализация NAPI */
netif_napi_add(adapter->netdev, &q_vector->napi, igb_poll, 64);This is where the NAPI structure is registered, with a weight of 64 registered in the code. Let's see how this weight is used in the net_rx_action processing cycle.
Adapted from net / core / dev.c :
weight = n->weight;
work = 0;
if (test_bit(NAPI_STATE_SCHED, &n->state)) {
work = n->poll(n, weight);
trace_napi_poll(n);
}
WARN_ON_ONCE(work > weight);
budget -= work;We get the weight registered to the NAPI structure and pass it to the poll function, also registered to the NAPI structure (in the above example, igb_poll).
The poll function returns the number of frames processed. It is saved as work, and then subtracted from the total budget.
Let's say:
- you use weight 64 from your driver (all drivers in Linux 3.13.0 have this value hardcoded)
- and your default budget size is 300.
Your system will stop processing data if:
- The igb_poll function was called at most 5 times (maybe even less if no data was processed, as we will see later),
- OR at least 2 jiffies have passed.
Contract between NAPI and Network Device Driver
Important information regarding the contract between the NAPI subsystem and device drivers has not yet been mentioned. We are talking about the conditions for stopping NAPI.
- If the poll driver function consumes all its weight (64), it should not change the state of the NAPI. The net_rx_action cycle will come into operation.
- If the poll driver function does NOT use all its weight, it should disable NAPI. NAPI will be reenabled upon receiving the next IRQ when the driver interrupt handler calls napi_schedule.
Now let's see how net_rx_action works with the first part of the contract. Then, after considering the poll function, we find out how the second part of the contract is implemented.
Net_rx_action loop completion
The net_rx_action processing cycle ends with a code section that executes the first part of the contract with NAPI. Adapted from net / core / dev.c :
/* Драйверы не должны изменять состояние NAPI, если они
* расходуют весь свой вес. В таких случаях код всё ещё
* «владеет» экземпляром NAPI, и, следовательно, может
* перемещать его по списку по своему желанию.
*/
if (unlikely(work == weight)) {
if (unlikely(napi_disable_pending(n))) {
local_irq_enable();
napi_complete(n);
local_irq_disable();
} else {
if (n->gro_list) {
/* сбрасываем слишком старые пакеты
* Если HZ < 1000, сбрасываем все пакеты.
*/
local_irq_enable();
napi_gro_flush(n, HZ >= 1000);
local_irq_disable();
}
list_move_tail(&n->poll_list, &sd->poll_list);
}
}If all the work is consumed, then net_rx_action handles two situations:
- The network device must be turned off (for example, because the user has executed ifconfig eth0 down).
- If the device does not turn off, then check if there is a generic receive offload (GRO) list. If the timer tick rate > = 1000, then all network streams with GRO that were recently updated will be reset. Later we will take a closer look at GRO. The NAPI structure moves to the end of the list of the given CPU, so the next iteration of the loop will receive the next registered NAPI structure.
Thus, the packet processing loop invokes the registered poll driver function, which will handle the processing. Further we will see that this function collects network data and sends them to the stack for further processing.
Exiting the loop when limits are reached
The net_rx_action loop will exit if:
- the poll list registered for this CPU no longer contains NAPI structures (! list_empty (& sd-> poll_list))
- or budget balance <= 0,
- or a time limit of two jiffies was reached.
Here is one of the above code examples
/* Если исчерпалось окно SoftIRQ - отфутболиваем.
* Выполняйте это в течение двух тиков, это позволит сделать
* среднюю задержку на уровне 1.5/Гц.
*/
if (unlikely(budget <= 0 || time_after_eq(jiffies, time_limit)))
goto softnet_break;If you track the label softnet_break, you'll come across something interesting. Adapted from net / core / dev.c :
softnet_break:
sd->time_squeeze++;
__raise_softirq_irqoff(NET_RX_SOFTIRQ);
goto out;Some statistics of the struct softnet_data structure are incremented and SoftIRQ NET_RX_SOFTIRQ is closed. The time_squeeze field is the number of times net_rx_action had a job, but there was not enough budget or a time limit was reached before the job was completed. This counter is extremely useful for understanding bottlenecks in network processing. Soon we will touch on this. NET_RX_SOFTIRQ is disabled to free up processing time for other tasks. This makes sense, because this small piece of code is only executed when more work can be done, but we do not need to monopolize the CPU.
Then, execution is passed to the label out. It can be passed out even if there are no more NAPI structures left for processing, that is, there is more budget than network activity, all drivers have closed NAPI, and net_rx_action has nothing to do.
Before returning from net_rx_action, the important thing is done in the out section: net_rps_action_and_irq_enable is called. If Receive Packet Steering is enabled , this function wakes up the remote CPUs so that they can start processing network data.
Below we will analyze the operation of RPS. In the meantime, let's see how to monitor the correctness of the net_rx_action processing cycle, and move on to the "internals" of the NAPI poll functions, moving along the network stack.
NAPI poll function
Let me remind you that drivers allocate a memory area into which the device can directly send incoming packets. The driver is responsible not only for allocating memory, but also for freeing them, collecting data, and sending them to the network stack.
How does the igb driver do all this?
igb_poll
Finally, we can analyze the operation of igb_poll. Her code is deceptively simple. Adapted from drivers / net / ethernet / intel / igb / igb_main.c :
/**
* igb_poll – NAPI Rx polling callback
* @napi: структура опроса (polling) NAPI
* @budget: счётчик количества пакетов, которые нужно обработать
**/
static int igb_poll(struct napi_struct *napi, int budget)
{
struct igb_q_vector *q_vector = container_of(napi,
struct igb_q_vector,
napi);
bool clean_complete = true;
#ifdef CONFIG_IGB_DCA
if (q_vector->adapter->flags & IGB_FLAG_DCA_ENABLED)
igb_update_dca(q_vector);
#endif
/* ... */
if (q_vector->rx.ring)
clean_complete &= igb_clean_rx_irq(q_vector, budget);
/* Если вся работа не завершена, бюджета возвращается и продолжается опрос */
if (!clean_complete)
return budget;
/* Если выполнено недостаточно работы по приёму, выходит из режима опроса */
napi_complete(napi);
igb_ring_irq_enable(q_vector);
return 0;
}There are a number of interesting things here:
- If the kernel includes support for direct cache access ( Direct Cache Access (DCA) ), then the CPU cache is “warmed up” so that calls to the RX ring get into it. You can read more about this in the chapter with ext. materials at the end of the article.
- Then the igb_clean_rx_irq function is called, which performs a difficult task. About it below.
- Next, clean_complete is checked to determine if there is still work that can be done. If so, then the budget is returned (the value 64 is hard-coded). net_rx_action moves the NAPI structure to the end of the poll list.
- Otherwise, the driver will turn off NAPI by calling napi_complete, and using igb_ring_irq_enable will enable interrupts again. The next interrupt that arrives will enable NAPI again.
Let's see how igb_clean_rx_irq sends network data up the stack.
igb_clean_rx_irq
The igb_clean_rx_irq function is a loop that processes one packet at a time until budget or data to run out runs out.
The following things are done in this loop:
- Additional buffers for receiving data are located in the memory in case the buffers used are cleared. They are added by IGB_RX_BUFFER_WRITE (16) at a time.
- Из очереди приёма извлекается буфер и сохраняется в структуре skb.
- Проверяется, является буфер “End of Packet”. Если да, то обработка продолжается. В противном случае продолжается извлечение дополнительных буферов из очереди приёма и добавление в skb. Это необходимо на тот случай, если полученный фрейм превышает размер буфера.
- Проверяется корректность схемы (layout) и заголовков данных.
- С помощью skb->len увеличивается счётчик количества обработанных байтов.
- В skb настраиваются хэш, контрольная сумма, временная метка, VLAN id и поля протокола. Первые четыре позиции предоставляются сетевой картой. Если она сообщает об ошибке контрольной суммы, то инкрементируется csum_error. Если с контрольной суммой всё в порядке, а данные получены по UDP или TCP, то skb помечается как CHECKSUM_UNNECESSARY. Если возникает сбой контрольной суммы, то с пакетом разбираются стеки протоколов. Нужный протокол вычисляется с помощью вызова eth_type_trans и сохраняется в структуре skb.
- Сделанный skb передаётся вверх по сетевому стеку с помощью вызова napi_gro_receive.
- Инкрементируется счётчик количества обработанных пакетов.
- Цикл продолжает выполняться до тех пор, пока количество обработанных пакетов не исчерпает бюджет.
When the loop is interrupted, the function writes counters of processed packets and bytes.
Before continuing with the discussion of the network stack, we will make a couple of digressions. First, let's see how to monitor and configure SoftIRQ network subsystem. Secondly, let's talk about Generic Receive Offloading (GRO). After we delve into napi_gro_receive, we will understand the work of the rest of the network stack.
Network data processing monitoring
/ proc / net / softnet_stat
As we saw in the previous chapter, statistical counters are incremented when you exit the net_rx_action cycle, or if there is work left that could be done, but we ran into a budget or time limit for SoftIRQ. These statistics are tracked as part of the struct softnet_data structure associated with the CPU. It is dumped to the / proc / net / softnet_stat file, which, unfortunately, has very little documentation. Fields are not named and may vary depending on the kernel version.
On Linux 3.13.0, you can figure out which values correspond to which fields in / proc / net / softnet_stat by reading the kernel sources. Adapted from net / core / net-procfs.c :
seq_printf(seq,
"%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
sd->processed, sd->dropped, sd->time_squeeze, 0,
0, 0, 0, 0, /* was fastroute */
sd->cpu_collision, sd->received_rps, flow_limit_count);Many of these counters have strange names and increment in unexpected places. Only by studying the network stack can we understand when and where each of them is incremented. Since squeeze_time statistics were found in net_rx_action, it seems to me that it makes sense to document this file now.
We read / proc / net / softnet_stat to monitor the statistics of processing network data:
$ cat /proc/net/softnet_stat
6dcad223 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
6f0e1565 00000000 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
660774ec 00000000 00000003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
61c99331 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
6794b1b3 00000000 00000005 00000000 00000000 00000000 00000000 00000000 00000000 00000000
6488cb92 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000Important details regarding / proc / net / softnet_stat:
- Each line / proc / net / softnet_stat corresponds to a struct softnet_data structure, one for each CPU.
- Values are separated by single spaces and displayed in hexadecimal.
- Первое значение, sd->processed, — это количество обработанных сетевых фреймов. Оно может превышать общее количество полученных фреймов, если вы используете связывание Ethernet (Ethernet bonding). Драйвер связывания может стать причиной повторной обработки данных, и тогда счётчик sd->processed для одного и того же пакета будет инкрементироваться больше одного раза.
- Второе значение, sd->dropped, — это количество отброшенных сетевых фреймов по причине нехватки места в очереди обработки. Об этом поговорим ниже.
- Третье значение, sd->time_squeeze, — это количество раз, когда цикл net_rx_action прерывался из-за истощения бюджета или достижения ограничения по времени, хотя работа ещё оставалась. Как объяснялось выше, помочь тут может увеличение budget.
- Следующие пять значений всегда равны 0.
- Девятое значение, sd->cpu_collision, — это количество коллизий, возникавших при попытках получения блокировки устройства в ходе передачи пакетов. Поскольку эта статья посвящена приёму пакетов, то мы эту статистику рассматривать не будем.
- Десятое значение, sd->received_rps, — это количество раз, когда посредством межпроцессорного прерывания будили CPU для обработки пакетов.
- Последнее значение, flow_limit_count, — это количество раз, когда было достигнуто ограничение потока (flow limit). Ограничение потока — это опциональная возможность управления принимаемыми пакетами (Receive Packet Steering), которую мы скоро рассмотрим.
If you decide to monitor this file and display the results graphically, then be extremely careful not to change the order of the fields and keep their meaning. To do this, check out the kernel sources.
Configuring network data processing
Net_rx_action budget setting
Setting the budget net_rx_action allows you to determine how much you can spend on processing packets for all NAPI structures registered on the CPU. To do this, use sysctl to configure the value of net.core.netdev_budget.
Example: Assign the total package processing budget to 600.
$ sudo sysctl -w net.core.netdev_budget=600You can also write this setting to the /etc/sysctl.conf file so that the change is saved after a reboot. On Linux 3.13.0, the default value is 300.
Generic Receive Offloading (GRO)
Generic Receive Offloading (GRO) is a software implementation of hardware optimization known as Large Receive Offloading (LRO).
The essence of both mechanisms is to reduce the number of packets transmitted over the network stack by combining “fairly similar” packets. This reduces the load on the CPU. Imagine that a large file is being transferred to us, and most packages contain data chunks from this file. Instead of sending small packets one at a time on the stack, incoming packets can be combined into one, with a huge payload. And then pass it along the stack. Thus, the protocol layers process the headers of one packet, while passing larger chunks to the user program.
But this optimization is inherent in the problem of information loss. If a package has an important option or flag configured, then this option or flag may be lost when combined with other packages. In general, LRO implementations have very loose package bundling rules.
GRO is a software implementation of LRO, but with stricter merging rules.
By the way: if you once used tcpdump and encountered too large sizes of incoming packets, then this was probably due to the inclusion of GRO in your system. As we will soon see, packet capture tapes are inserted further up the stack, after GRO.
Configure GRO settings with ethtool
ethtool can be used to check if GRO is enabled, as well as to configure it.
Verification of settings:
$ ethtool -k eth0 | grep generic-receive-offload
generic-receive-offload: onIn this case, generic-receive-offload is enabled. Turn on GRO:
$ sudo ethtool -K eth0 gro onNote: for most drivers, such changes will cause the interface to crash and reboot, because connections to it will be interrupted. Although a single change is not too important.
napi_gro_receive
The napi_gro_receive function handles network data for GRO (if GRO is enabled) and sends it along the stack to protocol levels. Most of the logic is in the dev_gro_receive function.
dev_gro_receive
This function first checks to see if GRO is enabled. If so, it’s getting ready for its use: it goes through the list of offload filters so that high-level protocol stacks can work with data intended for GRO. This is necessary so that the protocol layers can tell the network device layer whether the packet is part of the network stream that is currently free and can also handle everything related to the protocol that should happen within the GRO framework. For example, the TCP protocol needs to decide whether / when it is possible to confirm the combination of a packet with an existing one.
Here is an example of the code that does this, taken from net / core / dev.c :
list_for_each_entry_rcu(ptype, head, list) {
if (ptype->type != type || !ptype->callbacks.gro_receive)
continue;
skb_set_network_header(skb, skb_gro_offset(skb));
skb_reset_mac_len(skb);
NAPI_GRO_CB(skb)->same_flow = 0;
NAPI_GRO_CB(skb)->flush = 0;
NAPI_GRO_CB(skb)->free = 0;
pp = ptype->callbacks.gro_receive(&napi->gro_list, skb);
break;
}If the protocol layers report that it is time to drop the GRO packet, then this procedure is performed. In this case, napi_gro_complete is called, which calls callback gro_complete for the protocol layers, and then passes the packet along the stack by calling netif_receive_skb.
Sample code from net / core / dev.c :
if (pp) {
struct sk_buff *nskb = *pp;
*pp = nskb->next;
nskb->next = NULL;
napi_gro_complete(nskb);
napi->gro_count--;
}If the protocol layers combined this packet with an existing stream, then napi_gro_receive is constantly returned.
If the package has not been merged and the system has less than MAX_GRO_SKBS (8) GRO threads, then a new entry is added to the gro_list of the NAPI structure of this CPU.
Sample code from net / core / dev.c :
if (NAPI_GRO_CB(skb)->flush || napi->gro_count >= MAX_GRO_SKBS)
goto normal;
napi->gro_count++;
NAPI_GRO_CB(skb)->count = 1;
NAPI_GRO_CB(skb)->age = jiffies;
skb_shinfo(skb)->gso_size = skb_gro_len(skb);
skb->next = napi->gro_list;
napi->gro_list = skb;
ret = GRO_HELD;This is how the GRO system works in the Linux network stack.
napi_skb_finish
At the end of dev_gro_receive, napi_skb_finish is called, which frees up data structures that are not claimed due to packet merging, or netif_receive_skb is called to transfer data over the network stack (because GRO has already been applied to MAX_GRO_SKBS streams).
Now it's time to consider the Receive Packet Steering (RPS) mechanism.
3.4. Receive Packet Steering (RPS)
Remember, we discussed above how network device drivers register the poll NAPI function? Each instance of the NAPI poller runs in the context of SoftIRQ, one for each CPU. Now, recall that the CPU on which the driver interrupt handler is running will activate the SoftIRQ processing loop to process the packets.
In other words, as part of the processing of incoming data, a single CPU processes interrupt packets and poll functions for packets.
Some network cards (such as the Intel I350) support multiple queues at the hardware level. This means that incoming packets can be sent directly to different areas of memory allocated for each queue. In this case, polling of each region is performed using separate NAPI structures. So interrupts and packets will be processed by several CPUs.
This mechanism is called Receive Side Scaling (RSS).
Receive Packet Steering (RPS) is a software implementation of RSS. And once implemented in the code, it can be applied to any network card, even if it has only one receive queue. On the other hand, the programmatic nature leads to the fact that RPS can enter the stream only after a packet extracted from the DMA memory area.
This means that the CPU will not spend less time processing interrupts or the poll cycle, but you can distribute the load on processing packets after they have been collected and reduce the duration of the CPU in the network stack.
RPS generates a hash for incoming data to determine which CPU should process it. Then the data is placed in the backlog of this processor in anticipation of further processing. An interprocessor interrupt (IPI) is passed to the processor with the backlog , initiating queue processing, if it is not already done. / proc / net / softnet_stat contains a counter for the number of times each softnet_data structure received an IPI (received_rps field).
Therefore, netif_receive_skb will continue to send data over the network stack or pass it to RPS for processing by other CPUs.
Enable RPS
First you need to enable the RPS mechanism in the kernel configuration (on Ubuntu it is true for the kernel 3.13.0), and also create a bit mask that describes which CPUs should process packets for a specific interface or receive queue.
Information on these bitmasks can be found in the kernel documentation . In short, you can take and modify masks from here:
/sys/class/net/DEVICE_NAME/queues/QUEUE/rps_cpusFor example, for eth0 and receive queue 0, you need to add a hexadecimal number to the file / sys / class / net / eth0 / queues / rx-0 / rps_cpus indicating which CPU should process packets from queue 0 eth0. As indicated in the documentation , there is no need for RPS for certain configurations.
Note: enabling RPS to distribute packet processing among CPUs that have not done this before, for each of these CPUs will increase the number of SoftIRQ `NET_RX`, as well as` si` or `sitime` in the CPU consumption graph. You can compare the “before” and “after” indicators to find out if the RPS configuration matches your wishes.
3.5. Receive Flow Steering (RFS)
Receive flow steering (RFS) is used in conjunction with RPS. RPS tries to distribute incoming packets among several CPUs, but does not take into account data locality issues in order to increase the frequency of CPU cache hits. If you need to increase this frequency, the RFS mechanism will help in this, forwarding packets of one stream to the same CPU.
Enabling RFS
For RFS to work, you need to enable it and configure it. RFS keeps track of the global hash table of all threads. The size of the table is configured through sysctl with the parameter net.core.rps_sock_flow_entries.
Increase the stream hash size in the RFS socket:
$ sudo sysctl -w net.core.rps_sock_flow_entries=32768Now you can configure the number of threads for each receive queue. This is done in the rps_flow_cnt file for each queue.
Example: increase the number of threads to 2048 in eth0 for queue 0:
$ sudo bash -c 'echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt'3.6. Hardware Accelerated Receive Flow Steering (aRFS)
RFS can be hardware accelerated. The network card and the core can jointly determine which thread on which CPU needs to be processed. This function must be supported by the card and driver, refer to the specification. If your card driver provides the ndo_rx_flow_steer function, then it supports aRFS.
Enabling aRFS
Suppose your driver supports this mechanism. The order of its inclusion and settings:
- Turn on and configure RPS.
- Turn on and configure RFS.
- During kernel compilation, CONFIG_RFS_ACCEL is activated. In particular, in the core of Ubuntu 3.13.0.
- We enable ntuple support for our device, as described above. Check if it is enabled using ethtool.
- We adjust the IRQ parameters to make sure that each receive queue is processed by one of your CPUs that process network data.
When all this is done, aRFS is automatically used to move data to the receive queue assigned to the CPU core that processes data from this stream. You do not need to manually set ntuple filtering rules for each stream.
3.7. Moving up a network stack using netif_receive_skb
Let's go back to the place where we left netif_receive_skb, called from several places. Most often of the two (we have already examined them):
- napi_skb_finish - if the package is not merged with the existing GRO stream, OR
- napi_gro_complete - if protocol levels signal that it is time to drop the stream, OR
I remind you: netif_receive_skb and its descendants operate in the context of the SoftIRQ processing cycle. Tools like top take into account the time spent here as sitime or si.
netif_receive_skb will first check the sysctl value to determine if the user has timestamp enabled when receiving before or after the packet enters the backlog queue. If this setting is enabled, data is now assigned timestamps before they are exposed to the RPS mechanism (and the backlog queue associated with the CPU). If this setting is turned off, then timestamps are assigned to the data after they are queued. When RPS is enabled, this can be used to distribute the load by labeling among several CPUs, but it will give some delay.
Setup: Assigning Timestamps to Received Packets
Labeling order can be configured in sysctl using net.core.netdev_tstamp_prequeue.
Disable labeling for received packets:
$ sudo sysctl -w net.core.netdev_tstamp_prequeue=0The default value is 1. The meaning of this setting is explained in the previous chapter.
3.8. netif_receive_skb
Having dealt with timestamps, further netif_receive_skb acts differently, depending on whether RPS is enabled. Let's start with the simpler case when RPS is off.
No RPS (default)
In this case, __netif_receive_skb is called, which registers some data (bookkeeping), and then calls __netif_receive_skb_core to transfer the data closer to the protocol stacks.
Then we will look at how __netif_receive_skb_core works, but first we will analyze how the code works with RPS enabled, since in this case __netif_receive_skb_core is also called.
RPS enabled
After working with the timestamping options, netif_receive_skb performs a series of calculations to determine which CPU's backlog queue should be used. This is done using the get_rps_cpu function. Adapted from net / core / dev.c :
cpu = get_rps_cpu(skb->dev, skb, &rflow);
if (cpu >= 0) {
ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);
rcu_read_unlock();
return ret;
}get_rps_cpu also takes into account the settings of RFS and aRFS to make sure that by calling enqueue_to_backlog the data goes to the backlog of the desired CPU.
enqueue_to_backlog
This function first gets a pointer to the softnet_data structure of the remote CPU containing a pointer to input_pkt_queue. It then checks the length of the input_pkt_queue queue for this remote CPU. Adapted from net / core / dev.c :
qlen = skb_queue_len(&sd->input_pkt_queue);
if (qlen <= netdev_max_backlog && !skb_flow_limit(skb, qlen)) {First, the length of the input_pkt_queue queue is compared to netdev_max_backlog. If the length is greater than this value, then the data is discarded . Similarly, the flow restriction is checked, and if it is reached, then the data is discarded . In both cases, the drop counter of the softnet_data structure is incremented. Note that this is the structure of the CPU in whose queue the data should have been placed. Drop counter monitoring is described in the chapter on / proc / net / softnet_stat.
enqueue_to_backlog is called when processing packets with RPS enabled, as well as from netif_rx. Most drivers should not usenetif_rx, and netif_receive_skb. If you do not use RPS and your driver does not use netif_rx, then increasing the backlog will not have a noticeable effect on the system, since it is not used.
Note: check the driver used. If it calls netif_receive_skb and you are not using RPS, then increasing netdev_max_backlog will not improve performance in any way, because no data will get to input_pkt_queue.
Assume that input_pkt_queue is small enough and the flow restriction is not reached (or disabled), and the data can be queued. The logic here is a little ridiculous, it can be generalized as follows:
- If the queue is empty: it checks to see if NAPI is running on the remote CPU. If not, it checks to see if the IPI is in the queue. If not, the IPI is queued, and through a call to ____napi_schedule, the NAPI processing cycle starts. We proceed to transfer data to the queue.
- If the queue is not empty, or if the previous operation is completed, data is transferred to the queue.
Due to using goto, the code is pretty tricky, so read carefully. Adapted from net / core / dev.c :
if (skb_queue_len(&sd->input_pkt_queue)) {
enqueue:
__skb_queue_tail(&sd->input_pkt_queue, skb);
input_queue_tail_incr_save(sd, qtail);
rps_unlock(sd);
local_irq_restore(flags);
return NET_RX_SUCCESS;
}
/* Schedule NAPI for backlog device
* We can use non atomic operation since we own the queue lock
*/
if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) {
if (!rps_ipi_queued(sd))
____napi_schedule(sd, &sd->backlog);
}
goto enqueue;Thread limits
RPS distributes the burden of processing packets between multiple CPUs, but one large thread can monopolize the processing time and infringe on smaller threads. Thread limits allow you to limit the number of packets from a single thread placed in the backlog. As a result, small streams can be processed in parallel with much larger ones.
The if statement from net / core / dev.c by calling skb_flow_limit checks the flow restriction:
if (qlen <= netdev_max_backlog && !skb_flow_limit(skb, qlen)) {It checks if there is still space in the queue and if the limit is reached . By default, restrictions are disabled. To enable them, you need to set a bit mask (similar to RPS).
Drop monitoring due to filling input_pkt_queue or flow restriction
See the monitoring chapter / proc / net / softnet_stat. The dropped field is a counter incremented each time data is discarded instead of being placed in the input_pkt_queue CPU queue.
Customization
Setting netdev_max_backlog to prevent drops
Before setting this value, read the note in the previous chapter.
If you use RPS or your driver calls netif_rx, then you can help prevent drops in enqueue_to_backlog by increasing netdev_max_backlog.
Example: increase backlog to 3000:
$ sudo sysctl -w net.core.netdev_max_backlog=3000The default value is 1000.
NAPI weight setting in poll loop backlog
You can configure the weight of the NAPI poller in the backlog using net.core.dev_weight sysctl. This value determines how much of the total budget the poll backlog cycle can spend (see the chapter on setting net.core.netdev_budget).
Example: increase the poll processing loop in the backlog:
$ sudo sysctl -w net.core.dev_weight=600The default value is 64.
Remember that backlog processing is performed in the SoftIRQ context in the same way as the poll function registered on the device driver. It will be limited by the general budget and time limit, as described in the previous chapter.
Enabling stream restrictions and setting the size of the stream restriction hash table
Set the size of the thread restriction table:
$ sudo sysctl -w net.core.flow_limit_table_len=8192The default value is 4096.
This change only affects newly allocated tables. So first adjust the size of the table, and then enable thread limits.
To enable restrictions, you need to set a bitmask in / proc / sys / net / core / flow_limit_cpu_bitmap, similar to the RPS mask, which shows which CPUs have thread restrictions enabled.
Poller NAPI backlog queues
The backlog queues of each CPU use NAPI in the same way as the device driver. A poll function is provided that is used to process packets from the SoftIRQ context. As with the driver, weight is also used here.
The NAPI structure is provided during the initialization of the network subsystem.
Taken from net_dev_init to net / core / dev.c :
sd->backlog.poll = process_backlog;
sd->backlog.weight = weight_p;
sd->backlog.gro_list = NULL;
sd->backlog.gro_count = 0;The NAPI structure of the backlog differs from the structure of the driver by the ability to configure the weight parameter: the value of the driver is hard-coded. Further we will consider the process of adjusting the weight.
process_backlog
The process_backlog function is a loop executed until its weight (as described in the previous chapter) is used up or until more data remains in the backlog.
Data is taken in parts from the backlog queue and transferred to __netif_receive_skb. The code branch will be the same as in the case of disabled RPS. Namely, __netif_receive_skb performs the same procedures before calling __netif_receive_skb_core to transfer data to the protocol layers.
process_backlog follows the same NAPI agreement as device drivers: NAPI is disabled if all the weight is not consumed. The poller is restarted by calling ____napi_schedule from enqueue_to_backlog, as described above.
The function returns the completed work, which then net_rx_action (described above) will be deducted from the budget (configured using net.core.netdev_budget, described above).
__netif_receive_skb_core transfers data to packet taps and to protocol layers
__netif_receive_skb_core does a difficult job of passing data to protocol stacks. Before that, she first checks to see if any packet taps are installed that catch all incoming packets. An example is the AF_PACKET address family, commonly used through the libpcap library .
If there is such a tap, before the data is transferred there first, and then to the protocol layers.
Transmit to packet tap
The packet transfer is performed by the following code. Adapted from net / core / dev.c :
list_for_each_entry_rcu(ptype, &ptype_all, list) {
if (!ptype->dev || ptype->dev == skb->dev) {
if (pt_prev)
ret = deliver_skb(skb, pt_prev, orig_dev);
pt_prev = ptype;
}
}If you are interested in how the data goes through pcap, read net / packet / af_packet.c .
Transfer to protocol layers
After the tapes are satisfied, __netif_receive_skb_core transfers the data to the protocol layers. To do this, the protocol field is extracted from the data and iterated over the list of delivery functions registered for this type of protocol.
This can be viewed in __netif_receive_skb_core in net / core / dev.c :
type = skb->protocol;
list_for_each_entry_rcu(ptype,
&ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
if (ptype->type == type &&
(ptype->dev == null_or_dev || ptype->dev == skb->dev ||
ptype->dev == orig_dev)) {
if (pt_prev)
ret = deliver_skb(skb, pt_prev, orig_dev);
pt_prev = ptype;
}
}Here the ptype_base identifier is defined as a hash table of the list in net / core / dev.c :
struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;Each protocol level adds a filter to the list of a given hash table slot calculated by the ptype_head helper function:
static inline struct list_head *ptype_head(const struct packet_type *pt)
{
if (pt->type == htons(ETH_P_ALL))
return &ptype_all;
else
return &ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK];
}Adding a filter to the list ends with a call to dev_add_pack. In this way, protocol layers register themselves to transmit data to them.
Now you know how network data gets from the network card to the protocol layer.
3.9. Protocol Level Registration
Let's look at how protocol layers register themselves. In this article, we restrict ourselves to the IP protocol stack, since it is extremely widely used and will be understood by most readers.
IP layer
The IP protocol layer itself includes itself in the ptype_base hash table, and in the previous chapters we have already considered the data that is transmitted to this layer from the network device layer.
This happens in the inet_init function, taken from net / ipv4 / af_inet.c :
dev_add_pack(&ip_packet_type);
Здесь регистрируется структура типа IP-пакета, определённая в net/ipv4/af_inet.c:
static struct packet_type ip_packet_type __read_mostly = {
.type = cpu_to_be16(ETH_P_IP),
.func = ip_rcv,
};__netif_receive_skb_core calls deliver_skb (as we saw in the previous chapter), which calls func (in this case, ip_rcv).
ip_rcv
At a high level, the ip_rcv function is quite straightforward. It has several integrity checks to make sure the data is valid, as well as statistics counters twitch.
ip_rcv ends with passing the packet to ip_rcv_finish via netfilter . This is done so that any iptables rule that must be respected at the IP protocol level can verify the packet before it goes on.
Code passing data through netfilter at the end of ip_rcv to net / ipv4 / ip_input.c :
return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL, ip_rcv_finish);netfilter and iptables
For the sake of brevity, I decided to skip the detailed discussion of netfilter, iptables, and conntrack.
Short version: NF_HOOK_THRESH checks if any filters are installed and tries to return execution back to the IP protocol level so as not to go into netfilter and everything under it like iptables and conntrack.
Remember: if you have many netfilter or iptables rules, or they are too complicated, then these rules will be executed in the context of SoftIRQ, and this is a risk of delays in the network stack. You may not be able to avoid this if you need a specific set of established rules.
ip_rcv_finish
After netfilter gets an opportunity to look at the data and decide what to do with it, ip_rcv_finish is called. Of course, unless the data is discarded by netfilter.
ip_rcv_finish begins with optimization. To deliver the packet to the right place, the dst_entry structure from the routing system must be ready. To get it, the code first tries to call the early_demux function from a protocol of a higher level than the one for which this data is intended.
early_demux is an optimization during which we try to find the dst_entry structure necessary for package delivery. To do this, it checks to see if dst_entry is cached in the socket structure.
Here's what it looks like, taken from net / ipv4 / ip_input.c :
if (sysctl_ip_early_demux && !skb_dst(skb) && skb->sk == NULL) {
const struct net_protocol *ipprot;
int protocol = iph->protocol;
ipprot = rcu_dereference(inet_protos[protocol]);
if (ipprot && ipprot->early_demux) {
ipprot->early_demux(skb);
/* нужно перезагрузить iph, skb->head могла измениться */
iph = ip_hdr(skb);
}
}As you can see, this code is protected by sysctl_ip_early_demux. By default, early_demux is enabled. In the next chapter, you will learn how to disable it and why you might want to do this.
If optimization is enabled and not cached in the record (because it is the first arriving packet), then the packet will be transferred to the kernel routing system, where dst_entry will be computed and assigned.
After the routing layer is completed, statistics counters are updated, and the function ends with a call to dst_input (skb). That, in turn, calls the input function pointer to the dst_entry packet structure, attributed to the routing system.
If the packet destination is a local system, then the routing system will attach the ip_local_deliver function to the pointer to the input function in the dst_entry package structure.
Configuring early demux IP protocol
Turning off early_demux optimization:
$ sudo sysctl -w net.ipv4.ip_early_demux=0The default value is 1; early_demux is enabled.
In some cases, this sysctl degrades throughput by about 5% with the optimization of early_demux.
ip_local_deliver
We recall the following pattern at the IP protocol level:
- When ip_rcv is called, a series of procedures (bookkeeping) are performed.
- The packet is passed to netfilter for further processing, with a pointer to the execution of the callback when processing is complete.
- ip_rcv_finish is a callback that completes processing and continues to move the packet along the network stack.
In the case of ip_local_deliver, the same template is used. Taken from net / ipv4 / ip_input.c :
/*
* Доставляет IP-пакеты на более высокие уровни протоколов.
*/
int ip_local_deliver(struct sk_buff *skb)
{
/*
* Пересобирает IP-пакеты.
*/
if (ip_is_fragment(ip_hdr(skb))) {
if (ip_defrag(skb, IP_DEFRAG_LOCAL_DELIVER))
return 0;
}
return NF_HOOK(NFPROTO_IPV4, NF_INET_LOCAL_IN, skb, skb->dev, NULL,
ip_local_deliver_finish);
}After netfilter gets a chance to look at the data, ip_local_deliver_finish is called. Of course, unless the data is discarded by netfilter.
ip_local_deliver_finish
ip_local_deliver_finish extracts the protocol from the packet, looks for the net_protocol structure registered to this protocol, and calls the function pointed to by handler in this structure.
Thus, the packet is transmitted to a higher protocol level.
Monitoring IP protocol layer statistics
We read / proc / net / snmp to monitor the detailed statistics of the IP protocol:
$ cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 25922988125 0 0 15771700 0 0 25898327616 22789396404 12987882 51 1 10129840 2196520 1 0 0 0
...This file contains statistics for several protocol levels. The first is the IP level. The first line contains the space-separated names of the corresponding values from the second line.
At the IP protocol level, you can find a number of statistics counters that are used in the C-enumeration. All valid enum values and field names in / proc / net / snmp to which they correspond can be found in include / uapi / linux / snmp.h :
enum
{
IPSTATS_MIB_NUM = 0,
/* часто записываемые напрямую поля, хранятся в той же кэш-строке */
IPSTATS_MIB_INPKTS, /* InReceives */
IPSTATS_MIB_INOCTETS, /* InOctets */
IPSTATS_MIB_INDELIVERS, /* InDelivers */
IPSTATS_MIB_OUTFORWDATAGRAMS, /* OutForwDatagrams */
IPSTATS_MIB_OUTPKTS, /* OutRequests */
IPSTATS_MIB_OUTOCTETS, /* OutOctets */
/* ... */We read / proc / net / netstat to monitor the extended statistics of the IP protocol:
$ cat /proc/net/netstat | grep IpExt
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT0Pktsu InCEPkts
IpExt: 0 0 0 0 277959 0 14568040307695 32991309088496 0 0 58649349 0 0 0 0 0The format is similar to / proc / net / snmp, with the exception of lines with the IpExt prefix.
Some interesting metrics:
- InReceives: total number of IP packets that reached ip_rcv before all integrity checks.
- InHdrErrors: общее количество IP-пакетов с повреждёнными заголовками. Заголовок был слишком маленьким или длинным, вообще не существует, содержал неправильный номер версии протокола IP и так далее.
- InAddrErrors: общее количество IP-пакетов, когда хост был недоступен.
- ForwDatagrams: общее количество IP-пакетов, которые были переадресованы (forwarded).
- InUnknownProtos: общее количество IP-пакетов с неизвестным или неподдерживаемым протоколом, указанным в заголовке.
- InDiscards: общее количество IP-пакетов, отклонённых в результате сбоя выделения памяти, или в результате сбоя контрольной суммы при обрезании пакета.
- InDelivers: общее количество IP-пакетов, успешно доставленных на более высокие уровни протоколов. Помните, что эти уровни могут отбрасывать данные, даже если этого не сделал уровень IP.
- InCsumErrors: total number of IP packets with checksum errors.
All of these counters are incremented at specific IP level locations. From time to time, the code passes through them, and double-counting errors and other bugs may occur. If these statistics are important to you, I highly recommend that you study the source code of these metrics at the IP protocol level to understand how they are incremented.
Higher layer protocol registration
Here we look at UDP, but the TCP protocol handler is registered in the same way and at the same time as the UDP protocol handler.
In net / ipv4 / af_inet.c, you can find structure definitions that contain handler functions for connecting UDP, TCP, and ICMP protocols to the IP protocol layer. Taken from net / ipv4 / af_inet.c :
static const struct net_protocol tcp_protocol = {
.early_demux = tcp_v4_early_demux,
.handler = tcp_v4_rcv,
.err_handler = tcp_v4_err,
.no_policy = 1,
.netns_ok = 1,
};
static const struct net_protocol udp_protocol = {
.early_demux = udp_v4_early_demux,
.handler = udp_rcv,
.err_handler = udp_err,
.no_policy = 1,
.netns_ok = 1,
};
static const struct net_protocol icmp_protocol = {
.handler = icmp_rcv,
.err_handler = icmp_err,
.no_policy = 1,
.netns_ok = 1,
};These structures are registered in the inet address family initialization code. Taken from net / ipv4 / af_inet.c :
/*
* Добавляем все базовые протоколы.
*/
if (inet_add_protocol(&icmp_protocol, IPPROTO_ICMP) < 0)
pr_crit("%s: Cannot add ICMP protocol\n", __func__);
if (inet_add_protocol(&udp_protocol, IPPROTO_UDP) < 0)
pr_crit("%s: Cannot add UDP protocol\n", __func__);
if (inet_add_protocol(&tcp_protocol, IPPROTO_TCP) < 0)
pr_crit("%s: Cannot add TCP protocol\n", __func__);Consider the level of the UDP protocol. As we saw above, the handler function for UDP is called by udp_rcv. This is the entry point to the UPD layer, where data from the IP layer is transmitted.
UDP protocol layer
UDP protocol level code can be found here: net / ipv4 / udp.c .
udp_rcv
The code of the udp_rcv function consists of one line in which the __udp4_lib_rcv function is directly called to process received datagrams.
__udp4_lib_rcv
The __udp4_lib_rcv function verifies the validity of the packet and retrieves the UDP header, UDP datagram length, source address, and destination address. Next, integrity checks and checksums are performed.
We recall that in the part devoted to the level of the IP protocol, we saw optimization, as a result of which dst_entry is attached to the packet before it is transmitted to a higher level (in our case, UDP).
If a socket and the corresponding dst_entry are found, then the __udp4_lib_rcv function places the packet in the socket queue:
sk = skb_steal_sock(skb);
if (sk) {
struct dst_entry *dst = skb_dst(skb);
int ret;
if (unlikely(sk->sk_rx_dst != dst))
udp_sk_rx_dst_set(sk, dst);
ret = udp_queue_rcv_skb(sk, skb);
sock_put(sk);
/* возвращаемое значение > 0 означает повторный ввод данных,
* но он хочет вернуть –protocol или 0
*/
if (ret > 0)
return -ret;
return 0;
} else {If there is no attached receiving socket left from the early_demux operation, then it is searched by calling __udp4_lib_lookup_skb.
In both cases, the datagram will be placed in the socket queue:
ret = udp_queue_rcv_skb(sk, skb);
sock_put(sk);If the socket is still not found, then the datagram will be discarded:
/* Сокета нет. Пакет тихо отбрасывается, если контрольная сумма ошибочна */
if (udp_lib_checksum_complete(skb))
goto csum_error;
UDP_INC_STATS_BH(net, UDP_MIB_NOPORTS, proto == IPPROTO_UDPLITE);
icmp_send(skb, ICMP_DEST_UNREACH, ICMP_PORT_UNREACH, 0);
/*
* Хм. Мы получили UDP-пакет на порт, который
* не хотим прослушивать. Игнорируем его.
*/
kfree_skb(skb);
return 0;udp_queue_rcv_skb
The initial part of the function:
- Determines whether the socket associated with the datagram is an encapsulating socket. If so, then before processing the packet is passed to the processor function of this level.
- Determines whether the datagram is a UDP-Lite and integrity checks are performed.
- Checks the UDP checksum of the datagram and discards the latter in the event of a failure.
Finally we got to the logic of the receive queue. It starts by checking if the queue for the socket is full. Taken from net / ipv4 / udp.c :
if (sk_rcvqueues_full(sk, skb, sk->sk_rcvbuf))
goto drop;sk_rcvqueues_full
The sk_rcvqueues_full function checks the length of the backlog and sk_rmem_alloc of the socket to determine if the sum of their lengths exceeds the size of sk_rcvbuf for the socket (sk-> sk_rcvbuf in the above example):
/*
* Учитывает размер очереди приёма и backlog-очереди.
* Не учитывает этот skb truesize,
* чтобы мог прибыть даже одиночный большой пакет.
*/
static inline bool sk_rcvqueues_full(const struct sock *sk, const struct sk_buff *skb,
unsigned int limit)
{
unsigned int qsize = sk->sk_backlog.len + atomic_read(&sk->sk_rmem_alloc);
return qsize > limit;
}These values are tricky to configure, and you can tweak a lot of things.
Setup: socket receive queue memory
The value sk-> sk_rcvbuf (called restriction in sk_rcvqueues_full) can be increased to any level within sysctl net.core.rmem_max.
Increase the maximum size of the receive buffer:
$ sudo sysctl -w net.core.rmem_max=8388608sk-> sk_rcvbuf starts with the value net.core.rmem_default, which can also be configured using sysctl.
Set the default default receive buffer size:
$ sudo sysctl -w net.core.rmem_default=8388608You can also adjust the size sk-> sk_rcvbuf, by calling setsockopt from your application and passing SO_RCVBUF. The maximum value of setsockopt does not exceed net.core.rmem_max.
But you can exceed the net.core.rmem_max limit by calling setsockopt and passing SO_RCVBUFFORCE. But the user who is running the application will need the CAP_NET_ADMIN capability.
The value of sk-> sk_rmem_alloc is incremented by calls to skb_set_owner_r, which specify the socket that owns the datagram. Later we will encounter this at the UDP level.
The value sk-> sk_backlog.len is incremented by calling sk_add_backlog.
udp_queue_rcv_skb
After checking the queue is full, the process of placing the datagram in the queue continues. Taken from net / ipv4 / udp.c :
bh_lock_sock(sk);
if (!sock_owned_by_user(sk))
rc = __udp_queue_rcv_skb(sk, skb);
else if (sk_add_backlog(sk, skb, sk->sk_rcvbuf)) {
bh_unlock_sock(sk);
goto drop;
}
bh_unlock_sock(sk);
return rc;First, it turns out if there are currently system calls to the socket from the user space program. If not , the datagram can be added to the receive queue by calling __udp_queue_rcv_skb. If there is , then the datagram is placed in the backlog-queue using sk_add_backlog call.
Datagrams are added to the receive queue from the backlog when system calls release the socket using the release_sock call in the kernel.
__udp_queue_rcv_skb
The __udp_queue_rcv_skb function adds datagrams to the receive queue by calling sock_queue_rcv_skb. And if the datagram cannot be added to the socket receive queue, then __udp_queue_rcv_skb pulls the statistics counters.
Taken from net / ipv4 / udp.c :
rc = sock_queue_rcv_skb(sk, skb);
if (rc < 0) {
int is_udplite = IS_UDPLITE(sk);
/* Обратите внимание, что ошибка ENOMEM выдаётся дважды */
if (rc == -ENOMEM)
UDP_INC_STATS_BH(sock_net(sk), UDP_MIB_RCVBUFERRORS,is_udplite);
UDP_INC_STATS_BH(sock_net(sk), UDP_MIB_INERRORS, is_udplite);
kfree_skb(skb);
trace_udp_fail_queue_rcv_skb(rc, sk);
return -1;
}Monitoring UDP protocol level statistics
Two very useful files for getting statistics over UDP:
- / proc / net / snmp
- / proc / net / udp
/ proc / net / snmp
We read / proc / net / snmp to monitor the detailed statistics of the UDP protocol.
$ cat /proc/net/snmp | grep Udp\:
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors
Udp: 16314 0 0 17161 0 0Like the detailed statistics on IP protocol that is available in this file, you will need to read the protocol level sources to determine exactly when and where these values are incremented.
InDatagrams: increments when:
- recvmsg is used by the user space program to read the datagram.
- The UDP packet is encapsulated and returned for processing.
NoPorts: incremented when UDP packets arrive at a specific port that no program listens on.
InErrors: increments if:
- the memory of the receive queue has run out
- Incorrect checksum
- sk_add_backlog cannot add a datagram.
OutDatagrams: increments when a UDP packet is accurately transmitted down to the IP protocol layer for subsequent sending.
RcvbufErrors: incremented when sock_queue_rcv_skb reports no available memory; this happens if sk-> sk_rmem_alloc is greater than or equal to sk-> sk_rcvbuf.
SndbufErrors: increments if:
- the IP protocol layer reported an error while trying to send a packet,
- the kernel is out of memory
- out of place in the send buffer.
InCsumErrors: increments when a UDP checksum failure is detected. Please note that in all cases that I came across, InCsumErrors increments simultaneously with InErrors. Therefore, the InErrors - InCsumErros link should reflect the number of memory errors.
/ proc / net / udp
We read / proc / net / udp to monitor the detailed UDP socket statistics.
$ cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
515: 00000000:B346 00000000:0000 07 00000000:00000000 00:00000000 00000000 104 0 7518 2 0000000000000000 0
558: 00000000:0371 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 7408 2 0000000000000000 0
588: 0100007F:038F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 7511 2 0000000000000000 0
769: 00000000:0044 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 7673 2 0000000000000000 0
812: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 7407 2 0000000000000000 0The first line describes each of the fields from the following lines:
- sl: hash of the kernel for the socket.
- local_address: hexadecimal local socket address and port number, separated by:.
- rem_address: hexadecimal remote socket address and port number, separated by:.
- st: socket state. Oddly enough, the UDP protocol layer seems to be using TCP socket states. In the above example, 7 is TCP_CLOSE.
- tx_queue: the amount of memory allocated in the kernel for outgoing UDP datagrams.
- rx_queue: The amount of memory allocated in the kernel for incoming UDP datagrams.
- tr, tm-> when, retrnsmt: these fields are not used by the UDP protocol layer.
- uid: valid identifier of the user who created this socket.
- timeout: not used by the UDP protocol layer.
- inode: inode number corresponding to this socket. It can be used to determine which user process opened this socket. Look at / proc / [pid] / fd, it contains symlinks on socket [: inode].
- ref: current counter of references to this socket.
- pointer: address of struct sock in kernel memory.
- drops: number of discarded datagrams associated with this socket.
Displaying all of this code can be found in net / ipv4 / udp.c .
Putting data in a socket queue
Network data is put into the socket queue by calling sock_queue_rcv. Before adding a datagram to the queue, this function does a few things:
- The memory allocated for the socket is checked to determine if the size of the receive buffer size has been reached. If so, then a drop counter is incremented for this socket.
- sk_filter is used to process Berkeley Packet Filters applied to a socket.
- sk_rmem_schedule allows you to make sure that there is enough space in the receive buffer to receive the datagram.
- Then, by calling skb_set_owner_r, the size of the datagram is entered into the socket. Increments sk-> sk_rmem_alloc.
- By calling __skb_queue_tail, data is added to the queue.
- Finally, by calling the notification function sk_data_ready, all processes that are waiting for data to arrive on the socket are notified.
In this way, the data arrives at the system and goes through the network stack until it reaches the socket. Now they are ready for use by the user program.
3.10. Additional Information
It is necessary to mention a few additional things about which there was no reason to tell above.
Assigning Timestamps
I already wrote that the network stack can collect timestamps of incoming data. In sysctl there are values, together with RPS, that allow you to control the moment and method of collecting labels. Read more about this in the chapters on RPS and timestamps. Some network cards even support tagging in hardware.
This is a useful feature if you want to determine the amount of delay added to the received packets by the network kernel stack.
In the kernel documentation, the issue of assigning timestamps is well covered , even a sample program and an assembly file are included there !
Determine which timestamp modes your driver and device support:
$ sudo ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: noneUnfortunately, this network card does not support timestamps in hardware, but you can still use them programmatically. They will help determine what kind of delay the kernel adds to my packet path.
Loaded polling for low latency sockets
You can use the socket option called SO_BUSY_POLL. It forces the kernel to apply loaded polling to new data when blocking reception is completed and there is no data.
IMPORTANT NOTE: for this option to work, it must be supported by your device driver. The igb driver of kernel 3.13.0 does not support it. And ixgbe - supports. If your driver has a function configured in the ndo_busy_poll field of the struct net_device_ops structure (mentioned above), then it supports SO_BUSY_POLL.
Intel has an excellent description of how it works and how to use it.
When using this option for a single socket, it is necessary to send the value of time in microseconds as the amount of time for polling for new data to the device driver's receive queue. When, after setting this value, a block reading of the socket is performed, the kernel will apply polling to the new data.
You can also set sysctl to net.core.busy_poll - how long calls with poll or select should wait for new data to arrive under loaded polling conditions (in microseconds).
This option allows you to reduce the delay, but increases the load on the CPU and power consumption.
Netpoll: Support for Networking in Critical Contexts
The Linux kernel provides a way to use device drivers to send and receive data on a network card if the kernel crashes. The API for this functionality is called Netpoll. It is used by different systems, but especially kgdb and netconsole .
Netpoll is supported by most drivers. Your driver must implement the ndo_poll_controller function and attach it to the struct net_device_ops structure registered during probe.
When the subsystem of a network device performs operations with incoming and outgoing data, the Netpoll system is first checked to determine if the package is intended for it.
The following code can be found in __netif_receive_skb_core from net / dev / core.c :
static int __netif_receive_skb_core(struct sk_buff *skb, bool pfmemalloc)
{
/* ... */
/* если мы попали сюда через NAPI, проверяем netpoll */
if (netpoll_receive_skb(skb))
goto out;
/* ... */
}Netpoll checks are pretty early on in most Linux network device subsystems that work with sending or receiving network data.
Netpoll API consumers can register struct netpoll structures by calling netpoll_setup. These structures contain function pointers for attaching hooks, and the API exports a function for sending data.
If you are interested in using the Netpoll API, then check out the netconsole driver , the Netpoll API header file, 'include / linux / netpoll.h` and this wonderful text .
SO_INCOMING_CPU
The SO_INCOMING_CPU flag was not present on Linux until version 3.19, but it is quite useful, so I will mention it.
To determine which CPU processes network packets for a particular socket, you can use getsockopt and the SO_INCOMING_CPU option. Then your application will be able to use this information to transfer sockets to threads running on the desired CPU. This will help improve data locality and CPU hit rates.
A short example of architecture when this option is useful is given here: patchwork.ozlabs.org/patch/408257 .
DMA engines
The DMA engine is the hardware that allows you to free the CPU from large copy operations that are transferred to another hardware. So turning on the use of the DMA engine and running code that takes advantage of it should reduce the load on the CPU.
The Linux kernel contains a generic DMA engine interface that can be used by the authors of the engine drivers. You can read more about this interface in the kernel documentation .
The kernel supports several DMA engines, but we will talk about one of the most common - Intel IOAT DMA engine .
Intel's I / O Acceleration Technology (IOAT)
Many servers include the Intel I / O AT package , which introduces a number of changes to the performance. One of them is the use of the DMA hardware engine. Check the output of your dmesg for ioatdma to determine if the module is loaded and if it has found supported hardware. The DMA engine is used in a number of places, but especially actively - in the TCP stack.
Support for the Intel IOAT engine was included in Linux 2.6.18, but in 3.13.11.10 it was abandoned due to unexpected bugs that damaged the data . Users of kernels prior to version 3.13.11.10 can use the ioatdma module by default. Perhaps in future releases it will be fixed.
Direct cache access (DCA)
Another interesting feature that comes with Intel I / O AT is Direct Cache Access (DCA).
It allows network devices (via their drivers) to put network data directly into the CPU cache. The specific implementation depends on the driver. In the case of igb, you can see the code for the igb_update_dca function , as well as igb_update_rx_dca . The igb driver uses DCA when writing register values to the network card.
To use DCA, you need to enable it in the BIOS, check if the dca module is loaded and your network card and driver support it.
IOAT DMA engine monitoring
If, despite the risk of data corruption, you are using the ioatdma module, then you can monitor it through some entries in sysfs.
We monitor the total number of unloaded memcpy operations in the DMA channel:
$ cat /sys/class/dma/dma0chan0/memcpy_count
123205655We monitor the total number of bytes transmitted through the DMA channel:
$ cat /sys/class/dma/dma0chan0/bytes_transferred
131791916307Configuring the IOAT DMA Engine
The IOAT DMA engine is used only when the packet size exceeds a certain threshold - copybreak. This check is necessary because for small copies, the overhead of installing and using the DMA engine does not cover the benefits of acceleration.
Configure copybreak for the DMA engine:
$ sudo sysctl -w net.ipv4.tcp_dma_copybreak=2048The default value is 4096.
4. Conclusion
The Linux network stack is quite complex. Without a deep understanding of the processes, you cannot monitor or configure it (like any other complex software). On the Internet, you can find examples of sysctl.conf containing sets of values that are proposed to be copied and pasted onto your computer. This is not the best way to optimize your network stack.
Monitoring the network stack requires careful management of network data at every level, starting with drivers. Then you can determine where the drops occur and errors occur, and then make adjustments, reducing these harmful effects.
Unfortunately, there is no easy way.
