October 24, 2017 at 19:14The Tale of sysctls (folk penguin story)
Very often, administrators configure the system simply by setting up the basic things - ip, dns, hostname, install software, and everything else is already application settings. In most cases, this is the case, since linux has very reasonable defaults and, in most cases, these defaults are enough and everyone lives happily. Among beginners there are legends about certain sysctls, and those who saw with more experience and even ruled something.
But the moment comes when the administrator in his campaigns on the system meets this beast - sysctl. Most likely he meets someone from the net.ipv4 or vm family, even most likely net.ipv4.ip_forward, if he is going for a router or vm.swappinness, if he is concerned about the growing swap of his penguin. The first beast allows the penguin to receive packets with one wing and give it to the other (allows routing), and the second helps to cope with the use of swap in a quiet system and to regulate its use - in a loaded one.
A meeting with any of these “animals” opens the door to a whole world of system settings and kernel components, there are several families, the names of most of which speak for themselves: net - network; kern - kernel; vm - memory and caches. But this is all a "fairy tale", the reality is much more interesting.
Acquainted with this host of settings, an inexperienced administrator, like a moth into the light, flies and wants to rather quickly configure them all, to make the system better and more optimal. Oh yeah! His temptation is great and the goal is good, but the chosen path, sometimes, leads in a completely different direction. The name of that path is “google tuning”.
After all, what is the temptation to quickly google “optimal sysctl settings” and apply some recipe from the beginning of the article without really going into what is written below - TL; DR. In most cases, the system does not get worse, because the load or configuration is not so that problems pop up. This is later, with experience, delving into the system settings for a specific case, you will understand that this is some kind of nonsense that was written.
Here, for example, the parameters net.ipv4.tcp_mem, rmem, wmem - look very similar, three numbers are of the type “4096 87380 6291456”, only, this is bad luck, for [rw] mem they are bytes, and for mem they are bytes and, if set all three parameters to the same value, then there will be an “explosive" configuration, because tcp_mem is responsible for managing the memory consumed by tcp, and [rw] mem for socket buffers.
And there are sysctl'i which, until you meet and rake - you won’t think about what this sysctl is, like, for example, net.ipv4.conf.all.rp_filter. The worst thing that shoots only when you need to do asymmetric routes, that is, your router has 2+ interfaces and traffic can come from one interface and return through another, that is, quite rarely. Called Reverse Path Filtering, it blocks packets that come from an address that is not routed through the interface from where they come.
There are very useful parameters, but they require a careful study of the documentation and calculations to find out how they can help you in your specific situation. I repeat that the defaults in linux are good enough. This is especially true for tcp settings and the network as a whole.
The parameters that you have to play often enough are:
vm.swappiness - setting the aggressiveness of the “free" memory. This is necessary in order to maintain the largest possible amount of RAM available for applications and work. In the default value - 60, the system translates into memory pages that have not been used for some long time, in the value 0 or 1 - the system tries to use the swap only when it cannot allocate physical memory or when the amount of available memory matches the specified in vm.min_free_kbytes. This is not to say that using a swap is definitely good or bad. It all depends on the situation and the profile of memory usage, and this knob allows us to control the ratio of the system to the swap from 0 - I do not like it at all, up to 100 - oh yes, svoop!
vm.min_free_kbytes - (once mentioned in vm.swappiness) Defines the minimum free memory size that must be supported. If set too small - the system will break, if too large - the OOM (Out Of Memory) killer will often come.
vm.overcommit_memory- allows / disallows “allocate” more memory than it is: 0 - the system checks every time if there is a sufficient amount of free memory; 1 - the system always believes that there is memory and allows allocation while the memory really is; 2 - prohibit asking for more memory than is - you can allocate no more than RAM + SWAP. This can fire when you have an application, for example redis, that consumes> ½ of memory and decides to write data to disk, for which it fork and copy all the data, but because there may not be enough available memory, it may either break the recording to disk, or come oom and kill something you need.
net.ipv4.ip_forward- permission or prohibition on packet routing. We are faced with the need to twist this knob when we configure the router. Here everything is more or less clear: 0 - turn off; 1 - enable.
net.ipv4. {all, default, interface} .rp_filter - controls the Reverse Path Filtering option: 0 - do not check, disable; 1 - “strict” mode, when packets are discarded answers that would not go through the interface from which the packet came; 2 - “relaxed” mode - only those packets to which the route is unknown are discarded (if there is a default route, in my opinion, it should lead to the same effect as 0).
net.ipv4.ip_local_port_range- defines the minimum and maximum ports, which is used to create a local client socket. If your system makes a large number of calls to network resources, then you may have a problem of lack of local ports for establishing connections. This option allows you to adjust the port range that is used to establish client connections. It can also be useful in order to secure your services that “listen” to high ports.
net.ipv4.ip_default_ttl - packet lifetime (TTL) by default. It may be necessary when it is necessary to mislead the CC operator using the phone as a modem, or when you need to make sure that packets from this host do not go beyond the network perimeter.
net.core.netdev_max_backlog- adjusts the size of the packet queue between the network card and the kernel. If the kernel does not manage to process packets and the queue is full, then new packets are discarded. it may be necessary to tighten in certain situations to cope with peak loads and not experience network problems.
Below are two parameters that are responsible for connection queues, i.e. TCP. These two parameters are primarily tuned on loaded web-services, in case there are problems.
To understand, you need to know how connections work in tcp:
net.core.somaxconn - the size of the queue of established connections awaiting acceptance () processing. If we experience small peaks in the load on the application, perhaps increasing this parameter will help you.
net.ipv4.tcp_max_syn_backlog - the size of the queue of not established connections .
For most TCP settings, and indeed, for everyone else, an understanding of the mechanisms that regulate these settings is required.
For example, how tcp transfers data. Since this protocol “guarantees delivery”, it requires confirmation of delivery from the second side of the connection. These confirmations are called ACKnoweledgement. The received byte range is confirmed. We also know that we can transfer data to the network in blocks equal to the size of the MTU (for simplicity, 1,500 bytes), and we need to transfer more, say 1,500,000 bytes, that is, 1000 frames, the data will be fragmented. If we have a server on the same network at a distance of one patch cord from each other - we won’t notice any problems, but if we are exchanging with a remote system, then it will take a very long time to confirm each packet, because we have to wait there our package will reach and confirmation of receipt will return from there, this will greatly affect the data transfer speed. To solve this problem, tcp_window was introduced, under the size of which 16 bits are allocated in the tcp header. Roughly speaking, this is the number of bytes that we can send without confirmation. In 16 bits, we can store a maximum value of 2 ^ 16 = 65536 (65Kb), which is not a lot of multigigabit networks in our age.
To understand how we can transmit data, say, from Moscow to Novosibirsk (RTT (Round Trip Time) = 50ms) through a 1Gbit channel, let's do some calculations (VERY rough calculations below).
net.ipv4.tcp_window_scaling - 0 - turn off window scaling; 1 - enables window scaling. This should be supported on both sides and serves to optimize the use of the channel band.
But it’s not enough to be able to specify a window larger than 65K, you still need to be able to keep all the necessary data in memory allocated for the socket, in our case in tcp_buffer'ah:
net.ipv4.tcp_wmem, net.ipv4.tcp_rmem- Read and Write buffers look the same. These are three numbers in bytes, “min default max” - the minimum guaranteed buffer size, the default size and the maximum size, which the system will not allow to grow larger than the buffer. It’s worth approaching the setting of these parameters with the understanding of how many connections you are expecting, how much data you are going to transfer and how “slow” your clients / servers are.
net.ipv4.tcp_mem- tcp stack memory management settings. 3 numbers in the PAGES “min pressure max”, which describe: min - the threshold for using memory with buffers below which the system will not care about crowding out the buffers; pressure - the threshold at which the system will take care of reducing memory consumption by buffers, as long as this is possible; max - the threshold upon reaching which the memory will not be allocated and the buffers cannot grow.
Sometimes applications fall, so to speak, “into the crust”. This crust (“memory dump”) can be used to debug the problem with gdb. The thing is definitely useful when set up. By default, coredump is saved in the faceless core file in the working directory of the application, it may still be accompanied by the pid of the application, but you can do everything much more convenient:
kernel.core_pattern- allows you to specify the format of the name (and path) that will be used to save core dump. For example, “/var/core/%E.%t.%p” will save the “peel” to the / var / core / directory using the full path to the program that crashed (replacing / with!) And adding timestamp events and pid applications. You can even redirect core to an external program for analysis. For more information, see man 5 core.
This is all just the tip of the iceberg, if you describe everything in more detail - you can write a whole book.
Good luck in the console.
Here is such a small note that we prepared as part of our Linux Administrator course . The topic was chosen by our listeners by voting, so we hope that the note will be interesting and useful to you.
But the moment comes when the administrator in his campaigns on the system meets this beast - sysctl. Most likely he meets someone from the net.ipv4 or vm family, even most likely net.ipv4.ip_forward, if he is going for a router or vm.swappinness, if he is concerned about the growing swap of his penguin. The first beast allows the penguin to receive packets with one wing and give it to the other (allows routing), and the second helps to cope with the use of swap in a quiet system and to regulate its use - in a loaded one.
A meeting with any of these “animals” opens the door to a whole world of system settings and kernel components, there are several families, the names of most of which speak for themselves: net - network; kern - kernel; vm - memory and caches. But this is all a "fairy tale", the reality is much more interesting.
Acquainted with this host of settings, an inexperienced administrator, like a moth into the light, flies and wants to rather quickly configure them all, to make the system better and more optimal. Oh yeah! His temptation is great and the goal is good, but the chosen path, sometimes, leads in a completely different direction. The name of that path is “google tuning”.
After all, what is the temptation to quickly google “optimal sysctl settings” and apply some recipe from the beginning of the article without really going into what is written below - TL; DR. In most cases, the system does not get worse, because the load or configuration is not so that problems pop up. This is later, with experience, delving into the system settings for a specific case, you will understand that this is some kind of nonsense that was written.
Here, for example, the parameters net.ipv4.tcp_mem, rmem, wmem - look very similar, three numbers are of the type “4096 87380 6291456”, only, this is bad luck, for [rw] mem they are bytes, and for mem they are bytes and, if set all three parameters to the same value, then there will be an “explosive" configuration, because tcp_mem is responsible for managing the memory consumed by tcp, and [rw] mem for socket buffers.
And there are sysctl'i which, until you meet and rake - you won’t think about what this sysctl is, like, for example, net.ipv4.conf.all.rp_filter. The worst thing that shoots only when you need to do asymmetric routes, that is, your router has 2+ interfaces and traffic can come from one interface and return through another, that is, quite rarely. Called Reverse Path Filtering, it blocks packets that come from an address that is not routed through the interface from where they come.
There are very useful parameters, but they require a careful study of the documentation and calculations to find out how they can help you in your specific situation. I repeat that the defaults in linux are good enough. This is especially true for tcp settings and the network as a whole.
The parameters that you have to play often enough are:
vm.swappiness - setting the aggressiveness of the “free" memory. This is necessary in order to maintain the largest possible amount of RAM available for applications and work. In the default value - 60, the system translates into memory pages that have not been used for some long time, in the value 0 or 1 - the system tries to use the swap only when it cannot allocate physical memory or when the amount of available memory matches the specified in vm.min_free_kbytes. This is not to say that using a swap is definitely good or bad. It all depends on the situation and the profile of memory usage, and this knob allows us to control the ratio of the system to the swap from 0 - I do not like it at all, up to 100 - oh yes, svoop!
vm.min_free_kbytes - (once mentioned in vm.swappiness) Defines the minimum free memory size that must be supported. If set too small - the system will break, if too large - the OOM (Out Of Memory) killer will often come.
vm.overcommit_memory- allows / disallows “allocate” more memory than it is: 0 - the system checks every time if there is a sufficient amount of free memory; 1 - the system always believes that there is memory and allows allocation while the memory really is; 2 - prohibit asking for more memory than is - you can allocate no more than RAM + SWAP. This can fire when you have an application, for example redis, that consumes> ½ of memory and decides to write data to disk, for which it fork and copy all the data, but because there may not be enough available memory, it may either break the recording to disk, or come oom and kill something you need.
net.ipv4.ip_forward- permission or prohibition on packet routing. We are faced with the need to twist this knob when we configure the router. Here everything is more or less clear: 0 - turn off; 1 - enable.
net.ipv4. {all, default, interface} .rp_filter - controls the Reverse Path Filtering option: 0 - do not check, disable; 1 - “strict” mode, when packets are discarded answers that would not go through the interface from which the packet came; 2 - “relaxed” mode - only those packets to which the route is unknown are discarded (if there is a default route, in my opinion, it should lead to the same effect as 0).
net.ipv4.ip_local_port_range- defines the minimum and maximum ports, which is used to create a local client socket. If your system makes a large number of calls to network resources, then you may have a problem of lack of local ports for establishing connections. This option allows you to adjust the port range that is used to establish client connections. It can also be useful in order to secure your services that “listen” to high ports.
net.ipv4.ip_default_ttl - packet lifetime (TTL) by default. It may be necessary when it is necessary to mislead the CC operator using the phone as a modem, or when you need to make sure that packets from this host do not go beyond the network perimeter.
net.core.netdev_max_backlog- adjusts the size of the packet queue between the network card and the kernel. If the kernel does not manage to process packets and the queue is full, then new packets are discarded. it may be necessary to tighten in certain situations to cope with peak loads and not experience network problems.
Below are two parameters that are responsible for connection queues, i.e. TCP. These two parameters are primarily tuned on loaded web-services, in case there are problems.
To understand, you need to know how connections work in tcp:
- The program opens a listening socket: socket () -> listen (). As a result, it gets, for example *: 80 (the 80th port on all interfaces).
- The client establishes a connection: a) sends a syn-packet to the server (tcp_max_syn_backlog works in this place); b) receives syn-ack from the server; c) sends ack to the server (and here somaxconn already works)
- Through an accept call, the connection is processed and passed to the process to work with a specific client.
net.core.somaxconn - the size of the queue of established connections awaiting acceptance () processing. If we experience small peaks in the load on the application, perhaps increasing this parameter will help you.
net.ipv4.tcp_max_syn_backlog - the size of the queue of not established connections .
For most TCP settings, and indeed, for everyone else, an understanding of the mechanisms that regulate these settings is required.
For example, how tcp transfers data. Since this protocol “guarantees delivery”, it requires confirmation of delivery from the second side of the connection. These confirmations are called ACKnoweledgement. The received byte range is confirmed. We also know that we can transfer data to the network in blocks equal to the size of the MTU (for simplicity, 1,500 bytes), and we need to transfer more, say 1,500,000 bytes, that is, 1000 frames, the data will be fragmented. If we have a server on the same network at a distance of one patch cord from each other - we won’t notice any problems, but if we are exchanging with a remote system, then it will take a very long time to confirm each packet, because we have to wait there our package will reach and confirmation of receipt will return from there, this will greatly affect the data transfer speed. To solve this problem, tcp_window was introduced, under the size of which 16 bits are allocated in the tcp header. Roughly speaking, this is the number of bytes that we can send without confirmation. In 16 bits, we can store a maximum value of 2 ^ 16 = 65536 (65Kb), which is not a lot of multigigabit networks in our age.
To understand how we can transmit data, say, from Moscow to Novosibirsk (RTT (Round Trip Time) = 50ms) through a 1Gbit channel, let's do some calculations (VERY rough calculations below).
- Without tcp_window. It turns out that we can only send 1500 bytes in .05s, 1500 / 0.05 = 30000 bytes / second. Not much, despite the fact that the channel speed is 1Gbit / s, which is roughly equal to 100Mb / s. ~ 30Kb / s vs ~ 100Mb / s - there is a problem, we will not recycle the available band.
- With tcp_window equal to 65536 (which maximum can be specified in the header). Those. we can immediately send all of our 65K. 65536 / 0.05 = 1310720 = 1.25Mb / s. 1.25Mb / s vs ~ 100Mb / s - it's still not enough, the difference is about 80 times.
- So how much do we need in order to utilize at least 900Mbit? carry out the inverse calculations. (900000000/8) b / s * 0.05s = 5625000b ~ = 5.36 Mb. This is the window size we need in order to efficiently transfer data. But since we have a very long link, then we may have problems => losses there. This will also affect bandwidth. In order to be able to notify of a window size larger than 65K through a 16-bit field, the tcp_window_scaling option was introduced.
net.ipv4.tcp_window_scaling - 0 - turn off window scaling; 1 - enables window scaling. This should be supported on both sides and serves to optimize the use of the channel band.
But it’s not enough to be able to specify a window larger than 65K, you still need to be able to keep all the necessary data in memory allocated for the socket, in our case in tcp_buffer'ah:
net.ipv4.tcp_wmem, net.ipv4.tcp_rmem- Read and Write buffers look the same. These are three numbers in bytes, “min default max” - the minimum guaranteed buffer size, the default size and the maximum size, which the system will not allow to grow larger than the buffer. It’s worth approaching the setting of these parameters with the understanding of how many connections you are expecting, how much data you are going to transfer and how “slow” your clients / servers are.
net.ipv4.tcp_mem- tcp stack memory management settings. 3 numbers in the PAGES “min pressure max”, which describe: min - the threshold for using memory with buffers below which the system will not care about crowding out the buffers; pressure - the threshold at which the system will take care of reducing memory consumption by buffers, as long as this is possible; max - the threshold upon reaching which the memory will not be allocated and the buffers cannot grow.
Sometimes applications fall, so to speak, “into the crust”. This crust (“memory dump”) can be used to debug the problem with gdb. The thing is definitely useful when set up. By default, coredump is saved in the faceless core file in the working directory of the application, it may still be accompanied by the pid of the application, but you can do everything much more convenient:
kernel.core_pattern- allows you to specify the format of the name (and path) that will be used to save core dump. For example, “/var/core/%E.%t.%p” will save the “peel” to the / var / core / directory using the full path to the program that crashed (replacing / with!) And adding timestamp events and pid applications. You can even redirect core to an external program for analysis. For more information, see man 5 core.
This is all just the tip of the iceberg, if you describe everything in more detail - you can write a whole book.
Good luck in the console.
Here is such a small note that we prepared as part of our Linux Administrator course . The topic was chosen by our listeners by voting, so we hope that the note will be interesting and useful to you.
