October 23, 2012 at 10:21Overcome remote connection disconnection in the absence of user actions
- Tutorial
When working with the GUI and terminal applications, it often happens that a user, working in remote access mode (usually via the Internet), after leaving the computer for about 15 minutes, upon returning finds that the program has crashed. It responds to any action with an error containing approximately the following phrases: "Lost connection with the server", "[WINSOCK] virtual circuit reset by host", etc. This is also observed when executing "long-playing" methods (server requests), in which there is no output of a progress bar or any interactivity.
This problem is typical not only for GUI and terminal solutions based on InterSystems Caché and Ensemble database systems, but in general for any client-server interaction via TCP / IP. Usually it is solved at the application level by periodically exchanging empty messages of a special kind, intended only to signal that the application is "alive".
Below is how to solve this problem without programming.
The source of the problem lies in the nature of the TCP / IP protocol. Typically, the source of the TCP / IP session and its receiver are on different networks, and there are several routers in the path of the session. At least one of them usually performs NAT address translation. Router resources are always limited, so some of them clean up NAT tables from dead sessions. A session is considered “dead” if packets were not transmitted over it for some specified time interval (let's call it the cleaning interval). Thus, a “silent” session can be mistaken for a “dead” one and removed from the NAT table. At the same time, neither the source nor the receiver is notified about this (“not a royal case”), and both of them remain confident that the session is still “alive” (which is easy to verify with the netstat command by executing it on the client or server at the time of occurrence mistakes but before clicking on OK). When the user who receives the error message clicks OK, the client will find out about the session break; the server process will end when the "dead" session recognizes the OS.
It has been experimentally established that the cleaning interval for many routers (at least with flashed Linux 2.4 / iptables) is about 10 minutes. We will try to make our TCP session automatically maintain itself in an active state, even when no data packets are transmitted.
At the OS level, the detection of broken TCP connections is controlled by the following kernel parameters that control the operation of the tcp_keepalive mechanism [1] :
• tcp_keepalive_time - time interval since the last data packet was sent; after this period, the connection is marked as requiring verification; after the start of the check, the parameter is not used;
• tcp_keepalive_intvl - the interval between test packets (sending of which begins after tcp_keepalive_time expires);
• tcp_keepalive_probes - the number of unacknowledged test packets; after this counter has been used up, the connection is considered broken.
I must say that the tcp_keepalive mechanism has a dual purpose: it can be used both to artificially maintain the activity of the compound, and to detect broken (so-called "half-open") compounds. This article mainly discusses the first application, and the second application, perhaps, will be discussed in the next article on this topic.
In order for the tcp_keepalive mechanism to be used for TCP connections, two conditions are necessary:
• support at the OS level; fortunately, it is available on both Windows and Linux;
• at one end of the connection, the socket must be open with the SO_KEEPALIVE parameter. As it turned out, Caché services open sockets with this parameter, and the OpenSSH service is easy to get to do the same.
The first parameter ( tcp_keepalive_time ) is of most interest to us , since it depends on it how often inactive connections will be checked (from the point of view of lack of traffic). Its default value, in both Windows and Linux, is two hours (7200 s). The typical time of inactivity, after which there is a break, is about 10 minutes. Therefore, it is proposed to set the parameter value to 5 minutes, which will allow artificially maintaining the activity of TCP sessions without overloading the network with excessive traffic (5 minutes is not 5 seconds).
You must have Administrator rights to the server. In the registry key
HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ Tcpip \ Parameters,
create a DWORD parameter named KeepAliveTime with a value of 300000 (decimal). The parameter is set in milliseconds, so the proposed value is 5 minutes. Then stop Caché and reboot the server.
As for the two other tcp_keepalive parameters, their defaults on Windows are:
KeepAliveInterval
Key: Tcpip \ Parameters
Value Type: REG_DWORD — time in milliseconds
Valid Range: 0–0xFFFFFFFE
Default: 1000 (1 second)
KeepAliveProbes
Such a parameter (setting the number of unconfirmed test packets) does not exist in the registry. According to [2] , in Windows 2000 / XP / 2003, the TcpMaxDataRetransmission parameter value (default is 5) is used as such, and in later versions [3] - a fixed value equal to 10. Therefore, if only the value of the first parameter is changed (with 7200 by 300), keeping the default for the second, the Windows 2008 server will learn about the disconnection of the TCP connection after 1 * 10 + 300 = 310 seconds.
You can change the parameter values on the go without restarting the server. Log in as root and run:
To make the change durable with respect to possible server reboots, the easiest way is to edit the kernel parameters file /etc/sysctl.conf, adding a line to it (two are better):
Please note that unlike Windows, the parameter value is set in seconds.
As for the other two tcp_keepalive parameters, their defaults in Linux are as follows:
If you change only the value of the first parameter (from 7200 to 300), while preserving the defaults for the other two, the Linux server will only learn about the disconnection after 75 * 9 + 300 = 975 seconds.
Starting with version 2008.2, in Caché for Windows and Linux platforms, it became possible to set tcp_keepalive_time at the socket level, which is convenient since it avoids changing the operating system settings. However, "in its pure form" this feature is mainly of interest only to independent socket server developers. Fortunately, it was supplemented by the configuration parameter TCPKeepAlive = n in the [SQL] section, which is editable from the System Management Portal page: Configuration> General SQL Settings. The default value is 300 seconds (what the doctor ordered). The effect of the parameter extends not only to SQL, but, as you might guess, to any connections to Caché served by% Service_Bindings. These include, in particular, object access via CacheActiveX.Factory, so if your application can use this protocol as a transport, do not miss this opportunity.
If you use SSH [4] (to work in shell mode or as a transport for your GUI application), then ... most likely, the done kernel tuning will be enough, since the OpenSSH service (at least in version 5.x) -default opens a socket with parameter SO_KEEPALIVE.
Just in case, you should check the configuration file / etc / ssh / sshd_config. Find the line in it
If you find, then you do not need to do anything, since the default parameter values are provided in a commented form.
The SSH v.2 protocol has alternative means of monitoring session activity, for example, by configuring the OpenSSH service settings ClientAliveInterval and ClientAliveCountMax.
Using these parameters, unlike TCPKeepAlive, KeepAlive requests are sent through a secure SSH channel and cannot be substituted. One has to admit that alternative tools are safer than the traditional TCPKeepAlive mechanism, for which there is a danger of analyzing KeepAlive packets and organizing DoS attacks [5] .
There are categories of network problems in which the described approach may be ineffective.
One of them occurs when due to the poor quality of network services, communications can physically disappear for short periods of time. If the session is idle, and the connection is temporarily lost and restored before the client or server tries to send something to each other, then none of them “notices” anything, and the TCP session is saved. In the case of periodic checks of TCPKeepAlive, the likelihood of a server contacting the client increases when the connection is temporarily lost, which can lead to forced disconnections of the TCP connection. In this situation, you can try to increase the KeepAliveInterval server to 60-75 seconds (remembering that Windows defaults to 1 second) with a maximum number of retries of 10, in the hope that in 10 minutes any temporary network problem will resolve itself. True,
KeepAliveTime + (KeepAliveInterval * number of retries)> 10 minutes
then a TCP session, despite all the efforts made, can be mistaken for “dead” and cleared from the NAT table.
Another category of problems is related to the insufficient bandwidth of the routers and / or communication channels used, when any packets, including KeepAlive, can be lost during congestion. In the case of routers, such problems are sometimes solved by changing the firmware (for example, it helped me defeat Acorp ADSL XXXX), or, in the worst case, replacing it with a more productive model. In the case of “too narrow” communication channels, there is nothing left to do but expand them.
The proposed approach allows artificially maintaining the activity of TCP / IP sessions, which currently do not transmit any data, exclusively at the system level, without making any changes to the application code. Today it is tested and successfully used in Caché for UNIX (Red Hat Enterprise Linux 5 for x86-64) 2010.1.4 (Build 803), Caché for Windows (x86-64) 2010.1.4 (Build 803), as well as in more later versions.
It should be recognized that it works effectively if the network connection is physically stable, and in addition to breaking inactive sessions, you have no other network problems.
When deploying an application in an aggressive environment (remote access, distributed systems, etc.), think about implementing KeepAlive not at the TCP level, but at the level of a secure protocol of a higher level; SSH is a good candidate.
[1] Fabio Bussato. TCP Keepalive HOWTO
[2] How to harden the TCP / IP stack against denial of service attacks in Windows Server 2003
[3] TCP / IP Registry Values for Microsoft Windows Vista and Windows Server 2008
[4] Interactive system manual review system (man- s)
[5] OpenSSH: installing and configuring sshd
This problem is typical not only for GUI and terminal solutions based on InterSystems Caché and Ensemble database systems, but in general for any client-server interaction via TCP / IP. Usually it is solved at the application level by periodically exchanging empty messages of a special kind, intended only to signal that the application is "alive".
Below is how to solve this problem without programming.
Source of problem
The source of the problem lies in the nature of the TCP / IP protocol. Typically, the source of the TCP / IP session and its receiver are on different networks, and there are several routers in the path of the session. At least one of them usually performs NAT address translation. Router resources are always limited, so some of them clean up NAT tables from dead sessions. A session is considered “dead” if packets were not transmitted over it for some specified time interval (let's call it the cleaning interval). Thus, a “silent” session can be mistaken for a “dead” one and removed from the NAT table. At the same time, neither the source nor the receiver is notified about this (“not a royal case”), and both of them remain confident that the session is still “alive” (which is easy to verify with the netstat command by executing it on the client or server at the time of occurrence mistakes but before clicking on OK). When the user who receives the error message clicks OK, the client will find out about the session break; the server process will end when the "dead" session recognizes the OS.
It has been experimentally established that the cleaning interval for many routers (at least with flashed Linux 2.4 / iptables) is about 10 minutes. We will try to make our TCP session automatically maintain itself in an active state, even when no data packets are transmitted.
Proposed solution
At the OS level, the detection of broken TCP connections is controlled by the following kernel parameters that control the operation of the tcp_keepalive mechanism [1] :
• tcp_keepalive_time - time interval since the last data packet was sent; after this period, the connection is marked as requiring verification; after the start of the check, the parameter is not used;
• tcp_keepalive_intvl - the interval between test packets (sending of which begins after tcp_keepalive_time expires);
• tcp_keepalive_probes - the number of unacknowledged test packets; after this counter has been used up, the connection is considered broken.
I must say that the tcp_keepalive mechanism has a dual purpose: it can be used both to artificially maintain the activity of the compound, and to detect broken (so-called "half-open") compounds. This article mainly discusses the first application, and the second application, perhaps, will be discussed in the next article on this topic.
In order for the tcp_keepalive mechanism to be used for TCP connections, two conditions are necessary:
• support at the OS level; fortunately, it is available on both Windows and Linux;
• at one end of the connection, the socket must be open with the SO_KEEPALIVE parameter. As it turned out, Caché services open sockets with this parameter, and the OpenSSH service is easy to get to do the same.
The first parameter ( tcp_keepalive_time ) is of most interest to us , since it depends on it how often inactive connections will be checked (from the point of view of lack of traffic). Its default value, in both Windows and Linux, is two hours (7200 s). The typical time of inactivity, after which there is a break, is about 10 minutes. Therefore, it is proposed to set the parameter value to 5 minutes, which will allow artificially maintaining the activity of TCP sessions without overloading the network with excessive traffic (5 minutes is not 5 seconds).
Setting tcp_keepalive options on a Windows server
You must have Administrator rights to the server. In the registry key
HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ Tcpip \ Parameters,
create a DWORD parameter named KeepAliveTime with a value of 300000 (decimal). The parameter is set in milliseconds, so the proposed value is 5 minutes. Then stop Caché and reboot the server.
As for the two other tcp_keepalive parameters, their defaults on Windows are:
KeepAliveInterval
Key: Tcpip \ Parameters
Value Type: REG_DWORD — time in milliseconds
Valid Range: 0–0xFFFFFFFE
Default: 1000 (1 second)
KeepAliveProbes
Such a parameter (setting the number of unconfirmed test packets) does not exist in the registry. According to [2] , in Windows 2000 / XP / 2003, the TcpMaxDataRetransmission parameter value (default is 5) is used as such, and in later versions [3] - a fixed value equal to 10. Therefore, if only the value of the first parameter is changed (with 7200 by 300), keeping the default for the second, the Windows 2008 server will learn about the disconnection of the TCP connection after 1 * 10 + 300 = 310 seconds.
Setting tcp_keepalive options on a Linux server
You can change the parameter values on the go without restarting the server. Log in as root and run:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
To make the change durable with respect to possible server reboots, the easiest way is to edit the kernel parameters file /etc/sysctl.conf, adding a line to it (two are better):
# help to prevent disconnects
net.ipv4.tcp_keepalive_time = 300
Please note that unlike Windows, the parameter value is set in seconds.
As for the other two tcp_keepalive parameters, their defaults in Linux are as follows:
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
If you change only the value of the first parameter (from 7200 to 300), while preserving the defaults for the other two, the Linux server will only learn about the disconnection after 75 * 9 + 300 = 975 seconds.
Setting the TCPKeepAlive parameter in the Caché DBMS configuration
Starting with version 2008.2, in Caché for Windows and Linux platforms, it became possible to set tcp_keepalive_time at the socket level, which is convenient since it avoids changing the operating system settings. However, "in its pure form" this feature is mainly of interest only to independent socket server developers. Fortunately, it was supplemented by the configuration parameter TCPKeepAlive = n in the [SQL] section, which is editable from the System Management Portal page: Configuration> General SQL Settings. The default value is 300 seconds (what the doctor ordered). The effect of the parameter extends not only to SQL, but, as you might guess, to any connections to Caché served by% Service_Bindings. These include, in particular, object access via CacheActiveX.Factory, so if your application can use this protocol as a transport, do not miss this opportunity.
Setting KeepAlive Settings in OpenSSH Server Configuration
If you use SSH [4] (to work in shell mode or as a transport for your GUI application), then ... most likely, the done kernel tuning will be enough, since the OpenSSH service (at least in version 5.x) -default opens a socket with parameter SO_KEEPALIVE.
Just in case, you should check the configuration file / etc / ssh / sshd_config. Find the line in it
#TCPKeepAlive yes
If you find, then you do not need to do anything, since the default parameter values are provided in a commented form.
The SSH v.2 protocol has alternative means of monitoring session activity, for example, by configuring the OpenSSH service settings ClientAliveInterval and ClientAliveCountMax.
Using these parameters, unlike TCPKeepAlive, KeepAlive requests are sent through a secure SSH channel and cannot be substituted. One has to admit that alternative tools are safer than the traditional TCPKeepAlive mechanism, for which there is a danger of analyzing KeepAlive packets and organizing DoS attacks [5] .
ClientAliveInterval 0
ClientAliveCountMax 3
TCPKeepAlive no
Does it always work?
There are categories of network problems in which the described approach may be ineffective.
One of them occurs when due to the poor quality of network services, communications can physically disappear for short periods of time. If the session is idle, and the connection is temporarily lost and restored before the client or server tries to send something to each other, then none of them “notices” anything, and the TCP session is saved. In the case of periodic checks of TCPKeepAlive, the likelihood of a server contacting the client increases when the connection is temporarily lost, which can lead to forced disconnections of the TCP connection. In this situation, you can try to increase the KeepAliveInterval server to 60-75 seconds (remembering that Windows defaults to 1 second) with a maximum number of retries of 10, in the hope that in 10 minutes any temporary network problem will resolve itself. True,
KeepAliveTime + (KeepAliveInterval * number of retries)> 10 minutes
then a TCP session, despite all the efforts made, can be mistaken for “dead” and cleared from the NAT table.
Another category of problems is related to the insufficient bandwidth of the routers and / or communication channels used, when any packets, including KeepAlive, can be lost during congestion. In the case of routers, such problems are sometimes solved by changing the firmware (for example, it helped me defeat Acorp ADSL XXXX), or, in the worst case, replacing it with a more productive model. In the case of “too narrow” communication channels, there is nothing left to do but expand them.
Conclusion
The proposed approach allows artificially maintaining the activity of TCP / IP sessions, which currently do not transmit any data, exclusively at the system level, without making any changes to the application code. Today it is tested and successfully used in Caché for UNIX (Red Hat Enterprise Linux 5 for x86-64) 2010.1.4 (Build 803), Caché for Windows (x86-64) 2010.1.4 (Build 803), as well as in more later versions.
It should be recognized that it works effectively if the network connection is physically stable, and in addition to breaking inactive sessions, you have no other network problems.
When deploying an application in an aggressive environment (remote access, distributed systems, etc.), think about implementing KeepAlive not at the TCP level, but at the level of a secure protocol of a higher level; SSH is a good candidate.
Literature
[1] Fabio Bussato. TCP Keepalive HOWTO
[2] How to harden the TCP / IP stack against denial of service attacks in Windows Server 2003
[3] TCP / IP Registry Values for Microsoft Windows Vista and Windows Server 2008
[4] Interactive system manual review system (man- s)
[5] OpenSSH: installing and configuring sshd
