Netstat, where are my datagrams?

    It is likely that it is difficult to find another program that is just as useful and poorly documented as the Netstatoption to display network data flow statistics. When we inspect the network status on a single Linux host, you can always be sure that this utility is available. And now we want to understand whether the network stack is coping with the load, or the problem on the upper floors of OSI , in fact, where the wheels of the business logic of our application are spinning.




    (5:562)$ netstat -s |wc -l
    124

    Hooray, we have a lot of useful information, now we quickly figure out what's what. Just to understand what kind of beast this is timeout in transit, obviously something bad.


    Icmp:
    11475275 ICMP messages received
    327527 input ICMP message failed.
    ICMP input histogram:
    detination unreachable: 2233840
    timeout in transit: 5612259

    Now I will look at the manual.


    # man netstat |grep timeout
    #

    An empty set suggests that it is time to find out from Yandex, Google and even DuckDuckGo. The search result was about the same, but only Google gave 111 thousand variants of the empty set, and Yandex was succinctly limited to 326 variants.


    But we didn’t choose open source software in vain, you can always check with the source. In our case, the source code is in the nertstat.cpackage file Net-tools.


    Here is the part of the function int mainthat defines the action for displaying statistics.


        case '?':
            usage(E_OPTERR);
        case 'h':
            usage(E_USAGE);
        case 's':
            flag_sta++;
    ...
        if (flag_sta) {
            if (!afname[0])
                safe_strncpy(afname, DFLT_AF, sizeof(afname));
            if (!strcmp(afname, "inet")) {
    #if HAVE_AFINET
                parsesnmp(flag_raw, flag_tcp, flag_udp, flag_sctp);

    The function is parsesnmpdefined in a file statistics.cand parses files:


    /proc/net/netstat
    /proc/net/snmp
    /proc/net/sctp/snmp

    Clearly clear, so you can just look in / proc / net / snmp and see what is behind timeout in transit.


    # cat /proc/net/snmp |grep -iw icmp
    Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
    Icmp: 127 2 0 90 0 0 0 0 25 0 8 0 4 0 1763 0 1730 0 0 0 0 0 25 0 8 0 0

    Comparing these values ​​with the output of the command netstat, we can conclude that it timeout in transitcorresponds to a variable InTimeExcds, and the chain further leads to a file /usr/include/linux/snmp.hwhere all these fields are defined.


    /* icmp mib definitions */
    /*
     * RFC 1213:  MIB-II ICMP Group
     * RFC 2011 (updates 1213):  SNMPv2 MIB for IP: ICMP group
     */
    enum
    {
        ICMP_MIB_NUM = 0,
        ICMP_MIB_INMSGS,            /* InMsgs */
        ICMP_MIB_INERRORS,          /* InErrors */
        ICMP_MIB_INDESTUNREACHS,        /* InDestUnreachs */
        ICMP_MIB_INTIMEEXCDS,           /* InTimeExcds */
        ICMP_MIB_INPARMPROBS,           /* InParmProbs */
        ICMP_MIB_INSRCQUENCHS,          /* InSrcQuenchs */
        ICMP_MIB_INREDIRECTS,           /* InRedirects */
        ICMP_MIB_INECHOS,           /* InEchos */
        ICMP_MIB_INECHOREPS,            /* InEchoReps */
        ICMP_MIB_INTIMESTAMPS,          /* InTimestamps */
        ICMP_MIB_INTIMESTAMPREPS,       /* InTimestampReps */
        ICMP_MIB_INADDRMASKS,           /* InAddrMasks */
        ICMP_MIB_INADDRMASKREPS,        /* InAddrMaskReps */
        ICMP_MIB_OUTMSGS,           /* OutMsgs */
        ICMP_MIB_OUTERRORS,         /* OutErrors */
        ICMP_MIB_OUTDESTUNREACHS,       /* OutDestUnreachs */
        ICMP_MIB_OUTTIMEEXCDS,          /* OutTimeExcds */
        ICMP_MIB_OUTPARMPROBS,          /* OutParmProbs */
        ICMP_MIB_OUTSRCQUENCHS,         /* OutSrcQuenchs */
        ICMP_MIB_OUTREDIRECTS,          /* OutRedirects */
        ICMP_MIB_OUTECHOS,          /* OutEchos */
        ICMP_MIB_OUTECHOREPS,           /* OutEchoReps */
        ICMP_MIB_OUTTIMESTAMPS,         /* OutTimestamps */
        ICMP_MIB_OUTTIMESTAMPREPS,      /* OutTimestampReps */
        ICMP_MIB_OUTADDRMASKS,          /* OutAddrMasks */
        ICMP_MIB_OUTADDRMASKREPS,       /* OutAddrMaskReps */
        ICMP_MIB_CSUMERRORS,            /* InCsumErrors */
        __ICMP_MIB_MAX
    };

    It’s like who, but for me RFC 1213 as a chemist, you need to know the periodic table, because this is one of the cornerstones of monitoring network devices. Here is an element definition from RFC 1213writing from memory.


    icmpInTimeExcds OBJECT-TYPE
                  SYNTAX  Counter
                  ACCESS  read-only
                  STATUS  mandatory
                  DESCRIPTION
                          "The number of ICMP Time Exceeded messages received."
                  ::= { icmp 4 }

    So far, it seems that all these definitions as butter are not disclosing the essence, but fortunately they ICMP Time Exceededare part of the ICMP protocol definition .


    If the gateway processing the datagram sees that the TTL field contains a null value, the datagram should be discarded. The gateway can notify the sender of the datagram with a time exceeded message.

    So, we found what we were looking for. The output of the command netstatthat contained this mysterious field means the number of dropped ICMP packets with zero TTL . IRL [1] this means that there is a ring in the network where the packets profiled the entire TTL [2] .


    timeout in transit: 5612259

    Linux MIB


    During this dive into the jungle of the Linux kernel, I was surprised to find the SNMP subsystem. No, not an SNMP agent, save us Alan Cox, namely a small SNMP stack. This is written in the comment on the file /usr/src/linux/include/net/snmp.h.


    /*
     *              SNMP MIB entries for the IP subsystem.
     *              Alan Cox 
     *
     *              We don't chose to implement SNMP in the kernel (this would
     *              be silly as SNMP is a pain in the backside in places). We do
     *              however need to collect the MIB statistics and export them
     *              out of /proc (eventually)
     */

    SNMP stands for Simple Network Management Protocol , but as a bearded joke says, the letter S has never lied in the abbreviation. I plan to write about this brain grinder separately, since all the monitoring software for computer networks and network nodes is spinning on it . In the meantime, the necessary minimum is to make it clear where the statistics variables come from netstat.




    In the simplest case, we have a client-server architecture, where the client can be MRTG or Munin , and the server part is an SNMP agent on a network node. Almost all modern network devices have an SNMP agent on board, even home WiFi routers. Windows has its own SNMP Service, while Linux and open Unix systems use it Net-SNMP.


    The SNMP client and server exchange messages: the client sends a request, and the server returns a response. Usually, this exchange looks as follows.


    K. - Cheburashka, appliances!
    S. - Forty
    K. - What is forty?
    S. - What about appliances?

    The client asks for the value of a variable from the Great Dictionary. The server looks in the sun, finds a variable, looks in its registry and returns a value. The Great Dictionary is the MIB [3] Database from the figure.


    Here's what polling for system variables uptimeand others looks like on the man page .


    snmpwalk -Os -c public -v 1 zeus system 
    sysDescr.0 = STRING: "SunOS zeus.net.cmu.edu 4.1.3_U1 1 sun4m"
    sysObjectID.0 = OID: enterprises.hp.nm.hpsystem.10.1.1
    sysUpTime.0 = Timeticks: (155274552) 17 days, 23:19:05
    sysContact.0 = STRING: ""
    sysName.0 = STRING: "zeus.net.cmu.edu"
    sysLocation.0 = STRING: ""
    sysServices.0 = INTEGER: 72

    It should be understood that dozens of RFCs and the abyss of their possible implementations are hidden behind the euphemisms of the Great Dictionary, registry and the SNMP protocol itself. Returning to our file snmp.h, only the surface part of the iceberg is visible:


    # grep RFC /usr/include/linux/snmp.h
    * RFC 1213:  MIB-II
    * RFC 2011 (updates 1213):  SNMPv2-MIB-IP
    * RFC 2863:  Interfaces Group MIB
    * RFC 2465:  IPv6 MIB: General Group
    * RFC 1213:  MIB-II ICMP Group
    * RFC 2011 (updates 1213):  SNMPv2 MIB for IP: ICMP group
    * RFC 2466:  ICMPv6-MIB
    * RFC 1213:  MIB-II TCP group
    * RFC 2012 (updates 1213):  SNMPv2-MIB-TCP
    * RFC 1213:  MIB-II UDP group
    * RFC 2013 (updates 1213):  SNMPv2-MIB-UDP
     LINUX_MIB_TCPMINTTLDROP, /* RFC 5082 */

    And if that was all, the decryption of the statistics netstatcould be like this:


    переменная из netstat --> соответствующая ей строка из /proc:
    * /proc/net/snmp
    * /proc/net/netstat
    * /proc/net/sctp/snmp
    --> соответствующее RFC --> определение.

    However, even such a multi-stage decryption is not enough, since a significant part snmp.his occupied by a mysterious one Linux MIB, the variables of which are not defined in any RFC, and in the Great Dictionary - MIB Database, they are not there either.


    /* linux mib definitions */
    enum
    {
        LINUX_MIB_NUM = 0,
        LINUX_MIB_SYNCOOKIESSENT,       /* SyncookiesSent */
        LINUX_MIB_SYNCOOKIESRECV,       /* SyncookiesRecv */
        LINUX_MIB_SYNCOOKIESFAILED,     /* SyncookiesFailed */
        LINUX_MIB_EMBRYONICRSTS,        /* EmbryonicRsts */
        LINUX_MIB_PRUNECALLED,          /* PruneCalled */
        LINUX_MIB_RCVPRUNED,            /* RcvPruned */
        LINUX_MIB_OFOPRUNED,            /* OfoPruned */
    ...
        LINUX_MIB_TCPMTUPSUCCESS,       /* TCPMTUPSuccess */
        __LINUX_MIB_MAX
    };

    We are ready for the next stage of the dive, in the next article we will try to find out what is hidden behind the buffer memory errors and how queue breakers collapse in tcp.




    1. In real life, from English in real life.
    2. Package lifetime, from English time to live
    3. Management Information Base - a structured set of control information for node objects. More details here .

    Also popular now: