Principles of organizing IP traffic accounting

Sooner or later, any administrator receives instructions from the management: "calculate who goes to the network, and how many downloads." For providers, it is supplemented with the tasks of "letting anyone in, taking payment, restricting access." What to consider? How? Where? There is a lot of fragmentary information, they are not structured. We will save the novice admin from the tiresome searches, providing him with general knowledge, and useful links to the materiel.
In this article I will try to describe the principles of organizing the collection, accounting and control of network traffic. We will consider the issue, and we list the possible ways to retrieve information from network devices.

This is the first theoretical article in a series of articles devoted to the collection, accounting, management and billing of traffic and IT resources.

Internet access structure


In general, the network access structure is as follows:



  • External resources - the Internet, with all sites, servers, addresses and other things that do not belong to the network that you control.
  • An access device is a router (hardware, or PC-based), a switch, a VPN server, or a hub.
  • Internal resources - a set of computers, subnets, subscribers, the operation of which in the network must be taken into account or controlled.
  • Management or accounting server - a device on which specialized software runs. It can be functionally combined with a software router.

In this structure, network traffic flows from external resources to internal, and vice versa, through an access device. It sends traffic information to the management server. The management server processes this information, stores in the database, displays, issues lock commands. However, not all combinations of access devices, and methods of collection and management are compatible. Various options will be discussed below.

Network traffic


First you need to determine what is meant by “network traffic” and what useful statistical information can be extracted from the stream of user data.
The dominant internetworking protocol is still IP version 4 . IP protocol corresponds to the 3rd level of the OSI model(L3). Information (data) between the sender and the receiver is packaged in packets - having a header and a “payload”. The header determines where the packet comes from and where (IP address of the sender and recipient), packet size, type of payload. The main part of network traffic consists of packets with a payload of UDP and TCP - these are protocols of the 4th level (L4). In addition to addresses, the header of these two protocols contains port numbers that determine the type of service (application) that transfers data.



To transmit an IP packet over wires (or radio), network devices are forced to "wrap" (encapsulate) it in a protocol packet of the 2nd level (L2). The most common protocol of this type is Ethernet.. The actual transmission "to the wire" is at the 1st level. Usually, an access device (router) does not analyze packet headers at a level higher than the 4th (the exception is smart firewalls).
The information from the fields of addresses, ports, protocols and length counters from the L3 and L4 headers of the data packets makes up the “source material” that is used in accounting and traffic management. The actual amount of transmitted information is in the Length field of the IP header (including the length of the header itself). By the way, due to packet fragmentation due to the MTU mechanism, the total amount of transmitted data is always larger than the payload size.

The total length of the IP and TCP / UDP packet fields of interest to us in this context is 2 ... 10% of the total packet length. If you process and store all this information in batches, there will not be enough resources. Fortunately, the overwhelming amount of traffic is structured so that it consists of a set of “dialogs” between external and internal network devices, the so-called “flows”. For example, as part of a single email forwarding operation (SMTP protocol), a TCP session is opened between the client and the server. It is characterized by a constant set of parameters {source IP address, source TCP port, destination IP address, destination TCP port}. Instead of processing and storing information on a batch basis, it is much more convenient to store stream parameters (addresses and ports), as well as additional information - the number and sum of the lengths of transmitted packets to each side, optionally the duration of the session, the indexes of the router interfaces, the value of the ToS field, etc. This approach is beneficial for connection-oriented protocols (TCP), where you can explicitly capture the moment the session ends. However, for non-session-oriented protocols, it is possible to aggregate and logically terminate a stream record by, for example, a timeout. The following is an excerpt from the SQL database of our own billing system that logs information about traffic flows:



It is necessary to note the case when the access device translates addresses (NAT , masquerading) for organizing Internet access of computers on a local network using a single, external, public IP address. In this case, a special mechanism performs the substitution of IP addresses and TCP / UDP ports of traffic packets, replacing internal (non-routable on the Internet) addresses according to its dynamic translation table. In this configuration, it is necessary to remember that for the correct accounting of data on the internal hosts of the network, statistics should be taken in a way and in the place where the result of the broadcast does not yet "depersonalize" the internal addresses.

Methods for collecting traffic / statistics information


It is possible to remove and process information about the passing traffic directly on the access device itself (PC router, VPN server), from this device transferring it to a separate server (NetFlow, SNMP), or “from the wire” (tap, SPAN). We will analyze all the options in order.

PC router

Consider the simplest case - an access device (router) based on a PC running Linux. Much has been written

about how to configure such a server, address translation, and routing . We are interested in the next logical step - information on how to get information about traffic passing through such a server. There are three common ways:
  • interception (copying) of packets passing through the server’s network card using the libpcap library
  • interception of packets passing through the integrated firewall
  • using third-party tools for converting package statistics (obtained by one of the two previous methods) into the netflow aggregated information stream

Libpcap


In the first case, a copy of the package passing through the interface, after passing through the filter ( man pcap-filter ), can be requested by the client program on the server written using this library. The packet arrives with the 2nd level header (Ethernet). You can limit the length of the captured information (if we are only interested in the information from its header). Examples of such programs are tcpdump and Wireshark . There is a libpcap implementation for Windows. In the case of using address translation on a PC router, such interception can only be performed on its internal interface, connected to local users. On the external interface, after broadcast, IP packets do not contain information about the internal hosts of the network. However, with this method, it is impossible to take into account the traffic generated by the server itself on the Internet (which is important if a web or mail service is running on it).

The work of libpcap requires support from the operating system, which currently boils down to installing a single library. At the same time, the application (user) program that collects the packets should:
  • open the necessary interface
  • specify the filter through which to pass the received packets, the size of the captured part (snaplen), the size of the buffer,
  • set the promisc parameter, which puts the network interface in the capture mode for all packets passing by, not just addressed to the MAC address of this interface
  • install a function (callback) called on each received packet.


When a packet is transmitted through the selected interface, after passing through the filter, this function receives a buffer containing Ethernet, (VLAN), IP, etc. headers, total size up to snaplen. Since the libcap library copies packages, it is impossible to block their passage with its help. In this case, the traffic collection and processing program will have to use alternative methods, for example, calling a script to put the specified IP address in the traffic blocking rule.

Firewall


Capturing data passing through the firewall allows you to take into account both the traffic of the server itself and the traffic of network users, even when address translation is working. The main thing in this case is to correctly formulate the capture rule, and put it in the right place. This rule activates the transfer of the package towards the system library, from where the traffic accounting and control application can receive it. For Linux OS iptables are used as a firewall, and ipq, netfliter_queue or ulog are used as interception means . For OC FreeBSD - ipfw with rules like tee or divert . In any case, the firewall mechanism is supplemented by the ability to work with the user program in the following way:
  • A user program - a traffic handler registers itself in the system using a system call or library.
  • A user program or an external script sets a rule in the firewall, which “wraps” the selected traffic (according to the rule) inside the handler.
  • For each passing packet, the processor receives its contents in the form of a memory buffer (with IP headers, etc. After processing (accounting), the program also needs to tell the kernel of the operating system what to do next with such a package - drop it or pass it on. As an option, it’s possible pass the modified package to the kernel.


Since the IP packet is not copied, but sent to the analysis software, it becomes possible to “eject” it and, therefore, completely or partially restricts traffic of a certain type (for example, to the selected local network subscriber). However, if the application program stopped responding to the kernel about its decision (it hung, for example), traffic through the server is simply blocked.
It should be noted that the described mechanisms with significant amounts of transmitted traffic create an excessive load on the server, which is associated with the constant copying of data from the kernel to the user program. This drawback is deprived of the method of collecting statistics at the OS kernel level, with the issuance of aggregated statistics to the application program using the NetFlow protocol .

Netflow

This protocol was developed by Cisco Systems to export traffic information from routers for traffic accounting and analysis. The most popular version 5 now provides the recipient with a stream of structured data in the form of UDP packets containing information about past traffic in the form of so-called flow records:


The amount of traffic information is less than the traffic by several orders of magnitude, which is especially true in large and distributed networks. Of course, it is impossible to block the transfer of information when collecting statistics on netflow (unless additional mechanisms are used).
Currently, the further development of this protocol is becoming popular - version 9, based on the template structure of the flow record, implementation for devices from other manufacturers ( sFlow) Recently, the IPFIX standard has been adopted, which allows statistics to be transmitted over protocols of deeper levels (for example, by application type).
Implementation of netflow sources (agents, probe) is available for PC routers, both in the form of utilities running on the above mechanisms (flowprobe, fprobe , softflowd ) and directly embedded in the OS kernel (FreeBSD: ng_netgraph , Linux: ipt_neflow ). For software routers, the netflow statistic stream can be received and processed locally on the router itself, or sent over the network (transmission protocol over UDP) to the receiving device (collector).

A collector program can collect information from many sources at once, being able to distinguish their traffic even with intersecting address spaces. Using additional tools, such as nprobe, it is also possible to carry out additional data aggregation, stream splitting or protocol conversion, which is important when managing a large and distributed network with dozens of routers.

Netflow export features are supported by Cisco Systems, Mikrotik, and several others. Similar functionality (with other export protocols) is supported by all major manufacturers of network equipment.


Libpcap “outside”

Let's complicate the task a bit. What if your access device is a third-party hardware router? For example, D-Link, ASUS, Trendnet, etc. On it, most likely, it is impossible to put additional software for data retrieval. Alternatively, you have an intelligent access device, but it is not possible to configure it (there are no rights, or it is controlled by your provider). In this case, you can collect traffic information directly at the junction of the access device with the internal network, using "hardware" means of copying packets. In this case, you will certainly need a separate server with a dedicated network card to receive copies of Ethernet packets.
The server should use the packet collection mechanism according to the libpcap method described above, and our task is to send a data stream identical to the one leaving the access server to the input of the network card allocated for this. To do this, you can use:
  • Ethernet hub (hub): a device that simply forwards packets between all its ports indiscriminately. In modern realities, it can be found somewhere in a dusty warehouse, and it is not recommended to use this method: unreliable, low speed (there are no hubs at a speed of 1 Gbit / s)
  • Ethernet - switch with the ability to mirror (Mirroring, SPAN ports . Modern intelligent (and expensive) switches allow you to copy to the specified port all traffic (incoming, outgoing, both) of another physical interface, VLAN, including remote (RSPAN)
  • A hardware splitter , which may require installation to collect two network cards instead of one - and this is in addition to the main system one.



Naturally, you can configure the SPAN port on the access device (router), if it allows it - Cisco Catalyst 6500, Cisco ASA. Here is an example of this configuration for a Cisco switch:
monitor session 1 source vlan 100 ! откуда берем пакеты
monitor session 1 destination interface Gi6/3! куда выдаем пакеты


SNMP

What if there is no router under our control, there is no desire to communicate with netflow, we are not interested in the details of the traffic of our users. They are simply connected to the network through a managed switch, and we just need to roughly estimate the amount of traffic that falls on each of its ports. As you know, network devices with remote control support, and can display the counters of packets (bytes) passing through the network interfaces. To poll them, it will be correct to use the standardized SNMP remote management protocol . Using it, you can simply get not only the values ​​of the specified counters, but also other parameters, such as the name and description of the interface, the MAC addresses visible through it, and other useful information. This is done as command line utilities (snmpwalk ), graphical SNMP browsers, and more complex network monitoring programs ( rrdtools , cacti , zabbix , whats up gold , etc.). However, this method has two significant drawbacks:
  • traffic can be blocked only by completely disabling the interface using the same SNMP
  • SNMP traffic counters refer to the sum of the Ethernet packet lengths (moreover, unicast, broadcast, and multicast separately), while the rest of the previously described tools give values ​​relative to IP packets. This creates a noticeable discrepancy (especially on short packets) due to the overhead caused by the length of the Ethernet header (however, this can be approximately dealt with: L3_byte = L2_byte - L2_packets * 38).

VPN

Separately, it is worth considering the case of user access to the network by explicitly establishing a connection to the access server. A good example is the good old dial-up, the analogue of which in the modern world is remote access VPN services (PPTP, PPPoE, L2TP, OpenVPN, IPSEC)

The access device not only routes the IP traffic of users, but also is a specialized VPN server, and terminates logical tunnels (often encrypted), within which user traffic is transmitted.
To account for such traffic, you can use both all the tools described above (and they are well suited for deep analysis by ports / protocols), and additional mechanisms that provide VPN access control tools. First of all, we will talk about the RADIUS protocol. His work is a rather complicated topic. We briefly mention that the control (authorization) of access to the VPN server (RADIUS client) is controlled by a special application (RADIUS server), which has a database (text file, SQL, Active Directory) of valid users with their attributes (restrictions on connection speeds, assigned IP addresses). In addition to the authorization process, the client periodically transmits accounting messages to the server, information about the status of each current working VPN session, including counters of transmitted bytes and packets.


Conclusion


Let us summarize all the methods for collecting traffic information described above together:


To summarize a small result. In practice, there are a large number of methods for connecting the network you manage (with clients or office subscribers) to the external network infrastructure, using a number of access tools - software and hardware routers, switches, VPN servers. However, in almost any case, you can come up with a scheme where information about traffic transmitted over the network can be directed to software or hardware for its analysis and control. It is also possible that this tool will allow for feedback with the access device, using intelligent access restriction algorithms for individual clients, protocols, and more.
On this I will finish the analysis of the materiel. Of the unselected topics, there were:
  • how and where the traffic data gets to
  • traffic accounting software
  • how billing differs from a simple “reader”
  • how can you restrict traffic
  • accounting and restriction of visited websites

Also popular now: