Counting network traffic with netflow and lightsquid

Once in our small office I needed to configure a traffic accounting system for linux. Quickly looking at ready-made solutions, I decided to use something simpler. Googling, I liked the approach of using a self-written script that would convert netflow statistics to a squid-compatible log, which can then be analyzed with anything. I did not use the finished script, walking on Google, because it uses ipcad, which is absent in debian and ubuntu. I wrote my own version using any netflow sensor and flow-tools as a collector. The script gradually became more complicated and, as a result, grew into a small traffic accounting system. The archive of this pleasure weighs about 50kb, while:

• installs using deb for debian-based distributions or using its own installer for the rest
• able to read Internet traffic on the router for all ip v4 protocols and ports
• can restrict access to non-registered users
• instead of bare ip statistics, netflow can drill down to reports using dig and squid
• can selectively block traffic
• has a web interface for managing users and user groups
• configures a little more complicated lightsquid, which uses in work
• does not use the intermediate log access.log, its own parser converts netflow statistics directly to the lightsquid report

It looks like this: user management and access to statistics is carried out through a web interface ( demo ). Each user can be authorized through ip, mac or ip + mac. Unauthorized users see the "access denied" error. Each user can be assigned a squid flag to redirect web traffic to transparent squid. Each user can be assigned the ipblock flag to block any traffic through ipset. Block list management and other fine-tuning are done through the console. It is not necessary to install and configure everything, you can configure only the necessary components.
The name lightwrapper (derived from lightsquid wrapper) was chosen as the name and a project on sourceforge was created . How to try:

1. install recommended dependencies: apache2 softflowd ipset lightsquid perl iptables flow-tools dnsutils conntrack iptables-persistent
2. Download the archive , install deb or run the installer.

Actually, there are a lot of dependencies, and many of them require tuning - this is a minus. But it is unlikely that with this approach, something else is possible. Briefly, the setup boils down to the following:

1. configure netflow sensor and collector (one line in the config of both)

   Configure flow-tools

   Enter in the configuration file (in debian - /etc/flow-tools/flow-capture.conf)
   -N 0 -w / tmp -R / usr / sbin / lw-export -n 95 0 / 127.0.0.1 / 2055

   
   

   Configure softflowd

   Enter in the configuration file (in debian - / etc / default / softflowd):
   INTERFACE = "the name of the local network interface, for example, eth0"
   OPTIONS = "- n 127.0.0.1:2055"

   Softflowd is easy to configure, but may lose packets. If you are lacking in performance, try ipt-netflow .
2. customize lightsquid

   Configure apache2

   The archive contains examples: lightwrapper.conf - virtualhost configuration for apache2 and access_denied.html - a stub for showing to non-users. Examples are in / usr / share / doc / lightwrapper / examples. You must turn off the virtualhost settings for lightsquid (because they give access without a password) and enable the settings for lightwrapper. In debian you can replace the /etc/apache2/sites-enabled/000-default.conf file with the given one. You must also enable the cgi module. In debian, you need to make a link with /etc/apache2/mods-available/cgi.load in / etc / apache2 / mods-enabled. After that, you must enable access with password protection. Install the apache2-utils package, if you do not have the htpasswd utility, create the etc directory in the root of the site and set the administrator password: htpasswd -c / www / etc / passwd administrator
   

   
   

   Customize lightsquid

   You must delete the cron jobs for lightparser.pl, if any are present on the system.
   The configuration file is configured automatically by installation scripts. You can customize lightsquid to your liking, except for the following parameters that cannot be changed:
   $ graphmaxuser = 1
   $ graphmaxall = 1
   $ showgroupid = 0
   $ showputpost = 1
   $ templatename = 'lightwrapper'
   $ logpath = '/ var / log / lightwrapper'
   $ lang = 'lw-eng'
   or another language (lw-ru)
   
3. edit the config file. In the simplest case, you only need to enter the names of 2 network interfaces

   / etc / lightwrapper / cfg:
   in_if and out_if
   
4. run the iptables configuration script (does everything automatically, this is necessary once)

   Launch lw-geniprules. Make sure ip v4 forwarding is enabled (add net.ipv4.ip_forward = 1 to /etc/sysctl.conf and apply the sysctl --system changes).
5. install and configure additional dependencies, if any

   Installation of transparent squid is not described; the ipset setting for blocking traffic is not described. Detailed instructions can be found in man or on the wiki.
6. Restart all services to apply the settings or reboot.

Statistics is replenished by default once every 15 minutes. The sensor may give streams with some delay.
All this is described in detail step by step in broken English in the wiki and in man lightwrapper. There are examples of configuration files of everything that is possible; in most cases, you can simply copy them to your system.

A bit about the detail of the reports. Netflow provides the following information: ip addresses, source and destination ports, protocol.



If for a pair (protocol: destination port) there is a service name in / etc / services, then lightwrapper displays it in statistics. If the dig utility is installed, then the IP addresses are automatically resolved into names (if there is such a match in dns).



If transparent squid is installed and some users are redirected to it, then lightwrapper scans the squid log and matches the found ip addresses with the names. You can configure lightwrapper to resolve IP addresses to names for users who are not redirected to squid. At the same time, for obvious reasons, you cannot vouch for the accuracy of such recognition.



I will be glad to any feedback and bug reports.