Parse NetFlow v.9 packages in C #

NetFlow is a network protocol created by Cisco Systems to account for network traffic. The most common versions of this protocol are 5 and 9. The ninth version is more flexible since templates are used according to which data is sent. In the fifth version, data is sent according to the specification.


The NetFlow traffic information collection systemconsists of the following components:

• Sensor . A device (router, L3 switch) that collects statistics on traffic passing through it;
• Collector . It collects data from the sensor and puts it in the storage;
• Analyzer . It analyzes the data that the collector has collected and generates reports.

I’ll tell you about the development of part of the analyzer functions in C # , more precisely, the analysis of NetFlow packets . A MikroTik router

was used as a sensor . We enable NetFlow on it for the ether1 interface: And we add a collector (as a rule, the collector listens on port 2055, 9555 or 9995): Or the same, but through WinBox : Now on a computer with IP address 192.168.0.100 to port 9995 via UDP (or SCTP ) NetFlow 9 version packages will come . Packages come and have something to work with.


/ip traffic-flow set enabled=yes interfaces=ether1


/ip traffic-flow target add disabled=no version=9 address=192.168.0.100:9995

Parse incoming packets

Having studied the protocol specification, we learn that each NetFlow packet (N bytes) consists of:

1. Packet Header (20 bytes) - the packet header, in a single copy with fields:
   
   • Version Number ( UInt16 - 2 bytes) - NetFlow version number , we always have 9;
   • Count ( UInt16 - 2 bytes) - the total number of records. Further, articles with this field were an adventure ;
   • sysUpTime ( UInt32 - 4 bytes) - time in milliseconds from the start of the device - UpTime;
   • UNIX Secs ( UInt32 - 4 bytes) - time in seconds since 0000 UTC 1970 at which the packet was sent;
   • Sequence Number ( UInt32 - 4 bytes) - counter of transmitted packets, it is constantly increasing from packet to packet, thereby it is possible to check if packets between them were lost;
   • Source ID ( UInt32 - 4 bytes) - data stream number, the fact is that several data streams can go from the sensor side.
   
   
2. FlowSet (N-20 bytes) - templates, data ... FlowSet`s can be several or one. In each FlowSet there are two fields that are unchanged from the type of transmitted data (template, data):
   
   • FlowSet ID ( UInt16 2 bytes) - for a template it is always 0, for an optional template 1, for data it is equal to Template ID and therefore more than 255 (from 256 to 65535);
   • Length ( UInt16 2 bytes) - the size of the entire FlowSet along with the FlowSet ID and Length fields ;
   • Other fields depending on the type of data transferred.
   
   

We look at the FlowSet ID containing the template, begins with the FlowSet ID fields , then Length , then:

• Template ID ( UInt16 2 bytes) - a unique ID for each template through which data is transmitted. Number from 256 to 65535;
• Field Count ( UInt16 2 bytes) - the number of fields in the template. Next come the field type ( Field Type ) and size ( Field Length );
• Field Type ( UInt16 2 bytes) - a number defining the type of field. All types are in the protocol specification ;
• Field Length - the length of the field in bytes.

We look at the FlowSet ID containing the data, begins with the FlowSet ID fields , then Length , then:

• Data ... data that correspond to fields and their sizes;
• Padding - zeros filling to the border of 4 bytes.

There is also the so-called optional template and data on it. I will not consider them, they have not met me, for this reason there are no libraries in the implementation, but everything can be added.

Compiled a UML class diagram (using NClass ): or in pdf And wrote a library for parsing incoming packages. The main class that starts it all is Packet . Its only constructor accepts the incoming NetFlow packet in bytes and an object of the Templates class , which is a list of the current Template (s). Next , the Parse function is called in the constructor of the Packet class.





which accepts an object of the Templates class .
This function breaks the package into a header - 20 bytes and further works with it through the Header class ; on FlowSet`s and transfer of each FlowSet`s for processing to the corresponding FlowSet class .

Due to the fact that there may be several FlowSets , the second part of the packet (without 20 bytes of the header) has to be analyzed and split into different Flowsets . It is noteworthy that in MikroTik `s FlowSet`s in a single copy in the package, but using Netflow Simulator in C # it was possible to work with packages with severalFlowSet `s in the package. In addition, thanks to him, a funny bug was found in the implementation of NetFlow v9 on MikroTik `s, more about that here .

Netflow Simulator in C # :


Here is a piece of code that breaks down part of the package on FlowSet`s :

this._flowset = new List();
Int32 length = _bytes.Length - 20;
Byte[] flowset = new Byte[length];
Array.Copy(_bytes, 20, flowset, 0, length);
byte[] reverse = flowset.Reverse().ToArray();
int templengh = 0;
while ((templengh + 2) < flowset.Length)
{
	UInt16 lengths = BitConverter.ToUInt16(reverse, flowset.Length - sizeof(Int16) - (templengh+2));
	Byte[] bflowsets = new Byte[lengths];
	Array.Copy(flowset, templengh, bflowsets, 0, lengths);
	FlowSet flowsets = new FlowSet(bflowsets, templates);
	this._flowset.Add(flowsets);
	templengh += lengths;
}

In the Header class, the package header is parsed into its fields. Before doing this, the header is reversed:

this._bytes.Reverse().ToArray();

Next, convert the bits to the type of field that it is, for example, the version field:

this._version = BitConverter.ToUInt16(reverse, this._bytes.Length - sizeof(Int16) - 0);

Yes, in the Header field sysUpTime is of type TimeSpan , we can convert to this type:

get
{
	return new TimeSpan((long)this._uptime * 10000);
}

and the UNIX Secs field is of type DateTime :

get
{
	return new DateTime(1970, 1, 1).AddSeconds(this._secs);
}

Let's move on to processing FlowSet`s . After receiving the FlowSet ID and Length fields, the remaining fields are parsed depending on the FlowSet ID . If it is 0 or 1 then this is a pattern, and if it is a number from 256 to 65535 then this is data.

If this is a template, then we pass its processing to the Template class and then check our template store (object of the Templates class ) for the presence of a template with the same ID and replace it, otherwise just add the template.

If this data, then we check whether there is such a template ( FlowSet ID == Template ID in the storage (object of the Templates class )) and if there is, then copy this template with the DeepClone function and fill its fields - Field , otherwise we do nothing, because without a template it is just a set of bytes. DeepClone

function :

public static object DeepClone(object obj)
{
    object objResult = null;
    using (MemoryStream ms = new MemoryStream())
    {
        BinaryFormatter bf = new BinaryFormatter();
        bf.Serialize(ms, obj);
        ms.Position = 0;
        objResult = bf.Deserialize(ms);
    }
    return objResult;
}

Field is a field; it has the following parameters:

• Type - type;
• Length - size;
• Value - value.

And Field in the Template in the repository are without Value parameters i.e. Value is empty, but when processing Template packages in FlowSet , the Packet object already contains Value fields .

Besides all this, there is also the FieldType enumeration - an enumeration in which the type name corresponds to the number of this type. ( Type parameter in Field )

An example was written for this library to work:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.NetFlow;
using System.Net.Sockets;
using System.Text;
using System.Threading.Tasks;
namespace Consoles
{
    class Program
    {
        static void Main(string[] args)
        {
            Templates _templates = new Templates();
            Socket sock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
            IPEndPoint iep = new IPEndPoint(IPAddress.Any, 9995);
            sock.Bind(iep);
            EndPoint ep = (EndPoint)iep;
            byte[] data = new byte[2048];
            while (true)
            {
                int recv = sock.ReceiveFrom(data, ref ep);
                Console.ReadKey();
                Console.Clear();
                byte[] bytes = new byte[recv];
                for (int i = 0; i < recv; i++)
                    bytes[i] = data[i];
                Packet packet = new Packet(bytes, _templates);
                Console.ForegroundColor = ConsoleColor.Yellow;
                Console.WriteLine(packet.ToString());
            }
            sock.Close();
            Console.ReadKey();
        }
    }
}

We create a socket and listen on UDP 9995 port on our PC. _templates is our template repository. We feed each packet that arrives to the packet object of the Packet class, passing our template storage as well. And then we execute packet.ToString (). This overloaded function displays the contents of the packet and is needed only to verify that everything works out for us.

That's all with the library, now it can be used for further writing the Traffic Analyzer using the NetFlow protocol.

Example with MikroTik :

Received a package without a template in the repository:


Received a template from the sensor:


Received data for which there is a template in the repository:

Error implementing NetFlow v9 in MikroTik `s

In the process of analyzing this topic, an error was found in the implementation of NetFlow v9 in MikroTik . The essence of the error:
The Count field in the packet header ( Packet Header ) carries:

Count
The total number of records in the Export Packet, which is the
sum of Options FlowSet records, Template FlowSet records, and
Data FlowSet records.

those. contains all records, in all FlowSet `s, and in MikroTik` s this field is always equal to 1 (see screenshots above), even if several templates or data are transferred. those. according to MikroTik’s logic, the Count field = the number of FlowSets (which they wrote to me in the letter and can be seen from the screenshots), and it should be equal to the total number of all templates and data, as it sounds in the specification. For this reason, the use of the Count field in parsing packages is fraught.

Here is an example from Netflow Simulator in C # (I would like to receive data from Cisco as well , but I don’t have such an opportunity, can any of the readers check this):

Получили пакет без наличия шаблона в хранилище (обратите внимание на Count):


Получили шаблон от сенсора (тут одновременно два FlowSet`а пришло, что в MikroTik`е не бывает. Обратите внимание на Count он равен 7 = 1 шаблон и 6 записей с данными. По логике MikroTik`а Count должен был бы равен 2 = 2 FlowSet`а):


Получили данные, для которых есть шаблон в хранилище (обратите внимание на Count):


Ну и еще раз пакет в Wireshark Помечено поле Count:


Еще раз: буду очень благодарен всем кто пришлет скрин с Wireshark`ohm from Cisco . I'll put it in here.

Source code is available here .

The guide was guided by:
Wikipedia material: Netflow
Caligare: WHAT IS NETFLOW?
Protocol Specification Version 9

Today (19:05 07/30/2013)

MikroTik replied:

MikroTik support [Dzintars] support@mikrotik.com

Hello,

Thank you for reporting a problem (the author of the article correctly pointed out
that count value in netflow packet header not always is set correctly). The
problem will be fixed in the next version of RouterOS.

Regards,
Dzintars

UPD: In RouterOS 6.2 this bug has been fixed.

Today (13:22 09/13/2013)

MikroTik wrote:

MikroTik support [Dzintars] support@mikrotik.com

Hello,

The NetFlow V9 count field problem was fixed in version 6.2

Regards,
Dzintars