Visualization of NFS traffic with elasticsearch + kibana

  • Tutorial
image

On duty, I often have to analyze NFS traffic. Wireshark is my main tool and for him I even created an extension on lua. But something was missing. And two weeks ago I came across a new Packetbeat tool for me . Unfortunately, paketbeat does not support, did not support NFS, but I managed to fix this shortcoming.

Packetbeat



Paketbeat is one of the beats tools from the creators of elasticsearch, logstash and kibana. This is a data shipper in elasticsearch that listens for network traffic, converts it to json records, and sends it to elasticsearch. If you use Kibana4, then there are standard panels for visualizing the collected traffic. At the moment, packetbeat recognizes TCP, UDP, DNS, ICMP, HTTP, memcache, MongoDB, redis, PostgreSQL, MySQL, thrift and, now, NFS. Somewhere inside, packetbeat is using libpcap.



How to add your protocol



Packetbeat is written in go. The code is on the github and contains a file with instructions on how to add a new protocol. What is missing is the 'desired' format of the json object being created.

NFS traffic processing



Processing NFS (as well as probably everything else) traffic occurs according to the following scheme:
  • we collect tcp packets until the message is completely received
  • parsing the rpc header
  • if this is a request, create a new record and put it in a special cache.
    key in cache is xid (rpc transaction id)
  • if this is the answer, we take the corresponding record from the cache, add
    information from the response, add the time it took the server
    to process the request and send the record to elasticsearch.


The created record looks like this:
{
  "@timestamp": "2016-03-28T06:18:18.431Z",
  "beat": {
    "hostname": "localhost",
    "name": "localhost"
  },
  "count": 1,
  "dst": "127.0.0.1",
  "dst_port": 2049,
  "nfs": {
    "minor_version": 1,
    "opcode": "GETATTR",
    "status": "NFSERR_NOENT",
    "tag": "",
    "version": 4
  },
  "rpc": {
    "auth_flavor": "unix",
    "call_size": 200,
    "cred": {
      "gid": 500,
      "gids": [
        491,
        499,
        500
      ],
      "machinename": "localhost",
      "stamp": 4597002,
      "uid": 500
    },
    "reply_size": 96,
    "status": "success",
    "time": 25631000,
    "time_str": "25.631ms",
    "xid": "2cf0c876"
  },
  "src": "127.0.0.1",
  "src_port": 975,
  "type": "nfs"
}


Having this data, you can get the following information:
  • the number of different requests
  • number and type of errors
  • top N customers
  • top N users and groups
  • server response time

(your options)

Listening to traffic



The easiest option would be to run packetbeat on an NFS server. If this is not possible, then port mirroring on the switch can be used . You can read more about this here .

Packetbeat has a configuration file, where we need to say what it should do:

interfaces:
  device: any
protocols:
  nfs:
    ports: [2049]
logging:
  level: info
output:
  elasticsearch:
    hosts: ["elasticsearch.node.name:9200"]


The configuration file is specified with the '-c' option.

Instead of a conclusion



Hopefully after reading up to this place you have learned something new.

Also popular now: