Collect metrics from .NET applications using Telegraf

One of the most important tasks in the design of systems is the organization of monitoring the status of all nodes, including a large number of services. In conditions when additional forces and funds are not allocated for this, it is necessary to use ready-made solutions to the maximum.

I think that for many people the picture in the project is about this:

Something is being sent somewhere, somehow processed and held on one nail. The task was to collect data processing statistics from all points and put them in one place, then to build graphs and write reports.

Thanks to the article, the choice fell on a bunch of Telegraf-Elasticsearch-Grafana. Telegraf fit perfectly into the project for organizing monitoring of the state of both iron and publicly available third-party software, but I will separately discuss the issue of measuring the load on my own services. In this case, we are talking about .NET services running in docker containers under Linux. All services form several stages of processing incoming information, and I needed to measure the number of successfully processed and rejected packets with additional labels for the processing stage, source, etc. for the possibility of subsequent statistical processing.

I will omit the installation process and proceed immediately to the configuration. So, Telegraf can receive messages with metrics on tcp, udp channels, as well as through unixsocket:

[[inputs.socket_listener]]
        #service_address = "unixgram:///tmp/telegraf.sock"
        service_address = "udp4://:14230"
        data_format = "json"
        json_name_key = "name"
        namepass = ["query_pass"]
        tag_keys = ["appname","fromip"]

Services will send messages every time the next packet is processed, so we’ll additionally configure aggregation. An interval of 10 seconds is quite enough, depending on the load of a particular system.

[[aggregators.basicstats]]
        period = "10s"
        drop_original = true
        stats = ["sum"]
        namepass = ["query_pass"]
        fieldpass = ["pass","fail"]

Let's go through the parameters: query_pass - the name of the metric that combines future measurements, pass - successful processing, fail - no. Additionally, metrics will be tagged with appname and fromip tags .

Now a little code. I like to send metrics via udp and unixsocket, although other options might work for you.

UdpClient udpClient = new UdpClient("127.0.0.1", 14230);
byte[] datagramBytes= Encoding.UTF8.GetBytes("{\"name\":\"query_pass\",\"pass\":1,\"fromip\":\"127.0.0.1\",\"appname\":\"application\"}");
udpClient.Send(datagramBytes, datagramBytes.Length);
datagramBytes= Encoding.UTF8.GetBytes("{\"name\":\"query_pass\",\"fail\":1,\"fromip\":\"127.0.0.1\",\"appname\":\"application\"}");
udpClient.Send(datagramBytes, datagramBytes.Length);

All this is perfectly summarized and added to the database, in my case it is elasticsearch ( screenshot ).

[[outputs.elasticsearch]]
        urls = [ "http://localhost:9200" ] # required.
        timeout = "5s"
        health_check_interval = "5s"
        index_name = "telegraf-%Y.%m.%d" # required.
        manage_template = true
        template_name = "tp_telegraf"
        overwrite_template = true

All cats.

PS: here is the final project for sending metrics under net.core