Structural Logging with Serilog and Seq as an Example

Structural logging takes one small step forward compared to regular logging.

Idea

Regular log entries consist of strings and we have to use regular expressions to search for entries among an array of text

_logger.Warning("Запись в лог файл. Пост на хабре ID"+postId); 

Structural logging stores records in the form of objects (structures). For example, using JSON

_logger.Warning("Запись в лог файл. Пост на хабре {ID}", postId); 

Two entries will be saved:

1. Template for the final output of the line “Write to the log file. Post on Habr {ID} »
2. The object that we will substitute in the template: {"ID": 1, type: Number}

The main advantage of this approach is that you can now store the integer ID object as a separate parameter, for example, in a NoSql database. And carry out a convenient and quick search using typed comparison operations instead of writing regular expressions.

Serilog

One of the convenient .NET libraries that support structural logging is Serilog .
The library supports all the basic logging functions that log4net, Nlog, and other well-known libraries have:

• A few common types of entries:
  Verbose - the lowest level and most detailed logging (for example, arguments to the method)
  Debug - data for debugging code one level higher than Verbose (for example, which method was launched and the execution result)
  Warning - warning for the business process must not contain Debug data (for example, launched the calculation of wages)
  the error - error in the application, which is not expected
  Fatal - exclusive error stops the business application processes (such as the user redirected to the PayPal payment and the buyer is not equal to idaemoy amount).
  
• Different types of storage called in Serilog stock: text file, relational databases, NoSql databases, Windows Events, http / s requests, etc.
• Convenient configuration through code and through .config files

So, after installing the Serilog Nuget package, we configure the configuration. We will do this in the code instead of the .config file.

 var logger = new LoggerConfiguration()
             .MinimumLevel.Verbose() // ставим минимальный уровень в Verbose для теста, по умолчанию стоит Information 
             .WriteTo.ColoredConsole()  // выводим данные на консоль
             .WriteTo.RollingFile(@"C:\Logs\Log-{Date}.txt") // а также пишем лог файл, разбивая его по дате
              // есть возможность писать Verbose уровень в текстовый файл, а например, Error в Windows Event Logs
             .CreateLogger();

You can now pass the logger object through a Dipendency Injection, or use it right away.
Suppose we create a program for diagnosing a car and we need information about the manufacturer, series and date of intake. And of course, so that everything is stored in a structural form.

            logger.Debug("Request for vehicles {Manufature}, {Series}, {ProductionYear}", "BMW", "F02", new DateTime(2015, 1,1));
            // search returns 10 vehicles
            logger.Verbose("Found {FoundVehiclesCount}", 10);

Since we used the output to the console and the file, the result will be in the form of a familiar line.


One convenient feature is the addition of commonly used properties. For a web application, this can be a user session, a visited URL, browser data. In a more general case, you can add an application version. You can do this using the Enrich property and already built-in classes, or write your own.

 var logger = new LoggerConfiguration()
                .Enrich.With()
                .Enrich.With()
                .Enrich.With()
                .Enrich.With();
                .Enrich.WithProperty("ApplicationVersion", config.ApplicationVersion)
                .CreateLogger();

For convenient output to the console or a text file of complex objects, it is better to tell the Serilog template that we are using a complex object, rather than a primitive type, by adding the @ symbol. Otherwise, typeof (MyClass) .ToSting () will be displayed.

            logger.Debug("We are using anonymous object instead of simple string/integer {@Car}", new {Manufacture="Toyota", Production = new DateTime(2015, 1,1)});

Seq

The Seq application comes to the rescue for convenient storage and retrieval of structural logs.
Seq works as a windows service that accepts REST requests, and internally stores data in a NoSql database.

After installing Seq, you must also add the Serilog.Sinks.Seq Nuget package to the application . And then make friends our Serilog with Seq.

 
            var logger = new LoggerConfiguration()
             .MinimumLevel.Verbose()
             .WriteTo.Seq("http://localhost:5341")
             .CreateLogger();

Now you can conveniently search our fields, including comparing numbers and dates.


Search parameters can be saved and used in other parts of the application. Adding Environment to the Dashboard

And you can add a real-time Dashboard displaying errors that came precisely from the MyApp application, version 1.2.1 and occurred in the Repository.GetUserByIdAndPassword () method.

Business requirements

Like any new, fashionable, luminous program, you need to check it by asking the questions:
“What benefit does this program bring for business?”, “What problem does it solve and for what price?”.
Often in relation to logging, the goals come down to

• Reduce the effort to find errors in the program.
• Have the ability to restore the history of user actions. For example, to find out if the goods are really paid.
• Be aware of errors, as they lead to user losses and profits.

If the first two points are solved by detailed records and a convenient search, then reporting errors is often skipped. I call this "calm in ignorance." We open the Windows event logs and it turns out that almost every server has programs that pour in with critical errors.

With the help of Seq extensions, you can inform about ongoing events. For example, immediately send a message to the programmer / administrator if a critical error occurs. And send a list of all errors that occurred in a day or a week. You can configure extensions in the Seq -> Settings -> Apps section

Cons of structural logging

The disadvantage of structural logging is the need to determine the parameters by which the search will be performed. They will then be used to create NoSql indices. If you forgot to define a parameter, then the search comes down to the old scan of strings through regular expressions.
A giant step in this direction makes the Splunk application . The idea is to collect string data, absolutely any and in any format (application logs, OS operation events, etc.). And then the amazing line break depending on the queries and the dynamic construction of the result through Map / Reduce. Splunk is a great infrastructure for collecting and analyzing data, and goes beyond the topic of logging and this article.