Streaming and analysis of Java application logs in MS Azure using Log4j and Stream Analytics

In this article I will show some working solutions to the problem of transferring and analyzing logs from Java applications to MS Azure. We will consider solutions for both windows and linux virtual machines located both in the cloud and on-premise. As a logging subsystem for Java we will use log4j2 .

We will use Azure Stream Analytics to analyze the logs .

To understand what this article is all about, it is advisable to have basic knowledge of log4j2 and some Azure resources, namely stream analytics, event hub, blob storage.
If you have a desire to refresh them (knowledge), here are the
Apache Log4j 2
Azure Stream Analytics Documentation
Azure Event Hubs Azure Storage links

Why java?

Perhaps the idea of ​​hosting java applications in Azure VM will seem strange to someone, but according to the data from the IaaS 20011-2026 market analysis report in Germany from Colorbridge Gmbh , Azure IaaS use only a little less than AWS. Therefore, if prejudice is left, such a statement of the question will turn out to be quite reasonable, and to someone even topical.

Although Azure has supported Java for a long time, especially at the PaaS level, providing an SDK for Java, hosting Java applications in the Web App and the popular SaaS software in the java and opensource world. But at the IaaS level (generally abstracted from the software itself), the specifics of working with Java are not much covered. And it is, at least in the field of logging. Try to fix it.

Why log4j?

For java, there are several subsystems for logging. We will use log4j because

• He is very popular.
• Its configuration capabilities are sufficient to integrate with the Azure resources we need
• All configuration can be done through an external configuration file.
• The configuration file with the new settings can also be specified for an already built application ( -Dlog4j.configurationFile = log4j2.xml ), so all scripts in the article can be implemented without changing the application code at all.

What I tested on

As a test application, I took the most common springboot starter app with the spring-boot-starter-log4j2 module but the latest version 2.0.0.M5, because one of the scenarios will need the latest version of log4j.

4.0.0com.examplelog4j2-demo0.0.1-SNAPSHOTjarlog4j2-demoDemo project for Spring Bootorg.springframework.bootspring-boot-starter-parent2.0.0.M5UTF-8UTF-81.8sbootyour custom repohttps://repo.spring.io/libs-milestonefalseorg.springframework.bootspring-boot-starter-web2.0.0.M5org.springframework.bootspring-boot-starter-testtest2.0.0.M5org.springframework.bootspring-boot-starterorg.springframework.bootspring-boot-starter-loggingorg.springframework.bootspring-boot-starter-log4j22.0.0.M5sbootplughttps://repo.spring.io/libs-milestonefalseorg.springframework.bootspring-boot-maven-plugin2.0.0.M5

To test the ability to add functionality for log streaming for a ready-made application, I experimented with a java minecraft server :)

What's next?

And then there will be several scenarios indicating the method of their implementation and their inherent limitations.

The meaning of the scenarios is the organization of a pipeline on

• collecting logs from java applications
• transferring them to some storage in Azure (we will use either Blob Storage or EventHub)
• transferring them to Stream Analytics and initial analysis (basically I will show how to get from Stream Analytics to the data transferred to log4j),
  but the step of consuming data (creating all kinds of dashboards, etc.) will not be covered in this article.

Scenario 1, universal

Features: Windows or Linux OS, VM in Azure or On-premise
Limitations: you need a <defined> version of log4j2

Work algorithm:

• Logs using the HTTP appender are pushed directly into Azure in EventHub
• EventHub monitors the Stream Analytics job and when messages appear it begins to parse and analyze them in accordance with a given request

Implementation Details:

Configuring Log4j
In the log4j2 config, an HTTP appender must be defined

A little more about Authorization header.
In order to work with the EventHub REST API, authorization on the so-called SaS token . This is essentially a hash of the resource url and token lifetime.

To generate a sas token, in addition to the event hub namespace and event hub, you also need to know the name and key of the policy event hub with the rights to send messages. All information is on portal.azure.com.

Microsoft provides code examples for generating Authorization header in various languages, including in Java.

I use this html snippet , which I found on the Internet and slightly improved - it generates an Authorization header for EventHub with a lifetime of one year.

Setting up Stream Analytics
In Stream Analytics, the job configures input with EventHub with the parameters
Event serialization format = JSON, Encoding = UTF-8, Event compression type = None
At the same time, access to the data that the http appender log4j pushes is simple, directly via select * from ( and so it will not always be)

WITH errors as (
    SELECT
        *
    FROM
        javahub
    WHERE level='FATAL' OR level='ERROR'
),
activity as (
    SELECT
       System.TimeStamp AS WindowEnd, level, COUNT(*)
    FROM
        javahub
    GROUP BY TumblingWindow( second , 10 ), level
)
select * into pbierrors from errors;
select * into pbiactivity from activity;

With this request we

• Forward to pbierrors output all log messages with FATAL or ERROR level
• Every 10 seconds we send to pbiactivity the number of messages received during these 10 seconds, grouped by log level

A couple of mouse clicks in power bi and we can monitor not only application errors, but also monitor the overall activity.

Scenario 2, Windows only, budget

Features: no EventHub needed. Less traffic (logs are archived before transmission), no need to bother with SaS tokens.
Limitations: VM only in Azure. Logs arrive with a slight delay.

Work algorithm

• Logs are written to a file using the RollingRandomAccessFile appender
• Upon reaching certain conditions (log4j triggers) the logs are archived in a separate folder
• The folder is monitored by Azure Monitoring & Diagnostics Extension for VM, and when a new archive file appears, transfers it to Blob storage in Azure
• Blob storage monitors the Stream Analytics job, and when a new archive file appears, it begins to parse and analyze it in accordance with the given request

Implementation details

Configuring Log4j
Be sure to note that

• Archiving must be done NOT in the folder where the current logs are written
• Each file must have a header - a list of field names
• If there is a desire to organize the hierarchy of directories in the folder where the logs are archived, only the date (yyyy-MM-dd) (delimiters can be arbitrary) and / or hour (HH) can be used as directory names (from the dynamics)
  Example of determining "correct "appender'a log4j
  

  TS;LEVEL;MESSAGE%n

Configure Azure Monitoring & Diagnostics Extension for VM

• create two config files (examples below)
• using Azure CLI 2 execute

az vm extension set --name IaaSDiagnostics \
                    --publisher "Microsoft.Azure.Diagnostics" \
                    --resource-group  \
                    --vm-name  \
                    --protected-settings "privateSettings.json" \
                    --settings "publicSettings.json" \
                    --version "1.11.1.0"

{
    "WadCfg": {
        "DiagnosticMonitorConfiguration": {
            "overallQuotaInMB": 10000,
            "DiagnosticInfrastructureLogs": {
                "scheduledTransferLogLevelFilter": "Error"
            },
            "Directories": {
                "scheduledTransferPeriod": "PT1M",
                "DataSources": [
                    {
                        "containerName": "",
                        "Absolute": {
                            "path": "C:\\\\",
                            "expandEnvironment": false
                        }
                    }
                ]
            }
        }
    },
    "StorageAccount": "",
    "StorageType": "Table"
}

{
    "storageAccountName": "",
    "storageAccountKey": ""
}

Stream Analytics setup
Blob storage with parameters is used as input:
PathPattern - path to files with archived logs in Blob storage. If you made a hierarchy of directories (as in the example above) - then here it should also be taken into account.

Example: WAD / be7f1c92-2841-4ea1-b9d8-ec83c211b8ea / IaaS / _minesrv / {date} / {time} /
DateFormat must be set in accordance with the% d pattern format in log4j
Event serialization format = CSV, Delimeter = semicolon, Encoding = UTF-8, Event compression type = GZIP

Access to data in Steam Analytics requests is also direct.
select * from will return a table with fields TS, LEVEL, MESSAGE (according to the header defined in log4j)

WITH 
SessionInfo AS (  
    SELECT
        TS, 'START' as EVENT, SUBSTRING(MESSAGE, 0, REGEXMATCH(MESSAGE, '[ ]*joined the game')) as PLAYER
    FROM
        logslob
    TIMESTAMP BY TS
    WHERE REGEXMATCH(MESSAGE, 'joined the game') > 0
UNION
    SELECT
        TS, 'END' as EVENT, SUBSTRING(MESSAGE, 0, REGEXMATCH(MESSAGE, '[ ]*left the game')) as PLAYER
    FROM
        logslob
    TIMESTAMP BY TS
    WHERE REGEXMATCH(MESSAGE, 'left the game') > 0
),
RawLogs AS (
    SELECT
        TS, LEVEL, MESSAGE
    FROM
        logslob
    TIMESTAMP BY TS
)
SELECT * INTO sbq from SessionInfo;
SELECT * INTO pbi from RawLogs;

Here we send all the logs to the RawLogs output, but in SessionInfo there are separate entries about the start and stop of the session with the name of the player - for subsequent notifications

Scenario 3, Linux only

Features: the minimum configuration of Log4j (and the minimum requirements for the version of log4j) - just write to a file. All new entries in the file are processed (without delay)
Limitations: VM only in Azure, write to file only in json, more difficult access to data from stream analytics job

Work algorithm:

• Logs are written to file using File appender
• The file monitors the Azure Linux Diagnostics Extension for VM and pushes them to EventHub when new entries appear in the file
• EventHub monitors the Stream Analytics job and when messages appear it begins to parse and analyze them in accordance with a given request

Implementation details

Configuring Log4j
Logs should be written in json format and be sure to - each object on one line of the
log file Fortunately, with log4j this can be configured simply

Configure Azure Linux Diagnostics Extension for VM

• create two config files (examples below)
• using Azure CLI 2 execute

az vm extension set --name LinuxDiagnostic \
                    --publisher "Microsoft.Azure.Diagnostics" \
                    --resource-group \
                    --vm-name \
                    --protected-settings "linux_privateSettings.json" \
                    --settings "linux_publicSettings.json" \
                    --version "3.0.109"

• in the configs we prescribe a storage account, although it is not used to transfer logs. It is only needed for diagnostic records of the Linux Diagnostics Extension itself.
• in the configs we prescribe the sas token for the sotage account (and not just the key, as in the case of the Diagnostics Extension for Windows), fortunately it can be generated through the portal
• in the configs we prescribe the sas token for the event hub - it almost does not differ in format from what we generated for the first script (and is generated by the same code or snippet )

linux_publicSettings.json:
  {
    "StorageAccount": "",
    "sampleRateInSeconds": 15,
    "ladCfg": {
        "diagnosticMonitorConfiguration": {
          "metrics": {
            "metricAggregation": [
              {
                "scheduledTransferPeriod": "PT1H"
              },
              {
                "scheduledTransferPeriod": "PT1M"
              }
            ],
            "resourceId": "/subscriptions//resourceGroups//providers/Microsoft.Compute/virtualMachines/"
          }
        }
      },
    "fileLogs": [
      {
        "file": "///",
        "sinks": "LinuxEH"
      }
    ]
  }

{
    "storageAccountName" : "",
    "storageAccountSasToken": "",
    "sinksConfig": {
      "sink": [
        {
          "name": "LinuxEH",
          "type": "EventHub",
          "sasURL": "https://.servicebus.windows.net/?sr=xxxxxx&sig=yyyy&se=zzzz&skn="
        }
      ]
    }
}

Setting up Stream Analytics
In Stream Analytics, the input is configured with an EventHub with the parameters
Event serialization format = JSON, Encoding = UTF-8, Event compression type = None.
However, access to the data that is logged is not so simple. If we just do select * from we will see something like this

those. we need the data hidden somewhere in the json property of the object stored in the PROPERTIES field
But this problem can also be solved beautifully in stream analytics (yes, I really like this thing :)), for example with such a request

with events as (
 select UDF.to_json(properties.MSG) as obj
 from ehtest
)
select obj.* from events

where UDF.to_json is the function we wrote for converting a string into a JSON object (yes, there you can also write functions in javascript ...)

As a result, we get easy access to the log data

Finally

I hope this article is useful to someone.
She already brought me great benefit, because only through the implementation of practical cases can you really understand the capabilities and maturity of a particular technology.
If suddenly I missed any scripts - please write about this in the comments.