Splunk Scripted Input. Or how to use scripts to obtain data on the operation of systems and analyze them in Splunk

Earlier we wrote how you can upload logs to Splunk from a catalog or using syslog, told how to pick up standard Windows and Linux events, but what if we need to get more granular information about the operation of our systems?
In this case, scripts come to the rescue!



When, what and how you can use scripts in Splunk to receive data - you can find out under the cut.

Typical Use Cases

Scripts are often used in cases where:

• You need access to information that is not written to the log ;
• We need data generated by the command line , for example, using vmstat or iostat;
• We need specific data or the results of databases, web services or APIs;
• Data needs pre-processing to more easily parse events and fields;
• Data sources with slow and resource-intensive startup procedures are used ;
• And etc.

For the script, you can set the interval at which it will be played and transfer data to Splunk.

As scripts, you can use shell scripts, python scripts, Windows batch files, PowerShell, or any other utility that can generate and transmit data.

Example

In the article, we will consider an example of loading data using a script.

Suppose we have a file server and a directory whose size we need to monitor for some reason, and we also want it to not exceed the threshold value (for a test of 45 mb). Let's write a script that will calculate the size of this directory with an interval of 30 seconds, and also make an alert that will notify us when the threshold value is exceeded.
We will read the size of the folder using the script below, which will give the timestamp, the path to the folder and its size in bytes at the output.

import os
import time
from datetime import datetime
dir_path="///for_script"
def get_size(start_path = '.'):
    total_size = 0
    for dirpath, dirnames, filenames in os.walk(start_path):
        for f in filenames:
            fp = os.path.join(dirpath, f)
            total_size += os.path.getsize(fp)
    return total_size
time_of_event=datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S")
print time_of_event, dir_path, get_size(dir_path)

In more detail, we analyzed data loading from remote sources in previous articles ( here and here ). Therefore, we will now discuss this briefly.

We need:

• A remote machine on which Splunk Universal Forwarder is installed
• Splunk-indexer, on which we create the send-to-indexer application , transfer it to deployment-apps and configure Forwarder managemen t.

We also create the monitor_scripts application on the Splunk indexer , transfer it to the deployment-apps folder . In the application, create the local folder and in it the inputs.conf file with the following contents:

[script://./bin/scripts/foldersize.py]
disabled = false
index = test_script
interval = 30.0
sourcetype = test_script

We also add our script to the / bin directory . We

reboot the Deployment server.
.../splunk/bin/splunk reload deploy-server

And ... we get the data!

Data processing and alert creation

Splunk automatically selected a timestamp, but the rest of the information remained in the form of raw data, so you need to select the fields (We wrote about this in the previous article ) In this case, we selected 2 fields: folder path (folder_path) and size (size)

The size of the folder is in bytes, let's translate this number into MB. (This can be done in the script, but we will show how to do it in Splunk)

Create a new calculated field (Settings - Fields - Calculated fields - New)
Specify the source type of our data, the name of the new field and the expression for the calculation. Now this calculated field will be added to the data with the specified source type.



We got all the fields that interest us, let's create a graph, which will show the dynamics of changing the size of the folder and whether it reaches a threshold value.



Create an alert. Let, when the folder size exceeds 45 mb, Splunk will send us an email. For more information about how to send an alert to an email we wrote here , and Slack - here .

The alert will be based on a new request so that fields from the request can be inserted into messages.



We save the request as an alert and prescribe its conditions:




And we get a letter:



In the alert settings, we have established that if the folder size does not decrease within 15 minutes, the notification will come again.

Conclusion

In this simple example, we showed the principle of loading data into Splunk through scripts. You can create a script that will solve your problem: upload the necessary information to Splunk and quickly get the result.

We hope you find this information useful.

We are happy to answer all your questions and comments on this topic. Also, if you are interested in something specifically in this area, or in the field of machine data analysis in general, we are ready to finalize the existing solutions for you, for your specific task. To do this, you can write about it in the comments or simply send us a request through the form on our website .