Splunk - Installing agents for collecting Windows and Linux logs
- Tutorial
We are often asked questions about how to load various data into Splunk . One of the most common sources of interest was the Windows and Linux logs , which allow you to track and manage operating system problems. By uploading data to Splunk, you can analyze the operation of all systems in one place, even when you have dozens or hundreds of different sources.
In this article, we will step by step explain to you how to download data from Windows and Linux to Splunk, for further processing and analysis.
In order to start collecting data, we need the following system elements:
In order to upload logs to Splunk, you must first configure the indexer, for this you will need:
At this stage, we finish the preliminary settings of the indexer and proceed to install the agents on Windows and Linux machines.
A universal tool for downloading logs is a special agent - Splunk Universal Forwarder . Universal Forwarder is a version of Splunk Enterprise with significantly limited functionality, the only task of which is to collect data from the host and send them.
You can download it here .
The picture above shows that Universal Forwarder can be installed on both Windows and Linux, Solaris and other operating systems.
1. Install Universal Forwarder
As the Deployment Server, specify the IP address or name of the Splunk indexer, where we created the “Send to indexer” application. Default Port: 8089. The Receiving Indexer section will be left blank, since the Send to indexer function will perform these functions.
2. The next step is to return to Splunk and define the server class for the “Send to indexer” application.
The server class is something similar to a rule in which we indicate which applications we will distribute between which target client machines. The criteria for the formation of different classes of the server can be the type of machine, OS, geographic area or type of application, and the classes can intersect with each other. (You can read more on the official website )
Settings - Forwarder Management - edit action - add new classes.
3. After saving, you will be asked to add applications that we will distribute and target systems, the so-called clients, to which we will distribute them.
Add “ Send to indexer ” to the application section.
4. Then we add the client. The client will be our Windows machine on which we installed Universal Forwarder. If Universal Forwarder was installed correctly, then the machine should appear in the list of clients connected to the Deployment Server . Put it in Include (whitelist) .
5. You can check whether everything works correctly by looking at the contents of the _internal index. After adding "Send to indexer" to the server class, Universal Forwarder starts sending its internal logs there. Also in this index, we can further monitor whether our agents are working properly.
6. Next, download the special Add-on from the SplunkBase website , which allows you to collect data on the operation of Windows.
7. Install the application on the Splunk-Indexer ( Apps - Manage Apps - Install app from file )
By default, it is installed in the directory ... \ Splunk \ etc \ apps \ Splunk_TA_windows , but we need to copy it to the deployment-apps folder so that it the application was available for deployment server, so that later we could send it to other machines as well as "Send to indexer". ( Important: it should also remain in the apps folder so that the indexes we need for the data are formed on the indexer).
8. Then you need to pre-configure the application.
Go to the directory ... \ Splunk \ etc \ deployment-apps \ Splunk_TA_windows
Create a subdirectory “local” in it ( Important : You must always make changes to the configuration files in the local directory).
Copy the inputs.conf file from. .. \ Splunk \ etc \ deployment-apps \ Splunk_TA_windows \ default \ inputs.conf to the local directory .
Turn on the indexing of the required data. To do this, in the inputs.conf file from the local directorythrough a text editor we make some changes. Replace the disabled = 1 values with disabled = 0 in the necessary blocks of the file. Let's add the system logs for Application, Security, System.
9. Next, on Splunk-indexer, we add to the application the server class created earlier. ( Settings - Forwarder Management - Apps - Splunk_TA_Windows - “+” - Windows Forwarder )
10. Reboot the deployment server , this can be done through the command line from the directory ... / splunk / bin :
Checks if data is being uploaded. ( Settings - Indexes ) They should be in the wineventlog index. As you can see in our figure, the last data that was downloaded at the moment has a timestamp 3 minutes ago.
One of the tools to improve Linux security is the auditd audit subsystem . With its help, you can get detailed information about all system events. It is the data generated by this system that we will index in Splunk.
(The code will be presented for Linux CentOS)
1. Check if the machine has a pre-installed audit system, if not, install it.
Add a new rule that we will track.
You can check its availability using the function.
Logs generated by auditd fall into the file:
2. Next, install Universal Forwarder . You can find the distribution by the link.
You should download the .rpm file, after downloading which you will be able to get a wget link.
3. Next, we will create a new user who will be responsible for working with splunk.
4. We will give permissions to the user we just created and run UniversalForwarder on his behalf.
5. We will configure the forwarder and indicate the Deployment Server , as in the part with Windows, this is the IP address or the name Splunk-indexer /
6. You can check whether the forwarder works as follows:
7. Next, go to Splunk-indexer and install a special Add-on on it, which allows you to transfer logs from Linux. You can download the distribution link .
8. After installation, we find the folder with the application at the following address ../splunk/etc/apps/Splunk_TA_nix . Copy the Splunk_TA_nix folder from apps to deployment-apps . For this application to appear as available for deployment server.
In the directory ... / deployment-apps / Splunk_TA_nix, create the local folder and copy the input.conf file from the ../Splunk_TA_nix/default folder into it.
In the ... / deployment-apps / Splunk_TA_nix / local / input.conf file through a text editor we make changes that show the data from which folders we want to collect. In our case, this is / var / log / audit.
In input.conf there is a section [monitor: /// var / log] in which you need to change disabled = 1 to disabled = 0 (Important: make sure that the necessary folder is in whitelist, if it is not there, but you need to add it)
9 Next, check to see if the Deployment server saw a new client, our Linux machine. ( Settings - Forwarder Management - Clients ).
If it is not there, then you need to check the host name of the machine, if it matches the name of the machine of the indexer, then you need to change it, otherwise an error occurs.
10. Then create a new server class related to Linux.
Settings - Forwarder Management - Server Classes - New Server Class
11. Add “Send to indexer” and “Splunk_TA_nix” applications to this class, and add a Linux machine as a client.
Please note that the files will not be downloaded if Universal Forwarder (the user under whom we use Universal Forwarder) does not have access to the folders that need to be monitored. So you need to take this point into account and allow access.
12. At the end, you need to restart deployment serve r, this can be done through the command line from the directory ... / splunk / bin :
After carrying out the above operations, you will receive Linux logs that will be loaded into the OS index.
Thus, we showed you how to upload your logs from Windows and Linux to Splunk for further analysis and processing. We hope you find this information useful.
We are happy to answer all your questions and comments on this topic. Also, if you are interested in something specifically in this area, or in the field of machine data analysis in general, we are ready to modify the existing solutions for you, for your specific task. To do this, you can write about it in the comments or simply send us a request through the form on our website .
Configure basic infrastructure
In order to start collecting data, we need the following system elements:
- Splunk - Indexer
- Windows server
- Linux server
In order to upload logs to Splunk, you must first configure the indexer, for this you will need:
• Install and configure Splunk-indexer to receive data;
First of all, you will need Splunk on the machine, which is our indexer. If you do not have Splunk installed, then read more about how and on which systems you can install it, you can read here .
After installation, you need to configure the indexer to receive data:
Settings -Forwarding and Receiving , then in the Receive data section add a new configuration: Configure receiving .
After installation, you need to configure the indexer to receive data:
Settings -Forwarding and Receiving , then in the Receive data section add a new configuration: Configure receiving .
• Create a “Send to indexer” application that will configure forwarding on all sources sending data to the indexer;
This application is necessary in order to simplify the management of data sources when there are many of these sources or access to them for making changes is difficult. Also, the application will allow you not to make potentially erroneous configuration changes on many hosts, limiting them to changes in only one place.
We create the application: Apps - Manage Apps - Add New
We create the application: Apps - Manage Apps - Add New
• Form outputs.conf configuration file
After creating the application, it is necessary to configure the outputs.conf configuration file (You can read more about that file on the official Splunk website )
In the text editor, enter the following text, replacing indexer_hostname_or_ip_address with the host name or IP address of the indexer and the receiving port set in the previous step:
Save as outputs.conf and add to the folder \ etc \ apps \ sendtoindexer \ local (The local folder must be created).
In the text editor, enter the following text, replacing indexer_hostname_or_ip_address with the host name or IP address of the indexer and the receiving port set in the previous step:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = :9997
[tcpout-server://:9997] Save as outputs.conf and add to the folder \ etc \ apps \ sendtoindexer \ local (The local folder must be created).
• Configure Deployment Server to manage the Send to indexer application and other applications;
Deployment Server is required to distribute applications and configurations to all related Splunk instances on other hosts. To activate Deployment Server, you must place at least one application in the % SPLUNK_HOME% \ etc \ deployment-apps folder . In our case, we moved the Send to indexer application there . (It was moved, not copied, as we will do next with other applications.)
At this stage, we finish the preliminary settings of the indexer and proceed to install the agents on Windows and Linux machines.
WINDOWS
A universal tool for downloading logs is a special agent - Splunk Universal Forwarder . Universal Forwarder is a version of Splunk Enterprise with significantly limited functionality, the only task of which is to collect data from the host and send them.
You can download it here .
The picture above shows that Universal Forwarder can be installed on both Windows and Linux, Solaris and other operating systems.
1. Install Universal Forwarder
As the Deployment Server, specify the IP address or name of the Splunk indexer, where we created the “Send to indexer” application. Default Port: 8089. The Receiving Indexer section will be left blank, since the Send to indexer function will perform these functions.
2. The next step is to return to Splunk and define the server class for the “Send to indexer” application.
The server class is something similar to a rule in which we indicate which applications we will distribute between which target client machines. The criteria for the formation of different classes of the server can be the type of machine, OS, geographic area or type of application, and the classes can intersect with each other. (You can read more on the official website )
Settings - Forwarder Management - edit action - add new classes.
3. After saving, you will be asked to add applications that we will distribute and target systems, the so-called clients, to which we will distribute them.
Add “ Send to indexer ” to the application section.
4. Then we add the client. The client will be our Windows machine on which we installed Universal Forwarder. If Universal Forwarder was installed correctly, then the machine should appear in the list of clients connected to the Deployment Server . Put it in Include (whitelist) .
5. You can check whether everything works correctly by looking at the contents of the _internal index. After adding "Send to indexer" to the server class, Universal Forwarder starts sending its internal logs there. Also in this index, we can further monitor whether our agents are working properly.
6. Next, download the special Add-on from the SplunkBase website , which allows you to collect data on the operation of Windows.
7. Install the application on the Splunk-Indexer ( Apps - Manage Apps - Install app from file )
By default, it is installed in the directory ... \ Splunk \ etc \ apps \ Splunk_TA_windows , but we need to copy it to the deployment-apps folder so that it the application was available for deployment server, so that later we could send it to other machines as well as "Send to indexer". ( Important: it should also remain in the apps folder so that the indexes we need for the data are formed on the indexer).
8. Then you need to pre-configure the application.
Go to the directory ... \ Splunk \ etc \ deployment-apps \ Splunk_TA_windows
Create a subdirectory “local” in it ( Important : You must always make changes to the configuration files in the local directory).
Copy the inputs.conf file from. .. \ Splunk \ etc \ deployment-apps \ Splunk_TA_windows \ default \ inputs.conf to the local directory .
Turn on the indexing of the required data. To do this, in the inputs.conf file from the local directorythrough a text editor we make some changes. Replace the disabled = 1 values with disabled = 0 in the necessary blocks of the file. Let's add the system logs for Application, Security, System.
9. Next, on Splunk-indexer, we add to the application the server class created earlier. ( Settings - Forwarder Management - Apps - Splunk_TA_Windows - “+” - Windows Forwarder )
10. Reboot the deployment server , this can be done through the command line from the directory ... / splunk / bin :
./splunk reload deploy-server Checks if data is being uploaded. ( Settings - Indexes ) They should be in the wineventlog index. As you can see in our figure, the last data that was downloaded at the moment has a timestamp 3 minutes ago.
LINUX
One of the tools to improve Linux security is the auditd audit subsystem . With its help, you can get detailed information about all system events. It is the data generated by this system that we will index in Splunk.
(The code will be presented for Linux CentOS)
1. Check if the machine has a pre-installed audit system, if not, install it.
sudo yum list audit audit-libs
sudo yum install audit audit-libs Add a new rule that we will track.
sudo auditctl -w /etc/ -p wa -k test_auditYou can check its availability using the function.
auditctl -lLogs generated by auditd fall into the file:
cd /var/log/audit/audit.log
cat audit.log2. Next, install Universal Forwarder . You can find the distribution by the link.
You should download the .rpm file, after downloading which you will be able to get a wget link.
yum install wget
cd /tmp/
wget -O splunkforwarder-7.0.3-fa31da744b51-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.3&product=universalforwarder&filename=splunkforwarder-7.0.3-fa31da744b51-linux-2.6-x86_64.rpm&wget=true'
rpm -i splunkforwarder-7.0.3-fa31da744b51-linux-2.6-x86_64.rpm3. Next, we will create a new user who will be responsible for working with splunk.
adduser splunk4. We will give permissions to the user we just created and run UniversalForwarder on his behalf.
chown -R splunk:splunk /opt/splunkforwarder/
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk5. We will configure the forwarder and indicate the Deployment Server , as in the part with Windows, this is the IP address or the name Splunk-indexer /
/opt/splunkforwarder/bin/splunk set deploy-poll :8089 -auth admin:changeme
/opt/splunkforwarder/bin/splunk edit user admin -password <Укажите новый пароль> -auth admin:changeme
/opt/splunkforwarder/bin/splunk restart 6. You can check whether the forwarder works as follows:
cd /opt/splunkforwarder/bin/
./splunk status7. Next, go to Splunk-indexer and install a special Add-on on it, which allows you to transfer logs from Linux. You can download the distribution link .
8. After installation, we find the folder with the application at the following address ../splunk/etc/apps/Splunk_TA_nix . Copy the Splunk_TA_nix folder from apps to deployment-apps . For this application to appear as available for deployment server.
In the directory ... / deployment-apps / Splunk_TA_nix, create the local folder and copy the input.conf file from the ../Splunk_TA_nix/default folder into it.
In the ... / deployment-apps / Splunk_TA_nix / local / input.conf file through a text editor we make changes that show the data from which folders we want to collect. In our case, this is / var / log / audit.
In input.conf there is a section [monitor: /// var / log] in which you need to change disabled = 1 to disabled = 0 (Important: make sure that the necessary folder is in whitelist, if it is not there, but you need to add it)
9 Next, check to see if the Deployment server saw a new client, our Linux machine. ( Settings - Forwarder Management - Clients ).
If it is not there, then you need to check the host name of the machine, if it matches the name of the machine of the indexer, then you need to change it, otherwise an error occurs.
cd /etc/hosts
cat hosts
hostname test.testdomain.com10. Then create a new server class related to Linux.
Settings - Forwarder Management - Server Classes - New Server Class
11. Add “Send to indexer” and “Splunk_TA_nix” applications to this class, and add a Linux machine as a client.
Please note that the files will not be downloaded if Universal Forwarder (the user under whom we use Universal Forwarder) does not have access to the folders that need to be monitored. So you need to take this point into account and allow access.
12. At the end, you need to restart deployment serve r, this can be done through the command line from the directory ... / splunk / bin :
./splunk reload deploy-server After carrying out the above operations, you will receive Linux logs that will be loaded into the OS index.
Conclusion
Thus, we showed you how to upload your logs from Windows and Linux to Splunk for further analysis and processing. We hope you find this information useful.
We are happy to answer all your questions and comments on this topic. Also, if you are interested in something specifically in this area, or in the field of machine data analysis in general, we are ready to modify the existing solutions for you, for your specific task. To do this, you can write about it in the comments or simply send us a request through the form on our website .