Splunk. Easy Troubleshooting application work

How often do you use already created add-ins with Splukbase, instead of writing your own? Do they always have what you would like to see? Do they all work fine, even if it was installed exactly according to the instructions? Today we will talk about the features of working with Splunk applications.
Namely, we will learn to understand how requests are formed in applications, so that later on, on the basis of them, we can do something concrete for our task. And also we will consider several reasons because of which often applications start to work crookedly or do not work at all.
Why do we talk about this? Because many applications have a more complex query structure than ordinary ordinary users use when writing their own applications. But at the same time, there is often a need to understand how a particular table is constructed in order to correct it for yourself.
What chips are used by advanced application developers:
- Eventtype
- Macros
- Alert
- Lookups
Now, using examples of several queries, we will try to show what it is and how to understand them.
Example number 1
Take the request from the application Splunk App for Windows Infrastructure , which should give us the number of new users in Active Directory. We get the following table as a result:

Suppose we are not satisfied with something in this table: we need to add another field, do additional filtering by Subject User, or something else. It seems that here is difficult? Now open the search query and edit everything!
But ... something went wrong.

Instead of a standard and understandable query, a strange construct appeared in single quotes. Let's get acquainted, this is a macro.
Macroin Splunk is a reusable search query. Macros are created when, in the framework of a single application, you often have to make similar transformations, and in order not to repeat them in each request, save them separately. The macro in the request can be recognized by the `macros_name` structure. Also, macros may require some kind of input variable, such macros in the query have the structure `macros_name (value)`.
Next, open the macro text, you can find it here Settings - Advanced Search - Search macros .

In general, such a request looks more ordinary and can already be worked with. But there is one nuance, if you try to find such a field as eventtype in the logs, you will not find anything. This is another Splunk tool.
Eventtype is a filtering tool. It is used in cases where the necessary events are selected by a variety of criteria. Therefore, with the help of eventtype, we can once configure which events will be of this type, give them an adequate name and use them in any future requests.
You can find Eventtype in Settings - Event types.
Under “eventtype = wineventlog_security” the following search is hidden:

Thus, we “unwound” one of the requests in the application. Now we know on the basis of what data and what transformations were carried out in order to get the result, which means that if necessary we can correct it or create our own similar one. Just be careful, do not try to edit the macros and event types that are embedded in the application. They can be referenced by other dashboards and then the results will change too. To prevent this from happening, it is better to collect all the fragments in one search or create new macros and event types.
I note immediately that this is not the most complicated and complicated story. Eventtype can refer to any other eventtype, and this other to the third and so on. Some macros can refer to directories that are generated using other macros and so on.
Example number 2
Consider another example: the formation of a list of domains.

Let's look at the query that forms this list for us.

And we again see a macro that replaces almost the entire request.
Go to the menu with macros ( Settings - Advanced Search - Search macros ) and see that our macro takes the basis of the directory with domains.

This begs the question: what is this directory, how and from where did it get our data, if we downloaded the application from the Internet and did not give him any reference information.
We recall that reference books can be created not only from downloaded files, but also with the help of searches or alerts. Go to the appropriate section ( Settings - Searches, Reports, and Alerts) and really find an object there, which by its name looks like something we need. (DomainSelector_Lookup)
This is an alert that runs on a schedule, fires at a certain frequency and updates the directory.

But let's see where it takes the domain name, and for this we will open a search query. And in it ... again a macro.

In which there is again a link to the eventtype.

And in which there is another link to the eventtype

And just now we got to the end of this confusing chain:

In two examples, we examined what lies behind the signs and graphs of such an application as the Splunk App for Windows Infrastructure. This architecture shows a high level of data model development, and therefore the quality of the application, but it gives a lot of pain to users who want to understand what is happening there.
Troubleshooting
As stated at the beginning of the article, let's say a couple of words about why the application may not work as we expect.
Consider this example: we installed the Splunk App for Windows Infrastructure application, installed the add-ons that are required for it, set up data loading on the splank. But when we go into the application, we see the following picture:

Why can this be? To be honest, there can be many reasons, but we will now consider two of the most likely and common ones.
1. Default Indexes
As you can see, in all search queries that were in the first part of the article there is no classical part of index = .... When the query does not indicate which index is being searched for, then the search is only for those indices that are in the Default Indexes list. By default, only the main index is included there, so we need to add the indices we need to this group.
You can do this as follows:
Settings - Users and authentication - Access controls - Roles - <those roles that will work with this application> - Indexes searched by default
Then, add the necessary indexes to Selected indexes.

2. Scheduled Alerts
Many filters are based on directories, and directories on alerts. Perhaps these alerts have not yet worked and have not filled out reference books with the necessary information. We need to look at when they will work and, if necessary, change the time and interval for updating the directory. ( Settings - Searches, Reports, and Alerts )
In this article, we all looked at the example of a Windows Infrastructure application, but the same thing is found in many applications created by experienced spunkers.
If you want to understand them better, so that this is not a black box for you, which transforms information in order to create high-quality your own analytics based on this data, then you need to learn more about macros and eventtypes .
Splunk Course at Splunk Fundamentals 2 Training Center .
