Loading data into Splunk: Universal Forwarder vs Heavy Forwarder. What is the difference?



    Today we will talk about agents (forwarders) for loading data into Splunk . In the article we will briefly describe what it is, what types are there, what is the difference between them and in what situations it is better to use one or another forwarder.

    Correct data loading is the most problematic issue in any data handling system. Data transfer can be carried out in various ways, but the most common of these is the use of forwarders.

    Splunk forwarder has several advantages:

    • Metadata labeling (source, source type and host)
    • Custom buffering
    • Data compression
    • SSL
    • Using any available network ports

    After you have decided that you will send data using forwarders, the following question arises: which is the best forwarder to use?

    In total there are 2 types of forwarders :

    1. Universal Forwarder , which contains only those components that are required to transfer data.
    2. Heavy Forwarder , which is a full-fledged Splunk Enterprise, which, in addition to data transfer, can index, perform search queries and modify data.

    Universal forwarder


    Universal Forwarder has several advantages over using Heavy Forwarder. Therefore, it is often recommended to use it, if there are no specific prerequisites for using Heavy Forwarder, which we will discuss below.

    The most notable advantage is that Universal Forwarder uses significantly less hardware resources than other Splunk software products. It loads less CPU, uses less memory and takes up less disk space. It is also more scalable than other Splunk products, since you can install over a thousand instances that will not greatly affect network and host performance.

    Another advantage is its availability for installation on many different platforms. It can be installed not only on Windows, Linux and Mac OS, like Splunk Enterprise, but also on Solaris, FreeBSD and AIX.

    Universal Forwarder is available as a separate installation package and includes only the necessary components necessary for sending data to other instances of the Splunk platform. Although it does not have a web interface, it can still be customized, managed and scaled by editing configuration files.

    To achieve better performance, Universal Forwarder has several limitations:

    • It is not possible to locally perform indexing and search queries.
    • Cannot configure sending alerts.
    • You can analyze the incoming data stream before indexing only if it is structured data.
    • Does not include Python.

    How to install and configure Universal Forwarder can be found here .

    Heavy forwarder


    Although Universal Forwarder is the preferred way to send data, you may need Heavy Forwarder if you need to analyze or make changes to the data before sending it, or you will need to control where the data is going, based on its content.

    One of the key advantages of Heavy Forwarder is that it can filter unwanted events, even in unstructured data, which will reduce the amount of indexing, and the size of the license depends on it.

    However, it should be noted that the use of Heavy Forwarder increases network traffic, CPU and memory usage. This is due to the fact that Heavy Forwarder sends the analyzed data over the network not just with raw events, but with all the fields that are highlighted during indexing and additional metadata.

    To compare the performance of Heavy and Universal Forwarder, a test was conducted.
    There were 367,463,625 events in the test file.

    Network traffic (GB)Average bit rate (kbps)Average Indexing Rate
    (kbps)    		Duration (s)
    Heavy forwarder38.41922513920998
    Universal forwarder6.41015174666662

    Results of the experiment


    When using Universal Forwarder:

    • The amount of data sent over the network was 6 times lower.
    • The amount of data indexed per second was about 3 times higher
    • The total data loading time was 3 times faster.

    Recommendations


    Use Heavy Forwarder only when:

    • It is possible to filter a significant part of the data by conducting a preliminary analysis of unstructured events.
    • There are special requirements for the user interface or add-on , for example, DBconnect, Checkpoint, Cisco IPS
    • Difficult (by the content of the event) data routing

    In other cases, it is better to use Universal Forwarder.


    If you still have not tried Splunk, then it's time to start, the free version up to 500MB per day is available to everyone. And if you have questions or problems with Splunk - you can ask them to us , and we will help.

    We are the official Premier Splunk Partner .

    Tags:

    Also popular now: