April 11, 2019 at 09:44

Splunk Universal Forwarder in docker as a system logger

  • Tutorial


Splunk is one of several of the most recognized commercial products for collecting and analyzing logs. Even now, when sales in Russia are no longer made, this is not a reason not to write instructions / how-to on this product.

Task : to collect system logs from the docker nodes in Splunk without changing the host machine configuration

I would like to start with the official approach, which looks strange when using docker.
Link to the Docker hub
What do we have:

1. Pullim image

$ docker pull splunk/universalforwarder:latest

2. We start the container with the necessary parameters

$ docker run -d  -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=' splunk/universalforwarder:latest

3. We go into the container

docker exec -it  /bin/bash

Next, we are asked to go to the known address in the documentation.

And configure the container after it starts:


./splunk add forward-server :
./splunk add monitor /var/log
./splunk restart

Wait. What?

But the surprises do not end there. If you run the container from the official image in interactive mode, you will see the following:

A bit of disappointment

$ docker run -it -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=password' splunk/universalforwarder:latest
PLAY [Run default Splunk provisioning] *******************************************************************************************************************************************************************************************************
Tuesday 09 April 2019  13:40:38 +0000 (0:00:00.096)       0:00:00.096 *********
TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019  13:40:39 +0000 (0:00:01.520)       0:00:01.616 *********
TASK [Get actual hostname] *******************************************************************************************************************************************************************************************************************
changed: [localhost]
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.599)       0:00:02.215 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.054)       0:00:02.270 *********
TASK [set_fact] ******************************************************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.075)       0:00:02.346 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.067)       0:00:02.413 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.060)       0:00:02.473 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.051)       0:00:02.525 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.056)       0:00:02.582 *********
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.216)       0:00:02.798 *********
included: /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml for localhost
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.087)       0:00:02.886 *********
TASK [splunk_common : Update Splunk directory owner] *****************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.324)       0:00:03.210 *********
included: /opt/ansible/roles/splunk_common/tasks/get_facts.yml for localhost
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.094)       0:00:03.305 *********
ну и так далее...

Excellent. The image does not even have an artifact. That is, each time you start it will take time to download the archive with binaries, unpack and configure.
But what about docker-way and all that?

No thanks. We will go the other way. What if we perform all these operations at the assembly stage? Then let's go!

In order not to pull for a long time, I will immediately show the final image:

Dockerfile
# Тут у кого какие предпочтения
FROM centos:7
# Задаём переменные, чтобы каждый раз при старте не указывать их
ENV SPLUNK_HOME /splunkforwarder
ENV SPLUNK_ROLE splunk_heavy_forwarder
ENV SPLUNK_PASSWORD changeme
ENV SPLUNK_START_ARGS --accept-license
# Ставим пакеты
# wget - чтобы скачать артефакты
# expect - понадобится для первоначального запуска Splunk на этапе сборки
# jq - используется в скриптах, которые собирают статистику докера
RUN yum install -y epel-release \
    && yum install -y wget expect jq
# Качаем, распаковываем, удаляем
RUN wget -O splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.4&product=universalforwarder&filename=splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz&wget=true' \
    && wget -O docker-18.09.3.tgz 'https://download.docker.com/linux/static/stable/x86_64/docker-18.09.3.tgz' \
    && tar -xvf splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz \
    && tar -xvf docker-18.09.3.tgz  \
    && rm -f splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz \
    && rm -f docker-18.09.3.tgz
# С shell скриптами всё понятно, а вот inputs.conf, splunkclouduf.spl и first_start.sh нуждаются в пояснении. Об этом расскажу после source тэга.
COPY [ "inputs.conf", "docker-stats/props.conf", "/splunkforwarder/etc/system/local/" ]
COPY [ "docker-stats/docker_events.sh", "docker-stats/docker_inspect.sh", "docker-stats/docker_stats.sh", "docker-stats/docker_top.sh", "/splunkforwarder/bin/scripts/" ]
COPY splunkclouduf.spl /splunkclouduf.spl
COPY first_start.sh /splunkforwarder/bin/
#  Даём права на исполнение, добавляем пользователя и выполняем первоначальную настройку
RUN chmod +x /splunkforwarder/bin/scripts/*.sh \
    && groupadd -r splunk \
    && useradd -r -m -g splunk splunk \
    && echo "%sudo ALL=NOPASSWD:ALL" >> /etc/sudoers \
    && chown -R splunk:splunk $SPLUNK_HOME \
    && /splunkforwarder/bin/first_start.sh \
    && /splunkforwarder/bin/splunk install app /splunkclouduf.spl -auth admin:changeme \
    && /splunkforwarder/bin/splunk restart
# Копируем инит скрипты
COPY [ "init/entrypoint.sh", "init/checkstate.sh", "/sbin/" ]
# По желанию. Кому нужно локально иметь конфиги/логи, кому нет.
VOLUME [ "/splunkforwarder/etc", "/splunkforwarder/var" ]
HEALTHCHECK --interval=30s --timeout=30s --start-period=3m --retries=5 CMD /sbin/checkstate.sh || exit 1
ENTRYPOINT [ "/sbin/entrypoint.sh" ]
CMD [ "start-service" ]

And so, what is contained in

first_start.sh
#!/usr/bin/expect -f
set timeout -1
spawn /splunkforwarder/bin/splunk start --accept-license
expect "Please enter an administrator username: "
send -- "admin\r"
expect "Please enter a new password: "
send -- "changeme\r"
expect "Please confirm new password: "
send -- "changeme\r"
expect eof

At the first start, Splunk asks for a username / password, BUT this data is used only to execute administrative commands of this particular installation, that is, inside the container. In our case, we just want to launch the container so that everything works and the logs flow like water. Of course, this is hardcode, but I have not found other ways.

Further on the script is executed

/splunkforwarder/bin/splunk install app /splunkclouduf.spl -auth admin:changeme

splunkclouduf.spl - This is a credits file for Splunk Universal Forwarder, which can be downloaded from the web interface.

Where to click to download (in pictures)



This is a regular archive that you can unzip. Inside - certificates and password for connecting to our SplunkCloud and outputs.conf with a list of our input instances. This file will be relevant until you reinstall your Splunk installation or add an input node if the installation is on-premise. Therefore, it’s okay to add it inside the container.

And the last one is restart. Yes, to apply the changes, you need to restart it.

In our inputs.conf we add the logs that we want to send to Splunk. It is not necessary to add this file to the image if, for example, you scatter configs through puppet. The main thing is that Forwarder sees configs when the daemon starts, otherwise ./splunk restart will be needed .

What docker stats scripts? The github has an old solution from outcoldman , the scripts are taken from there and finalized to work with the current versions of Docker (ce-17. *) And Splunk (7. *).

With the data obtained, you can build such

dashboards: (a couple of pictures)



The source code of the deshes lies in the turnip indicated at the end of the article. Please note that there are 2 select fields: 1 - select the index (searched by mask), select the host / container. You probably have to update the index mask, depending on the names you use.

In conclusion, I want to pay attention to the start () function in

entrypoint.sh
start() {
    trap teardown EXIT
	if [ -z $SPLUNK_INDEX ]; then
	echo "'SPLUNK_INDEX' env variable is empty or not defined. Should be 'dev' or 'prd'." >&2
	exit 1
	else
	sed -e "s/@index@/$SPLUNK_INDEX/" -i ${SPLUNK_HOME}/etc/system/local/inputs.conf
	fi
	sed -e "s/@hostname@/$(cat /etc/hostname)/" -i ${SPLUNK_HOME}/etc/system/local/inputs.conf
    sh -c "echo 'starting' > /tmp/splunk-container.state"
	${SPLUNK_HOME}/bin/splunk start
    watch_for_failure
}

In my case, for each environment and each individual entity, whether it be an application in a container or a host machine, we use a separate index. So the search speed will not suffer with a significant accumulation of data. A simple rule is used to name indexes:_. Therefore, to make the container universal, before starting the daemon directly, we replace the sed wildcard with the environment name. A variable with an environment name is passed through environment variables. That sounds funny.

It’s also worth noting that for some reason, Splunk is not affected by the docker hostname parameter . Anyway, he will persistently send logs from the id of his container to the host field. As a solution, you can mount / etc / hostname from the host machine and, at startup, make a replacement similar to index names.

Docker-compose.yml example
version: '2'
services:
  splunk-forwarder:
    image: "${IMAGE_REPO}/docker-stats-splunk-forwarder:${IMAGE_VERSION}"
    environment:
      SPLUNK_INDEX: ${ENVIRONMENT}
    volumes:
    - /etc/hostname:/etc/hostname:ro
    - /var/log:/var/log
    - /var/run/docker.sock:/var/run/docker.sock:ro


Summary

Yes, perhaps the solution is not perfect and certainly not universal for everyone, since there is a lot of "hardcode" . But on the basis of it, everyone can collect their image and put it in their own private artifact, if, as it happened, you need Splunk Forwarder in the docker.

Links:

The solution from the article
The decision from outcoldman inspired to reuse part of the functional
Of. Universal Forwarder setup documentation
Tags:

Also popular now: