How did this sidecar container get here [at Kubernetes]?
- Transfer
Note trans. A: This article, written by Scott Rahner - an engineer at Dow Jones, we continue the cycle of numerous materials that are available to tell about how Kubernetes works, how they work, are interconnected and use its basic components. This time it is a practical note with an example of code for creating a hook in Kubernetes, demonstrated by the author “under the pretext” of automatic creation of sidecar-containers.
(The author of the photo - Gordon A. Maxwell, found on the Internet.)
When I began to study sidecar-containers and service mesh, I needed to understand how the key mechanism works - automatic insertion of a sidecar-container. Indeed, in the case of using systems like Istio or Consul, when a container with an application is deployed, an Envoy container that is already configured appears suddenly in its pod (a similar situation occurs with Conduit, which we wrote about at the beginning of the year - approx. Transl.) . What? How? So began my research ...
For those who do not know, a sidecar-container is a container that is deployed next to the containers of the application in order to “help” this application in any way. An example of such use is a proxy for managing traffic and terminating TLS sessions, a container for streaming logs and metrics, a container for scanning security problems ... The idea is to isolate various aspects of the entire application from business logic by using separate containers for each functions.
Before continuing, I will outline my expectations. The purpose of this article is not to explain the intricacies and scenarios of using Docker, Kubernetes, service meshes, etc., but to demonstrate one powerful approach to expanding the capabilities of these technologies. The article is for those who are already familiar with the use of these technologies, or at least have read a lot about them. To try the practical part in action, you need a machine with already configured Docker and Kubernetes. The easiest way to do this is https://docs.docker.com/docker-for-windows/kubernetes/ (a Windows instruction that works in Docker for Mac). (Note. Trans .: As an alternative to users of Linux and * nix-systems, we can offer Minikube .)
First, let's get a little closer with Kubernetes:
Kube Arch , licensed under CC BY 4.0.
When you are going to commit anything to Kubernetes, you need to send the object to kube-apiserver. Most often this is done by passing arguments or a YAML file to kubectl. In this case, the API server goes through several stages before directly placing data on etcd and scheduling the corresponding tasks:
This sequence is important in order to understand how the insertion of sidecar containers works. In particular, attention should be paid to the Admission Control , under which Kubernetes validates and, if necessary, modifies the objects before storing them (for more about this stage, see the “Admission Control” chapter of this article- approx. Trans.) . Kubernetes also allows you to register webhooks that can perform user-defined validation and changes (mutations) .
However, the process of creating and registering your hooks is not so simple and well documented. I had to spend several days reading and re-reading the documentation, as well as analyzing the Istio and Consul code. And when it came to the code for some of the API responses, I spent at least half a day doing random trials and errors.
After the result has been achieved, I think it will be dishonest not to share it with all of you. It is simple and at the same time effective.
The name webhook speaks for itself - this is the HTTP endpoint that implements the API defined in Kubernetes. You create an API server that Kubernetes can call before it deals with Deployment. Here I had to face difficulties, since only a few examples are available, some of which are just Kubernetes unit tests, others are hidden in the middle of a huge code base ... and all are written in Go. But I chose a more affordable option - Node.js:
( index.js )
The path to the API - in this case, this
Once this is done, all you need to do is send the response to the API server with a very simple object. In this case, we add the label to
Well, we have code that accepts requests from the Kubernetes API server and responds to them, but how can we fix it? And how to force Kubernetes to redirect us these requests? You can deploy such an endpoint everywhere that the Kubernetes API server can reach. The simplest way is to deploy the code into the Kubernetes cluster itself, which we will do in this example. I tried to make the example as simple as possible, so for all actions I use only Docker and kubectl. Let's start by creating a container in which the code will be run:
( Dockerfile )
Apparently, everything is very simple here. Take the image from the node from the community and drop the code into it. Now you can perform a simple build:
The next step is to create the Deployment in Kubernetes:
( deployment.yaml )
Notice how we referred to the newly created image? It could just as well have been a pod, and anything else to which we can connect the service to Kubernetes. Now we define this Service:
This way, an endpoint with an internal name that indicates our container will appear in Kubernetes. The final step is to inform Kubernetes that we want the API server to call this service when it is ready to make changes (mutations) :
The name and the path here can be any, but I tried to make them as meaningful as possible. Changing the path will mean the need to modify the corresponding code in javascript. The webhook
That's all! So simple ... but what about security? RBAC is one aspect that is not covered in the article. I assume that you run the example in Minikube or in Kubernetes that comes with the Docker for Windows / Mac. However, I will tell about one more necessary element. The Kubernetes API server only accesses HTTPS endpoints, so the application will require SSL certificates. You will also need to tell Kubernetes who is the certification authority for the root certificate.
For demonstration purposes only (!!!), I added
( Dockerfile )
Note: the last stage - displays a single line with root CA encoded in base64. This is exactly what is required for the hook configuration, so in your further tests be sure to copy this string into the
Now the code supports the launch of HTTPS, and also told Kubernetes where to find us and which authentication center to trust. It remains only to enclose all this into a cluster:
Everything is deployed in a cluster - it's time to try the code in action, which we will do by adding a new pod / Deployment. If everything works correctly, the hook will have to add an additional label
( test.yaml )
Ok, we saw
Wonderful! Although u
But wait! Were we going to use this code to create a sidecar container? I warned that I’ll show you how to add a sidecar ... And now, with the knowledge and code you received: https://github.com/dowjones/k8s-webhook - experiment and explore how to make your sidecar automatically inserted. It's quite simple: you only need to prepare the correct JSON Patch, which will add an additional container in the test Deployment. Happy orchestration!
Read also in our blog:
(The author of the photo - Gordon A. Maxwell, found on the Internet.)
When I began to study sidecar-containers and service mesh, I needed to understand how the key mechanism works - automatic insertion of a sidecar-container. Indeed, in the case of using systems like Istio or Consul, when a container with an application is deployed, an Envoy container that is already configured appears suddenly in its pod (a similar situation occurs with Conduit, which we wrote about at the beginning of the year - approx. Transl.) . What? How? So began my research ...
For those who do not know, a sidecar-container is a container that is deployed next to the containers of the application in order to “help” this application in any way. An example of such use is a proxy for managing traffic and terminating TLS sessions, a container for streaming logs and metrics, a container for scanning security problems ... The idea is to isolate various aspects of the entire application from business logic by using separate containers for each functions.
Before continuing, I will outline my expectations. The purpose of this article is not to explain the intricacies and scenarios of using Docker, Kubernetes, service meshes, etc., but to demonstrate one powerful approach to expanding the capabilities of these technologies. The article is for those who are already familiar with the use of these technologies, or at least have read a lot about them. To try the practical part in action, you need a machine with already configured Docker and Kubernetes. The easiest way to do this is https://docs.docker.com/docker-for-windows/kubernetes/ (a Windows instruction that works in Docker for Mac). (Note. Trans .: As an alternative to users of Linux and * nix-systems, we can offer Minikube .)
Overall picture
First, let's get a little closer with Kubernetes:
Kube Arch , licensed under CC BY 4.0.
When you are going to commit anything to Kubernetes, you need to send the object to kube-apiserver. Most often this is done by passing arguments or a YAML file to kubectl. In this case, the API server goes through several stages before directly placing data on etcd and scheduling the corresponding tasks:
This sequence is important in order to understand how the insertion of sidecar containers works. In particular, attention should be paid to the Admission Control , under which Kubernetes validates and, if necessary, modifies the objects before storing them (for more about this stage, see the “Admission Control” chapter of this article- approx. Trans.) . Kubernetes also allows you to register webhooks that can perform user-defined validation and changes (mutations) .
However, the process of creating and registering your hooks is not so simple and well documented. I had to spend several days reading and re-reading the documentation, as well as analyzing the Istio and Consul code. And when it came to the code for some of the API responses, I spent at least half a day doing random trials and errors.
After the result has been achieved, I think it will be dishonest not to share it with all of you. It is simple and at the same time effective.
Code
The name webhook speaks for itself - this is the HTTP endpoint that implements the API defined in Kubernetes. You create an API server that Kubernetes can call before it deals with Deployment. Here I had to face difficulties, since only a few examples are available, some of which are just Kubernetes unit tests, others are hidden in the middle of a huge code base ... and all are written in Go. But I chose a more affordable option - Node.js:
const app = express();
app.use(bodyParser.json());
app.post('/mutate', (req, res) => {
console.log(req.body)
console.log(req.body.request.object)
let adminResp = {response:{
allowed: true,
patch: Buffer.from("[{ \"op\": \"add\", \"path\": \"/metadata/labels/foo\", \"value\": \"bar\" }]").toString('base64'),
patchType: "JSONPatch",
}}
console.log(adminResp)
res.send(adminResp)
})
const server = https.createServer(options, app);( index.js )
The path to the API - in this case, this
/mutate- can be arbitrary (it should only later correspond to the YAML passed to Kubernetes). It is important for it to see and understand the JSON received from the API server. In this case, we do not pull anything out of JSON, but this may be useful in other scripts. In the above code, we update JSON. Two things are needed for this:- Learn and understand JSON Patch .
- Correctly convert the JSON Patch expression to a byte array encoded with base64.
Once this is done, all you need to do is send the response to the API server with a very simple object. In this case, we add the label to
foo=barany pod that comes to us.Deployment
Well, we have code that accepts requests from the Kubernetes API server and responds to them, but how can we fix it? And how to force Kubernetes to redirect us these requests? You can deploy such an endpoint everywhere that the Kubernetes API server can reach. The simplest way is to deploy the code into the Kubernetes cluster itself, which we will do in this example. I tried to make the example as simple as possible, so for all actions I use only Docker and kubectl. Let's start by creating a container in which the code will be run:
FROM node:8
USER node
WORKDIR /home/node
COPY index.js .
COPY package.json .
RUN npm install
# позже сюда добавятся дополнительные команды для TLS
CMD node index.js( Dockerfile )
Apparently, everything is very simple here. Take the image from the node from the community and drop the code into it. Now you can perform a simple build:
docker build . -t localserverThe next step is to create the Deployment in Kubernetes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: webhook-server
spec:
replicas: 1
selector:
matchLabels:
component: webhook-server
template:
metadata:
labels:
component: webhook-server
spec:
containers:
- name: webhook-server
imagePullPolicy: Never
image: localserver( deployment.yaml )
Notice how we referred to the newly created image? It could just as well have been a pod, and anything else to which we can connect the service to Kubernetes. Now we define this Service:
apiVersion: v1
kind: Service
metadata:
name: webhook-service
spec:
ports:
- port: 443
targetPort: 8443
selector:
component: webhook-serverThis way, an endpoint with an internal name that indicates our container will appear in Kubernetes. The final step is to inform Kubernetes that we want the API server to call this service when it is ready to make changes (mutations) :
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: webhook
webhooks:
- name: webhook-service.default.svc
failurePolicy: Fail
clientConfig:
service:
name: webhook-service
namespace: default
path: "/mutate"
# далее записан результат base64-кодирования файла rootCA.crt
# с помощью команды `cat rootCA.crt | base64 | tr -d '\n'`
# подробнее об этом см. ниже
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdHekNDQkFPZ0F3SUJBZ0lKQU1jcTN6UHZDQUd0TUEwR0NTcUdTSWIzRFFFQkN3VUFNSUdqTVFzd0NRWUQKVlFRR0V3SlZVekVUTUJFR0ExVUVDQXdLVG1WM0lFcGxjbk5sZVRFVE1CRUdBMVVFQnd3S1VISnBibU5sZEc5dQpJREVTTUJBR0ExVUVDZ3dKUkc5M0lFcHZibVZ6TVF3d0NnWURWUVFMREFOUVNVSXhIakFjQmdOVkJBTU1GWGRsClltaHZiMnN0S2k1a1pXWmhkV3gwTG5OMll6RW9NQ1lHQ1NxR1NJYjNEUUVKQVJZWmMyTnZkSFF1Y21Gb2JtVnkKUUdSdmQycHZibVZ6TG1OdmJUQWVGdzB4T0RFd016RXhOalU1TURWYUZ3MHlNVEE0TWpBeE5qVTVNRFZhTUlHagpNUXN3Q1FZRFZRUUdFd0pWVXpFVE1CRUdBMVVFQ0F3S1RtVjNJRXBsY25ObGVURVRNQkVHQTFVRUJ3d0tVSEpwCmJtTmxkRzl1SURFU01CQUdBMVVFQ2d3SlJHOTNJRXB2Ym1Wek1Rd3dDZ1lEVlFRTERBTlFTVUl4SGpBY0JnTlYKQkFNTUZYZGxZbWh2YjJzdEtpNWtaV1poZFd4MExuTjJZekVvTUNZR0NTcUdTSWIzRFFFSkFSWVpjMk52ZEhRdQpjbUZvYm1WeVFHUnZkMnB2Ym1WekxtTnZiVENDQWlJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dJUEFEQ0NBZ29DCmdnSUJBTHRpTU5mL1l3d0RkcHlPSUhja2FQK3J6NmdxYXBhWmZ2a0JndHVZK3BYQVZnNWc5M1RISmlPdlJYUnAKeG9UZ1o0RlA4N0V3R0NXRUZxZTRFRjh5UUxCK1NvWHBxUmRrWlVLYlM3eDVJNnNDb0h1dFJXaURpd3piV3lGawp3UnppeXpyMTQzN2wzYWxadU9VNkl5bU9mVDlETzdRaDNnY01HOEprQ09aVlVOelVIN3J4WmtieGg3M1lXNW5ZCjhSMU5tZDJ3cm1IWkVWc2JmS21GTlhvZjFueWtRcXMyMUQxT1FwQ3A1VDB5QU9penZlaW9OS3VsQVVpcjNVQ0EKSmNYYWpMMGZVS1ZIcGVTbGlhWXdKZmZNSDFqOElqSDZTdm5TdG9qQWlWdnJHb1ZKUlFqRXFLQkpYVGMyaHZCWQpCcjJqdGdQb25WWnBBTFphbktha0JTV1cyZ25oZVFKaHpKOGhkMXlEU0x6dFFKb2JkOHZUMEZ5bHZaQzY3aURnCmROb1NWbHBaQlpDSVIxTldaRVdGbTlTWWtKLzZ6emVqMFZpWnp2aFBYdm9GelZEVGZoMEwzQWljUTZlWTNzcEMKV0Fmb2VTcFUxaEVJeG92SmdwVkpMbnRaWkhyN1RJQ05CNlV5QnFVUzhEa0lTMkhnWkh2MTd1VjA3bTFzZDZDMApDUnV5YmZHQ0l2RGNwMCtzMjF6TENXemJuS3BzaFo5UkYvYWhXMW11cVN2dGt0WXlYOFVySlpKT1h3Z0NKenhLCmdwZGs3YlA4Y3ZkRWxUZDduQXRJbjZPcm42VWlVUnFpSXY1VSt0bmIvOVlrNDIxVzdlT2NxZ3JqTEY4eUo5ckIKN0hBYlhGRjM5OW5NMlBtYkZIV2FROG1xeWo0L0kxNm9tTHVsUGZvekVWK0xvMXVwQWdNQkFBR2pVREJPTUIwRwpBMVVkRGdRV0JCUnVKaTcyS0U5bWhpejZvYVhkSXlpbGpTeXhkVEFmQmdOVkhTTUVHREFXZ0JSdUppNzJLRTltCmhpejZvYVhkSXlpbGpTeXhkVEFNQmdOVkhSTUVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQ0FRQlQKS28wczJTTWZkSzdkRS9ZdFBwQ2lQNDVBK0xJSjVKd0l2dWdiUlNGeVRUSEU0akhVRTdQdWc3VHdGNC93YnJFZwpNN1F3OWUxbDA1M2lheWRFOS9sUlVDbzN4TnVVcU5jU2lCK3RIOE54dURHUUw5NHBuWTdTR3FuRjBDMlZ2d2x2CmxaYUQxNU41cVdvTVJrQU54VXRPRGFaWEdLcS94VVBSQWdNMHFtbXc5ZnIwaXAvQzFjVGMyVVhlejlGNTMvV2cKV1FNempWbUNTNGlnckR1a1FBNWxodFRlYUlzK3pxNk9ZeWNiN01KR1JBL0NhcnpDL1VuZExMbmhsdEtITkJhMwp0TDFVVUJCTzBMdmdMaE8zVk9nRENOazJYVmZzVHFueEUrTGp6R2dmUnRqYjE5L0p1d2V2OW00Y3ZzUlZESGVMCk9oQ0lvenorUHRLWHBwVDFWd1VRbFZlOG5ic2RiVnNZWmt4Q3llcGpMUTJ5TXNUUXdoa2NncGRiTnYzbTMvRC8Kc3N5ZS9iZnphUGFXVEE1R0d5emhXdXlENDZPT1lCUFlhZzd0aFFneXRvOWRpSWNDSHNMQ3BVZm1FQ1d6TERBYgozK2NadnZnYXZybFJCZjN2cVhrVlZxT1NLNGxna25iUEZJc0YvbnFIanM2WXI5Tktiai9sRGlBalRYaVdQdFRmClJzd0JodndveDJnK21zd0prQytId0cvckZ1RXFDdklTaFJGWlEvMDgyL0F5ekpYRlE3SlV3eHluL0dTQXlGZUsKL1Y3T01XTEhUeVd4Vkg4eVBCZ1JSVE1CK3NrOEVQQndveFRLSjZnLytTbmdkNXM1ZEx6ZDhpSTlsVHdxWDZBTApzNU1OY2NobFRWVU9RYnFGWXBKc3FTUTlIVlB2bjZDckRlTGlxTlNKQVE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]The name and the path here can be any, but I tried to make them as meaningful as possible. Changing the path will mean the need to modify the corresponding code in javascript. The webhook
failurePolicyis also important - it determines whether the object should be saved if the hook returns an error or does not work. We in this case tell Kubernetes not to continue processing. Finally, the rules ( rules): they will change depending on what API calls you expect actions from Kubernetes. In this case, since we are trying to emulate the insertion of a sidecar container, we need to intercept requests to create a pod.That's all! So simple ... but what about security? RBAC is one aspect that is not covered in the article. I assume that you run the example in Minikube or in Kubernetes that comes with the Docker for Windows / Mac. However, I will tell about one more necessary element. The Kubernetes API server only accesses HTTPS endpoints, so the application will require SSL certificates. You will also need to tell Kubernetes who is the certification authority for the root certificate.
Tls
For demonstration purposes only (!!!), I added
Dockerfilesome code to create a root CA and use it to sign the certificate:RUN openssl genrsa -out rootCA.key 4096
RUN openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt \
-subj "/C=US/ST=New Jersey/L=Princeton /O=Dow Jones/OU=PIB/CN=*.default.svc/emailAddress=scott.rahner@dowjones.com"
RUN openssl genrsa -out webhook.key 4096
RUN openssl req -new -key webhook.key -out webhook.csr \
-subj "/C=US/ST=New Jersey/L=Princeton /O=Dow Jones/OU=PIB/CN=webhook-service.default.svc/emailAddress=scott.rahner@dowjones.com"
RUN openssl x509 -req -in webhook.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out webhook.crt -days 1024 -sha256
RUN cat rootCA.crt | base64 | tr -d '\n'( Dockerfile )
Note: the last stage - displays a single line with root CA encoded in base64. This is exactly what is required for the hook configuration, so in your further tests be sure to copy this string into the
caBundlefile field hook.yaml. Dockerfileit throws certificates right in WORKDIR, so javascript just takes them from there and uses it for the server:const privateKey = fs.readFileSync('webhook.key').toString();
const certificate = fs.readFileSync('webhook.crt').toString();
//…const options = {key: privateKey, cert: certificate};
const server = https.createServer(options, app);Now the code supports the launch of HTTPS, and also told Kubernetes where to find us and which authentication center to trust. It remains only to enclose all this into a cluster:
kubectl create -f deployment.yaml
kubectl create -f service.yaml
kubectl create -f hook.yamlWe summarize
Deployment.yamllaunches a container that serves the API's hook over HTTPS and returns a JSON Patch to modify the object.Service.yaml- provides for an endpoint containerwebhook-service.default.svc.Hook.yamlsaid API-server, where we find:https://webhook-service.default.svc/mutate.
Let's try in business!
Everything is deployed in a cluster - it's time to try the code in action, which we will do by adding a new pod / Deployment. If everything works correctly, the hook will have to add an additional label
foo:apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
replicas: 1
selector:
matchLabels:
component: test
template:
metadata:
labels:
component: test
spec:
containers:
- name: test
image: node:8
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]( test.yaml )
kubectl create -f test.yamlOk, we saw
deployment.apps test created... but did it work out?kubectl describe pods test
Name: test-6f79f9f8bd-r7tbd
Namespace: default
Node: docker-for-desktop/192.168.65.3
Start Time: Sat, 10 Nov 2018 16:08:47 -0500
Labels: component=test
foo=barWonderful! Although u
test.yamlhad a single label set ( component), the resulting pod received two: componentand foo.Homework
But wait! Were we going to use this code to create a sidecar container? I warned that I’ll show you how to add a sidecar ... And now, with the knowledge and code you received: https://github.com/dowjones/k8s-webhook - experiment and explore how to make your sidecar automatically inserted. It's quite simple: you only need to prepare the correct JSON Patch, which will add an additional container in the test Deployment. Happy orchestration!
PS from translator
Read also in our blog:
- “ So what is the pod in Kubernetes? ";
- " How to ensure high availability in Kubernetes ";
- “ How does the Kubernetes scheduler actually work? ";
- “What happens in Kubernetes when starting the kubectl run?” Part 1 and Part 2 ;
- “ Understand RBAC in Kubernetes ”;
- " Our experience with Kubernetes in small projects " (video of the report, which includes an introduction to the technical device Kubernetes) ;
- “ What is a service mesh and why do I need it? ".