February 9, 2018 at 10:54Launch a full cluster on Kubernetes from scratch on Ubuntu 16.04
Quite a lot of articles have already been written on installing and running Kubernetes , however, not everything is so smooth (I spent several days launching my cluster).
This article is intended to provide comprehensive information not only on installing k8s, but also to explain each step: why and why we do exactly as it is written (this is very important for a successful launch).
Servers: The
cluster implies that you have more than one physical server, between which resources will be distributed. Servers are called nodes.
Drives:
Normal k8s hard drives are not supported. Work with disks is done using distributed file storage facilities. This is necessary so that k8s can "move" docker containers to other nodes if necessary, without losing data (files).
To start creating a cluster, you need to create your own distributed file storage. If you are sure that you will never need disks, then you can skip this step.
I chose Ceph . I also recommend reading this wonderful article .
The minimum reasonable number of servers for Ceph is 3 (you can build on one, but this makes little sense because of the high probability of losing data).
Network:
We need Flannel - it allows you to organize a software-defined network (Software Defined Network, SDN). It is SDN that allows all of our containers to communicate with each other within the cluster (Flannel installation is performed together with k8s and is described below).
In our example, we use 3 physical servers. Install Ubuntu 16.04 on all servers. Do not create swap partitions (k8s requirement).
Provide at least one disk (or partition) for Ceph in each server.
Do not enable SELinux support (in Ubuntu 16.04 it is turned off by default).
We named the server like this: kub01 kub02 kub03. The sda2 partition on each server is created for Ceph (formatting is optional).
I will describe Ceph installation quite briefly. There are many examples online and the Ceph site itself has pretty good documentation.
All operations are performed from under the privileged root user.
Create a temporary directory:
Install Ceph:
You must create a key and decompose it across all servers. This is needed for the ceph-deploy utility:
(you may need to fix the ssh config to allow login as root).
Check that kub01 is not registered in your / etc / hosts as 127.0.0.1 (if registered, delete this line).
Create a disk cluster and initialize it:
Checking our disk cluster:
The following useful commands can be adopted:
Now that we have verified that Ceph is working, we will create a separate pool for k8s:
(you can see all existing pools with the command: ceph df)
Now we will create a separate user for our kube pool and save the keys:
(keys will be needed to access k8s to the repository)
Add the k8s repository to our system:
Now install the main packages:
Initialize and run k8s
(just such a network 10.244.0.0/16 is necessary for flannel to work - do not change it)
Save the command printed by the script to join the nodes to the cluster.
It is convenient to use a separate unprivileged user to work with k8s. Create it and copy the k8s configuration file into it:
To work with k8s, use the utility: kubectl . We use it only from under our user kube . To go under the user, do:
Allow the launch of containers on the wizard:
Configure the rights:
Install flannel (network subsystem):
Create a user to access the web interface:
Run kube-proxy, you can do it like this:
And forward port 8001 from your working machine to the kub01 server:
Now we can go to the web interface from our working machine at:
http://127.0.0.1:8001/ui
(the web interface opens, where you need to specify the token)
You can get the token for access to the web interface like this:
The current controller kube-controller-manager-amd64: v1.9.2 is not binary rbd , needed to work with Ceph , so create your own kube-controller-manager:
To create our kube-controller-manager do the following:
(we execute all commands from under the root user)
(be sure to indicate the current version of k8s and the OS distribution) We
check that our controller has been successfully created:
Check that there is rbd in our image:
You should see something like: rbd: / usr / bin / rbd /usr/share/man/man8/rbd.8.gz
Replace the standard controller with ours, for this we edit the file:
/ etc / kubernetes / manifests / kube-controller- manager.yaml
Replace the line:
image: gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2
with:
image: my-kube-controller-manager: v1.9.2
imagePullPolicy: IfNotPresent
(be sure to add the imagePullPolicy directive so that k8s did not try to download this image from the Internet)
We go under the user kube and wait until our controller starts up (nothing needs to be done).
You should see that our image is being used:
Image: my-kube-controller-manager: v1.9.2
Now that our RBD -enabled controller has started, we can begin to configure the connection between k8s and Ceph.
Add keys to k8s to access Ceph:
Create StorageClass (default):
On the new server, run the following commands:
Connect the node to the master:
We need a key. You can get it on the wizard by running the command:
Or, create it:
An example of a command to join (run on a new node):
Helm was invented for quick and easy installation of applications in k8s.
A list of available applications can be found here .
Install and initialize Helm:
PS: it took 6 hours to write this article. Do not judge strictly if there are typos somewhere. Ask questions, I will be happy to answer and help.
This article is intended to provide comprehensive information not only on installing k8s, but also to explain each step: why and why we do exactly as it is written (this is very important for a successful launch).
What you need to know
Servers: The
cluster implies that you have more than one physical server, between which resources will be distributed. Servers are called nodes.
Drives:
Normal k8s hard drives are not supported. Work with disks is done using distributed file storage facilities. This is necessary so that k8s can "move" docker containers to other nodes if necessary, without losing data (files).
To start creating a cluster, you need to create your own distributed file storage. If you are sure that you will never need disks, then you can skip this step.
I chose Ceph . I also recommend reading this wonderful article .
The minimum reasonable number of servers for Ceph is 3 (you can build on one, but this makes little sense because of the high probability of losing data).
Network:
We need Flannel - it allows you to organize a software-defined network (Software Defined Network, SDN). It is SDN that allows all of our containers to communicate with each other within the cluster (Flannel installation is performed together with k8s and is described below).
Server Preparation
In our example, we use 3 physical servers. Install Ubuntu 16.04 on all servers. Do not create swap partitions (k8s requirement).
Provide at least one disk (or partition) for Ceph in each server.
Do not enable SELinux support (in Ubuntu 16.04 it is turned off by default).
We named the server like this: kub01 kub02 kub03. The sda2 partition on each server is created for Ceph (formatting is optional).
Install and configure Ceph
I will describe Ceph installation quite briefly. There are many examples online and the Ceph site itself has pretty good documentation.
All operations are performed from under the privileged root user.
Create a temporary directory:
mkdir ~/ceph-admin
cd ~/ceph-admin
Install Ceph:
apt install ceph-deploy ceph-common
You must create a key and decompose it across all servers. This is needed for the ceph-deploy utility:
ssh-keygen
ssh-copy-id kub01
ssh-copy-id kub02
ssh-copy-id kub03
(you may need to fix the ssh config to allow login as root).
Check that kub01 is not registered in your / etc / hosts as 127.0.0.1 (if registered, delete this line).
Create a disk cluster and initialize it:
ceph-deploy new kub01 kub02 kub03
ceph-deploy install kub01 kub02 kub03
ceph-deploy mon create-initial
ceph-deploy osd prepare kub01:sda2 kub02:sda2 kub03:sda2
ceph-deploy osd activate kub01:sda2 kub02:sda2 kub03:sda2
Checking our disk cluster:
ceph -s
cluster 363a4cd8-4cb3-4955-96b2-73da72b63cf5
health HEALTH_OK
The following useful commands can be adopted:
ceph -s
ceph df
ceph osd tree
Now that we have verified that Ceph is working, we will create a separate pool for k8s:
ceph osd pool create kube 100 100
(you can see all existing pools with the command: ceph df)
Now we will create a separate user for our kube pool and save the keys:
ceph auth get-or-create client.kube mon 'allow r' osd 'allow rwx pool=kube'
ceph auth get-key client.admin > /etc/ceph/client.admin
ceph auth get-key client.kube > /etc/ceph/client.kube
(keys will be needed to access k8s to the repository)
Install Kubernetes
Add the k8s repository to our system:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat </etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
Now install the main packages:
apt update
apt install -y docker.io kubelet kubeadm kubernetes-cni
Initialize and run k8s
kubeadm init --pod-network-cidr=10.244.0.0/16
(just such a network 10.244.0.0/16 is necessary for flannel to work - do not change it)
Save the command printed by the script to join the nodes to the cluster.
It is convenient to use a separate unprivileged user to work with k8s. Create it and copy the k8s configuration file into it:
useradd -s /bin/bash -m kube
mkdir ~kube/.kube
cp /etc/kubernetes/admin.conf ~kube/.kube/config
chown kube: ~kube/.kube/config
To work with k8s, use the utility: kubectl . We use it only from under our user kube . To go under the user, do:
su - kube
Allow the launch of containers on the wizard:
kubectl taint nodes --all node-role.kubernetes.io/master-
Configure the rights:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
Install flannel (network subsystem):
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
How to check that everything works
Run the command:
The output should be something like this:
kubectl -n kube-system get pods
The output should be something like this:
NAME READY STATUS RESTARTS AGE
etcd-kub01.domain.com 1/1 Running 1 4d
kube-apiserver-kub01.domain.com 1/1 Running 1 4d
kube-controller-manager-kub01.domain.com 1/1 Running 0 4d
kube-dns-7c6d8859cb-dmqrn 3/3 Running 0 1d
kube-flannel-ds-j948h 1/1 Running 0 1d
kube-proxy-rmbqq 1/1 Running 0 1d
kube-scheduler-kub01.domain.com 1/1 Running 1 4d
Install and configure the web interface
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
Create a user to access the web interface:
cat << EOF > account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
EOF
kubectl -n kube-system create -f account.yaml
Run kube-proxy, you can do it like this:
kubectl proxy &
And forward port 8001 from your working machine to the kub01 server:
ssh -L 8001:127.0.0.1:8001 -N kub01 &
Now we can go to the web interface from our working machine at:
http://127.0.0.1:8001/ui
(the web interface opens, where you need to specify the token)
You can get the token for access to the web interface like this:
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Customize Kubernetes and Ceph
The current controller kube-controller-manager-amd64: v1.9.2 is not binary rbd , needed to work with Ceph , so create your own kube-controller-manager:
What is RBD, why is it needed and why is it missing?
RBD (Rados Block Device) is a block device that k8s uses to create and mount partitions of Docker containers. In this case, this is the binary included in the package: ceph-common .
k8s does not include this package in its controller, apparently because it depends on the distribution of the operating system that you are using. Therefore, when assembling your controller, be sure to specify your distribution to keep RBD up to date.
k8s does not include this package in its controller, apparently because it depends on the distribution of the operating system that you are using. Therefore, when assembling your controller, be sure to specify your distribution to keep RBD up to date.
To create our kube-controller-manager do the following:
(we execute all commands from under the root user)
mkdir docker
cat << EOF > docker/Dockerfile
FROM ubuntu:16.04
ARG KUBERNETES_VERSION=v1.9.2
ENV DEBIAN_FRONTEND=noninteractive \
container=docker \
KUBERNETES_DOWNLOAD_ROOT=https://storage.googleapis.com/kubernetes-release/release/${KUBERNETES_VERSION}/bin/linux/amd64 \
KUBERNETES_COMPONENT=kube-controller-manager
RUN set -x \
&& apt-get update \
&& apt-get install -y \
ceph-common \
curl \
&& curl -L ${KUBERNETES_DOWNLOAD_ROOT}/${KUBERNETES_COMPONENT} -o /usr/bin/${KUBERNETES_COMPONENT} \
&& chmod +x /usr/bin/${KUBERNETES_COMPONENT} \
&& apt-get purge -y --auto-remove \
curl \
&& rm -rf /var/lib/apt/lists/*
EOF
docker build -t "my-kube-controller-manager:v1.9.2" docker/
(be sure to indicate the current version of k8s and the OS distribution) We
check that our controller has been successfully created:
docker images | grep my-kube-controller-manager
Check that there is rbd in our image:
docker run my-kube-controller-manager:v1.9.2 whereis rbd
You should see something like: rbd: / usr / bin / rbd /usr/share/man/man8/rbd.8.gz
Replace the standard controller with ours, for this we edit the file:
/ etc / kubernetes / manifests / kube-controller- manager.yaml
Replace the line:
image: gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2
with:
image: my-kube-controller-manager: v1.9.2
imagePullPolicy: IfNotPresent
(be sure to add the imagePullPolicy directive so that k8s did not try to download this image from the Internet)
We go under the user kube and wait until our controller starts up (nothing needs to be done).
kubectl -n kube-system describe pods | grep kube-controller
You should see that our image is being used:
Image: my-kube-controller-manager: v1.9.2
Now that our RBD -enabled controller has started, we can begin to configure the connection between k8s and Ceph.
Configuring a bunch of disk subsystems (k8s + Ceph)
Add keys to k8s to access Ceph:
kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-file=/etc/ceph/client.admin --namespace=kube-system
kubectl create secret generic ceph-secret-kube --type="kubernetes.io/rbd" --from-file=/etc/ceph/client.kube --namespace=default
Create StorageClass (default):
cat << EOF > ceph_storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd
annotations: {"storageclass.kubernetes.io/is-default-class":"true"}
provisioner: kubernetes.io/rbd
parameters:
monitors: kub01:6789,kub02:6789,kub03:6789
pool: kube
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: "kube-system"
userId: kube
userSecretName: ceph-secret-kube
fsType: ext4
imageFormat: "2"
imageFeatures: "layering"
EOF
kubectl create -f ceph_storage.yaml
How to check the operation of the disk subsystem?
Checking the presence of StorageClass:
Create a test pod with a disk:
Check that the Pod is created (you need to wait for the creation and launch):
Check that Claim has been created (a request for a drive):
Check that the disk itself is created:
And finally, check that the disk is mounted on the system:
kube@kub01:~$ kubectl get storageclass
NAME PROVISIONER AGE
ceph-rbd (default) kubernetes.io/rbd 4d
Create a test pod with a disk:
cat << EOF > test_pod.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: claim1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: test-pod-with-pvc
spec:
volumes:
- name: test-pvc-storage
persistentVolumeClaim:
claimName: claim1
containers:
- name: test-container
image: kubernetes/pause
volumeMounts:
- name: test-pvc-storage
mountPath: /var/lib/www/html
EOF
kubectl create -f test_pod.yaml
Check that the Pod is created (you need to wait for the creation and launch):
kube@kub01:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod-with-pvc 1/1 Running 0 15m
Check that Claim has been created (a request for a drive):
kube@kub01:~$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
claim1 Bound pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b 1Gi RWO ceph-rbd 12m
Check that the disk itself is created:
kube@kub01:~$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b 1Gi RWO Delete Bound default/claim1 ceph-rbd
And finally, check that the disk is mounted on the system:
root@kub01:~$ mount | grep pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b
/dev/rbd0 on /var/lib/kubelet/pods/076fff13-0ce9-11e8-8b93-901b0e8fc39b/volumes/kubernetes.io~rbd/pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b type ext4 (rw,relatime,stripe=1024,data=ordered)
Adding new (additional) nodes to the k8s cluster
On the new server, run the following commands:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat </etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt update
apt install -y docker.io kubelet kubeadm kubernetes-cni ceph-common python
Connect the node to the master:
We need a key. You can get it on the wizard by running the command:
kubeadm token list
Or, create it:
kubeadm token create --print-join-command
An example of a command to join (run on a new node):
kubeadm join --token cb9141.6a912d1dd7f66ff5 8.8.8.8:6443 --discovery-token-ca-cert-hash sha256:f0ec6d8f9699169089c89112e0e6b5905b4e1b42db22815186240777970dc6fd
Install and configure Helm
Helm was invented for quick and easy installation of applications in k8s.
A list of available applications can be found here .
Install and initialize Helm:
curl https://storage.googleapis.com/kubernetes-helm/helm-v2.8.0-linux-amd64.tar.gz | tar -xz
./linux-amd64/helm init
PS: it took 6 hours to write this article. Do not judge strictly if there are typos somewhere. Ask questions, I will be happy to answer and help.