Run the kubernetes inside the LXC

Original author: kvaps
  • Transfer

In this article I will tell how you can run the kubernetes master inside the LXC container.
This method works quite well with Proxmox and can be considered as an alternative to the classic deployment of kubernetes with several wizards.

Why proxmox?

Because proxmox out of the box provides many useful features.
Proxmox includes high availability features, migration, automatic backups, access control, and everything is available through a simple graphical interface.

You can also deploy the simplest configuration with one master kubernetes, high availability will be ensured by proxmox itself.
This method is perfect for deploying test clusters or small clusters.

For larger deployments, it is recommended that you put etcd into separate containers that will use fast local disks and cluster them.
Nevertheless, the kubernetes-master itself can still remain an ordinary ha-container, it does not need to have fast storage.

In addition, if you are reading this article, I assume that in most cases you already have some kind of infrastructure on Proxmox, and perhaps you want to have a single interface for managing your services.

Why LXC?

Kubernetes will run without problems inside a normal virtual machine. But LXC containers provide the flexibility that is not available when using conventional virtual machines.

In fact, LXC containers do not provide complete isolation of containers from the host; on the contrary, all processes inside containers are started as normal host processes, just in a separate namespace for them.

This method gives you good performance but imposes some limitations in our case.

About these restrictions and how to cope with them I will discuss in this article.


Since, by default, containers are not allowed to load kernel modules themselves, you must configure them to load directly on hypervisors.

We will use the overlaydriver for docker, so this is all we need:

echo overlay >> /etc/modules

Now we need to add more privileges for our container in order to allow it to run other containers inside, add these lines to your container config:

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a proc:rw sys:rw

Starting with version v11.0, kubelet requires shared mode for all mounts from the host.

This dirty hack will allow you to achieve this, inside the LXC container, run:

echo '#!/bin/sh -e
mount --make-rshared /' > /etc/rc.local

This action will add a command mount --make-rshared /to /etc/rc.localand will launch it each time the container is loaded.

Also, if you plan to use HA-manager in proxmox, know that at the moment there is an unpleasant bug # 1842 , which forcibly kills container processes during migration, which can spawn zombie processes or even block your storage.

This is not good, fortunately there is a simple solution:

sed -i 's/forceStop => 1/forceStop => 0/' /usr/share/perl5/PVE/HA/Resources/

In addition, you can add the following options for your docker:

--storage-driver overlay2

Copy docker.servicefrom /libto /etcto override its parameters:

cp /{lib,etc}/systemd/system/docker.service

Now add these options to the ExecStartsection.

That's all, after these steps, the standard kubeadm installation should work without problems.

Also popular now: