How to make an LXC container on a router running LEDE

  • Tutorial

From time to time, people feel like doing something strange. Why - in fact, it doesn’t matter, any argument that needs to be done that way - will be. But the main thing is the desire to do something strange. Well, since I decided to do something strange, why not describe this process.


So, how to start the LXC container with Debian not somewhere else, but on a piece of hardware running LEDE (also known as the OpenWRT fork)?


(It is assumed that the reader himself will figure out why he needs LXC on the router and is already familiar with this container technology)


Optimistic attempt


For the desired fraud, we take a piece of iron (or better yet x86 virtualka for the test) with installed LEDE and see:


root@lede:~# opkg find lxc*
lxc - 1.1.5-3 - LXC is the userspace control package for Linux Containers, a lightweight
 virtual system mechanism sometimes described as "chroot on steroids".
lxc-attach - 1.1.5-3 - Utility lxc-attach from the LXC userspace tools
lxc-auto - 1.1.5-3 - LXC is the userspace control package for Linux Containers, a lightweight
 virtual system mechanism sometimes described as "chroot on steroids".
 This package adds and initscript for starting and stopping the containers
 on boot and shutdown.
lxc-autostart - 1.1.5-3 - Utility lxc-autostart from the LXC userspace tools
lxc-cgroup - 1.1.5-3 - Utility lxc-cgroup from the LXC userspace tools
lxc-checkconfig - 1.1.5-3 - Utility lxc-checkconfig from the LXC userspace tools
lxc-clone - 1.1.5-3 - Utility lxc-clone from the LXC userspace tools
lxc-common - 1.1.5-3 - LXC common files
lxc-config - 1.1.5-3 - Utility lxc-config from the LXC userspace tools
lxc-configs - 1.1.5-3 - LXC virtual machine common config files
lxc-console - 1.1.5-3 - Utility lxc-console from the LXC userspace tools
lxc-create - 1.1.5-3 - Utility lxc-create from the LXC userspace tools
lxc-destroy - 1.1.5-3 - Utility lxc-destroy from the LXC userspace tools
lxc-device - 1.1.5-3 - Utility lxc-device from the LXC userspace tools
lxc-execute - 1.1.5-3 - Utility lxc-execute from the LXC userspace tools
lxc-freeze - 1.1.5-3 - Utility lxc-freeze from the LXC userspace tools
lxc-hooks - 1.1.5-3 - LXC virtual machine hooks
lxc-info - 1.1.5-3 - Utility lxc-info from the LXC userspace tools
lxc-init - 1.1.5-3 - LXC Lua bindings
lxc-ls - 1.1.5-3 - Utility lxc-ls from the LXC userspace tools
lxc-lua - 1.1.5-3 - LXC Lua bindings
lxc-monitor - 1.1.5-3 - Utility lxc-monitor from the LXC userspace tools
lxc-monitord - 1.1.5-3 - Utility lxc-monitord from the LXC userspace tools
lxc-snapshot - 1.1.5-3 - Utility lxc-snapshot from the LXC userspace tools
lxc-start - 1.1.5-3 - Utility lxc-start from the LXC userspace tools
lxc-stop - 1.1.5-3 - Utility lxc-stop from the LXC userspace tools
lxc-templates - 1.1.5-3 - LXC virtual machine templates
lxc-unfreeze - 1.1.5-3 - Utility lxc-unfreeze from the LXC userspace tools
lxc-unshare - 1.1.5-3 - Utility lxc-unshare from the LXC userspace tools
lxc-user-nic - 1.1.5-3 - Utility lxc-user-nic from the LXC userspace tools
lxc-usernsexec - 1.1.5-3 - Utility lxc-usernsexec from the LXC userspace tools
lxc-wait - 1.1.5-3 - Utility lxc-wait from the LXC userspace tools

Hm. It seems easier than expected. True, everything is divided into separate utilities, but oh well. We set it for a start lxc-checkconfigand see what it thinks about our core


root@lede:~# lxc-checkconfig 
Kernel configuration not found at /proc/config.gz; searching...
lxc-checkconfig: unable to retrieve kernel configuration
Try recompiling with IKCONFIG_PROC, installing the kernel headers,
or specifying the kernel configuration path with:
  CONFIG= lxc-checkconfig

Oops Doesn’t think anything. Some time later, it turns out that the default LEDE kernel is built without LXC support, and the packages in the repository are only for manual manipulation. But there are packages, and the kernel is quite fresh - which means you can rebuild the kernel and use it. So let's do it.


We deal with the assembly of LEDE


Let's go pick up the LEDE assembly short guide . We put the necessary packages under our OS. I recommend that you first start with the default config to check that the errors are caused by not enabling LXC support and you need to dig the reason separately from this note.


git clone https://git.lede-project.org/source.git build
cd build
git checkout v17.01.4
./scripts/feeds update -a
./scripts/feeds install -a

You can assemble LEDE using the official image settings for the desired hardware. To do this, go to https://downloads.lede-project.org/releases/, select the current release (at the time of writing this is January 17 ,.4), then in targets/your architecture and find the LEDE file next to the image config.seed. For example, for x86 we get the addresshttps://downloads.lede-project.org/releases/17.01.4/targets/x86/generic/config.seed


Download this file to the assembly directory under the name .configand invoke make defconfigit into a full assembly file.


wget -O .config https://downloads.lede-project.org/releases/17.01.4/targets/x86/generic/config.seed
make defconfig
time make

I had everything going for an hour and a half, ready-made images to look for in bin/targets/.


In general, makeyou can call in several threads, but I was impressed by the warning in the assembly documentation But this might expose bugs!and decided not to rush anywhere.


And pay attention to enough disk space. To occupy 20GB is quite simple.


Add LXC


A little familiar with the assembly, now it's time to add kernel options to support LXC. Many thanks to almost the only article on building OpenWRT with LXC for a starting point. But the article is old, and the options for the current LXC are not all. Add to our .configbundle of options:


CONFIG_KERNEL_AIO=y
CONFIG_KERNEL_BLK_CGROUP=y
CONFIG_KERNEL_BLK_DEV_BSG=y
CONFIG_KERNEL_CGROUPS=y
CONFIG_KERNEL_CGROUP_CPUACCT=y
CONFIG_KERNEL_CGROUP_DEVICE=y
CONFIG_KERNEL_CGROUP_FREEZER=y
CONFIG_KERNEL_CGROUP_PIDS=y
CONFIG_KERNEL_CGROUP_SCHED=y
CONFIG_KERNEL_CPUSETS=y
# CONFIG_KERNEL_DEBUG_FS is not set
# CONFIG_KERNEL_DEBUG_INFO is not set
# CONFIG_KERNEL_DEBUG_KERNEL is not set
CONFIG_KERNEL_DEVPTS_MULTIPLE_INSTANCES=y
CONFIG_KERNEL_DEVTMPFS=y
CONFIG_KERNEL_DEVTMPFS_MOUNT=y
CONFIG_KERNEL_DIRECT_IO=y
CONFIG_KERNEL_FANOTIFY=y
CONFIG_KERNEL_FHANDLE=y
CONFIG_KERNEL_FREEZER=y
CONFIG_KERNEL_IPC_NS=y
# CONFIG_KERNEL_KALLSYMS is not set
CONFIG_KERNEL_LXC_MISC=y
CONFIG_KERNEL_MEMCG=y
CONFIG_KERNEL_MEMCG_SWAP=y
CONFIG_KERNEL_MM_OWNER=y
CONFIG_KERNEL_NAMESPACES=y
CONFIG_KERNEL_NETPRIO_CGROUP=y
CONFIG_KERNEL_NET_CLS_CGROUP=y
CONFIG_KERNEL_NET_NS=y
CONFIG_KERNEL_PID_NS=y
CONFIG_KERNEL_POSIX_MQUEUE=y
CONFIG_KERNEL_PROC_PID_CPUSET=y
CONFIG_KERNEL_RESOURCE_COUNTERS=y
CONFIG_KERNEL_USER_NS=y
CONFIG_KERNEL_UTS_NS=y
CONFIG_PACKAGE_ip-tiny=y
CONFIG_PACKAGE_ipset=y
CONFIG_PACKAGE_iptables-mod-conntrack-extra=y
CONFIG_PACKAGE_iptables-mod-ipopt=y
CONFIG_PACKAGE_kmod-8021q=y
CONFIG_PACKAGE_kmod-fuse=y
CONFIG_PACKAGE_kmod-ip6tables-extra=y
CONFIG_PACKAGE_kmod-ipt-conntrack-extra=y
CONFIG_PACKAGE_kmod-ipt-ipopt=y
CONFIG_PACKAGE_kmod-ipt-ipset=y
CONFIG_PACKAGE_kmod-ipt-nat-extra=m
CONFIG_PACKAGE_kmod-ipt-nat6=m
CONFIG_PACKAGE_kmod-macvlan=y
CONFIG_PACKAGE_kmod-nf-nat6=m
CONFIG_PACKAGE_kmod-nfnetlink=y
CONFIG_PACKAGE_kmod-veth=y
CONFIG_PACKAGE_terminfo=y

In principle, all this can be found in the ordinary make menuconfig. Now it can be called, these parameters will not be lost and you can customize the assembly for yourself.


But, unfortunately, there are more options. Some are not in make menuconfigand you need to change the config of the kernel itself. In theory, the right way is to


make kernel_menuconfig

And putting down all the relevant marks there. But it’s kind of dreary to look for them, so the way is wrong: on the updated one we .configrun


make defconfig # если вы что-то поменяли в menuconfig то не нужно, но и не навредит. Наверное
make prepare

We find somewhere approximately in the build_dir/target-i386_pentium4_musl-1.1.16/linux-x86_generic/linux-4.4.92/.configgenerated kernel configuration file (the path, as you can see, differs from the heap of parameters). And add a bunch of settings to the end


CONFIG_CHECKPOINT_RESTORE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_FREEZER=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETLINK_DIAG=y
CONFIG_PACKET_DIAG=y
CONFIG_INET_DIAG=y
CONFIG_UNIX_DIAG=y
CONFIG_FHANDLE=y
CONFIG_INET_UDP_DIAG=m
# CONFIG_PROC_STRIPPED is not set
CONFIG_CFQ_GROUP_IOSCHED=y

CONFIG_IKCONFIG_PROCand CONFIG_IKCONFIGjust the thing because of which lxc-checkconfigI could not say anything. A CONFIG_PROC_STRIPPED- these are OpenWRT patches that remove some files from the pseudo-system /proc. The setup should reduce the amount of memory consumed, but the resulting warnings at lxc-start. However, LXC with a lack of memory is not needed.


After such an intervention (and it must be repeated after make prepareeach time), we do


time make

Now a bunch of everything has already been compiled and will not be recompiled, so it works much faster. In about 15 minutes I’ve got an update.


We load the received image on a piece of iron or a virtualka and we try to execute lxc-checkconfig. You should see a neat picture like this:


root@lede:~# lxc-checkconfig 
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled
--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Great, here is the core and done.


Container preparation


Now we put a small pack of packages (in fact, you can roll them right into the image that is going to make, but I did not understand it)


opkg install lxc-start lxc-attach lxc-stop lxc-ls lxc-monitor lxc-monitord lxc-checkconfig

I prefer not to use templates from lxc-templates- my scripts have been around since openvz, and the LXC config is pretty typical. But I checked the debian assembly in LEDE for the article, more packages are needed:


opkg install lxc-create lxc-templates debootstrap bash getopt rsync

Then a surprise, out of the box it still doesn't work. /var/cacheexists in LEDE as nodev and debootstrap are not allowed there. Therefore, we do instead /var/cacheof somewhere simlink.


mkdir /mnt/cache
ln -s /mnt/cache /var/cache
lxc-create -n testlxc -t debian -- -r jessie

So, container rootfs is ready. Let's try to run:


lxc-start -n testlxc -d

There was a problem with the task of connecting to the container. lxc-consolecomplains about bad file descriptor. But the container itself is up and running. Well, they wanted a strange, a strange received. We do:


lxc-attach -n testlxc -- bash

And here we are with console access inside the container. What lxc-consolewent wrong did not work out, but uncritically, left it that way.


By default, you can place the directory with containers anywhere, if only the /etc/lxc/lxc.confpath to this directory was indicated.


Autostart containers


The last detail remains: configure the autostart of the container (s) with the system. Install the packagelxc-auto


opkg install lxc-auto

And we get the init script and the reference configuration file /etc/config/lxc-auto. We bring it to the desired form from the required number of transfers of container names:


config container
    option name testlxc

And we send the system to reboot for verification. The init script unfortunately turns out to be rather stupid: it calls lxc-stop &all containers from the config and goes to sleep by default for 300 seconds. So even if the containers turned off quickly, the script will still sleep.


PS: first I tried to run LEDE in the LXC container on Debian. After some time of the proceedings with gdb, this was even possible, but on the task it was normal to start firewallstalled from the regular configs and went from the opposite.


Also popular now: