Diskless network boot and afterlife

Invariant December 27, 2012 at 16:29

Diskless network boot and afterlife

From the sandbox

History

Once we got (well, not ourselves ...) servers with 14 hard drives of 2 TB each. Having got rid of the hardware raid (why - a separate issue), we thought about what it would be nice to do for them to download over the network in order to get rid of fuss with partitions. The disks were supposed to be exported via iSCSI, and I did not want to allocate some disks to the Special System Disks, and some to everything else. Thus, the problem arose of making a boot over the network with the placement of the root directory in RAM.

Theory

In fact, in order for the system to boot, it needs 3 components - the kernel, the initial environment of initramfs and the root directory in which the system will work.

Practice

All actions are carried out by car with ubuntu precise.

Pxe

First, configure PXE. There are a lot of manuals on this subject, so I will tell only the essence.
We put your favorite dhcp server, for example, isc-dhcp-server, which will distribute ip addresses to the machines and indicate the path to the pxelinux.0 file that the tftp server (tftp-hpa or atftp) will give.

aptitude install isc-dhcp-server tftpd-hpa

An example of a dhcp server config. In the example, the pxe server is located at 10.0.0.1.

option domain-name-servers 8.8.8.8;
server-name "pxe";
subnet 10.0.0.0 netmask 255.255.255.0 {
        range dynamic-bootp 10.0.0.2 10.0.0.10;
        option subnet-mask 255.255.255.0;
        option routers 10.0.0.1;
        option root-path "10.0.0.1:/var/lib/tftpboot/";
        filename "pxelinux.0";
}

We start the tftp server (in ubuntu it has an init script, but it is likely that you will have to run it through inetd / xinetd as well).
We check the performance. We put the file in the directory / var / lib / tftpboot and try to pull it with the tftp client.

tftp 10.0.0.1
tftp> get pxelinux.0

In principle, it does not matter where you get the pxelinux.0 file, since it is just the bootloader into which we transfer what needs to be loaded further.
You can make a beautiful menu in the bootloader, but now we don’t need it, so my pxelinux.cfg / default looks like this

default vesamenu.c32
aprompt 1
timeout 2
label ubuntu 12.04
menu label Ubuntu precise
kernel vmlinuz
append initrd=initrd.img boot=ram rooturl=http://10.0.0.1/rootfs.squashfs ip=dhcp

rootfs

We collect the rootfs image through debootstrap, go into it and install the necessary programs. We configure the network, hostname, firewall, etc., the more settings we make, the larger the image. The main thing is not to forget to change the password to root.

mkdir -p /mnt/rootfs
debootstrap precise /mnt/rootfs/ http://mirror.yandex.ru/ubuntu/
chroot /mnt/rootfs /bin/bash
aptitude install vim 
...

With our minimum set, the system turned out to weigh 200 MB.

Initramfs

In this example, we will take the root fs image from the web server located on our network boot server, that is, at 10.0.0.1. The solution was so simple because there was a wget utility in our initramfs. In order not to pull a large amount of data over the network, we decided to compress the image. This could be done with a regular tar, but you can try squashfs, especially since usually tar is not built into initramfs, on the other hand, nothing prevents it from being added there.

Squashfs
squashfsIs a compression file system that has been included in the kernel since version 2.6.29. With its help, you can archive the directory, mount the device on the loop and read from it; to write, you must carry out the procedure of adding files to the archive. Since when you read squashfs, you read from the archive, this gives an additional load on cpu.

 mksquashfs /mnt/rootfs/ rootfs.squashfs -noappend -always-use-fragments
 du -hs rootfs.squashfs 
 92M	rootfs.squashfs

For more efficient compression, you can use the -comp option to set the type of compression, gzip is used by default.

Next, you need to teach init from initramfs to take the root image and put it in RAM.

init in initramfs is a sh script that parses options from cmdline, mounts fs, makes switch_root and starts the main init process of the system.
We will use this and add our options for cmdline. We write a ram script that will be called when the boot = ram option is set.

vim / usr / share / initramfs-tools / scripts / ram

#!/bin/bash
retry_nr=0
do_rammount()
{
        log_begin_msg "Configuring networking"
        configure_networking
        log_end_msg
        log_begin_msg "Downloading rootfs image"
        mkdir -p /tmp/squashfs
        wget ${rooturl} -O /tmp/squashfs/rootfs.squashfs
        log_end_msg
        log_begin_msg "Mounting rootfs image to /mnt/squashfs"
        mkdir -p /mnt/squashfs
        mount -t squashfs -o loop /tmp/squashfs/rootfs.squashfs /mnt/squashfs
        log_end_msg
        log_begin_msg "Mounting tmpfs and copy rootfs image"
        mkdir -p ${rootmnt}
        mount -t tmpfs -o size=1G none ${rootmnt}
        cp -r -v /mnt/squashfs/* ${rootmnt} || exit 2
        log_end_msg
        log_begin_msg "Umount squashfs"
        umount /mnt/squashfs || exit 2
        log_end_msg
}
mountroot()
{
        for x in $(cat /proc/cmdline); do
                case $x in
                rooturl=*)
                        export rooturl=${x#rooturl=}
                        ;;
                esac
        done
        log_begin_msg "Loading module squashfs"
        modprobe squashfs
        log_end_msg        # For DHCP
        modprobe af_packet
        wait_for_udev 10
        # Default delay is around 180s
        delay=${ROOTDELAY:-180}
        # loop until rammount succeeds
        do_rammount
        while [ ${retry_nr} -lt ${delay} ] && [ ! -e ${rootmnt}${init} ]; do
                log_begin_msg "Retrying rammount"
                /bin/sleep 1
                do_rammount
                retry_nr=$(( ${retry_nr} + 1 ))
                log_end_msg
        done
}

Through the rooturl parameter, you can specify where to download the root fs image from. To work with squashfs, you must load its module into the kernel. We point in /etc/initramfs-tools/initramfs.conf BOOT = ram and rebuild initramfs

mkinitramfs -o /var/lib/tftpboot/initrd.img

We turn on the machine, on which we will test, and look at what is happening. After a successful boot, we got a diskless system, which takes about 300 MB in memory, and we can write to it, but after reboot, the system will return to its original state.

In this example, we used squashfs just to compress the image, but why don’t we try to mount the root partition in squashfs and see what happens? We change our script, in do_rammount () function we leave only mounting squashfs.

do_rammount()
{
        log_begin_msg "Configuring networking"
        configure_networking
        log_end_msg
        log_begin_msg "Downloading rootfs image"
        mkdir -p /tmp/squashfs
        wget ${rooturl} -O /tmp/squashfs/rootfs.squashfs
        log_end_msg
        log_begin_msg "Mounting rootfs image to /mnt/squashfs"
        mkdir -p /mnt/squashfs
        mount -t squashfs -o loop /tmp/squashfs/rootfs.squashfs ${rootmnt}
        log_end_msg
}

We rebuild initramfs, run, look. The system boots in ro mode, but it only takes about 180MB of memory.
In some cases, mounting in ro mode is good, but it does not suit us, but we also do not want to waste RAM just like that. The solution was found with the help of Aufs.

Aufs
Aufs allows you to cascade-mount file systems - one in read-only mode and one in rw. It works in copy-on-write mode, that is, all changes are written to the rw system and after that reading is done from it.
We rewrite our script again.
In the mountroot () function, add

        log_begin_msg "Loading module aufs"
        modprobe aufs
        log_end_msg

And the do_rammount () function is reduced to the following form:

do_rammount()
{
        log_begin_msg "Configuring networking"
        configure_networking
        log_end_msg
        log_begin_msg "Downloading rootfs image"
        mkdir -p /tmp/squashfs
        wget ${rooturl} -O /tmp/squashfs/rootfs.squashfs
        log_end_msg
        log_begin_msg "Mounting rootfs image to /mnt/ro"
        mkdir -p /mnt/ro
        mount -t squashfs -o loop /tmp/squashfs/rootfs.squashfs /mnt/ro
        log_end_msg
        log_begin_msg "Mounting tmpfs to /mnt/rw"
        mkdir -p /mnt/rw
        mount -t tmpfs -o size=1G none /mnt/rw
        log_end_msg
        log_begin_msg "Mounting aufs to /mnt/aufs"
        mkdir -p /mnt/aufs
        mount -t aufs -o dirs=/mnt/rw=rw:/mnt/ro=ro aufs /mnt/aufs
        log_end_msg
        [ -d /mnt/aufs/mnt/ro ] || mkdir -p /mnt/aufs/mnt/ro
        [ -d /mnt/aufs/mnt/rw ] || mkdir -p /mnt/aufs/mnt/rw
        mount --move /mnt/ro /mnt/aufs/mnt/ro #сдвигаем точку squashfs монтирования в aufs
        mount --move /mnt/rw /mnt/aufs/mnt/rw #сдвигаем точку монтирования tmpfs в aufs 
        mount --move /mnt/aufs ${rootmnt} #сдвигаем точку монтирования aufs в ${rootmnt}
}

We rebuild initramfs, run, look. The system takes up 181MB of memory, while we can change it, write, read. All changes are stored separately in / mnt / rw, and the system itself is stored in / mnt / ro.

As a result, we got a system that is loaded on the network, occupies a small amount of memory, and after each reboot all changes disappear (therefore, you must collect all the necessary products of the system’s vital functions in a safe place).

All of the above methods have the right to life. I hope that this information is useful to you, but it will be interesting for me to read / listen to your comments.
Thanks for attention.

References

Ubuntu boot to ram
Squashfs home page
PXE

Tags:

Diskless network boot and afterlife