Thin diskless client based on Ubuntu, which does not require mounting FS over the network

Image from getwallpapers.com

History

Back in 2013, one bank used thin clients based on DisklessUbuntu . There were some problems with them, in my opinion, mounting the root FS over the network in large branches with a weak network did not work very well. Then my good friend @deadroot made the first version of the thin client, which was loaded entirely in memory, without requiring anything to be mounted on the network to work.

Then I actively finished this client, there were made a lot of useful things specific to our use case. Then the bank closed (the license was revoked), the rest of the client’s source code moved to my github: thunclient . A couple of times I lightly finished it to order.

Recently, I got a hand on how to make this heap of scary unreliable scripts a solution quite convenient for use:

• Vagrant raises a virtual machine that can be configured as a regular workstation.
• In one script, files that are ready for downloading over the network are collected from it, excess is cut out.
• Vagrant raises a virtual PXE server and network client to verify the resulting build.

What can

• Entirely loaded into memory, it does not require mounting the root FS over the network for operation.
• Built on the basis of Ubuntu, almost any software can be installed from its rich repositories, and connect third-party if something was missing. It’s especially nice that security updates arrive quickly in Ubuntu.
• Able to mount additional overlays on top of the root FS. You can add some software only for some workstations without collecting a new image
• Able to zram - memory compression, useful for older clients with a small amount of RAM. Although for new ones it usually doesn’t hurt.
• An easy desktop (LXDE) with an RDP client is assembled from the box, the addresses and parameters of the RDP servers are simply transmitted from the PXE server through the parameters at boot time.
• You can change one parameter in the config and the minimal console system will be assembled without unnecessary software - the basis for any of your custom builds.
• If the download fails due to problems with the server or the network, it will not display an error message for long and try to boot again. Conveniently, when the problems are fixed, the workstations will rise themselves without unnecessary gestures.

In the bank, VNC was used x11vncto connect remotely to the user's thin client ( to connect to an already running Xorg session). Not everyone needs it (usually the ability to connect to an RDP session on a terminal server is enough), and here everything is very individual in terms of convenience / security requirements. Therefore, I did not spread this part.

Analogs

• Thinstation

If Thinstation is completely satisfied, then it is better to use it, this is an older and more mature project. Plus, it’s one and a half times smaller in size, after all, it’s an assembly specially sharpened for the minimum volume, and not slightly doped with ordinary Ubuntu.

But the software versions in it are quite ancient and there are few of them. If you need something additional, in addition to RDP / Citrix / ... clients, you will need to collect it by hand, and so with each update.

• Ltsp

kvaps pointed out in a comment that LTSP can copy the squashfs image to memory and work without mounting the file system over the network: this is configured by the LTSP_NBD_TO_RAM variable . Chroot is used for customization, which may be less convenient, especially for customizing the graphical environment and applications. Also a good mature project, can be considered as an alternative.

Vagrant vs chroot

Previous versions used chroot, like most similar projects, the same Thinstation, for example. This is not difficult, but nevertheless, a separate program launched in chroot does not correspond to what is happening on a real machine: there is no interaction with system init, with other programs and services. Plus, Vagrant made the process of creating a client as simple as possible: the virtual machine is configured like a regular machine.

Of course, using Vagrant brings some difficulties.

A machine must be running on the machine virtualbox-guest-utilsfor shared folders to work. In addition, you need a boot manager ( grub), which is required for a machine with a disk and useless for a client downloaded over a network. I solved these problems by excluding from the assembly all files of these packages. Therefore, they do not affect the size of the resulting image.

In addition, Vagrant is required to run on the ssh machine, starting up the user with the generated key. I exclude from the assembly the home directory of the vagrant user used for configuration, along with its ssh keys. The keys for the ubuntu user used during the work can be put in his home directory.

Well, for work, Vagrant generates network interface settings that will be erroneous for a real machine. I had to replace the file at the time of assembly interfaces, and write a script that generates a config on the real machine to configure all available interfaces via DHCP.

Provisioning is done using Ansible. This is a very convenient tool for configuring all kinds of software and hardware. But I would not want to include Ansible and the second python with the necessary libraries in the final image: useless ballast. You don’t feel like putting Ansible on a machine where a virtual environment is flocking: this will complicate the work.

Vagrant allows you to do a trick: put Ansible on one machine (PXE test server), and do the deployment of other machines from it, within the same playbook. For this, machines must have a static IP in order to register it in ansible inventory. Well, we solved the problem with the configuration of the interfaces in the last paragraph.

Naughty zucchini

Squashfs is a read-only compression file system. Underlies most existing Linux LiveCDs. It is it that allows you to create a fairly compact image of the system that fits in the RAM of a thin client.

From the final image should be cut a lot of things: /tmp, /run, /proc, /sys, /usr/share/docand so on.

The utility mksquashfssupports as many as 3 types of lists for excluding files: by the full path, by mask, and by regular expression. It would seem that everything is fine. But the last two options do not support paths starting with /. I could not exclude all the files inside some folder structure, not excluding the last folder.

I was quickly tired of fighting it, I just found findall the files and folders that need to be deleted, and stuffed it into one large file with exceptions along the full path. Crutches.jpg. But it works. The only artifact of this approach in the final image is the lonely folder /proc/NNNcorresponding to the process number mksquashfs, which was not there when creating the exception list. From above, procfs is still mounted.

Magic initrd

In order not to pull in the kernel all the necessary drivers and the logic for mounting the root FS, Linux uses initial ramdisk. Previously, the initrd format was used, in which this disk was a real image of the file system. A new format appeared in kernel 2.6 - initramfs, which is a cpio archive extracted to tmpfs. Both initrd and initramfs can be compressed to save load time. Many utility names and file names still mention initrd, although it is no longer in use.

Debian / Ubuntu uses the initramfs-tools package to create initramfs. It provides the following options for customization:

• hooks - scripts of a special format that allow you to add kernel modules and executable files with all the libraries they need to the image
• scripts directory init-bottom, init-premount, init-top, local-block, local-bottom, local-premount, local-top, carried out at the appropriate moment load. See man initramfs-tools (8)
• the most interesting thing is to add your own boot scripts that are responsible for mounting the root FS. They must define a shell function mountroot()that will be used by the main script /init. The structure already has localto mount the root on the local disk and nfsto mount the root over the network. The script used is selected by the download parameter boot.

In total, in order to mount the root FS in some very tricky way, you need to create your boot script, define a function in it mountroot(), pass the name of this script in the boot parameter, bootand remember to write hooks that tighten all the programs and kernel modules needed in the script to initramfs.

Overlay fight

OverlayFS is used to create a single root file system from several . In the first versions, AUFS was used (it is used by most Linux LiveCDs). But it was not accepted into the kernel, and now everyone is recommended to switch to OverlayFS.

After mounting the real root FS into the directory inside initramfs, the program run_initfrom the composition will be launched klibc-utils. She will verify that the root FS is mounted inside initramfs, clean up the initramfs (why should I lose memory?) And move the mount point of the root FS to /, and run system init. Details . This program is compiled as a separate executable file because a script using any external utilities will break after clearing initramfs.

If the root FS is assembled from several overlays mounted inside initramfs, run_initthese mount points disappear during operation and it breaks. This problem can be solved by moving the mount points of the overlays inside the root FS, where they no longer disappear. Recursion :) do so: mount --move olddir newdir.

AppArmor had to be disabled: its profiles are designed to directly mount the root FS from a single device. When using OverlayFS, she sees that /sbin/dhclientthis is in fact /AUFS/root/sbin/dhclient, and the profile breaks. The only option to use it is to rewrite all profiles for all applications, and update if necessary.

Where do I need recording

Under the idea, Linux can work quietly when all the FSs are mounted read-only. But many programs rely on the ability to write to disk, you have to mount tmpfs there:

• /tmp, /var/tmp- of course, very many need
• /var/log - write logs
• /run - almost all services will not start without it
• /media - mount connected media
• /var/lib/system - used by many programs from systemd, in particular systemd-timesyncd
• /var/lib/dhclient - here dhclient writes information about leases
• /etc/apparmor.d/cache- if you still beat AppArmor, then he will need to write files to /etc. IMHO disgusting, for such things are /var.

Total

If you want to build a Ubuntu build that is loaded on a network and works only from memory, there is a ready-made convenient constructor here: thinclient . If you need help, write to the PM or to tickets on the github, I'll tell you.

PS The English version of the article on my blog .