August 8, 2013 at 09:02Co-launch of Linux and baremetal OS
Recently, I posted on the network under a BSD license a small project for 8 kilo lines of C code. Officially, this is a collection of benchmarks for my clients - vendors of industrial automation. The code is very specific, and, at first glance, hardly applicable outside the narrow area of PLC and motion control . But there is a small highlight, which I did not really focus on in the article on the IDZ . The delivery of benchmarks included a baremetal environment for their execution. In this post I will describe what it is and how it can be used.
What is baremetal OS? This is a small code that initializes the kernel / kernels and transfers control to the user program, which for some reason needs to be run without a normal OS and even without an RTOS. Naturally, in the code of this program you have to statically link libraries and drivers for all the devices it needs ... This spring on Habré there was already a series of articles on the creation of baremetal OS. Therefore, I will not repeat myself and describe the loading procedures. However, for a complete understanding of this post, it is advisable to re-read two key articles of that series. In this explains how to assemble, download and run the free program instead of running directly from GRUB. And here is how to run your code on multiple cores at once. I can also recommend English-language primary sources - osdev and the most famous examples - tutorials from Bran and James Molloy .
In the second articledescribes in detail how to run the initialization code for a new kernel. To do this, send Init IPI (Inter-Processor-Interrupt) and then two more Startup IPI. IPI forwarding is initiated by simply writing a few bytes to a specific physical address. But what happens if you write a small Linux user mode bootloader, a program that copies a binary to a specific address and sends the Init IPI, 2xSIPI, to the other kernel, so that it boots into this binary?
Do you think there will be kernel panic? After all, on this kernel Linux already executes user processes, idle process or interruption. In fact, everything will not fall at once, but the system will begin to work unstably and will soon freeze. But what happens if you boot Linux with only one kernel, or off-line the other kernels on a running system? (For example, like this:
And repeat for every core except 0.
In this case, you can not be afraid of competition with the task scheduler. But all the available physical physical memory Linux mm at boot has already cut into bucket'y and rightly considers his own. Fortunately, there is a work round here. Even two - you can write a kernel module that politely asks for a bit of continuous physical memory, change the linker script to put the .text and .data areas of the binary into this memory. And you can make it even simpler - ask Linux to reserve a bit of physical memory at a specific address at boot time. For example, when loading the kernel with the option memmap = 0x80M $ 0x18000000, the OS will allocate 128 megabytes of contiguous physical memory starting at address 0x18000000. Unfortunately, Linux kernels from 3.1 sometimes crash at the very beginning of the boot with this option. Now I'm just trying to find the reason.
Voilà, now we can keep Linux on several kernels, and arbitrary baremetal programs on everyone else. But how can this come in handy? Suppose we already have a task to run some code without an OS. For example, I needed benchmarks running on different x86 platforms and measuring the spread between the average code execution time and the worst time. x86 - generally not a model of determinism, and if you also add the operating system ... Even if it is a compact and nimble RTOS, it is still not immediately obvious where latency and jitter come from. In addition, each of my clients has its own RTOS, some are slightly more difficult than baremetal. Or if there is a code from the microcontroller that is sensitive to delays, which already works without the OS, and you just need to port-throw it on one of the x86 processor cores. If you simply turn it on with the kernel thread on the kernel and disable everything else on it, it will still be interrupted. With the RT-Linux patch, everything works better, but it adds latency and is sometimes not compatible with other useful patches.
And why is Linux and baremetal together better than a clean baremetal image loaded by a bootloader (GRUB)? And with Linux it’s more convenient! Compare rebooting the entire piece of hardware or just a couple of shell commands on the Linux host command line. Similarly with debugging - with Linux you can read-write memory baremetal OS perfectly, diagnose glands in the process. (Of course, if careful!) There are some disadvantages to this approach. To run baremetal OS with Linux, you need to know the ACPI ID of the kernel to which the IPI is sent. If you think that the first core has number 0, the second has 1, etc., then you are an optimist. I saw very different options :). You will either have to enumerate, this is described in one of the posts to which I referred, or select options. (And don't forget to [turn off] Hyperthreading!). Another possible ambush is non-determinism. If you disable all kernels in Linux, except for the first one, and there is nothing serious to run on it, it is impossible to measure the impact of Linux on the performance of kernels that run baremetal code. But there is still no 100% guarantee - if you wish or by chance, you can greatly affect the performance of baremetal code through shared resources (cache, memory controller). For example, try switching VGA console (Ctrl-Alt-F1, Ctrl-Alt-F2), measuring the performance of baremetal OS.
If anyone is interested in how it all works, or if there are ideas on how to practically use it (BSD license allows) - download the source from IDZ or github (it’s fresh there). Baremetal OS itself lies in tools / src / baremetal. In the first directory - phymem lies a tool for reading and writing physical memory. And in the second - smallos - actually the runtime. Everything there is designed to run benchmarks, so you have to use a file to cut them off - just throw out the links to bench / *. O from smallos / Makefile and replace the code for calling benchmarks in smallos / apps / mwaitTester.c with your code. Linux usermode bootloader lies in smallos / linux_boot. Good luck
What is baremetal OS? This is a small code that initializes the kernel / kernels and transfers control to the user program, which for some reason needs to be run without a normal OS and even without an RTOS. Naturally, in the code of this program you have to statically link libraries and drivers for all the devices it needs ... This spring on Habré there was already a series of articles on the creation of baremetal OS. Therefore, I will not repeat myself and describe the loading procedures. However, for a complete understanding of this post, it is advisable to re-read two key articles of that series. In this explains how to assemble, download and run the free program instead of running directly from GRUB. And here is how to run your code on multiple cores at once. I can also recommend English-language primary sources - osdev and the most famous examples - tutorials from Bran and James Molloy .
In the second articledescribes in detail how to run the initialization code for a new kernel. To do this, send Init IPI (Inter-Processor-Interrupt) and then two more Startup IPI. IPI forwarding is initiated by simply writing a few bytes to a specific physical address. But what happens if you write a small Linux user mode bootloader, a program that copies a binary to a specific address and sends the Init IPI, 2xSIPI, to the other kernel, so that it boots into this binary?
Do you think there will be kernel panic? After all, on this kernel Linux already executes user processes, idle process or interruption. In fact, everything will not fall at once, but the system will begin to work unstably and will soon freeze. But what happens if you boot Linux with only one kernel, or off-line the other kernels on a running system? (For example, like this:
$ sudo echo 0 > /sys/devices/system/cpu/cpu1/onlineAnd repeat for every core except 0.
In this case, you can not be afraid of competition with the task scheduler. But all the available physical physical memory Linux mm at boot has already cut into bucket'y and rightly considers his own. Fortunately, there is a work round here. Even two - you can write a kernel module that politely asks for a bit of continuous physical memory, change the linker script to put the .text and .data areas of the binary into this memory. And you can make it even simpler - ask Linux to reserve a bit of physical memory at a specific address at boot time. For example, when loading the kernel with the option memmap = 0x80M $ 0x18000000, the OS will allocate 128 megabytes of contiguous physical memory starting at address 0x18000000. Unfortunately, Linux kernels from 3.1 sometimes crash at the very beginning of the boot with this option. Now I'm just trying to find the reason.
Voilà, now we can keep Linux on several kernels, and arbitrary baremetal programs on everyone else. But how can this come in handy? Suppose we already have a task to run some code without an OS. For example, I needed benchmarks running on different x86 platforms and measuring the spread between the average code execution time and the worst time. x86 - generally not a model of determinism, and if you also add the operating system ... Even if it is a compact and nimble RTOS, it is still not immediately obvious where latency and jitter come from. In addition, each of my clients has its own RTOS, some are slightly more difficult than baremetal. Or if there is a code from the microcontroller that is sensitive to delays, which already works without the OS, and you just need to port-throw it on one of the x86 processor cores. If you simply turn it on with the kernel thread on the kernel and disable everything else on it, it will still be interrupted. With the RT-Linux patch, everything works better, but it adds latency and is sometimes not compatible with other useful patches.
And why is Linux and baremetal together better than a clean baremetal image loaded by a bootloader (GRUB)? And with Linux it’s more convenient! Compare rebooting the entire piece of hardware or just a couple of shell commands on the Linux host command line. Similarly with debugging - with Linux you can read-write memory baremetal OS perfectly, diagnose glands in the process. (Of course, if careful!) There are some disadvantages to this approach. To run baremetal OS with Linux, you need to know the ACPI ID of the kernel to which the IPI is sent. If you think that the first core has number 0, the second has 1, etc., then you are an optimist. I saw very different options :). You will either have to enumerate, this is described in one of the posts to which I referred, or select options. (And don't forget to [turn off] Hyperthreading!). Another possible ambush is non-determinism. If you disable all kernels in Linux, except for the first one, and there is nothing serious to run on it, it is impossible to measure the impact of Linux on the performance of kernels that run baremetal code. But there is still no 100% guarantee - if you wish or by chance, you can greatly affect the performance of baremetal code through shared resources (cache, memory controller). For example, try switching VGA console (Ctrl-Alt-F1, Ctrl-Alt-F2), measuring the performance of baremetal OS.
If anyone is interested in how it all works, or if there are ideas on how to practically use it (BSD license allows) - download the source from IDZ or github (it’s fresh there). Baremetal OS itself lies in tools / src / baremetal. In the first directory - phymem lies a tool for reading and writing physical memory. And in the second - smallos - actually the runtime. Everything there is designed to run benchmarks, so you have to use a file to cut them off - just throw out the links to bench / *. O from smallos / Makefile and replace the code for calling benchmarks in smallos / apps / mwaitTester.c with your code. Linux usermode bootloader lies in smallos / linux_boot. Good luck