An Inside Look at OpenBMC for OpenPOWER Systems

    In a previous article, Maxim touched on the hardware of the BMC (Baseboard Management Controller) board. I want to continue the story and tell about our approach to BMC and participation in the OpenBMC project.

    To complete the story, you will have to start a little from afar and talk about the purpose of service processors and the role of BMC in the server’s work, affect the IPMI protocol and the software part. After that, I will briefly describe how BMC is involved in booting systems on POWER8. I will end with an overview of the OpenBMC project and our attitude to the issue. Readers experienced in the subject of service processors can immediately rewind to the lower sections.

    Service processors - what, why and how


    A service processor is a separate dedicated controller that is embedded in a server. Its chip can be soldered to the motherboard, located on a separate card, or, for example, placed in a blade chassis to manage the resources of the entire system as a whole (and then it can already be called SMC - System Management Controller). BMC is a special case of a service processor for managing a separate host, and hereinafter we will talk only about them and use the term “service processor” only in the meaning of “BMC”. At the same time, saying “BMC”, in general, we mean both the chip itself and the control firmware. In some cases, we will separately indicate that we are talking about hardware or software.

    The BMC is powered separately from the main system, it turns on automatically when the standby power is supplied to the server and works until the power is turned off. Almost all service processors can manage host power, provide access to the console of the main operating system via Serial Over LAN (SoL), read system sensors (fan speed, voltage on power supplies and VRM, component temperature), monitor component health, store hardware error log (SEL). Many provide remote KVM, virtual media (DVD, ISO), support various out-of-band connection protocols (IPMI / RMCP, SSH, RedFish, RESTful, SMASH) and more.

    Remote control is now widespread. It facilitates the management of a large fleet of servers, increases availability due to reduced downtime, and improves the operational efficiency of data centers. As a result, the availability of extensive remote management capabilities is taken into account by customers when choosing a supplier of a hardware platform.

    BMC users are mainly sysadmins for remote management, crash recovery, log collection, OS installation, etc. Technical support uses data from the service processor. For her, BMC is often the only source of information in troubleshooting and identifying defective replacement components.

    In the modern infrastructure, BMC is not just a nice additional option for remote server management (although not to go to the server room where it is cold, noisy, have nowhere to sit down and poorly catches your mobile is nice). In many situations, this is a critical component of the infrastructure. When the operating system or application does not respond or is in an incomprehensible state, the service processor is the only source of information and a way to quickly restore work.

    To connect to the service processor, use a dedicated network port (out-of-band), or BMC shares the network port with the main system (sideband). That is, one physical Ethernet connector, but two independent MACs and two IP addresses. For initial setup, an RS-232 console connection is often used.

    IPMI


    Quick reference


    Historically, the BMC software was developed along with the server hardware platform and the same developers. As a result, the service processor software was unique for each platform. The same vendor could have several BMC firmware options for different product lines. Despite the spread of open source, BMC firmware for a long time remained exclusively proprietary.

    Typically, a service processor is based on specialized systems on a chip (System-on-Chip, SoC), and the IPMI (Intelligent Platform Management Interface) specification is the de facto standard for describing hardware architecture requirements. This is a fairly old standard; back in 1998, a group of companies developed the first IPMI specification for standardizing server management.

    IPMI provides a common messaging interface for access to all managed components in the system and describes a large set of interfaces for various types of operations - for example, monitoring temperature, voltage, fan speed, or gaining access to the OS console. Methods are also provided for managing the power of the entire complex, receiving hardware SEL logs (System Event Log), reading sensor data (SDR), implementing hardware watchdogs. IPMI provides a replacement or abstraction for individual sensor access methods, such as System Management Bus (SMBus) or Inter Integrated Circuit (I2C). Most BMCs use a proprietary IPMI stack from a small number of vendors.

    Claims for IPMI


    The protocol has accumulated many complaints, including in terms of security when accessing over a network (IPMI over LAN). Periodically, the network is shocked by stories like this . Here's the thing: after gaining access to the service processor, we get full control over the server. Nothing prevents you from rebooting into recovery mode and changing the password for the 'root' account. The only reliable remedy for this vulnerability is the rule that IPMI traffic (UDP port 623) should not go beyond a dedicated network or VLAN. Activity in the control network must be closely monitored.

    In addition to security issues, the hardware landscape of data centers has changed dramatically over the years. Virtualization, component disaggregation, and clouds spread. It is difficult to add something new to IPMI. The more servers you need to administer, the greater the importance of process automation. API specifications are emerging to replace IPMI over LAN. Many have hopes for RedFish.

    This API uses modern JSON and HTTPS protocols and a RESTful interface to access 'out-of-band' data. The goal of developing a new API is to offer the industry a single standard that is suitable for heterogeneous data centers. And for single complex enterprise servers and for cloud data centers from a variety of commodity servers. And this API must meet current security requirements.

    At the same time, at the hardware level, the standard is IPMI support, which is involved in the entire server work cycle, from power on, loading of the operating system to recovery from a failure (freezing, panic, etc.).


    The role of BMC in server life. In this picture, SMS stands for “System Management Software”. The picture is taken from here .

    The role of BMC in booting the OpenPOWER system


    At the heart of the IPMI hardware is the BMC chip. He is involved in the operation of the server, starting from the moment of power supply and is involved in the process of initial loading of the server on OpenPOWER. That is, BMC is necessary to turn on the system. At the same time, a reboot / crash of the BMC does not affect an already running operating system.

    BMC and POWER8 processor are connected by LPC (Low Pin Count) bus. This bus is designed to connect the processor with peripheral, relatively slow devices. It operates at a frequency of up to 33 MHz. Through the LPC, the central processor (that is, the Hostboot / OPAL microcode ) communicates with the BMC IPMI stack via the BT (p. 104) interface. On the same bus, POWER8 receives bootable microcode from a PNOR (Processor NOR chip) via an LPC → SPI (Serial Peripheral Interface) connection.

    The role of BMC in the boot phase Power off -> Standby


    The first boot phase begins as soon as the power supplies are plugged in and ends at the stage when the BMC is fully turned on and ready to start booting the entire host. Looking ahead, I note that from here on I describe the operation of BMC software on OpenPOWER systems in general, but specifically in our server these functions are performed by OpenBMC. When powered up, the BMC starts to execute code from the SPI flash, loads the u-boot, and then the Linux kernel. At this stage, IPMI is running on BMC, the LPC bus is prepared for host access to PNOR memory (mounted via mtdblock). The POWER8 chip itself is turned off at this stage. In this state, you can connect to the BMC network interface and do something.

    Standby -> OS boot


    When the system is in 'standby' mode and the power button is pressed, BMC initiates the continuation of the boot and launches the “small” SBE (Self Boot Engine) controller on the master processor inside the POWER8 chip to load the Hostboot microcode from the PNOR flash into the L3 master cache the processor.

    Hostboot microcode is responsible for initializing the processor buses, SDRAM memory, other POWER8 processors, OPAL (Open Power Abstraction Layer) processors and another microcontroller built into POWER8 called OCC (On Chip Controller).

    We will talk about this controller in more detail, since it has special significance for BMC. When Hostboot finishes its work, the next component of the Skiboot microcode is launched from the PNOR flash. This level synchronizes processors, initializes PCIe buses, and also interacts with BMC via IPMI (for example, it updates the FW Boot progress sensor value). Skiboot is also responsible for launching the next Petitboot boot level, which selects where the main operating system will be loaded from and starts it through a kexec call.

    But take a step back and go back to OCC. The OCC chip is a PPC 405 coreembedded in the IBM POWER8 processor along with the main POWER8 cores (one OSS per chip). It has its own 512K SRAM, access to main memory. This is a real-time system responsible for temperature control (monitoring the temperature of the memory and the processor core), managing memory performance, monitoring the voltage and frequency of the processor. OCC provides all this information to the BMC via the I2C bus.

    What exactly does OCC do?

    • Monitors the power status of server components.
    • Monitors and controls the temperature of components; in case of overheating, it reduces the memory performance (memory throttling).
    • If necessary, OCC reduces the frequency / power consumption of the processor by reducing the maximum P-state (performance state, see ACPI ). However, OCC does not specify a P-state directly. It sets limits within which the operating system can change the P-state.
    • Provides BMC power and temperature information for efficient fan control.

    Thus, OCC is the information provider for the BMC to which it is connected via the I2C bus. The source code for most of the microcode for POWER8 (and in particular for OCC) was opened by IBM .

    In addition to interacting with the OCC and the central processor via the LPC bus, the BMC has other interfaces. For example, GPIO on a BMC chip is used to control power supplies and LEDs, I2C can be used to read sensors.


    The interconnection of all of the above is not so complicated

    . At the moment, most of the OpenPOWER microcode is open. At the same time, the software part of the service processor and the IPMI stack until recently remained proprietary. The first open source project for a service processor was OpenBMC. The community greeted him with enthusiasm and began to actively develop. About OpenBMC and talk further.

    OpenBMC, its history and features


    Finally, we come to the story of OpenBMC and how we use it.

    The birth of OpenBMC on Facebook


    OpenBMC appeared on Facebook when developing the Wedge switch as part of the OCP community in 2015. Initially, the BMC software was developed by the iron supplier. In the first months of the work, many new requirements for BMC arose, the coordination of which with the developers was difficult and caused delays. Under the influence of this, at one of the hackathons, four Facebook engineers implemented some of the basic functions of BMC in 24 hours. It was very far from productive, but it became clear that the task of implementing the software part of BMC can be solved separately from the hardware.

    A few months later, OpenBMC was officially released with the Wedge switch, and after a while, the OpenBMC source code was opened as part of the OCP partnership.

    Then Facebook adapted OpenBMC for the NVMe Lightning flash shelves, and then for the Yosemite microserver chassis. In the last change, Facebook abandoned RMCP / RMCP + (IPMI over LAN access), but there was a REST API via HTTP (S) and console access via SSH. Thus, Facebook got a single API for managing different types of equipment and great flexibility in implementing new features. With a proprietary approach to BMC, this would not be possible.

    In the concept of Facebook, BMC is a regular server, but runs on SoC with limited resources (slow processor, low memory, small flash). With this in mind, OpenBMC was conceived as a specialized Linux distribution, all packages of which are collected from source using the Yocto project. The descriptions of all the packages in Yocto are combined into 'recipes', which in turn are combined into 'layers'.

    OpenBMC has three layers:

    1. general level, user space applications (do not depend on hardware).
    2. SoC level (Linux kernel, bootloader, C library).
    3. platform / product level (packages specific to a particular product, kernel settings, libraries for sensors).

    Facebook is not the first to use Yocto in BMC. The proprietary Dell iDRAC is built on the same build system.

    OpenBMC can be easily ported from one platform to another by rebuilding with several bitbake commands . This allows you to use the same BMC and, as a result, the same API on different hardware platforms. This can break the established tradition of the dependence of the software stack on the hardware.

    Fork Project at IBM


    The OCP community quickly embraced the idea of ​​OpenBMC, and another OCP member, IBM, was actively involved in the development. Their efforts resulted in a fork of the project under OpenPOWER , and by August 2015 the first version of OpenBMC for the Rackspace Barreleye server under the AST2400 SoC was released . IBM engineers resolutely got down to business and not only adapted OpenBMC for the new platform, but significantly reworked it. At the same time, due to the tight deadlines and for ease of development, Python was actively used.

    The changes affected all layers of the project, including the Linux kernel was redesigned for SoC (installed drivers, a device tree was added), at the user level, D-Bus appeared for interprocess communication (facebook did not have D-Bus). It is through D-Bus that all the functions of OpenBMC are implemented. The main way to work with OpenBMC is a REST interface for accessing bus interfaces. In addition, there is Dropbear SSH.


    There is web access to the REST API (for debugging, for example) through the Python Bottle framework.
    Due to the easy portability of OpenBMC from one platform to another, development boards can be used for development, up to RaspberryPI. To simplify development, an assembly for the QEMU emulator is provided.

    Now OpenBMC has a rather ascetic console interface via SSH. IPMI is supported by the host to a minimum. The REST interface can be used by applications for remote management and monitoring. Some of the most popular functions are implemented through the team obmcutil.

    Probably 90% of the operations performed through the BMC are turning the server on / off. In OpenBMC, this is done by obmcutil poweronand obmcutil poweroff.

    Also, for example, throughobmcutil You can see the value of the sensors and detailed information about the server hardware (FRU), if supported on a specific platform:

    obmcutil getinventory
    obmcutil getsensors

    Now, not only IBM, but also many other vendors interested in avoiding the closed BMC stack are actively participating in the OpenBMC project. IBM itself is mainly focused on adapting to the P9 platform.

    Most of the development of OpenBMC is under the Apache-2.0 license, but OpenBMC includes many components with different licenses (for example, the Linux kernel and u-boot under GPLv2). The result is a mix of different open source licenses. In addition, developers can add their own proprietary components to the final assembly that work in parallel with Open Source.

    Our view on OpenBMC


    As you can see from the text above, we design the software part of our service processor based on OpenBMC. The product is still raw, but the minimum functions for administering the server are already implemented in it, partially implemented IPMI (for the most basic needs). Service processors in servers with this feature set were on the market several years ago.

    OpenBMC is constantly changing and improving, almost every day on the gerrit server you can see new commits. Therefore, focusing heavily on functionality at the moment is not very grateful. Refactoring is continuously performed, Python code is replaced by C / C ++, more functions are transferred to systemd.

    The attitude to the service processor as to a regular server is not typical for BMC due to its important role in server life. Using systemd and D-Bus has not been common in this area before. New time - new trends.

    The first task for us is to adapt the current state of OpenBMC to our platform. Further, we plan to refine it with options and interfaces in which our customers are interested. Given the limited functionality at the moment, there are a great many directions for development. As new features are realized, we will definitely commit changes to the OpenBMC project so that the community can use it.

    Also popular now: