SLES 12, watchdog timer and IBM / Lenovo servers

UPD: Recent findings are presented here: IBM / Lenovo Servers and Watchdog Timer: Episode II . Further presentation is a prologue to the article by reference.

Faced with a significant regression in the 12th version of SLES related to the support of the watchdog timer (device / dev / watchdog) on ​​IBM / Lenovo servers.

First, a short educational program, if someone is not in the subject. How should it work and why is it needed? Those who already know the subject can safely skip the next paragraph.

As part of server and industrial platforms, there is a special scheme - a watchdog. When activated, it starts to count down a preset time (for example, one minute). If you do not contact him again during this time, then at the end of the interval a hardware reload will be performed. If you turn, the interval begins to re-count. This is necessary in order to automatically restore the computer in the event of a freezing operating system or providing some important software service. Such a solution is mandatory in high availability (HA) clusters and other applications that require constant system availability. For computers with Intel architecture, several hardware watchdog timer interfaces are used, depending on the system manufacturer, Of these, the most common is Intel TCO (iTCO). On Linux, watchdog drivers are implemented as kernel modules that provide a programmatic interface to it in the form of the / dev / watchdog device.

In IBM's Intel servers, now available from Lenovo, the Intel TCO hardware layer and its supporting Linux kernel module iTCO_wdt are responsible for the interface to the watchdog timer. In SLES version 11, everything was fine with this and worked automatically, the device / dev / watchdog, supported by the iTCO_wdt driver, appeared on the system at default settings. However, in the 12th version of SLES, the iTCO_wdt driver was rewritten, reducing its size by 3 times, and something was broken. As it turned out, in SLES 11 everything was bad too, the / dev / watchdog file was created, but it was not connected to the driver and did not provide timer functionality.

Now (in SLES12) the following happens. The iTCO_wdt module loads, leaves the diagnostics in the system log: “iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware / BIOS”, remains loaded into memory, but does nothing, and the device / dev / watchdog does not appear. Manual loading and unloading a module does not change anything in this behavior. The BIOS and integrated service module (IMM) settings also do not affect this in any way. The problem manifests itself in exactly the same way on multiple IBM / Lenovo HS23 and x3250 servers. If you load SLES11 on the same machine, everything works fine.

A workaround for resolving the issue may be to write the softdog module in /etc/modules-load.d, which provides an interface to the watchdog timer by emulating it at the kernel level of the program. But in fact, this is just a stub that does not solve the issue of a possible failure of the operating system itself.

Worse, in a recent SLES12 interim update, the softdog driver was loaded by default. Although this behavior was turned off very soon, we can’t be sure if the hardware or software driver provides you with a watchdog service until we check a specific version of Linux.

I passed the diagnostic information and error description to the Novell kernel developers and have been working on the incident with IBM / Lenovo support, but the situation has not been resolved for two months, although formally SLES12 is a fully supported and recommended operating system for these servers. So, if the reader suddenly encounters a watchdog timer inoperative (leading, for example, to the inability to start the cluster) or incomplete implementation of its functions related to replacing the hardware driver with software, then at least it will know where to dig.

It seems that there was a way to solve the problem, wrote a new article about it, mentioned at the very top.