HP iLO 2 is vulnerable to Heartbleed - we are protected and updated!
Heartbleed is not scary for iLO 2 from the point of view of penetration into the system, however, some scanners of this vulnerability (or an attempt to exploit it) tightly cut down the iLO 2 interface of Hewlett Packard servers.
On Friday, April 11, I thus lost access to three hundred servers with iLO modules of the 2nd generation (iLO interfaces are publicly available, as these are hosting servers).
The bulk of the servers is a blade server, however, several DL servers also fell under the "distribution".
Neither iLO 3, nor SuperMicro IPMI, nor anything else on neighboring IP addresses were affected.
Symptoms:
1) the latest version of iLO 2 v.2.23 is vulnerable;
2) even PING disappears before iLO 2;
3) the basket (Onboard Administrator) ceases to see the server, does not know its type, does not know its current state, allocates power for it with a margin (attention! This moment may lead to the shutdown of a number of servers if Dynamic Power Capping is enabled);
4) oddly enough, iLO 2 on DL servers continued to respond to PING, however this is the maximum that could be achieved from them;
5) rebooting the server using OS means it will not lead to loading, because (apparently) he is stuck on the start-up diagnostics of components;
6) a regular shutdown does not help - the server is of course jammed, but iLO 2 does not come to life;
7) from the inside of a running server (from OS) it is neither possible to chat or communicate (via hponcfg for example) with the iLO 2 module, the module does not respond;
8) the situation is treated ONLY by completely disconnecting the power from the DL servers (via managed PDUs or by hands), or a little more conveniently (but actually with the same effect) for blade servers using the reset server XX command via Onboard Administrator;
9) Cold Restart does not help either.
The most unpleasant thing is that in the case of highly loaded blade baskets, OA modules, having lost contact with the servers, will not know their power requirements, and will begin to allocate obviously large values to them (2-3 times higher than real consumption). This will lead to the fact that the OA considers that there is not enough power and can cut down some servers (or prevent it from starting). At least this is true if you have dynamic power distribution control enabled.
HP did not immediately recognize the problem, and initially reported only the fact that only Onboard Administrator (ver.> = 4.11) were vulnerable to Heartbleed, and about iLO it was said in c04239413 that everything was OK . They even used the expression NOT Impacted by "HeartBleed" .
Yes, from the point of view of penetration into the system, iLO is not vulnerable, but, unfortunately, iLO 2 turned out to be Impacted, and how!
HP has no official position on cutting iLO 2 as a result of scanning on Heartbleed.
Fortunately, by Monday, an HP Oscar A. Perez employee made us happy with beta version 2.25 of the firmware and posted it for tests .
You can download from here .
After hard reset the affected servers and updating the firmware to 2.25 beta, while the flight is normal.
If your iLO 2 is publicly accessible from the Internet, then do not wait when you lose connection to the servers and cannot manage it in a critical situation (or just after a scheduled restart) - upgrade to at least 2.25 beta.
Alternatively, temporarily restrict public access to iLO 2 until the release and application of the official release of patched firmware.
On Friday, April 11, I thus lost access to three hundred servers with iLO modules of the 2nd generation (iLO interfaces are publicly available, as these are hosting servers).
The bulk of the servers is a blade server, however, several DL servers also fell under the "distribution".
Neither iLO 3, nor SuperMicro IPMI, nor anything else on neighboring IP addresses were affected.
Symptoms:
1) the latest version of iLO 2 v.2.23 is vulnerable;
2) even PING disappears before iLO 2;
3) the basket (Onboard Administrator) ceases to see the server, does not know its type, does not know its current state, allocates power for it with a margin (attention! This moment may lead to the shutdown of a number of servers if Dynamic Power Capping is enabled);
4) oddly enough, iLO 2 on DL servers continued to respond to PING, however this is the maximum that could be achieved from them;
5) rebooting the server using OS means it will not lead to loading, because (apparently) he is stuck on the start-up diagnostics of components;
6) a regular shutdown does not help - the server is of course jammed, but iLO 2 does not come to life;
7) from the inside of a running server (from OS) it is neither possible to chat or communicate (via hponcfg for example) with the iLO 2 module, the module does not respond;
8) the situation is treated ONLY by completely disconnecting the power from the DL servers (via managed PDUs or by hands), or a little more conveniently (but actually with the same effect) for blade servers using the reset server XX command via Onboard Administrator;
9) Cold Restart does not help either.
The most unpleasant thing is that in the case of highly loaded blade baskets, OA modules, having lost contact with the servers, will not know their power requirements, and will begin to allocate obviously large values to them (2-3 times higher than real consumption). This will lead to the fact that the OA considers that there is not enough power and can cut down some servers (or prevent it from starting). At least this is true if you have dynamic power distribution control enabled.
HP did not immediately recognize the problem, and initially reported only the fact that only Onboard Administrator (ver.> = 4.11) were vulnerable to Heartbleed, and about iLO it was said in c04239413 that everything was OK . They even used the expression NOT Impacted by "HeartBleed" .
Yes, from the point of view of penetration into the system, iLO is not vulnerable, but, unfortunately, iLO 2 turned out to be Impacted, and how!
HP has no official position on cutting iLO 2 as a result of scanning on Heartbleed.
Fortunately, by Monday, an HP Oscar A. Perez employee made us happy with beta version 2.25 of the firmware and posted it for tests .
You can download from here .
After hard reset the affected servers and updating the firmware to 2.25 beta, while the flight is normal.
If your iLO 2 is publicly accessible from the Internet, then do not wait when you lose connection to the servers and cannot manage it in a critical situation (or just after a scheduled restart) - upgrade to at least 2.25 beta.
Alternatively, temporarily restrict public access to iLO 2 until the release and application of the official release of patched firmware.
