Firmware security using the industrial switches Hirschmann and Phoenix Contact as an example / Digital Security Blog

Many modern computer systems (motherboards in PCs, smartphones, network equipment ...) work under the control of firmware, which, as a rule, is developed for a specific hardware, can use the whole range of available hardware resources and has the highest privileges. Particular attention must be paid to protecting firmware, otherwise it will be a real Achilles heel in system security.
A small series of articles will be devoted to this issue; we will show the weak and strong models of firmware security using real examples.

By the way, this article is based on our study of the security of industrial network switches, which was presented by me at the information security conference for practitioners of ZeroNights 2015 (the report “Modification of Industrial Switch Firmware”, the presentation ishere ).

Introduction

First, let's talk about the objects of our study - industrial switches.

The heart of any modern ICS infrastructure is the data medium. In the vast majority of cases, it is an industrial network such as Industrial Ethernet, i.e. based on the Ethernet technology family. Such a network differs from the usual one primarily in the use of a real-time protocol (for example, Profinet, EtherCAT, Ethernet / IP and others).

An example of an industrial network [picture from here ]:



Various kinds of devices (PLCs, HMI panels, control PCs, field devices, etc.) are connected to the network via industrial type network switches (switches). Unlike conventional switches, they are less demanding of the operating conditions (extended operating temperature range, surge protection, shock resistance, the possibility of mounting on a DIN rail, etc.) and support industrial network protocols.

Thus, industrial switches are a critical element in the structure of an industrial network. It is very important to properly protect them from being compromised by a potential attacker because this will provide almost unlimited possibilities for controlling the process: compromising other devices on the network, interfering and changing data inside different connections between PLC and SCADA, between gateways and PLC, faking data, transferred to HMI and to journaling systems, etc. All this can lead to a loss of control by the operator over the actual state of the process and, as a result, a stop or an accident.

To prevent this from happening:

• firstly, the territorial location of the switch should minimize the possibility of unauthorized physical access to it;
• secondly, the software and hardware architecture of the switch must have an information security model that takes into account modern threats.

We’ll talk about the latter.

Objects of study

For research, we took one model of managed industrial switches from the two most common manufacturers of industrial network equipment:

• Hirschmann rs20
• Phoenix Contact FL SWITCH MM HS

You can control the operation of these switches through the COM port (both have an RS-232 connector) or at the network level.


When working through the COM port, we are dealing with a console for entering commands:



Access to the console requires a login and password. Nevertheless, some options are available before the authentication screen, for example, displaying information about hardware and firmware, updating the firmware, etc.


When working with a switch over a network, you can reach (by its IP address) the web interface:



Authentication is required to access it.


Another option for managing a switch at the network level is SNMP (Simple Network Management Protocol), which also has authentication. However, SNMP support in the studied switches has several disadvantages:

• by default (and this is the most favorite configuration of most network devices), the most insecure version of this protocol is used - SNMP v1;
• in SNMP v1, the default username and password are strongly recommended by the vendor of these devices not to change;
• versions of SNMP v1 and SNMP v2c do not use encryption, which means that the protocol is vulnerable to attacks like MITM (man in the middle).

First, let's talk about the study of the Hirschmann RS20 switch, then we outline the features of Phoenix Contact FL SWITCH MM HS in comparison with this switch.

Study of the Hirschmann RS20 Industrial Switch

Firmware

First you need to get the firmware, understand what kind of processor it is running, and evaluate the ability to connect the programmer / debugger. To do this, we looked inside and saw:

1. CPU Digi NET + ARM NS9360B-0-I155, 32-bit ARM9, without internal memory (therefore, the firmware must be stored in external memory);
2. 16 MB Micron MT48LC8M16A2 SDRAM;
3. 8 MB Intel 28F640JD3D75 flash memory (most likely, firmware here);
4. CPLD Marvell 88E6095F-LG01, performs the function of an Ethernet switch, has an internal configuration memory.

The flash memory chip is made in a BGA package. If we remove it, it will be very difficult to solder it back without an infrared soldering station. Therefore, the firmware (version 8.0.07) for this switch was merged from the open Hirschmann ftp server .

One of the files in the downloaded archive is the rsL2E.bin binary (about 4 MB), this is the firmware image. Rummaging through it, we saw that it consists of two modules. Each module has a header (their structure is the same), as well as a compressed body.




The first module is zlib compressed (RFC 1951). It contains the executable code. When unpacked, the module weighs approximately 7 MB.

The second module is compressed using the gzip algorithm (RFC 1952), it contains a pack200 archive, after unpacking which a JAR (Java Archive) file is obtained, which is an implementation of the device’s web interface. After unpacking, the module weighs approximately 3 MB. By the way, this file is transferred to the host (control PC) when accessing the switch by its IP address (i.e., when trying to access the device’s web interface) and it is executed on it (on the host).

It can be seen that the total volume of the unpacked modules is 10 MB, and the amount of flash memory on the device’s system board is 8 MB. Therefore, we assumed that the firmware image is stored in a packed form. So, there must be a bootloader that unpacks and launches this firmware. But about the bootloader later.


The size of each module header is 256 bytes. Its structure is not documented anywhere, so I had to disassemble it with the help of a scientific poke. Fields that caught our attention:

• module signature
• type of module;
• module size;
• module checksum (CRC32);
• offset to the end of the file (eof offset);
• module checksum (repeat) (CRC32);
• Header checksum (CRC32).

Based on this, we can conclude that there is a control of the integrity of the firmware. But authentication, usually an electronic digital signature (digital signature), is missing.

The ability to fake these firmware must be checked, for this we wrote a small script that allows you to quickly extract both modules from the firmware image, and, after making changes to them, assemble them back into the finished image:




But, before making changes, you need to think about it. And turn it around.


It turned out that inside the firmware there is an RTOS code (real-time operating system) VxWorks, moreover, a rather ancient version 5.4.2 (around 1998). At the time of writing, the latest version is 7.

On the Chinese "Internet"we found the source for VxWorks 5.5, which greatly facilitated the identification of various OS functions, libc-procedures, and more. Anyone who has faced the task of analyzing VxWorks code will probably think now: “Look at the end, VxWorks images always have a symbol table!” It is, but in this form it is almost useless:




VxWorks is a modular product, and the composition of the modules is determined by the customer when he buys the sources. In this case, in addition to the basic components, we saw web server modules (EmWeb), SnmpOverHttp (yes, we saw SNMP packets inside HTTP requests in the captured traffic), and others.
The vendor definitely took care of the filling.

But about security - not really. Here the linear address space, code execution can be carried out from anywhere in the memory, there is no protection against overflows on the stack, etc. Elementary protection mechanisms against exploitation of binary vulnerabilities are completely absent. Maybe they are not needed? But there are a lot of registered vulnerabilities for this version of VxWorks :

1. CVE-2015-3963 spoof TCP sessions;
2. CVE-2010-2968 brute-force;
3. CVE-2010-2967 obtain access;
4. CVE-2010-2966 obtain access;
5. CVE-2010-2965 RCE;
6. CVE-2008-2476 DoS;
7. ...

Choose for every taste: DoS (Denial of Service), RCE (Remote Code Execution), brute-force capabilities, and authorization bypass options. The list is not complete.

But don’t be upset! Indeed, during the analysis of the firmware, very interesting code fragments were found:

• SNMP request handlers
• handlers of console commands;
• reading and rewriting flash memory;
• reading and overwriting Marvell CPLD configuration memory!

Revolved, now you can hang.

To do this, you should correctly choose a place for the code to be introduced: you must not allow it to interfere with the normal functioning of the switch, and it is best if it is called on demand.

The obvious choice was one of the console command handlers. More specifically, the logout command handler. In its place in the firmware, we wrote a code that reads the memory of a given size at the specified address (everything is set in the variables at the end of the inserted fragment), and displays the removed dump in the COM port:




We made a ready-made image, updated the switch with it (you can update the firmware via COM port, and through the web interface). Success! Indeed, after entering the logout command, instead of logging out, a memory dump was now output:




Thus, the ability to fake firmware was confirmed.

But how can an attacker inject their firmware into a switch that is actually used at an industrial site?


Consider the possibility of updating through the COM port (i.e. through the console interface). As already mentioned, to update the firmware of the switch, you do not need to know the username and password, but you must have physical access to the device (to plug into RS-232). This option is hard to believe if the device is already in use at the facility.

However, from the factory, the switch does not immediately hit the industrial facility. In the delivery scheme to the customer, there are always intermediate organizations that have the opportunity, as a free bonus, to download non-original firmware onto the switch.

Another option (though this is already a distribution vector): you can infect the firmware of the switch from a previously compromised control PC that is already connected to the switches via the COM port.


Remotely modifying the firmware of a switch is more difficult, but the attack surface is much wider. Recall that authentication is required to access the web interface of the switch. What can be done? To try:

• enter the default username and password;
• pick up based on known password restrictions. Fortunately, there is no protection against brute-force attacks;
• exploit a binary vulnerability (known or honestly find one), again, there is no protection against exploitation of binary vulnerabilities.

By the way, remotely - this is not only from a PC connected to the switch. And also from one of the field devices, which, sometimes, are located in an unguarded territory. So it must be sure to be hung up!


The conclusion is as follows. A potential attacker, when introducing his code into the firmware of the switch, is able to:

• execute your code on the switch CPU (using the whole range of its hardware resources);
• execute your code on the side of the host (control PC) or any other client PC that accidentally accesses the IP address of the switch. Of course, this code will be executed in a Java machine, but, as you know, it is not a bulwark of security.

Note that a compromised switch can be "cured" by a regular firmware update procedure. The main thing is that you are sure of the authenticity of the new image.


In any case, this immediately leads to the question of the possibility of fixing on the device so that changes made to the firmware “survive” any updates. And then we remembered the bootloader, which we did not have a dump.

Loader

Hoping that the bootloader left a trace in the RAM, we dumped it (using the above code fragment) in different places, but did not find anything. And we decided to try to get it from the flash memory of the switch, pulling the previously found procedures for reading flash memory. And again success.

So, the contents of flash memory can be divided into three regions:

1. Bootblock, the first 80000h bytes. This is the boot area, the bootloader is stored here, which consists of three parts:
   
   • start code;
   • main code (compressed according to the Huffman algorithm);
   • header (the structure is identical to the previously described header structure of the firmware module);
2. Application firmware, the largest area, the firmware image is stored here in a packaged form, as expected;
3. Storage, an area for storing service information, configurations, logs, etc.

The bootloader code allowed us to reconstruct the early stages of the switch boot process.

At start, the first 4 KB of flash memory (and this is just the size of the start code) is projected into the address space of the CPU at address 0. Execution starts from this address.
The task of the start part of the bootloader is to configure the memory card, unpack the main part of the bootloader into memory and transfer control to it.

The main bootloader code should:

1. Initialize CPU hardware resources;
2. Configure interrupt model;
3. Unpack the firmware and transfer control to it.

Immediately comes to mind a way to modify the bootloader. The switch is updated with a modified version of the firmware, which, using standard procedures for working with flash memory, will overwrite the bootblock area.



In the future, the modified bootloader will be able to make changes to the unpacked firmware image at each boot. The switch “compromised” thus compromised is no longer so simple.




Is it possible to "cure" the switch using regular means? To do this, you need to update both the firmware and the bootloader to the original ones.

There is no possibility to update the bootloader in the console interface, although procedures have been found in the firmware code for updating the bootblock area in the same way as updating the firmware. But these functions are not called anywhere, so this is undocumented debugging functionality.

But in the web interface of the switch a completely standard option was found for updating the bootloader: The




only question is how to update? But nothing!

In the archive with the firmware image for many models of its devices, Hirschmann does not apply the bootloader image. And the investigated model among them.

Getting the bootloader image by contacting tech support will fail either. We tried and got the answer that bootloader images are not distributed. The only thing that was possible to get from them was the proposal to send the device to them using the RMA form. However, they will in every possible way discourage you from doing this, sincerely considering this process useless.

Thus, a potential attacker has the ability to be fixed in the device loader through a firmware modification. And to get him out of there by regular means is almost impossible.

Is it possible to dig even deeper?

At the moment, we do not have an answer to this question, but the idea will be voiced worth it.

Recall that this device has another executable medium, with its own, in some way, firmware. This is FPGA (programmable logic integrated circuit), a CPLD type from Marvell with internal non-volatile configuration memory.

Programs for them are written in the logic description language (VHDL, Verilog are the most common). Recall that in the switch firmware code, we found functions for reading and overwriting this configuration memory.

Study Industrial Switch Phoenix Contact FL SWITCH MM HS

In exploring this switch, we went the same way, only faster. What's interesting inside:

1. CPU PMC RM5231A, 32-bit MIPS IV, without internal memory;
2. RAM SDRAM Micron MT48LC8M16A2 16 MB, two pieces;
3. Flash memory from Intel unknown model;
4. Chipset Galileo GT-64115.

We downloaded the firmware image for this switch from the official website of the vendor and disassembled its structure in the same way. It has a header and a zlib-compressed (RFC 1951) body:




Main fields in the header structure:

• Signature
• Header checksum (ADLER32)
• unpacked body checksum (ADLER32);
• size of the unpacked body;
• packed body checksum (ADLER32);
• packaged body size.

There is also no authentication of the firmware, we experimentally verified this.




VxWorks is also here, although a slightly newer version 6.1 has the same problems: lack of protection mechanisms against exploitation of binary vulnerabilities and a bunch of registered CVEs.

You can update the firmware in the same ways:

• via COM port, authentication is also not required
• via the web interface. To access it, you do not need to enter a login and password, but when making changes to the settings (or when updating the firmware), you need to know the password.

Speaking of password. According to the documentation, there is an engineering password, in case the current password is lost (well, or never acquired if you are a hacker). To do this, you need to contact technical support by providing them with the MAC address and serial number of the switch.
This suggests that there is an algorithm for converting these numbers into an engineering password.

This switch also has a bootloader that can be overwritten. And it is experimentally proven. There are no regular capabilities for updating the bootloader at all.

Conclusion

As a result of the study, the following deficiencies in the architecture of industrial switches were identified:

1. The ability to illegally update the firmware (for some regular procedures, authentication is not required);
2. The ability to fake the device’s firmware image (later the bootloader, CPLD firmware ...) lack of code authentication;
3. There are no mechanisms from exploitation of binary vulnerabilities, which opens up additional possibilities for remote code execution on the switch.

In the course of the study, we showed the ability to fake switch firmware, as well as the ability to pin modifications to the bootloader.


In short, awesome.

A real example of the correct, in our opinion, firmware security model will be considered in the next article.