History of migration of operating systems

Original author: Michael Steil
  • Transfer
From a translator: I present to you the translation of an article by Michael Steil . For a long time I wanted to prepare a similar overview of the methods of using virtualization for compatibility problems. I even published some notes on this subject: in the textbook on simulation, chapter 1 , and on Habr in a post about system VMs . However, I was not able to reveal the question as deeply as it is presented in this work. So I decided to share the translation with readers.


Manufacturers of operating systems face this problem once or twice a decade: they need to transfer their user base from the old operating system to their very different new OS, or they need to switch from one CPU architecture to another while maintaining the ability to run old applications without modifications, and also help third-party developers port their applications to the new OS.

Let's look at how this has happened in the last 30 years, using MS DOS / Windows, Apple Macintosh, Amiga OS, and Palm OS as examples.




From CP / M to PC-DOS / MS-DOS


CP / M is an 8-bit OS from Digital Research that runs on all versions of Intel 8080-based systems. 86-DOS from Seattle Computer Products, later known as MS-DOS (also referred to as PC-DOS on IBM machines), was a clone of CP / M for the later Intel 8086; A similar situation was with CP / M-86 from DR itself (this OS was later called DR-DOS).

While not binary compatible, the Intel 8086 was “assembler compatible” with the 8080. This meant that it was easy to convert an 8-bit program written in assembler for the 8080 or Z80 to an assembler program for the 8086, as these two the architectures were very similar (backward compatible memory model, sets of registers easily mapped onto each other), only machine encodings of commands differed.

Since MS-DOS used the same ABI conventions and a memory card, this OS was compatible at source with CP / M. On CP / M systems with access to a maximum of 64 kB of memory, the address range from 0x0000 to 0x0100 (zero page) was reserved for the needs of the OS and contained, among other things, command line arguments. The executable application was located starting at address 0x0100 and higher, and the OS was located at the top of the memory; the application stack grew down, starting just below the start of the OS. The 8086 memory model breaks the memory into (overlapping) contiguous 64-kbyte segments, so each of these segments is actually sufficient for the 8080 virtual machine. Therefore, MS-DOS .COM executable files are up to 64 kbytes in size and are loaded into memory starting from the address 0x100 The 0x0 - 0x100 range in MS-DOS is called the Program Segment Prefix and is very similar to the CP / M page zero. Due to the high degree of compatibility of these processors and their ABI, transferring the program from CP / M to DOS was relatively painless. However, such a port could use a maximum of 64 kbytes of memory (examples are 1982 Wordstar 3.0). But this made it possible to have a single code-based application base for both operating systems with just a few macro definitions and two different assembler programs.

MS-DOS 2.0 brought with it new, more powerful API conventions (file handles instead of FCB structures , subdirectories, .EXE format) and transferred most of the APIs from CP / M to obsolete ones. But DOS remained compatible with CP / M up to the latest version.
From CP / M to PC-DOS / MS-DOS
ChangeNew CPU
New OS code base
New application executionDirect
Execution of old applications
(no modification)
Not supported
Execution of old driversNot supported
Application Porting ToolsHigh source and ABI compatibility

From DOS to Windows


Microsoft Windows was originally designed as a graphical shell on top of MS-DOS. All work with devices and the file system occurred through DOS API calls. Therefore, all MS-DOS drivers were executed directly, and Windows could use them. DOS applications could still run after exiting from Windows to pure DOS.

On Windows / 386 2.1, the model has been changed. This was already the real kernel of the OS, executing many virtual machines in virtual 8086 (V86) mode side by side; one was allocated for the MS-DOS OS session, and one was allocated for each Windows application. The DOS virtual machine was used by Windows to access device drivers and the file system, so it was almost a driver compatibility layer that runs inside the VM. The user could run any number of additional DOS VMs for working with DOS applications; and each contained its own copy of DOS. Windows intercepted calls from the screen memory, as well as some system calls, and redirected them from the Windows graphics driver or to the first “main” DOS VM.

Windows 3.x started using native drivers for Windows, which replaced calls inside the DOS VM, and also began to redirect some DOS calls to devices inside Windows. The standard installation of Windows 95 no longer used calls to DOS VMs for the needs of working with drivers and the file system, however, if necessary, this mode could still be used.

DOS was not only a compatibility environment for older drivers and applications, it also provided a Windows command line, so when Windows 95 introduced support for long file names, I had to intercept DOS API calls to provide new functionality for command-line utilities.
From MS-DOS to Windows
ChangeNew OS
New application executionDirect
Execution of old applicationsVirtual machine with old OS
Execution of old driversVirtual machine with old OS
Application Porting ToolsNot provided

From DOS to Windows NT


Systems of the Windows NT family were not based on DOS, however, from the very first version, they allowed executing MS-DOS applications. As in the case of the non-NT version of Windows, in this case the DOS program worked in the V86-mode of the processor. However, instead of using a full copy of MS-DOS inside the VM, NT placed only application code in it, intercepted all system calls and device calls and wrapped them in NT API calls. In fact, this method is not pure VM: V86 mode was used only to create the memory model necessary for supporting DOS applications.

There is a widespread misconception that the Windows NT command line is a “DOS shell”. In fact, the interpreter and auxiliary utilities were native NT applications, and the NTDVM subsystem (DOS virtual machine) did not start at all until there really was a need to launch a DOS program from the command line.
From DOS to Windows NT
ChangeNew OS
New application executionDirect
Execution of old applicationsCompatible APIs
Execution of old driversNot supported
Application Porting ToolsNot provided

From Windows 3.1 (Win16) to Windows 95 (Win32)


Starting with the release of Windows NT 3.1 in 1993, it became clear that it would eventually replace the classic Windows. Despite the fact that NT had the same user interface with Win16 and good DOS compatibility, each new version that was released usually required a fairly powerful computer to work. Therefore, the migration from Windows to Windows NT went the opposite way: each version of Windows became more and more similar to NT until they became quite similar. By that time, even the weak computers of that era had become powerful enough to work under NT.

A big step towards rapprochement between Windows and Windows NT was support for the native Win32 API for NT. The first step was a free upgrade of “Win32S” to Windows 3.1, which provided a subset (hence the “S” - subset - in the name) of the Win32 API on classic Windows. Win32S expanded the Windows kernel with the ability to create 32-bit address spaces for all 32-bit applications (in NT, each application created its own space). It also made available versions of some libraries from NT (for example, RICHED32.DLL), as well as 32-bit DLLs that received calls from low-level Win32API calls (subsystems "GDI" and "USER"), were automatically translated into calls to the Win16 subsystem ("thunking ").

In Windows 95, this functionality was available by default. She also used isolated address spaces for 32-bit applications, supported a larger subset of the Win32 API. Some basic applications became 32-bit (for example, Explorer), however, a significant part of the base system was still 16-bit. With the advent of Windows 95, most developers switched to creating 32-bit applications, which also allowed compiling them on Windows NT systems.
From Windows 3.1 (Win16) to Windows 95 (Win32)
ChangeNew processor and addressing mode
New application executionCompatibility layers
Execution of old applicationsDirect
Execution of old driversDirect
Porting methodsHigh source compatibility

From Windows 9x to Windows NT


The second step in migrating from 16-bit Windows to Windows NT occurred when switching from Windows ME to Windows XP (NT-family) in 2001. Windows NT (2000, XP ...) was a completely 32-bit OS with Win32 API, but also allowed to execute 16-bit Windows applications through the translation of Win16 calls to Win32 (thunking).

The Windows NT 3.1 / 3.5 / 4.0 driver model (Windows NT Driver Model) was different from the classic Windows (VxD) approach, so Windows 98 (a successor to Windows 95) and Windows 2000 (a successor to Windows 4.0) both supported the new “Windows Driver Model”. The same driver could now work in both OSs, but each while continuing to maintain its original mode for older drivers.

When Microsoft transferred home users to NT systems, most applications, games, and drivers made money on Windows XP. The only thing I had to rewrite was the system utilities.
From Windows 9x to Windows NT
ChangeNew OS
New application executionDirect
Execution of old applicationsWin16:
Win32 API translation: live
Execution of old driversAPI support from the old OS
Application Porting MethodsHigh source code compatibility
API support from old OS

From Windows i386 (Win32) to Windows x64 / x86_64 (Win64)


The transition from 32-bit to 64-bit Windows is currently underway. Windows XP was the first Microsoft OS available for the Intel 64 / AMD64 architecture, all subsequent systems (Vista, 7, 8, 10) are available in versions for 32 and 64 bits. In 64-bit editions, the kernel works in 64-bit mode, like all system libraries and most applications. The 32-bit API is supported through the WOW64 subsystem (Windows-on-Windows 64). A 32-bit application links with 32-bit libraries, however, low-level API calls are translated via WOW64 to their counterparts for 64-bit DLLs.

Since drivers run in the same address space as the kernel, it was difficult to provide support for 32-bit drivers, so it is not. Support for DOS and Win16 applications in 64-bit versions of the OS was also discontinued.
From Windows i386 (Win32) to Windows x64 / x86_64 (Win64)
ChangeNew processor and addressing mode
New application executionStraight
Execution of old applicationsCall broadcast
Execution of old driversNot supported
Application Porting MethodsHigh source compatibility

From Apple Macintosh for 68K to Macintosh on PowerPC


Apple moved its computers from Motorola 68k processors to Motorola / IBM PowerPC between 1994 and 1996. Since the Macintosh operating system, System 7, was mostly written in assembler for 68k, it could not be easily converted to an OS for PowerPC. Instead, most of it was performed through a simulation. The new “nano-core” received and processed interrupts and performed minimal memory management, abstracting the features of PowerPC, and the integrated 68k emulator executed the old OS, which was modified to communicate with the nano-core. Thus, System 7.1.2 for PowerPC was actually paravirtualized and executed on top of a very thin layer of the hypervisor.

The first version of Mac OS for PowerPC ran most of the system code inside the 68k emulator, including drivers. Part of the code critical for performance has been rewritten for direct execution. The executable loader could detect programs with PowerPC and could execute them directly. Most of the communications with the OS still went through the emulator. In later versions of Mac OS, an increasing proportion of code was being copied from 68k to PowerPC.
From Macintosh for 68k to Macintosh for PowerPC
ChangeNew processor
New application executionCall broadcast
Execution of old applicationsParavirtualized old OS inside the emulator
Execution of old driversParavirtualized old OS inside the emulator
Application Porting MethodHigh source compatibility

From Classic Mac OS to Mac OS X


Similar to how Microsoft migrated users from Windows to Windows NT, Apple switched from the classic Mac OS to Mac OS X. Whereas the classic Mac OS was a hackish OS with cooperative multitasking, without memory protection, while still executing part system code inside the 68k emulator, Mac OS X was based on NEXTSTEP, a modern UNIX-like OS with a completely different API.

When Apple decided to migrate, they ported the system libraries of the classic Mac OS (“Toolbox”), while removing calls that could not be supported on a modern OS, replacing them with alternatives, and called the new API “Carbon”. The same API became available in classic Mac OS 8.1 in 1998, so developers were able to update their applications to run on Mac OS X, while maintaining compatibility with the classic Mac OS. When Mac OS X became publicly available in 2004, carbonized applications by this point could run on both OSs without recompilation. This is reminiscent of Microsoft's approach to bringing classic Windows to Windows NT.

But since it was impossible to expect that all applications would be converted to carbonized by the time Mac OS X arrived, the new OS also contained a Classic or Blue Box virtual machine in which unmodified Mac OS 9 was running and could support arbitrary number of old applications. Requests to the network and file system inside the VM were intercepted and redirected to the host OS, and integration with the window manager allowed the two desktop environments to look almost seamless.
From Classic Mac OS to Mac OS X
ChangeNew OS
New application executionDirect
Executions of old applicationsClassic: VMs with the old
Carbon OS : Common API for both systems
Execution of old driversVirtual machine for old OS
Application Porting MethodCommon API for both systems

Migrating Mac OS X from PowerPC to Intel IA-32


In 2005, Apple announced a second transition from one CPU architecture to another, this time from PowerPC to Intel IA-32. Since this OS was primarily written in C and Objective-C, it could easily be ported to IA-32. Apple claims that they always supported the OS version for IA-32, starting with the very first release.

In order to run old applications that have not yet been ported to IA-32, Apple has included the Rosetta emulator in the new OS. This time it was not closely integrated with the kernel, as was the case with the transition from 68k to PowerPC. The kernel only provided the ability to run an external recompiler for the application whenever it turned out to be intended for PowerPC. Rosetta broadcast all application code and library code and provided an interface for interacting with the OS kernel.
Mac OS X with PowerPC on Intel IA-32
ChangeNew CPU architecture
New application executionDirect
Execution of old applicationsUser mode emulator
Execution of old driversNot supported
Application Porting MethodHigh source portability

Mac OS X 32 bits on Mac OS X 64 bits


The next transition for Apple was to migrate from Intel's 32-bit architecture to its 64-bit version. It happened on Mac OS X 10.4 in 2006. Although the entire OS could be ported to 64 bits, as was the case with Windows, Apple decided to use an approach more similar to switching to Windows 95: the kernel remained 32-bit, but received support for 64-bit applications. All system applications and drivers remained 32-bit, but some libraries also received 64-bit options. Thus, a 64-bit application linked with 64-bit libraries and made 64-bit system calls, which were converted from 32-bit inside the kernel.

Mac OS X 10.5 began to ship all the libraries in the 64-bit version, the kernel remained 32-bit. In version 10.6, the kernel was converted to 64 bits, and the use of 64-bit drivers became mandatory.
Mac OS X 32 bits on Mac OS X 64 bits
ChangeNew CPU and addressing mode
Execution of new programsCall broadcast
Execution of old programsDirect
Execution of old driversDirect
Application Porting MethodCarbon: Not Supported
Cocoa: Source Compatibility

AmigaOS with 68k on PowerPC


The Amiga platform used the same OS for the 68k CPU in the Commodore era from 1985 to 1994. However, since 1997, there has been an upgrade for a motherboard with a PowerPC architecture CPU. This operating system did not provide its source code for third parties, so it was impossible to port it to PowerPC. For this reason, Amiga OS itself continued to run on a dedicated 68k chip, and the advanced program loader detected PowerPC code and transferred it to the second CPU. All system calls were routed through the library layer back to 68k.

AmigaOS 4 (2006) is the real AmigaOS port on PowerPC. This transition required a lot of effort, since a lot of the source code of the OS had to be converted first from BCPL to C. Application support for 68k is done through emulation of binary code and forward API calls.
AmigaOS c 68k on PowerPC (version 3.x)
ChangeNew CPU
New application executionBroadcast system calls (on the new CPU)
Execution of old applicationsDirect (on the old CPU)
Execution of old driversDirect
Application Porting MethodHigh source compatibility

AmigaOS c68k on PowerPC (version 4.0)
ChangeNew CPU
New application executionDirect
Execution of old applicationsUser level emulator
Execution of old driversNot supported
Application Porting MethodHigh source compatibility

Palm OS with 68k on ARM


Palm switched from 68k to ARM in Palm OS 5 in 2002. The OS was ported to ARM, and the 68k Palm Application Compatibility Environment (“PACE”) emulator was included to run older applications. However, Palm did not recommend developers to switch to writing applications for ARM and did not even provide an environment for the execution of such applications running on the OS. It was argued that most applications for Palm OS spent most of their time in the (already optimized "native") code of the operating system, and therefore they would not have received tangible gains in speed when porting.

However, for applications that use CPU intensively or contain code to compress or encrypt data, Palm provided a method for executing small pieces of ARM code inside an application for 68k. Similar ARMlets (later called PNOlets - “Palm Native Objects”) could be called from code for 68k. They provided a minimal data transfer interface (one integer per input and one integer per output). The developers themselves had to take care of the transfer of large structures in memory and solve the problems of byte order (endianness) and their alignment. Procedures from the ARM code could neither call the code for 68k nor make API calls to the operating system.

Binding to the 68k architecture (which is not native to the hardware used) as the main one for most applications actually meant using the virtual architecture of user applications, in the manner of Java or .NET. The transition to ARM was almost invisible to third-party developers, and in theory, this approach allowed Palm to again change the host architecture, if necessary in the future, at minimal cost.
Palm OS with 68k on ARM
ChangeNew CPU
New application executionNot supported (PNOlet for procedures)
Execution of old applicationsUser level emulation
Execution of old driversNot supported
Application Porting MethodNot supported (PNOlet for procedures)

conclusions


Let's take a look at the transitions between the OS and the CPU and the ways in which different vendors approached the migration task.

Bit depth (address space width)


The transition to a new processor mode is the easiest for the OS, as old applications can continue to be executed directly, and API calls can be translated. At the same time, the OS itself can either continue to use the old mode, converting system calls from new applications to the old format, or switch to a new bit capacity and convert calls to old applications. There are also two ways to intercept the calls themselves. It is possible to intercept high-level API calls (such as creating a GUI window), but this is difficult to perform because there are many such calls, and it is difficult to write the correct converter for so many functions.

Alternatively, the OS can convert low-level system calls. In this case, the intercepted interface is very narrow. But, since old applications link to old libraries, and new ones to new libraries, their storage and use consumes twice as much space (memory) in cases where the user runs old and new applications at the same time.
New processor mode or bit depth
OSOld modeNew modeCall Direction
Call pickup rate
Windows16 bit32 bitnew to oldlibraries
Windows NT32 bit64 bitold to newcore
Mac OS X32 bit64 bitnew to oldcore

In the case of switching from 16 to 32 bits in Windows, the OS remained 16-bit, and calls were converted from a 32-bit format to a 16-bit API. When Windows NT was converted from 32 to 64 bits, the entire OS became 64-bit, and old system calls were converted to new ones. The same transition was performed differently in the case of Mac OS X: the OS remained 32-bit, and 64-bit system calls were broadcast at the kernel level.

The approaches used for Windows NT and Mac OS X are similar, since in both cases, 32-bit applications continued to use 32-bit libraries, and 64-bit ones continued to use the new 64-bit libraries. The difference was in the new kernel mode. In the case of Windows, there was an advantage in the form of the possibility of using more than 4 GB of memory in kernel mode and some acceleration from using the new 64-bit set of registers. In the case of Mac OS X, it remains possible to continue to use the old 32-bit drivers without modifications. And yes, the second step Mac OS X later completely switched to the 64-bit kernel and drivers.

CPU


Changing the processor used is more difficult, since the old machine code can no longer be executed on the new architecture. Some operating systems were not so easy to transfer.
New processor
OSOld cpuNew CPUMethod for starting old applicationsCall pickup rate
CP/M, DOS8080/Z808086Перекомпиляция-
Macintosh68KPowerPCИсполнение ОС и приложения внутри эмуляции-
Mac OS XPowerPCi386Эмуляция уровня пользователяядро
Amiga68KPowerPCДвойной ЦПУ и трансляция вызововбиблиотеки
Palm68KARMЭмуляция уровня пользователябиблиотеки

Mac OS X for Intel and Palm OS for ARM were written in a platform-independent manner enough to be portable to the new architecture. Both included recompilers that ran the old code. This is the easiest approach. Amiga OS could not be ported this way, since its source code was not available. Therefore, the system included both processors - the original OS and its code were executed on the old processor, and the new code worked on the new processor, sending system calls to the old CPU.

For the classic Macintosh (from 68k to PowerPC), the source code was available, but still could not be easily changed. Therefore, the transition was performed similarly to the situation with Amiga, but on one CPU. Most of the OS was executed inside the emulator, new applications were launched directly, causing system calls back from the emulator.

MS-DOS was an implementation of an old OS made by another company. It did not support the execution of old binary code. Instead, developers were asked to recompile their applications.

Operating Systems


The transition to a new OS with the preservation of all users and programs is the most difficult of those considered.
New OS
Old OSNew OSMethod for executing old applications
CP / MDosCompatible API
DosWindowsVirtual machine with old OS
DosWindows NTAPI emulation
Windows 9XWindows NTCompatible API
Mac OSMac OS XClassic: virtual machine with old
Carbon OS : compatible API

The approach chosen depends on plans to support the old API from the old OS. If the API was good enough to be supported in the new OS, the new OS should just support the same API. This was done for transitions from CP / M to DOS and from Windows 9x to Windows NT. In a sense, this was done to switch from the classic Mac OS to Mac OS X, but in this case Carbon was not the main API of the new OS, but only one of the three supported (Carbon, Cocoa, Java). Virtually everything except Cocoa has now been deprecated.

If the old API is not good enough for the new OS, but it is necessary to support the work of old applications, it makes sense to keep the old OS inside the VM along with applications for it. This was used in the case of Windows for running DOS applications, and in Mac OS X for older Mac OS applications.

If the interface of the old OS is relatively small or a detailed repetition of the semantics of calls is not necessary, emulating the API may be the best solution: intercepting system calls of the old application with mapping them to new ones. This was used to enable DOS applications to work on Windows NT; the compatibility level was average.

Conclusion


It is unexpected to see how many different ways migration was performed for all the OSs examined, no approach repeated completely different. The reason for this, perhaps, is that the conditions for the transition in each case were slightly different, and it took a lot of time to develop the best solution for each specific case.

But there are general patterns. Over time, the OS has become more modern and less clumsy; in addition, migrations went through many small steps, avoiding big jumps. Modern operating systems, Windows NT and Mac OS X, can be ported to new architectures relatively easily, emulators help run old applications, and translation of system calls can be used to map them to OS-native calls. Thanks to the abstractions that your systems use, the OS can be ported to the new architecture or bit-wise in steps, and part of it can remain in the old environment, and part can be transferred to the new one. The same abstractions allow developers to replace entire subsystems and remake parts of the OS without a strong effect on users. These processes are becoming more streamlined and mature - and, alas,

Literature


Unfortunately, some links from the original post lead nowhere, and I could not find the documents.

1. Pat Villani. FreeDOS Kernel: An MS-DOS Emulator for Platform Independence & Embedded System Development
2. Broken link to Apple's PowerPC documentation
3. Brian Maas. Palm OS 5 ARM Programming
4. PNOLet Samples

Also popular now: