How to create an operating system

    According to the report of Ivan Ruzanov “Windows - how to create an operating system” from the conference “ Platform 2009 ”.

    As antonms already wrote , reports of reports from the Platform became available. I looked at a few of them and want to share the one I liked most. Considering the busyness of an ordinary habr-user and the duration of the initial report recording (1 hour 20 minutes), below I give a detailed presentation of the material.
    • Windows History
    • Windows today
    • Windows development
    • Windows 7 kernel changes
    • Windows Maintenance Updates / Fixes

    Windows History

    Few people know when the development of Windows NT began - more than 20 years ago, in November 1988. Rumor has it that Microsoft bought / stole Windows NT. Of course, this is not so. To work on the new operating system, Microsoft invited Dave Cutler, who worked at DEC and was one of the developers of the DEC VAX VMS operating system . In 1988, the new operating system development team was only 12 people, among whom, along with Dave Cutler, were also former DEC employees who participated in the development of the DEC VAX VMS. And all the code for the new Windows NT was written by Microsoft.
    The abbreviation "NT" stands for marketing as "New Technologies", but in the design documentation, it meant a completely different thing. The fact is that Windows NT was developed for the new Intel i860 processor , which was not yet released in 1988 . Its code name was "N10" ( NT en) .
    The first version - Windows NT 3.1, was released 5 years later, in 1993. At this point, the team already had 250 developers.

    Windows today

    • 1 billion users
    • 140 million lines of code (including test code and tools)
      Windows code is very different. Some parts were written 20 years ago, some appeared only in the current version. For example, the Web Services on Devices (WSD) code in Windows Vista exists in its first version, the GDI code is at the final stage of its development and is almost unchanged, DirectX code is already well developed, but it is actively changing now.
    • 8000 developers
    • 36 languages ​​of localization
    • 20 years of development

    Windows development

    20-30 years ago, only one Waterfall programming methodology was used. It is a sequence:
    Specifications → Design → Implementation → Testing → Delivery.
    But such a methodology only works for small projects. For a product like Windows today, other methodologies are needed:All of these methodologies have both advantages and disadvantages. Depending on the size of the team and the development stage of the component, different Windows development teams apply different development methodologies.
    For Windows, as a product in general, the Product Cycle Model is used:
    • 3-4 months periods
    • Inside the period there is a “waterfall”
    The biggest problem in developing a product of this magnitude is that development takes time. At the initial stage, those problems that exist in the current time and existing means are solved. But the only thing that is constant is that everything will change. Over the years of development:
    • Requirements will change
    • Opportunities will change
    • Work schedule will change
    • Project will change
    • Users will change
    Despite the fact that different teams conduct development in different ways, there are "universal" rules:
    • Release of intermediate versions (milestones, beta, CTP) for the masses of testers
    • Release of internal assemblies with short cycles (1 day)
    • Simplicity and reliability of design
    • Personal and team code readings
    • Unit tests
    • Verification Tests
    • Any intermediate assembly should be of high quality (what is written should work)
    On my own, I note that over the month of working with Windows 7 build 6801 as the main OS on my home computer, I had a positive impression about this assembly.

    The whole Windows development process is built around a daily build:
    • This is the pulse of the product.
    • Development never stops
    • Daily auto testing
    • Early integration
    • Developer Responsibility
    • Apparent Product Condition
    Once upon a time there was only one source code branch, and all developers made changes directly to it. Now the development team is so large that it does not work. Many branches are supported, among which there is the main one - WinMain. Each laboratory has its own local development branch into which changes are integrated. Validated changes over time integrate into WinMain.
    Daily development cycle:
    • 15:00 - Changes to the source code control system allowed for integration
    • Build 6 versions (Free / Checked - x86, x64, IA64)
    • 18:00 - New versions are available for testing
    • The new version is installed on several thousand workstations and servers for testing.
    • Automated Stress Test
    • 05:00 - Test reports are analyzed, failures are diagnosed
    • 09:00 - Summary reports are automatically sent to teams
    • 09:30 - Team Leadership Summit
    All project participants, including the most senior managers, use intermediate versions on their working (and usually home) computers.

    What is Windows written on?
    • C, C ++, C #, Assembler (x86, x64, IA64)
      Assemblers are used to a rather limited extent in those situations where you cannot do without it.
    • Visual Studio, Source Insight, build, nmake
    • Source Depot - source control system
    • WinDbg, KD, NTSD - debuggers
    Many internal tools, such as build, can be downloaded from microsoft.com/whdc/devtools .

    Windows 7 kernel changes

    The Windows 7 kernel has undergone the following changes:
    • Refactoring.
      Why can't I remove the graphics subsystem in Windows?
      The answer to this question from a technical point of view is that the graphics subsystem in Windows is not independent, it is part of the Win32 subsystem.
      Windows 7 refactored many low-level components in order to break down dependencies. This will not be noticeable to users, only new Dlls will appear, for example, kernel32.dll is divided into kernel32.dll and kernelbase.dll .
      This partition made it possible to allocate a minimal core called MinWin (20 megabytes per disk).
    • EFI support for x86 and x64 (as in Vista SP1)
      Many manufacturers are trying to get rid of the BIOS in favor of EFI.
    • Booting from VHD (Virtual Hard Disk)
    • Parallel device initialization and start of services
      When loading Windows, it takes quite a long time to build a device tree. The PNP manager should poll bus drivers (PCI, USB, FireWire, etc.) for what devices they have. And most of the time the processor waits for the devices to respond (or not). After all, in order to identify devices on the bus you need to interview them. If they are, then they will answer, but if not, then you have to wait, and the processor is idle. Performing these tasks in parallel reduces load times.
    • Removing Dispatcher lock from the scheduler and PFN lock from the memory manager Over the
      past few years, the processor clock speeds have not increased, and development is going towards increasing the number of parallel instructions both at the core level and at the system level (multicore). In this regard, a lot of work has been done to improve scaling.
      The two hottest locks that were in the kernel, Dispatcher lock and PFN lock, were removed.
      Dispatcher lock was used by the scheduler when changing the state of threads. This lock was deleted, and the status of the "wait" thread was divided into several:
      • Waiting: In progress
      • Waiting: Completed
      • Pending: Canceled
      PFN lock was used when changing the attributes of physical memory pages. In a multiprocessor system, each processor requested access to this lock, which led to a large investment of time.
    • Support for 256 logical processors
      Previously, in Windows, a machine word was used as an affinity mask. This was due to the fact that it was so easy to find free processors - each bit represents a processor. Accordingly, 32 logical processors were supported in a 32-bit system, and 64 in a 64-bit system.
      In Windows 7, as a result of the transition to the affinity mask segment model, it became possible to support 256 logical processors. Processors began to be grouped into groups / segments. Each group can contain up to 64 processors. The result is backward compatibility, old programs only see processors in one group, and new programs that use new interfaces work with all processors in the system.
    • Improved energy saving: disabling processor sockets Today, there is a serious problem of energy saving not only for laptop owners, but also for data center owners. In the United States, 2% of electricity is consumed by computer data centers. Many of them shut down part of their servers during low user activity (weekends).
      It was found that it is much more profitable to disconnect the entire processor socket than one core per several, because in this case, you can disable the entire socket support infrastructure (memory controller).

    Windows Maintenance Updates

    Previously, updates were often cumulative (cumulative). This meant that if the erroneous code was contained in an early update of the component, then later versions would also contain this code. But not all users need all the updates, they have a different configuration.
    Now, after the release (RTM) on Windows, there are 2 versions of the source code:
    • RTM GDR (General Distribution Release)
      Includes those few changes that are for everyone. Mostly security fixes.
    • RTM LDR (Limited Distribution Release)
      During the installation of the update, the Windows Update client selects the branch it needs and installs the code from it.

    Create a security update

    Work on creating a security update begins with a vulnerability discovery. There are tons of different detection methods — internal security teams, security partners, developers. When a vulnerability is discovered, 2 parallel processes begin:
    • Patch development for all platforms
    • Search for “options”
      Large-scale search for similar vulnerability options on all platforms. Search not for an identical code, but for a similar one.
    After the development of the patch, the verification of its code begins. When they are completed, the fix is ​​integrated into the assembly, and the assembly is sent for testing:
    • Manual and automatic component testing
    • Automatic testing of file format distortions, network components, etc. (more than a million options)
    • Testing the entire system, including backward compatibility testing
    Only patches that meet all quality criteria are allowed to be released on Windows Update and Download Center.

    Thanks to everyone who read to the end =)

    Also popular now: