July 15, 2015 at 14:44What's under the hood of virtual disks? (on the example of VHD and VHDX)
Have you ever worked with virtual machines, created virtual disks? If yes, then you probably noticed such convenient features as dynamic increase in disk size (the ability to store only what was recorded) and the ability to create snapshots - snapshots of the state of the disk. If you are interested in knowing exactly how these capabilities are achieved and how data is stored in VHD and VHDX files, welcome to cat.
Fixed drives
A virtual disk is usually a simple file inside which everything that the virtual machine writes to some disk device is stored. For fixed disks, a full-size file is immediately allocated, which does not subsequently change in size. However, here you should make a reservation, and remember about the ability of many file systems to create "compressed" files. Compression is usually achieved by not storing file blocks filled with zeros (for example, NTFS, XFS, and VMFS do this). Even if there is 500GB of free space on your file system, you can easily create a 1TB fixed virtual disk and work with it until you run out of free space.
Fixed disks can be stored in two ways:
- Simple image files . They are RAW, they are flat (* flat.vmdk for example). Everything is clear from the name. Such files (almost) do not contain any specific metadata. A fixed VHD can hold 512 bytes of its metadata at the end of a file. Otherwise, it cannot be distinguished from a copy of the “real” disk in a file.
- Degenerate dynamic disks . Such disks are stored in the same format as dynamic ones, only all the required space is allocated immediately upon creation. It is as if a dynamic disk was immediately prescribed after the creation from start to finish. You will get such a file if you create a fixed VHDX.
Dynamic vhd
File systems write to disk in a chaotic manner. In order to ensure a gradual increase in the virtual disk file under such conditions, a translation system is needed. One of the easiest translation methods is a table, which for each logical block will indicate its location inside the file or say that such a block has not yet been allocated.
This is precisely the idea laid down in the dynamic VHD format (and not only). The logical space of a virtual disk (what the OS inside the virtual machine sees as a disk) is divided into blocks of equal size, for example, 2 megabytes each, which are addressed using the BAT - Block Allocation table.
When creating snapshots, it may be necessary to give the status of "empty-selected" not a separate block, but a separate sector in the block. Therefore, each block is supplied with a bitmap, which is written before the block. With a logical sector size of 512 bytes and a block size of 2 megabytes, a bitmap for a block will occupy exactly 1 sector (512 * 8 = 2 097 152/512). Those. one generalized block will occupy 4097 sectors.
In addition to BAT and generalized blocks, the VHD file also contains Hard Disk Footer structures (512 bytes) at the very end and a copy at the very beginning. And the Dynamic Disk Header (1024 bytes) at the beginning of the file. They store various metadata about the virtual disk: its size, format version, timestamps, block size, BAT offset, number of records in it, etc.
To summarize, the contents of the VHD file look like this (proportions are arbitrary):
Dynamic VHDX
The dynamic VHDX format is similar in general to VHD - the logical space is also divided into blocks that are addressed by a special translation table, there are also bitmaps to clarify the status of a particular sector. But in the details there are many differences.
To begin with, in VHDX the size of one bitmap is fixed - 1 megabyte. And he already covers several blocks. For example, with a logical sector size of 512 bytes (VHDX can also “give” a sector of 4096 bytes) and a block size of 2 megabytes, one bitmap “covers” 2,048 blocks. This value is also called chunk ratio .
The second difference is that a block with a bitmap is independently addressed from BAT. First, there are 2048 cells (chunk ratio), which address the corresponding data blocks, then comes the cell addressing the bitmap block, and so on.
The next difference is that the entry in the BAT now also stores the status of the block. For a data block, this is:
- NOT_PRESENT - absent;
- UNDEFINED - indefinite (a place is allocated in the file, but there are irrelevant data);
- ZERO - filled with zeros (no place in the file is allocated);
- UNMAPPED - for all sectors of the block, the UNMAP command was executed;
- FULLY_PRESENT - the block is fully present:
- PARTIALLY_PRESENT - the block is partially present.
An entry for a bitmap block can have only two statuses:
- NOT_PRESENT - the bitmap block is missing;
- PRESENT - bitmap block is present.
Bitmaps and partially present data blocks can only be in differential disks that appear when snapshots are created (I will tell about them below). In other cases, bitmaps are absent, and data blocks have different statuses.
In general, the structure of a VHDX file looks something like this:
Briefly about the remaining sections:
- At the very beginning of the disk is the FileIdentifier structure - it contains the signature and comment about the application that created the VHDX file. It is she who is open in the HEX editor in the screenshot at the beginning of the article;
- Next come two versions (to protect against failure) of the header. It indicates the offset and size of the log, as well as some other parameters;
- A log is a circular buffer through which all operations on metadata are carried out, except for Headers. This is done, again, to protect against sudden outages;
- The concept of regions is introduced. The specification lists only 2 types of regions: BAT and Metadata, but, it is understood that there may be more. Regions are addressed using the table of regions, which is stored in duplicate (and here protection from failures);
- Metadata records are stored in the Metadata region and are addressed using a metadata table in which applications can add metadata of their types. According to the specification, here we must find records about block size, logical sector size, disk size, and others.
Snapshots
A snapshot is a snapshot of the state of a virtual disk at some point in time. Having such a snapshot, we can roll back all the changes made after this moment.
If we are talking about VHD and VHDX disks, then when creating a snapshot, a new file is created in which all subsequent changes are recorded. Such a file is called a "delta" or "differential disk" (from the English. Differencing).
We said earlier that the format of dynamic disks allows you to store only recorded data. The same format is perfect for storing only modified data. Yes, delta files are in exactly the same format as dynamic disks. Only the interpretation of free blocks changes. For a delta, a free block means that you should not just return zeros, but try to read the block from the previous layer — if there is a previous delta, then from it, and if not, from the base disk.
If we remove the upper delta, we get the previous recorded state, and if both, then the earliest.
View from the side of data recovery
From the point of view of data recovery, virtual disks are bad primarily because additional levels of translation are introduced between the file and the disk.
Each level is an additional mixing of data, which increases fragmentation, and a potential point of failure. Some BAD sector now has much more opportunities to do a lot of problems.
In general, to read the data of a file from a virtual disk, you must have
- physical disk file system metadata that describes the location of the virtual disk file;
- metadata of the virtual disk itself (BAT, etc.);
- file system metadata on a virtual disk that describes the location of the file.
If you are very “lucky”, then the task of data recovery can turn into a puzzle assembly without a sample. Not that I discouraged using virtual disks, but the simpler the data is stored, the easier it will be to recover.
