How is ntds.dit structured?

All Active Directory data is stored in the database in the ntds.dit file. The vast majority of applications interact with the directory through the DSA layer implemented in ntdsa.dll. In turn, the functions from ntdsa.dll do not work directly with ntds.dit, their functionality is limited by the needs of the directory service and they cannot give us an idea of ​​the internal structure of the Active Directory database. However, ntds.dit is nothing more than a JET Blue database . Each version of windows (starting with Windows 2000) has everything you need to work with this database.

In the article below I will try to highlight the following issues:

• What is the structure of the database?
• How does the data in ntds.dit produce a "tree"?
• How is group membership implemented?
• What is the format of the replPropertyMetaData attribute and how accurately are timestamps stored in replication metadata?

What do you need to look at ntds.dit?

At a minimum: esent.dll
For "comfortable" use from various programming languages ​​over the ESENT API, there are various "wrappers". I used Meneged Esent in conjunction with C #. There are many examples on the project’s website - so I’ll try to concentrate on the contents of ntds.dit below.

It should also be noted that the JET Blue database has such a parameter as PageSize. By default, it is 4096 (for those versions of esent.dll that I have encountered). So in ntds.dit, the page size is 8192 and this parameter should be correctly set before opening the database.
The ESENT API has several versions. These versions are incompatible! So ntds.dit from Windows Server 2008 R2 does not open on windows xp, only on Windoes 7. (I did not check backward compatibility)

Question one: What is the structure of the database?

The GetTableNames function will return a list of tables in the database:
datatable - the main table with all the data in the
hiddentable link_table
directory - according to the article on technet a table with information about related attributes (for example MemberOf)
quota_rebuild_progress_table
quota_table
sdpropcounttable
sd_table - according to the article on technet, the table contains information about inherited access rules for each directory object.

Let us dwell on the datatable table. The list of its columns looks like (in the screenshot the first few lines):

Deadlock? Not!
Most column names are in ATT format <single latin letter> <numeric code>. So this digital code is a unique identifier for the Active Directory attribute. There is one row in the table in which the value of the digital code of the column matches its value. If we deduce all values ​​of type Text from this line, we will see that this is the definition of the attribute “attributeID”.



There remains one “unencrypted” ATTc0 attribute - this is a reference to “objectClass”.
Please note that all columns that store Active Directory attributes are set up as Multi-value regardless of the scheme.

Question two: How does the data in ntds.dit produce a "tree"?

The column name “DNT_col” provokes that it is somehow related to the distinguishedName of the object (as it turned out later, they are equal).
The values ​​of the DNT_col column begin with 1, and the line with DNT_col = 1 corresponds to an interesting object with the attribute value name = "$ NOT_AN_OBJECT1 $ "
The line with DNT_col = 2 contains the attributes of an object with the name" $ ​​ROOT_OBJECT $ "(Not to be confused with RootDSE)
With DNT_col = 6, objectClass definitions start
again Is it a dead end? No again!
Let's go the other way.
Searching the ATTm589825 (name) column for a value of Text type “Administrator” returned a record with DNT_col = 3841 and PDNT_col = 1951
Search in the ATTm589825 (name) column for a value of the Text "Users" type (the container in which the standard administrator account is located) returned a record with DNT_col = 1951 and PDNT_col = 1944
Here it is! the PDNT_col column contains the DNT_col identifier of the parent object.
In the line with DNT_col = 1944 (PDNT_col = 1943) - there was a second-level domain object (ntds.dit was taken from the second-level domain controller)
In a line with DNT_col = 1943 (PDNT_col = 2) - a first-level domain object.


If you ever wondered: “Why are objects from cn = Configuration, dc = contoso, dc = com not displayed when connected to dc = contoso, dc = com?” Then you should pay attention to the NCDNT_col column - it contains links on the naming context object. (Moreover, the first-level domain object does not have a naming context - can it be so that it is not visible?). For Objects in the naming contexts “dc = contoso, dc = com” and “cn = Configuration, dc = contoso, dc = com”, the value of this column is different.

No less interesting is how from these numbers the complete distinguishedName of the object is obtained? Which attribute of the object is a relative unique name (RDN) and is used to build the full distinguishedName.
The RDNtyp_col column contains the attribute identifier (attributeID) containing the RDN.
The CNT_col column contains the number of objects associated with the current. This column is used by the Link Cleaner mechanism.
If the bit in the OBJ_col column is set to 1, then this line describes the directory object. Otherwise, it is a phantom object.

How is SubTree search implemented?
It is clear that with such a structure, for searching OneLavel it is enough to search among the records with PDNT_col equal to the DN of the search base, but how to search by SubTree? Go around all branches? No it's too complicated.
Let's look at the values ​​in the column with the speaking name Ancestors_col. It is striking that the deeper the object is in the hierarchy, the longer the value in this column. Each nesting level adds 4 bytes to the length. This is nothing more than a list of parent objects in order of precedence.
Moreover, the list starts with dn = 2, i.e. with "$ ROOT_OBJECT $"
Who is involved in keeping Ancestors_col up to date? According to the document, when moving an object to a new place, only its PDNT_col changes, and the Ancestors_col value is updated by the SDProp mechanism at the same time as the new rules for accessing the object are recounted.

Question three: How is group membership implemented?

Why since 2003 the server has the opportunity to include in the group a unlimited number of users, and even more than ~ 1200 values ​​cannot be written to the string or numeric multi-value attribute?
Yes, because “member” and “memberOf” are defined as links. There are no columns in the datatable table matching these attributes. The values ​​of these attributes are implemented as mappings in a separate link_table.
Perhaps this is why when we delete a group (without using RecucleBin) we lose information about its members - the object of the remote group receives a new identifier in DNT_col when moving deleted objects into the container (and the connection of the group and its member is built using this identifier)


The names and type of columns are hinted, and the analysis of the data contained in them confirms that:
link_DNT - Contains the identifier dn (corresponds to DNT_col from the datatable) of the object whose relationship is described (for example, a group object)
backlink_DNT - Contains the identifier dn of the associated object (for example, user object)
link_ncdnt - Contains the identifier dn of the naming context in which the communication participants are located.
link_usnchanged - Contains the USN of the last change.
link_data - I saw this field filled out only for NTDS Settings object links. Yes! NTDS Settings objects have links with directory partition objects. Apparently, through these links, replication parameters of various directory sections are set.
link_deltime - Contains the time the link was removed.
link_metadata - contains the metadata necessary to replicate the changes made. I saw the filled values ​​of this column only on Windows 2012. Christoffer Andersson writes that the column contains the structure DS_REPL_VALUE_META_DATAand here I strongly disagree with him. The DS_REPL_VALUE_META_DATA structure is used at a higher level - ntdsa.dll calls return it. The data contained in this column is smaller than necessary under the DS_REPL_VALUE_META_DATA structure. While this column remains a mystery to me. In principle, replication metadata for link attributes can be obtained from the constructed msDS-ReplValueMetaData attribute, but this does not reveal the internal mechanism for processing and storing this data.

Fourth question: What is the format of the replPropertyMetaData attribute and how accurately are timestamps stored in replication metadata?

This attribute stores binary replication metadata:

I have not figured out what is stored in the first 8 bytes of this structure.
You should pay attention to the fact that only replicated attributes with values ​​are listed in the replPropertyMetaData structure. So, for example, you will not see the logonTimestamp attribute identifier in replPropertyMetaData.
There is an interesting point: in the replPropertyMetaData structure for Active Directory security objects (users, groups, and computers), the objectSid attribute is present.
By the way, in order to decipher the value of this attribute - it’s not necessary to strain so much - there is a built attribute msDS-ReplAttributeMetaData - it represents nothing but the parsed value of replPropertyMetaData.

Active Directory timestamps are 64-bit unsigned integers that indicate the number of seconds that have passed since 00:00 on January 1, 1601. An interesting nuance follows from this: when editing one attribute of the same object on two domain controllers for 1 second (for example, when batch processing a large number of users), an unexpected result can be obtained.
According to an article on technet , the version of the attribute from the controller with a lower GUID wins.

Information used:

technet.microsoft.com/en-us/library/cc772829%28v=ws.10%29.aspx
blogs.chrisse.se/2012/02/11/how-the-active-directory-data-store-really-works -inside-ntds-dit-part-1
blogs.chrisse.se/2012/02/15/how-the-active-directory-data-store-really-works-inside-ntds-dit-part-2
blogs.chrisse .se / 2012/02/20 / how-the-active-directory-data-store-really-works-inside-ntds-dit-part-3
blogs.chrisse.se/2012/02/28/how-the- active-directory-data-store-really-works-inside-ntds-dit-part-4
gexeg.blogspot.ru/2009/12/active-directory.html
www.ntdsxtract.com

All experiments were performed on ntds.dit from Windows Server 2012 Eng

UPD . For starters, a small utility that I wrote to study ntds.dit (maybe someone will do).