Do not trust the memory information in the Task Manager
- Transfer
For many years of using Windows, I’m used to Task Manager. From there I killed hundreds of apps for bad behavior. In the same place looked, who from them eats resources. Until I started working with machines that have hundreds of gigabytes of memory, and applications have corresponding requests. This article will discuss why Task Manager poorly tracks memory and what to use in return. First, the memory allocation mechanism in Windows.
tl; dr: Task Manager hides paging memory information and process virtual space. Better use Process Explorer from the Sysinternals suite.
When a new process starts, the OS assigns a continuous address space to this process. On 32-bit systems, this space can be 4 GB, usually 2 GB for the kernel, and the rest for the process. In this article, we ignore the use of memory by the kernel. In 64-bit systems, process-reserved memory can grow to a whopping 64 TB. What will this process do with a few terabytes of memory when we actually have a measly 8 GB? First you need to understand what is reserved and transferred memory.
Not all parts of this huge address space are equal. Some parts of the process address space are actually supported either by physical RAM or by disk (see below). A reserved memory is considered to be transferred (Committed) if the OS offers you this memory when you try to use it. The rest of the address space, and this vast majority remains available for reservation. That is, the OS may not always offer you this memory block for use: it may make a copy on a disk (paging file), for example, or it may not. In C ++, memory is reserved by calling VirtualAlloc . So the transferred memory is a hardware limited resource in the OS. Let's get a look.
The swap file is a great idea. Basically, the OS understands that some parts of the memory are not particularly used by your application. Why waste real physical memory on him? Instead, a process in the kernel writes this unused fragment to disk. Until it is addressed again, only then will it be returned to memory.
For a more detailed explanation of how memory works in Windows, I recommend the lecture “Secrets of memory management” by Mark Russinovich.
There is a lot to follow and analyze. Who to contact? Of course, to the Task Manager!
RAM memory is usually called the Working Set, while all allocated memory is usually called Private Bytes. DLLs confuse definitions, so for now we’ll ignore them. In other words:
By default, the Task Manager shows the working set for any process:
And this is the number I was looking at all the time. How did I know that in the Task Manager information about the transferred memory is in the column
Task Manager allows you to add information about the transferred memory, if you right-click on the columns and select the appropriate item
Fortunately, there are many other resources for tracking resources. PerfMon (System Monitor) is installed on each machine under Windows, which gives very detailed information about each process and the system as a whole:
Interestingly, System Monitor can actually explore and compare metrics on two or more computers on the network. This is a very powerful tool, but Task Manager is obviously more convenient for users. As an interim solution, I recommend Process Explorer :
Boom! Visual Studio, why are you still in 32-bit mode (note its virtual size)? Peak memory usage on my computer at the level of 89% of the maximum is still tolerable. It will come in handy later.
Addition: many pointed to other convenient tools, includingVMMap and RAMMap .
Fortunately, this is not some unnecessary trivia OS. Actual information about memory consumption has helped me many times in debugging various problems.
The most important thing is to find the untouched parts of the allocated memory. Paging data is also important: this memory is transferred, but is rarely used or not used at all.
Even if the memory will sometimes be used, it is important to understand that this is an expensive resource, so it is by no means necessary to take this path. Memory leaks will appear here.
For these reasons, I had previously heard the suggestion to completely remove the paging file and equate the allocated memory to the working set. However, this is a double-edged idea. Then the OS is not able to reset the memory in case of malfunction of applications, which are sometimes wasted memory.
tl; dr: Task Manager hides paging memory information and process virtual space. Better use Process Explorer from the Sysinternals suite.
Windows Memory Allocation
When a new process starts, the OS assigns a continuous address space to this process. On 32-bit systems, this space can be 4 GB, usually 2 GB for the kernel, and the rest for the process. In this article, we ignore the use of memory by the kernel. In 64-bit systems, process-reserved memory can grow to a whopping 64 TB. What will this process do with a few terabytes of memory when we actually have a measly 8 GB? First you need to understand what is reserved and transferred memory.
Reserved and transferred memory
Not all parts of this huge address space are equal. Some parts of the process address space are actually supported either by physical RAM or by disk (see below). A reserved memory is considered to be transferred (Committed) if the OS offers you this memory when you try to use it. The rest of the address space, and this vast majority remains available for reservation. That is, the OS may not always offer you this memory block for use: it may make a copy on a disk (paging file), for example, or it may not. In C ++, memory is reserved by calling VirtualAlloc . So the transferred memory is a hardware limited resource in the OS. Let's get a look.
OS paging file
The swap file is a great idea. Basically, the OS understands that some parts of the memory are not particularly used by your application. Why waste real physical memory on him? Instead, a process in the kernel writes this unused fragment to disk. Until it is addressed again, only then will it be returned to memory.
For a more detailed explanation of how memory works in Windows, I recommend the lecture “Secrets of memory management” by Mark Russinovich.
Memory tracking
There is a lot to follow and analyze. Who to contact? Of course, to the Task Manager!
RAM memory is usually called the Working Set, while all allocated memory is usually called Private Bytes. DLLs confuse definitions, so for now we’ll ignore them. In other words:
Private Bytes [выделенная память] = рабочий набор + файл подкачки
By default, the Task Manager shows the working set for any process:
And this is the number I was looking at all the time. How did I know that in the Task Manager information about the transferred memory is in the column
Commit Size
. I could not find there information about virtual memory. Task Manager allows you to add information about the transferred memory, if you right-click on the columns and select the appropriate item
Effective memory metrics
Fortunately, there are many other resources for tracking resources. PerfMon (System Monitor) is installed on each machine under Windows, which gives very detailed information about each process and the system as a whole:
Interestingly, System Monitor can actually explore and compare metrics on two or more computers on the network. This is a very powerful tool, but Task Manager is obviously more convenient for users. As an interim solution, I recommend Process Explorer :
Boom! Visual Studio, why are you still in 32-bit mode (note its virtual size)? Peak memory usage on my computer at the level of 89% of the maximum is still tolerable. It will come in handy later.
Addition: many pointed to other convenient tools, includingVMMap and RAMMap .
Debugging through memory information
Fortunately, this is not some unnecessary trivia OS. Actual information about memory consumption has helped me many times in debugging various problems.
The most important thing is to find the untouched parts of the allocated memory. Paging data is also important: this memory is transferred, but is rarely used or not used at all.
Even if the memory will sometimes be used, it is important to understand that this is an expensive resource, so it is by no means necessary to take this path. Memory leaks will appear here.
For these reasons, I had previously heard the suggestion to completely remove the paging file and equate the allocated memory to the working set. However, this is a double-edged idea. Then the OS is not able to reset the memory in case of malfunction of applications, which are sometimes wasted memory.