Regression tests for memory leaks, or how to write a memory profiler for .NET applications

As a rule, memory profilers begin to use when the application is guaranteed to “flow”, users actively send letters, full of screenshots of the task manager, and you need to spend a lot of time profiling and searching for the cause. Finally, when developers detect and fix a leak, release a new, excellent version of the application that is free from previous flaws, there is a risk that the leak will return after a while, because the application is growing, and developers can still make mistakes.

Automated regression error testing has long been the mainstream of the high-quality software development industry. Such tests help prevent the user from getting an error, as well as quickly figure out which change in the code led to the error, thereby minimizing the time it takes to fix it.

Why don't we take the same approach to memory leaks?

We asked this question, once again having received OutOfMemoryException during passage of regression autotests on x86 agents.

A few words about our product: we are developing Pilot-ICE - an engineering data management system. The application is written in .NET / WPF, and for regression testing we use the Winium.Cruciatus framework based on UIAutomation. Tests “click” through the UI all the available functionality of the application, checking the logic of work.

The idea of ​​implementing tests for memory leaks is as follows: at certain points in the passage of tests, connect to the application and check the number of instances of objects of certain types in memory.

Analysis of existing solutions

We reviewed most of the popular .NET memory profilers, and all of them store memory snapshots in a proprietary format that can only be opened for analysis in the corresponding viewer. We found no opportunity for the automated analysis of snapshots in any of them.

Stand alone is the dotMemory Unit, a free unit testing framework for analyzing memory leaks in tests. Unfortunately, in it, memory analysis is limited to the process that runs the tests. There is currently no way to connect to an external process using dotMemory Unit.

Writing your profiler

So, not finding a suitable ready-made solution, it was decided to write your memory profiler. What should he be able to do:

1. Call garbage collection in the application
2. Get the number of objects of a given type in the application memory
3. Analyze what keeps these objects from being collected by GC (Garbage Collector).

At the same time, I wanted to make it so that I did not have to modify the tested application itself.

Garbage collection

As you know, to call garbage collection in a .NET application, the GC.Collect () method can be used, which starts garbage collection at once in all generations. This method is not recommended for use in production code, and memory profiling is almost the only adequate scenario for its use. Garbage collection before profiling is needed to eliminate the false positives of the profiler on unreachable objects that the GC simply did not manage to reach.
The difficulty is that garbage collection must be started in an external process, and there are several possible solutions for this:

1. Connect to the process via the debugger API and call garbage collection
2. Get involved in the process and start garbage collection from there
3. Through the ETW (Event Tracing for Windows) team GCHeapCollect
   We chose the second path as the easiest to implement. The Managed Injector was borrowed from the Snoop for WPF project . It allows specifying the path with the assembly, class and method in it to load this assembly into the domain of the external application and run the specified method. In our case, after being introduced into the process, the named pipe server starts, which, on command from the client (profiler), starts the garbage collection process.

Application memory analysis

To analyze application memory, we used the CLR MD library , which provides an API similar to the SOS debugging extension in WinDbg. Using it, you can connect to the process, bypass all objects in heaps, get a list of root links (GC root) and objects dependent on them. By and large, all that we need is already implemented, we just need to use all of this correctly.

This is how you can get the number of objects of a certain type in memory using the CLR MD:

public int CountObjects(int pid, string type)
{
    using (var dataTarget = DataTarget.AttachToProcess(pid, msecTimeout: 5000))
    {
        var runtime = dataTarget.ClrVersions.First().CreateRuntime();
        return runtime.Heap.EnumerateObjects().Count(o => o.Type.Name == type);
    }
}

The most difficult, but quite solvable moment is to obtain information about what keeps the object from being collected by the garbage collector. To do this, it is necessary to go around all the dependency trees of root links, remembering as you bypass the retention path.

Continuous integration

Next, we built all the best practices into the regression test code. Information about the names of periodically leaking types and the maximum number of instances of this type that can be in memory was added to the tests. The verification algorithm is as follows: after the test is completed, garbage collection starts, then the analysis of the number of objects of types of interest to us starts, if their number is greater than the reference, the problem is reported and the build is marked as “dropped”. In addition, diagnostic information is collected on what keeps these objects from garbage collection and added to build artifacts. Here's what it looks like for TeamCity:

Sharing is caring. Meet Ascon.NetMemoryProfiler

The resulting solution turned out to be quite general, and we decided to share it with the community. The project code can be found in the github repository , in addition, the ready-to-use solution is available as a nuget package called Ascon.NetMemoryProfiler. Distributed under the Apache 2.0 license.
Below is an example of using the API. Minimalistic, but describing almost all the functionality provided:

// Присоединяемся с процессу MyApp
// После присоединения, в приложении будет вызвана сборка мусора
using (var session = Profiler.AttachToProcess("MyApp"))
{
    // Ищем в памяти живые объекты типа "MyApp.Foo"
    var objects = session.GetAliveObjects(x => x.Type == "MyApp.Foo");
    // Получаем информацию, что удерживает объекты от сборки мусора
    var retentions = session.FindRetentions(objects);
}

Consider a simple application as an example of how to write a test for memory leaks. Let's make a test project, add the Ascon.NetMemoryProfiler package to it.

Install-Package Ascon.NetMemoryProfiler

Let's write the basis for the test:

[TestFixture]
public class MemoryLeakTests
{
    [Test]
    public void MemoryLeakTest()
    {
        using (var session = Profiler.AttachToProcess("LeakingApp"))
        {
            var objects = session.GetAliveObjects(x => x.Type.EndsWith("LeakingObjectTypeName"));
            if (objects.Any())
            {
                var retentions = session.FindRetentions(objects);
                Assert.Fail(DumpRetentions(retentions));
            }
        }
    }
    private static string DumpRetentions(IEnumerable retentions)
    {
        StringBuilder sb = new StringBuilder();
        foreach (var group in retentions.GroupBy(x => x.Instance.TypeName))
        {
            var instances = group.ToList();
            sb.AppendLine($"Found {instances.Count} instances of {group.Key}");
            for (int i = 0; i < instances.Count; i++)
            {
                var instance = instances[i];
                sb.AppendLine($"Instance {i + 1}:");
                foreach (var retentionPath in instance.RetentionPaths)
                {
                    sb.AppendLine(retentionPath);
                    sb.AppendLine("----------------------------");
                }
            }
        }
        return sb.ToString();
    }
}

We’ll create a new WPF application and add several windows and a view-model to it, into which we will intentionally introduce different types of memory leaks:

Leak through EventHandler

Perhaps the most common type of memory leak. After the subscription, the object owner of the event begins to keep a strict reference to the subscriber, thereby preventing the garbage collector from removing the subscriber for the entire lifetime of the event owner. Example:

public class EventHandlerLeakViewModel : INotifyPropertyChanged
{
    public EventHandlerLeakViewModel()
    {
        Dispatcher.CurrentDispatcher.ShutdownStarted += OnShutdownStarted;
    }
    private void OnShutdownStarted(object sender, EventArgs e)
    {
    }
    //...
}

In this case, the lifetime of Dispatcher.CurrentDispatcher coincides with the lifetime of the application, and the EventHandlerLeakViewModel will not be released even after closing the window associated with it.
Check it out. We launch the application, open the window, close it, run the test, after specifying in it the name of the process and the type name for the search. We get the result:

Found 1 instances of LeakingApp.EventHandlerLeakViewModel
Instance 1:
static var System.Windows.Application._appInstance
LeakingApp.App
MS.Win32.HwndWrapper
System.Windows.Threading.Dispatcher
System.EventHandler

You can fix the leak by unsubscribing from the event in time (for example, when the window is closed), or by using weak events.

WPF binding leak

A rather unobvious way to get a memory leak in a WPF application. If the binding target is not a DependencyObject and does not support the INotifyPropertyChanged interface, then this object will live in memory forever. Example:

public class BindingLeakViewModel
{
    public BindingLeakViewModel()
    {
        Title = "Hello world.";
    }
    public string Title { get; set; }
}

Run the test. We get the following result:

Found 1 LeakingApp.BindingLeakViewModel of instances
Instance 1:
static var System.ComponentModel.ReflectTypeDescriptionProvider._propertyCache
System.Collections.Hashtable
System.Collections.Hashtable bucket + []
System.ComponentModel.PropertyDescriptor []
System.ComponentModel.ReflectPropertyDescriptor
System.Collections.Hashtable
System.Collections.Hashtable + bucket []

To eliminate this leak, you must support the INotifyPropertyChanged interface of the BindingLeakViewModel class, or define the binding as one-time (OneTime).

Leak through WPF collection binding

When linking to a collection that does not support the INotifyCollectionChanged interface, the collection will never be compiled by GC. Example:

public class CollectionLeakViewModel : INotifyPropertyChanged
{
    public List