May 10, 2017 at 10:3112 tools for debugging .NET applications by performance and memory
In a recent interview with John Skeet, we came to the conclusion that professional work with any technology implies the ability to diagnose problems and understand how your applications work under the hood. Following that conversation, I found out from Sasha goldshtn Goldshtein, one of the best .NET performance experts in the world, the author of the book “Pro .NET Performance” , on which tools .NET developers should pay attention to.
Some of them are relatively simple, others will take time and effort to master. Under the cutscene you will find a brief overview of 12 tools that will be useful to performance engineers, top-level support specialists, and generally developers who care about the performance of their applications.
More information about all the tools and how to use them to solve problems with the performance of your applications in practice, Sasha will tell on May 21 at a training in St. Petersburg .
Windows performance counters are the first step to getting a high-level overview of what your system is doing. Monitoring CPU and memory usage, disk and I / O, network packets and HTTP requests allows you to get a bird's eye view of the system with a small overhead and understand where to dig further. Perfmon (Performance Monitor) is a built-in tool in Windows that not only provides access to the real-time performance counter panel, but also allows you to record the counter data so that you can view it later on another computer, and even set up automatic notifications in case something happens. it will go wrong. For lovers of the command line there is Typeperf- a tool that writes counter data to a CSV file, which can later be easily parsed and automatically analyzed. These two tools allow you to quickly understand what you should pay attention to first of all, as well as whether your system works normally. However, of course, they are not suitable for in-depth research, because the counters simply show you numbers that display certain aspects of the operating system.
Over the past 10 years, ETW (Event Tracing for Windows) has become a very common tool and in fact has become the de facto standard among performance analysis tools for Windows. By recording and analyzing ETW data, you can profile the CPU at the OS level, examine locks, find out which parts of your application create a high load on the disk or network, and even track garbage collection, memory allocations, and NET assembly load events. XPerf- An older console tool for recording ETW events, which has several analytical modules for measuring I / O performance, reporting CPU performance and calculating the "cost" of launching applications. In addition, he is able to convert ETW records to CSV format, which can be easily parsed with other tools and scripts. WPR (Windows Performance Recorder) - a graphical shell that allows you to select the events that you would like to record.
There is also WPA (Windows Performance Analyzer), a modern tool for viewing ETW records, capable of building graphs, pivot tables with different filters and groupings, as well as separate views for certain events: CPU stacks, memory allocations, I / O requests and loading steps. More recently, support has appeared for flame graphs, a new method for displaying large stack trees, which makes it easy to analyze them.
In general, ETW tools are great for analysis on production, although they can also be used in development. The main drawback of these solutions is the relative inconvenience of their use compared to commercial profilers.
Windows Performance Toolkit documentation on MSDN
The main problem with the Windows Performance Toolkit is that they are not monitored for managed processes and CLR-specific events, such as garbage collection, managed exceptions, loading assemblies, or JIT compilation. PerfviewIt is an open source tool from Microsoft that can record, collect and display CLR events, including those messages that you trigger yourself using the EventSource API, available since .NET 3.5. PerfView has several unique features: for example, displaying dumps of a heap of managed processes in a very convenient form (in which it is very easy to detect memory leaks); folding and grouping call stacks, allowing you to quickly deal with thousands of data points, and collecting data on .NET events like GC and allocations. Unfortunately, the PerfView interface was made by a performance engineer, so it is not too intuitive and requires a certain period of getting used to, after which you can work with the tool for 100.
Sasha talked about PerfView at the last DotNext in St. Petersburg.
PerfView tutorial video series on Channel 9
At DotNext 2017 Piter, Sasha will present several of his own open source developments, which he himself uses for a “full-time” performance study and which do not require the generation of large amounts of data and their subsequent analysis. The first of these is etrace , which generates a live log of interesting ETW events and provides many options for filtering them. Example: you can ask etrace to output a message when using a particular file, when the process is completely (generation 2) garbage collected, when a particular .NET assembly is loaded, or when an ASP.NET application throws a managed exception. Working from the command line allows you to use etrace in scripts and quickly get results without digging into the GUI.
The second tool isLiveStacks , a profiler-like tool designed to capture and combine real-time event stacks. More precisely, LiveStacks collects managed call stacks (method names), examining the memory of the target process, in contrast to the traditional ETW approach, which requires the completion of a profiling session to generate and parse CLR rundown events. As a result of this approach, it is a fast, lightweight and effective tool that can save the stack in a compact form, suitable for creating a flame graph. If you need fast results on production and you do not hesitate to work in the console, I hope these tools will be useful to you.
The tools described above are primarily aimed at optimizing performance, although using the same ETW I caught a lot of bugs and memory leaks. However, in some cases, you may need a complete memory dump of a broken process, which could be studied on a production system or later in your development environment. There are several tools for creating dumps in Windows, we’ll talk about two of them here: Procdump , a very flexible free console application from Sysinternals that generates dumps for various triggers (% of CPU usage, amount of memory used, non-responding windows, etc.); and debugdiag, a monitoring tool that can be very finely tuned to track your processes in the background while waiting for problems. There are so many mistakes that you can find only by studying the dump, or even several dumps made with a certain frequency, so creating infrastructure for receiving such dumps can be the number one task.
Procdump documentation on TechNet
DebugDiag documentation on MSDN
In general, you can use Visual Studio to analyze dumps and get the most out of them, but it is better to pay attention to WinDbg and its various extensions. WinDbg, Windows DebuggerIt is a powerful and very unfriendly tool for debugging working processes and analyzing dumps. Armed with extensions (such as SOS, SOSEX, NetExt, and MEX), you will get enormous opportunities: exploring a managed heap, finding unused and undeleted objects, finding stubbed threads with one command, detecting ASP.NET requests in real time, and doing many other studies. It is important to note that WinDbg can be controlled both by external scripts (by launching it from the command line with certain parameters) and by internal ones (written in scripting languages or C / C ++ / C # DLL). All this gives significantly more flexibility for debugging compared to VS or other IDEs, and its lightness allows you to put it on production and use it for real-time monitoring. Believe me
By the way, at the last DotNext , Sasha already told a lot of interesting things about WinDbg.
WinDbg reference on MSDN
This list would be incomplete if it were not for another application (which Sasha wrote himself) designed to solve one very unpleasant problem with dump analysis: I had files with Windows Phone, and for the WinPhone CLR there is no public SOS extension that is so necessary for any kind of controlled dump analysis in WinDbg. Therefore, I wrote my open-source debugger based on the Windows Debugging API (DbgEng) and the Microsoft CLRMD library. Msos is an open-source framework that provides a managed shell for SOS interfaces. Over time, msos acquired unique features that separated it from WinDbg and SOS:
msos github repository
Hope it was helpful. And if you want more details -there are still a few tickets left for the training .
And if you don’t want to devote all day to debugging and performance, then you can meet with Sasha and 30 more .NET gurus from around the world at DotNext 2017 Piter .
UPD. We are doing the training again with Sasha, this time in Moscow: registration and conditions of participation are on the site .
Some of them are relatively simple, others will take time and effort to master. Under the cutscene you will find a brief overview of 12 tools that will be useful to performance engineers, top-level support specialists, and generally developers who care about the performance of their applications.
More information about all the tools and how to use them to solve problems with the performance of your applications in practice, Sasha will tell on May 21 at a training in St. Petersburg .
Typeperf and Perfmon
Windows performance counters are the first step to getting a high-level overview of what your system is doing. Monitoring CPU and memory usage, disk and I / O, network packets and HTTP requests allows you to get a bird's eye view of the system with a small overhead and understand where to dig further. Perfmon (Performance Monitor) is a built-in tool in Windows that not only provides access to the real-time performance counter panel, but also allows you to record the counter data so that you can view it later on another computer, and even set up automatic notifications in case something happens. it will go wrong. For lovers of the command line there is Typeperf- a tool that writes counter data to a CSV file, which can later be easily parsed and automatically analyzed. These two tools allow you to quickly understand what you should pay attention to first of all, as well as whether your system works normally. However, of course, they are not suitable for in-depth research, because the counters simply show you numbers that display certain aspects of the operating system.
XPerf, WPR, and WPA (Windows Performance Toolkit)
Over the past 10 years, ETW (Event Tracing for Windows) has become a very common tool and in fact has become the de facto standard among performance analysis tools for Windows. By recording and analyzing ETW data, you can profile the CPU at the OS level, examine locks, find out which parts of your application create a high load on the disk or network, and even track garbage collection, memory allocations, and NET assembly load events. XPerf- An older console tool for recording ETW events, which has several analytical modules for measuring I / O performance, reporting CPU performance and calculating the "cost" of launching applications. In addition, he is able to convert ETW records to CSV format, which can be easily parsed with other tools and scripts. WPR (Windows Performance Recorder) - a graphical shell that allows you to select the events that you would like to record.
There is also WPA (Windows Performance Analyzer), a modern tool for viewing ETW records, capable of building graphs, pivot tables with different filters and groupings, as well as separate views for certain events: CPU stacks, memory allocations, I / O requests and loading steps. More recently, support has appeared for flame graphs, a new method for displaying large stack trees, which makes it easy to analyze them.
In general, ETW tools are great for analysis on production, although they can also be used in development. The main drawback of these solutions is the relative inconvenience of their use compared to commercial profilers.
Windows Performance Toolkit documentation on MSDN
Perfview
The main problem with the Windows Performance Toolkit is that they are not monitored for managed processes and CLR-specific events, such as garbage collection, managed exceptions, loading assemblies, or JIT compilation. PerfviewIt is an open source tool from Microsoft that can record, collect and display CLR events, including those messages that you trigger yourself using the EventSource API, available since .NET 3.5. PerfView has several unique features: for example, displaying dumps of a heap of managed processes in a very convenient form (in which it is very easy to detect memory leaks); folding and grouping call stacks, allowing you to quickly deal with thousands of data points, and collecting data on .NET events like GC and allocations. Unfortunately, the PerfView interface was made by a performance engineer, so it is not too intuitive and requires a certain period of getting used to, after which you can work with the tool for 100.
Sasha talked about PerfView at the last DotNext in St. Petersburg.
PerfView tutorial video series on Channel 9
Etrace and LiveStacks
At DotNext 2017 Piter, Sasha will present several of his own open source developments, which he himself uses for a “full-time” performance study and which do not require the generation of large amounts of data and their subsequent analysis. The first of these is etrace , which generates a live log of interesting ETW events and provides many options for filtering them. Example: you can ask etrace to output a message when using a particular file, when the process is completely (generation 2) garbage collected, when a particular .NET assembly is loaded, or when an ASP.NET application throws a managed exception. Working from the command line allows you to use etrace in scripts and quickly get results without digging into the GUI.
The second tool isLiveStacks , a profiler-like tool designed to capture and combine real-time event stacks. More precisely, LiveStacks collects managed call stacks (method names), examining the memory of the target process, in contrast to the traditional ETW approach, which requires the completion of a profiling session to generate and parse CLR rundown events. As a result of this approach, it is a fast, lightweight and effective tool that can save the stack in a compact form, suitable for creating a flame graph. If you need fast results on production and you do not hesitate to work in the console, I hope these tools will be useful to you.
Procdump and DebugDiag
The tools described above are primarily aimed at optimizing performance, although using the same ETW I caught a lot of bugs and memory leaks. However, in some cases, you may need a complete memory dump of a broken process, which could be studied on a production system or later in your development environment. There are several tools for creating dumps in Windows, we’ll talk about two of them here: Procdump , a very flexible free console application from Sysinternals that generates dumps for various triggers (% of CPU usage, amount of memory used, non-responding windows, etc.); and debugdiag, a monitoring tool that can be very finely tuned to track your processes in the background while waiting for problems. There are so many mistakes that you can find only by studying the dump, or even several dumps made with a certain frequency, so creating infrastructure for receiving such dumps can be the number one task.
Procdump documentation on TechNet
DebugDiag documentation on MSDN
Windbg
In general, you can use Visual Studio to analyze dumps and get the most out of them, but it is better to pay attention to WinDbg and its various extensions. WinDbg, Windows DebuggerIt is a powerful and very unfriendly tool for debugging working processes and analyzing dumps. Armed with extensions (such as SOS, SOSEX, NetExt, and MEX), you will get enormous opportunities: exploring a managed heap, finding unused and undeleted objects, finding stubbed threads with one command, detecting ASP.NET requests in real time, and doing many other studies. It is important to note that WinDbg can be controlled both by external scripts (by launching it from the command line with certain parameters) and by internal ones (written in scripting languages or C / C ++ / C # DLL). All this gives significantly more flexibility for debugging compared to VS or other IDEs, and its lightness allows you to put it on production and use it for real-time monitoring. Believe me
By the way, at the last DotNext , Sasha already told a lot of interesting things about WinDbg.
WinDbg reference on MSDN
Msos
This list would be incomplete if it were not for another application (which Sasha wrote himself) designed to solve one very unpleasant problem with dump analysis: I had files with Windows Phone, and for the WinPhone CLR there is no public SOS extension that is so necessary for any kind of controlled dump analysis in WinDbg. Therefore, I wrote my open-source debugger based on the Windows Debugging API (DbgEng) and the Microsoft CLRMD library. Msos is an open-source framework that provides a managed shell for SOS interfaces. Over time, msos acquired unique features that separated it from WinDbg and SOS:
- Requests for fetching objects from the heap, for example, “Find all HTTP requests waiting more than 2 seconds” or “find all objects of type CustomerData where the AmountDue parameter is negative”;
- Deadlock detection in managed and native synchronization mechanisms;
- Hip indexing support for quick graph traversal;
- A custom memory dump generator that ignores memory regions that are unnecessary for the analysis of managed dumps (thus saving up to 80% of disk space);
- Automatic sort mode generating a concise JSON report with a minimum of data required for more in-depth research. I understand that most of you will continue to use VS for everyday debugging, and that's fine. However, my tool will allow you to cope with the most unpleasant situations.
msos github repository
Hope it was helpful. And if you want more details -
And if you don’t want to devote all day to debugging and performance, then you can meet with Sasha and 30 more .NET gurus from around the world at DotNext 2017 Piter .
UPD. We are doing the training again with Sasha, this time in Moscow: registration and conditions of participation are on the site .