December 6, 2012 at 12:48

View OpenCV images while debugging C ++ code in Visual Studio



If you write code for image processing in C ++, you are probably using the wonderful OpenCV library . I am sure that you have often wanted to look at images in the process of debugging your code. For this, you can use such convenient functions as imshow or imwrite . However, this requires modification of the source code, and any modern IDE during debugging allows you to watch the values ​​of variables on the fly. Would it be nice to also look at the images?

If you use Visual Studio as the IDE, then you know that with .NET everything is easier in this regard. However, we are talking about OpenCV, and this is only native C ++, only hardcore. In this article I will tell you how to make Visual Studio display images right in the process of debugging and give a link to a ready-made solution. And also briefly talk about ways to customize Visual Studio.

Introduction


For those who just want to use a ready-made solution, I’ll immediately give you links:


I will briefly tell the rest about the tools with which you can, in principle, customize Visual Studio, and in more detail about the implementation of the solution by one of them. All experiments were performed in Visual Studio 2010, the finished solution was tested in Visual Studio 2005, 2008, 2010, 2012 (and in theory should work in 2003).

Finding a solution


When I needed to debug C ++ code using OpenCV to work with images, I immediately wanted to be able to watch them directly during debugging. I happily climbed into Google, and began to look for ready-made solutions. And ... I didn’t find anything. Neither for OpenCV images, nor for generally rendering anything from native C ++ code. I asked the public what a trustworthy resident told me that , in principle, the native code debugger cannot be extended in this way. At that moment, I was sure that the bicycle had not yet been invented.

Visual Studio Customization Features


To choose a suitable tool for the visualizer, you need to at least approximately imagine from which sides it is generally possible to extend Visual Studio. First we need to know two concepts:

  • DTE Interface for interacting with Visual Studio, which was introduced in .NET versions (since 2003). With it, you can get objects that provide access to the most different parts of Visual Studio - the solution structure, shell settings, the active window, even the debugger.
  • The MEF Managed Extensibility Framework is a newer model available since Visual Studio 2010. It allows for tighter integration and interoperability between MEF components, which in turn can use the DTE.

The following is a list of entities used to customize Visual Studio. The list is compiled taking into account a specific task, therefore does not claim to be complete.

  • Macros (Macros) Great for automating any actions performed on the source code, project structure, etc. With their help, you can check the style of the code, generate templates, record and play back some user actions. Have access to DTE. Available in Tools → Macros.
  • Add-Ins. Added since Visual Studio .NET. They are used for any conceivable customization, from improved syntax highlighting to the integration of a version control system. Have access to DTE. Available through Tools → Add-In Manager.
  • Extensions. Introduced since Visual Studio 2010, they are a more advanced version of add-ons (list of differences here ). Interact with MEF. Available through Tools → Extension Manager.
  • Visualizers Designed to display objects during debugging. Available by clicking the small icon with a magnifying glass next to the variable name when displaying its contents.

It would seem that visualizers are exactly what you need. But here's the problem - they only work with managed C ++. The first three options (macros, additions, and extensions) can access the memory of the debugged process only with the help of the Debugger object , which of the suitable functions has only ExecuteStatement and GetExpression . However, both of them somehow return the results in the form of rows limited in size. Of course, you can try to tear the contents of the image in pieces and translate it back into binary form, but it turns out somehow very crookedly.

There is also such a magical file called autoexp.dat. It is installed with Visual Studio and usually lies at

С:\Program Files (x86)\Microsoft Visual Studio \Common7\Packages\Debugger\autoexp.dat

It describes the rules according to which the contents of variables are displayed when debugging native C ++ code. It is he who makes, for example, the contents of the std :: vector container look like on the left, and not like on the right:



And here is how he does it:

The std :: vector output rule in autoexp.dat
; ------------------------------------------------- -----------------------------
; std :: vector from
; ------------------------------------------------- -----------------------------
; vector is previewed with "[] () ".
; It has [size] and [capacity] children, followed by its elements.
; The other containers follow its example.
std :: vector <*> {
	preview (
		# (
			"[",
			$ e._Mylast - $ e._Myfirst,
			"] (",
			#array (
				expr: $ e._Myfirst [$ i],
				size: $ e._Mylast - $ e._Myfirst
			),
			")"
		)
	)
	children (
		# (
			# ([size]: $ e._Mylast - $ e._Myfirst),
			# ([capacity]: $ e._Myend - $ e._Myfirst),
			#array (
				expr: $ e._Myfirst [$ i],
				size: $ e._Mylast - $ e._Myfirst
			)
		)
	)
}
std :: _ Vector_iterator <*> | std :: _ Vector_const_iterator <*> {
	preview (
		* $ e._Ptr
	)
	children (
		# ([ptr]: $ e._Ptr)
	)
}

This is an old, poorly documented technology. Supported since Visual C ++ 6.0. A small usage guide is here . However, it turned out that it was she who would save the father of Russian democracy that would allow us to get normal access to the memory of the debugged process. In addition, if you often work with complex multi-level structures (for example, consider geometry using CGAL ), a couple of manually added rules in this file can seriously simplify your life.

It turns out that the syntax of the autoexp.dat file allows you to not only write expressions for, for example, address arithmetic, but also call any third-party code in general! This is done using the special $ ADDIN construct. In the absence of the best, I used this tool for my visualizer.

Expression Evaluator Add-In Solution


A small guide to writing libraries for the autoexp.dat file is available on MSDN , where they call it Expression Evaluator Add-In. To call the library function, just specify the path to the .dll file and the name of the function being called.

cv::Mat=$ADDIN(NativeViewer.dll,CvMatViewer)

The function prototype in the dll is as follows:

HRESULT WINAPI CvMatViewer(DWORD dwAddress, DEBUGHELPER* pHelper, 
  int nBase, BOOL bUniStrings, char* pResult, size_t max, DWORD reserved)

A pointer to the DEBUGHELPER structure is passed in the pHelper argument, which provides functions for accessing the memory of the debugged process and returns the address of the object displayed in the debugger:

DEBUGHELPER Structure Code
typedef struct tagDEBUGHELPER
{
	DWORD dwVersion;
	HRESULT (WINAPI *ReadDebuggeeMemory)( struct tagDEBUGHELPER *pThis, DWORD dwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
	// from here only when dwVersion >= 0x20000
	DWORDLONG (WINAPI *GetRealAddress)( struct tagDEBUGHELPER *pThis );
	HRESULT (WINAPI *ReadDebuggeeMemoryEx)( struct tagDEBUGHELPER *pThis, DWORDLONG qwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
	int (WINAPI *GetProcessorType)( struct tagDEBUGHELPER *pThis );
} DEBUGHELPER;

In pResult, you need to save the line shown by the debugger. In theory, this is all that this function should do. But who will stop us? We already have access to memory. And then, as they say, a matter of technology.

Extract image content

The only type information currently available to us is its name, cv :: Mat. This means that you have to calculate the field addresses based on the .h file of a specific version of OpenCV. However, I do not think that in the future the field structure of this fundamental class will change.

I will not dwell on technical issues, such as how to read an object from memory, knowing its address and having an .h file with its description, the sources are available on SourceForge. I only note that the pHelper structure allows you to know the bitness of the process being debugged. And this means that you need to take into account that pointers can be either 4 or 8 bytes in size. When we read all the fields of the cv :: Mat object, we can read the contents of the image in the same way, because its address is in the data field.

Formatting the display string

Well, since we're still here, let's really format the line displayed by the debugger. And then the standard insides do not look very pretty. Let's do something like this:



Image rendering

Actually, for what we all gathered here. Since we now have an image, we can render using .NET. I used the good old Windows Forms and C #. The form with the dialog for visualization is in a separate assembly, the System.Bitmap object is passed to it in the constructor. To build it, the original dll was built with .NET support, with the / clr switch. This allowed the use of C ++ / CLI.

I was worried that calling the .NET dialog from the native library would be difficult and you would have to create the AppDomain yourself. However, since the studio itself is a .NET application, nothing had to be done. Everything worked right away, so I didn’t understand in more detail. I only note that I had to show a dialog from a separate thread with the ApartmentState :: STA parameter, otherwise there were problems when opening additional dialogs, for example, to save the image to disk.

Now we are faced with the main drawback of the chosen approach - it is impossible to determine whether the function is called when formatting the output to the Watch window, or when the user hovers the mouse over a variable in the editor. If every time you show a window with an image, it will be impossible to use.

The only adequate solution that I could come up with is this: the window is shown only if any special key is pressed. Namely, Ctrl. Otherwise, the output string is only formatted. Thus, in order to see the image during debugging, the user needs to hold down Ctrl, point to a variable in the editor and then a window will pop up. Crooked, yes, but in fact it turned out to be quite convenient to use.



When the window with the image is displayed, the Visual Studio process is blocked, because formally, we are in the function of formatting the output of the debugger. This is unpleasant, but there is nothing fatal in this, because no need to simultaneously study the image and continue to interact with the studio.

Visual Studio Integration

There is a solution, now it needs to be packaged. The existing extension mechanism is great for this. Initially, I focused on Visual Studio 2010, so I chose more modern extensions, rather than more universal add-ons.

Creating extensions is well documented . First, we need the Visual Studio SDK (for 2010 or 2012). After installation, it will be possible to create a new Visual C # project → Extensibility → Visual Studio Package. This launches a wizard that asks for information about the extension:



By default, he suggests creating a menu command (Menu Comand), a toolbar (Tool Window) or tweaking the editor (Custom Editor). We will not need any of this. The fact is that the extension is a VSIX file, which is just a zip archive. When installed in Visual Studio, it unpacks it into one of the user directories (more details here ). There may be libraries that interact with MEF, but there may be any other files. For example, our library with the Expression Evaluator Add-In.

There is a small problem with loading the assembly with the form for visualization. If you add it to References, then its search will occur in the directory with the executable file devenv.exe and in the GAC. It is located in a directory with the extension, so you have to load it manually using the Assembly :: LoadFrom function . The dialog is then created and displayed using reflection methods.

I wanted to make settings that could be changed from the menu Tools → Options. This is well described in the manual on MSDN . It describes how to create a standard key / value grid, and an arbitrary form:





The last form is needed to automatically add a line to the autoexp.dat file. Unfortunately, it is not possible to execute any command at the time of installing the VSIX extension (see hereSupported Capabilities table). Therefore, after installing the extension, the user needs to go to this page and click the "Add entry" button.

To find the autoexp.dat file, you need to know where Visual Studio is installed. You can see it in the registry, but it’s better to ask it yourself using the DTE. How to get a DTE object from an extension is written here . The FullName property will return the full path to the devenv.exe file.

I was pleasantly surprised by Visual Studio 2012. It finally proposed the replacement of the obsolete autoexp.dat file - NATVIS technology . There is no documentation as such, but there is a good description.on the Microsoft blog. And, the glory of backward compatibility, they left the possibility of calling third-party code through the same mechanism, only now it is indicated in the LegacyAddin argument. The only description that I found is in the answer to this question . It is felt that the technology has recently been announced.

A huge advantage of NATIVS is that now visualization rules (designed as separate XML files with the natvis extension) can be scattered in different directories. Including custom, as well as they may be contained in extensions. To do this, just add the .nativs file to Assets when building the extension. Therefore, the NativeViewer extension for Visual Studio 2012 does not have an integration page and it works out of the box.

Regarding the debugging of VSIX extensions. It turned out to be implemented very conveniently. When you run a VSIX project, an experimental copy of Visual Studio opens, in which the current version of the extension is installed. Debugging works fine, I was able to debug native code in the dll in the source studio, which was loaded by the experimental copy.

Since extensions appeared only in Visual Studio 2010, they will have to be installed manually on earlier versions. While I wrote the manual , but in a good way, of course, you should make an installer.

Conclusion


In this article, I described a solution to a problem that I encountered during the development process. It is noteworthy that it is intended for popular development tools, but, oddly enough, has no analogues. Well, at least I couldn't find anything like it.

In this article, I tried to focus less on technical details and compensate for this with an abundance of links. Those interested can look at the source code and ask questions. If any aspect seems interesting enough to the public, I can write a more detailed article about it.

I really hope that this extension will be useful to many. If someone wants to take part in the development of the project - write.

References



Tags:

Also popular now: