Visual Studio extension for visualizing instances of custom classes in debug mode. Part 2
This article is a continuation of an article from the distant 2014. Let me remind you what was discussed in the previous article .
We write software in C ++ in the Visual Studio 2015 environment . Naturally, we have custom data types in our project. I can cite MbSolid class as an example of such types . This class is part of the C3D mathematical core and is an abstraction of a solid. The body is described by faces, faces by some surfaces, etc. Those. the structure of the class is quite complex, and in the process of debugging our own algorithms, I would like to visually figure out which body has turned out at the moment.
Picture from the last article. As an example of a custom class, the line segment class is used there.
To solve this problem, an extension was written for VisualStudio. There is nothing interesting here, the links are in the last article. But there was a problem - how in the VisualStudio extension to get data from the address space of a debugged (other) process?
In the case of simple data types, such as arrays, everything is not so complicated, and Microsoft even prepared an example for std :: vector . And in our case MbSolid has an impressive inheritance hierarchy and a large number of data fields.
An invasive solution to this problem was proposed in a previous article. A marker field is added to each custom class (which we want to render during debugging). And in each non-constant class method, code is added to serialize the class data into shared memory . The marker field stores the address in shared memory , where we saved the class data. At the time of debugging, when viewing the contents of a variable of interest to us, the VisualStudio extension finds a marker field and deserializes data from shared memory , and then somehow visualizes the received data.
For obvious reasons, this solution is not applicable in practice. Especially if there is no access to the source code of the classes that we want to debug in this way. It was not possible to come up with anything better at that time, and this topic stalled for several years.
And recently, the idea came up to write a simple server that will live in a user process in a separate stream and respond to requests from our VisualStudio extension. For the server, the Microsoft C ++ REST SDK project was taken as the basis . This project allowed you to quickly write your http server, which receives GET requests and returns a description of the user class instance in json format. Let me remind you that we are interested in the visual representation of instances of the MbSolid class (solids).
In the request to the server, the address of the variable in the address space of the debugged process is transmitted. Because Since the server lives in the same process, it gets access to data at the requested address without problems. The server, having received the class instance address, casts this pointer to a typeMbSolid * . Next, the server creates an approximation of this body in the form of a polygonal mesh. Serializes the calculated vertices and indices of the triangles in json and sends a response. On the VisualStudio side, the extension receives a response, deserializes the data, and draws the resulting polygon mesh in the VisualStudio window.
As a result, an extension in VisualStudio does not even need to know the structure of user data; it only needs to be able to send the correct GET requests, deserialize the response and draw triangles in the VisualStudio window. The server can be expanded. In this way, you can debug any custom classes that can be represented as a polygon mesh or a set of line segments, and the VisualStudio extension will be able to visualize them:
Moreover, in this way, you can even send requests to our server from a browser and visualize process data using WebGL.
Made a simple demo. We launch our application. We open the example page in the browser, enter the address of the variable on the page, send a request to the server and draw the response. I don’t know why this may be necessary, but a cool thing
All would be well. But there is one problem. When a breakpoint is triggered, the studio stops all threads of the user process. As a result, our server also stops and cannot respond to requests. To work around this problem, we use the following crutch: the current thread in which the breakpoint worked is frozen, and the user process starts. At this point, our server comes to life and the extension sends it a request with the address of the variable of interest to us. After receiving a response, the user process is paused again, and the current thread in which the breakpoint is triggered is set as the current thread for the debugger. For the user, it looks like nothing happened and the program stopped at the breakpoint.
In the VisualStudio extension code, this crutch looks like this. Breakpoint triggered. The user requests data of the variable of interest to him. At this moment, we freeze the current thread and start the debugger:
We send a request to the server. We get the answer. We stop the process, unfreeze our thread:
All! We received data from the server and rendered it in the VisualStudio window. Program execution is at the initial breakpoint.
In conclusion, I want to note that so far I do not understand the side effects that this approach generates. I would be glad if you share your thoughts in the comments on this article.
What problem will we solve
We write software in C ++ in the Visual Studio 2015 environment . Naturally, we have custom data types in our project. I can cite MbSolid class as an example of such types . This class is part of the C3D mathematical core and is an abstraction of a solid. The body is described by faces, faces by some surfaces, etc. Those. the structure of the class is quite complex, and in the process of debugging our own algorithms, I would like to visually figure out which body has turned out at the moment.
Picture from the last article. As an example of a custom class, the line segment class is used there.
To solve this problem, an extension was written for VisualStudio. There is nothing interesting here, the links are in the last article. But there was a problem - how in the VisualStudio extension to get data from the address space of a debugged (other) process?
In the case of simple data types, such as arrays, everything is not so complicated, and Microsoft even prepared an example for std :: vector . And in our case MbSolid has an impressive inheritance hierarchy and a large number of data fields.
Custom Type Modification
An invasive solution to this problem was proposed in a previous article. A marker field is added to each custom class (which we want to render during debugging). And in each non-constant class method, code is added to serialize the class data into shared memory . The marker field stores the address in shared memory , where we saved the class data. At the time of debugging, when viewing the contents of a variable of interest to us, the VisualStudio extension finds a marker field and deserializes data from shared memory , and then somehow visualizes the received data.
For obvious reasons, this solution is not applicable in practice. Especially if there is no access to the source code of the classes that we want to debug in this way. It was not possible to come up with anything better at that time, and this topic stalled for several years.
Server in user process
And recently, the idea came up to write a simple server that will live in a user process in a separate stream and respond to requests from our VisualStudio extension. For the server, the Microsoft C ++ REST SDK project was taken as the basis . This project allowed you to quickly write your http server, which receives GET requests and returns a description of the user class instance in json format. Let me remind you that we are interested in the visual representation of instances of the MbSolid class (solids).
In the request to the server, the address of the variable in the address space of the debugged process is transmitted. Because Since the server lives in the same process, it gets access to data at the requested address without problems. The server, having received the class instance address, casts this pointer to a typeMbSolid * . Next, the server creates an approximation of this body in the form of a polygonal mesh. Serializes the calculated vertices and indices of the triangles in json and sends a response. On the VisualStudio side, the extension receives a response, deserializes the data, and draws the resulting polygon mesh in the VisualStudio window.
As a result, an extension in VisualStudio does not even need to know the structure of user data; it only needs to be able to send the correct GET requests, deserialize the response and draw triangles in the VisualStudio window. The server can be expanded. In this way, you can debug any custom classes that can be represented as a polygon mesh or a set of line segments, and the VisualStudio extension will be able to visualize them:
Moreover, in this way, you can even send requests to our server from a browser and visualize process data using WebGL.
Made a simple demo. We launch our application. We open the example page in the browser, enter the address of the variable on the page, send a request to the server and draw the response. I don’t know why this may be necessary, but a cool thing
We revive the server
All would be well. But there is one problem. When a breakpoint is triggered, the studio stops all threads of the user process. As a result, our server also stops and cannot respond to requests. To work around this problem, we use the following crutch: the current thread in which the breakpoint worked is frozen, and the user process starts. At this point, our server comes to life and the extension sends it a request with the address of the variable of interest to us. After receiving a response, the user process is paused again, and the current thread in which the breakpoint is triggered is set as the current thread for the debugger. For the user, it looks like nothing happened and the program stopped at the breakpoint.
In the VisualStudio extension code, this crutch looks like this. Breakpoint triggered. The user requests data of the variable of interest to him. At this moment, we freeze the current thread and start the debugger:
if (dte.Debugger.CurrentMode != EnvDTE.dbgDebugMode.dbgBreakMode)
return;
currentThread = dte.Debugger.CurrentThread;
currentThread.Freeze();
dte.Debugger.Go(false);
We send a request to the server. We get the answer. We stop the process, unfreeze our thread:
if (dte.Debugger.CurrentMode == EnvDTE.dbgDebugMode.dbgBreakMode)
return;
dte.Debugger.Break();
if (currentThread != null)
{
currentThread.Thaw();
dte.Debugger.CurrentThread = currentThread;
}
All! We received data from the server and rendered it in the VisualStudio window. Program execution is at the initial breakpoint.
In conclusion, I want to note that so far I do not understand the side effects that this approach generates. I would be glad if you share your thoughts in the comments on this article.