May 29, 2018 at 06:04Automated python code visualization. Part Three: New Graphical Code Capabilities
Links to previous parts:
An example of an environment that supports this graphical representation is shown in the image below.
Environment that supports graphical representation of code
In the third part of the article, we will focus on new features of graphical representation of code.
Since the second part was written, an interface for changing colors used to display graphic primitives and an interface for changing text displayed by a graphic primitive have been added. In both cases, the interface is minimalistic and uses a context menu.
Context menu of selected graphic elements. The
user selects one or more primitives and presses the right mouse button. In the context menu that appears, you can select the desired item (the availability of menu items is determined dynamically).
If you select one of the first two menu items, an additional dialog is displayed for selecting a color or entering text, respectively. After completing the input, the necessary cml comments are added / modified to the source text and the entire diagram is immediately redrawn, taking into account the changes.
In the process of working with the environment, it was noticed that sometimes there is a desire to temporarily hide some graphic elements in order to reduce the amount of information displayed on the screen. This can help with certain program analysis scenarios. Of course, such functionality is quite simple to implement for graphical representations.
The following candidates were identified: documentation lines, comments, and except blocks. To control the suppression of primitives, three buttons have been added to the toolbar of the graphical representation, each of which can be in one of two positions: pressed / depressed.
Controlling primitive suppression in the diagram
Buttons are independent of each other, so any desired combination of suppression can be selected. By changing the state of the buttons, the chart is immediately redrawn, that is, literally with one click of the mouse, you can quickly change the number of parts that are currently displayed.
Python supports documentation lines for modules, classes, and functions. To display all these cases, Codimension uses a primitive of type “scope”, in which, after the header, the documentation line is located. For instance:
Function with a documentation line
However, the documentation lines are optional, and if the developer did not take care of the documentation, Codimension does not draw the corresponding section at all. For instance:
A function without a documentation line
When documentation lines are suppressed, it would be possible not to draw a section, but then, by looking at the diagram, it would be impossible to understand: there is no documentation at all or it is suppressed. Therefore, such a decision was made - instead of suppressed lines of documentation, draw a section in which instead of all the text only a badge is placed. The badge takes up less space than one line, it makes it possible to immediately understand that the documentation is there. And it makes it easy to add a tooltip for the section with the full text of the documentation. For instance:
Function with a suppressed documentation string
In cases where the documentation is voluminous and the initial familiarization with the module has already been completed, the docstring suppression functionality seems quite convenient.
Comments can also take up significant space on the chart. Sometimes there is a desire to suppress them. Do not draw comments at all seems unreasonable for the same reasons as for documentation lines, so it was decided to replace them with a small primitive that supports a tooltip with the full comment text.
Comments are shown
Comments suppressed
Typically, code located in except blocks does not lie on the main path of program execution, but serves to handle rare exceptions. Therefore, when analyzing the main purpose of the program, it seems convenient to be able to suppress the except blocks and present them as a small primitive, which makes it clear that they even exist.
Two except blocks shown
Suppressed except blocks
The tooltip for the primitive shows as many lines as there were except blocks indicating what is intercepted in each.
Codimension in its current implementation supports two types of chart scaling. Normal, or just scaling, which is based on the size of the selected font. With this scaling, the font size increases or decreases, and then all sizes are recalculated. Smart scaling involves several fixed levels, each of which changes the contents of the chart. Each next level hides more and more elements in the diagram.
It was not possible to come up with reasonable short names for each of the levels, so they are indicated by numbers. To control the level switching on the toolbar, an indicator of the current level and buttons for moving up and down are added.
Managing smart scaling
A tooltip has been added for the level indicator that describes what changes have been made to the chart.
Smart scaling works in conjunction with element suppression settings. That is, if a level involves showing, for example, except blocks, then they can be independently suppressed by pressing the corresponding button. Using the examples below, it is assumed that no comments, no documentation lines, or except blocks are suppressed.
This is the level of maximum detail. All graphic elements are shown.
At this level, comments and documentation lines disappear from the chart. The remaining graphic primitives remain on the diagram, but the text in them is suppressed, with the exception of class and function headers.
Smart scaling: level 1
This representation allows you to quickly get an idea of the amount of code and its complexity: the higher the concentration of “boxes” on the diagram, the more complex the code. A tooltip has been added for each graphic at this level, in which suppressed text is shown.
At this level, the diagram remains primitives that use the scope (with the exception of decorators) and conditional structures to display. This presentation is also intended for a quick assessment of complexity. In fact, nesting and branching remain on the diagram.
Smart Zoom: Level 2
At this level, only classes and functions remain on the diagram. This display was conceived as a “bird's-eye view”, only large objects in the module remain visible.
Smart Zoom: Level 3
At the moment, the experience of practical application of the levels of smart scaling is very small. Therefore, it is difficult to draw firm conclusions about the usefulness of any of the levels, as well as the idea of smart scaling in general. The plans, however, include the introduction of the next level with number 4. It is supposed to show a dependency diagram of the module on it, a kind of look at the file system. In the center, the primitive of the current module can be located, and around it the primitives of the imported modules. Thus, it will be possible to visually evaluate the current dependencies.
Interestingly, the implementation of smart scaling did not require much effort. On the graphical representation, it turned out to be quite simple to manipulate the content. Therefore, the introduction of some new intermediate levels for experimental purposes can be performed very quickly.
The basic idea of the functionality is simple - to give the opportunity to combine several graphic elements in a group, followed by folding and expanding the group into one graphic element with a new signature. A use case may be, for example, such. The developer begins to work with a new code for him written by another person. In the course of the analysis, understanding gradually comes, which makes a piece of code consisting, say, of ten blocks. Then the developer combines these blocks into a group and supplies it with suitable text. Upon further analysis, the group remains minimized on the screen, that is, on the diagram, instead of 10 elements, there is only one, which should facilitate subsequent analysis. Of course, groups can be nested and there are no restrictions on the number of groups in the module.
No one forbids the use of groups at the time of initial development. The programmer can create groups, which will facilitate code analysis for those who will work with him later.
Since groups support unlimited nesting, they can be considered as introducing a similarity of the third dimension to the code. The first two are a diagram, and the third is a collapsed group, inside which you can look.
To draw a group on a chart, Codimension needs to know where it starts and where it ends. To leave appropriate marks in the code, the micro markup language is used in the comments of the Codimension Markup Language.
The first comment has gb code (from the words group begin) and is located before the first element of the group. The second comment has the ge code (from the words group end) and is located after the last element of the group. Codimension understands that comments are paired by the value of the required id attribute, which can be an arbitrary string.
Obviously, there are a number of limitations to the formation of a group. A group must have one input and one output. When selecting elements for a group on the chart, all of them should be selected without gaps. If the if condition is selected, then all branches must also be selected. For loops, a similar requirement is associated with the else construct: if it is present, then both parts must be selected, and the loop itself and its else block. Similarly, you cannot tear apart the components of try-except-else-finally. All these restrictions are taken into account when the user interface allows grouping.
In addition, as part of the work of the groups, a change was needed in the python parser. With the initial design of the technology, it was decided not to support the final comments. However, for groups, such a concluding comment is absolutely necessary. For example, a group can combine all the elements in one of the if branches:
Here, the comment ending the group should belong to the if branch, which was not provided earlier.
For groups also added the ability to change the default colors. To do this, in the opening cml comment, optional attributes are supported for setting the background, ink and stroke colors.
To render groups, three types of primitives are required: a collapsed group, an expanded group, and an empty group.
For a folded group, such a primitive is desirable that it is immediately clear that several elements are hidden behind it and that it can be revealed. It is also desirable that the rendering of the primitive be simple regardless of whether it is selected or not. As a result, a rectangle with a double stroke was selected, inside which the group header is placed.
The minimized group
There is currently no complete satisfaction from the selected primitive, so there is a chance that it will be replaced with something else in the future.
The disclosed group should indicate its boundaries, plus there should be the ability to quickly see the group heading. To make the borders look “easier”, the rectangle of the opened group is drawn with a dashed line. There is no particular sense in making the fields of the open group wide, so the dotted line is drawn closer to the primitives than if it were another element.
Open group
In the upper left corner of the open group its “sensitive point” is drawn. When the mouse cursor is over it, the outline of the group is drawn with a solid line, and its title is shown above the group.
“Sensitive point” of the opened group
If necessary, in addition to the group title, you can add some control buttons. This will allow you to perform actions with a minimal number of mouse clicks.
There is nothing in the empty group, but it can, for example, contain a heading. In the future, when the editing functions on the graphical representation appear, new elements can be added to the empty group. Therefore, you still need to draw an empty group. A rectangle with a stroke is selected for it.
Empty group
The similarities between the primitives of the empty and collapsed groups are obvious. Therefore, if a primitive is changed for a collapsed group, then corresponding changes must be made for an empty group.
Group management is carried out from the context menu. To create a group, the necessary primitives are selected, and then the Group item is selected from the context menu ... After entering the group header, the necessary CML comments are added to the code and the chart is redrawn immediately. In this case, the newly created group is drawn in minimized form.
All other manipulations with groups - opening, hiding, deleting, editing a title, changing colors - are performed by choice from the group context menu.
You can try the new functionality by installing the Codimension IDE. Codimension supports Linux and Mac. Installation instructions are available at codimension.org
- Part one - discussed flowcharts, available tools for working with flowcharts, as well as all the graphical primitives needed to create a graphical representation of the code.
- Part two - discussed the implementation (mainly done in Python) of generating a graphical representation of the code, implemented and planned functionality, the proposed micro markup language.
An example of an environment that supports this graphical representation is shown in the image below.
Environment that supports graphical representation of code
In the third part of the article, we will focus on new features of graphical representation of code.
Little things
Since the second part was written, an interface for changing colors used to display graphic primitives and an interface for changing text displayed by a graphic primitive have been added. In both cases, the interface is minimalistic and uses a context menu.
Context menu of selected graphic elements. The
user selects one or more primitives and presses the right mouse button. In the context menu that appears, you can select the desired item (the availability of menu items is determined dynamically).
If you select one of the first two menu items, an additional dialog is displayed for selecting a color or entering text, respectively. After completing the input, the necessary cml comments are added / modified to the source text and the entire diagram is immediately redrawn, taking into account the changes.
Primitive suppression
In the process of working with the environment, it was noticed that sometimes there is a desire to temporarily hide some graphic elements in order to reduce the amount of information displayed on the screen. This can help with certain program analysis scenarios. Of course, such functionality is quite simple to implement for graphical representations.
The following candidates were identified: documentation lines, comments, and except blocks. To control the suppression of primitives, three buttons have been added to the toolbar of the graphical representation, each of which can be in one of two positions: pressed / depressed.
Controlling primitive suppression in the diagram
Buttons are independent of each other, so any desired combination of suppression can be selected. By changing the state of the buttons, the chart is immediately redrawn, that is, literally with one click of the mouse, you can quickly change the number of parts that are currently displayed.
Documentation line suppression
Python supports documentation lines for modules, classes, and functions. To display all these cases, Codimension uses a primitive of type “scope”, in which, after the header, the documentation line is located. For instance:
Function with a documentation line
However, the documentation lines are optional, and if the developer did not take care of the documentation, Codimension does not draw the corresponding section at all. For instance:
A function without a documentation line
When documentation lines are suppressed, it would be possible not to draw a section, but then, by looking at the diagram, it would be impossible to understand: there is no documentation at all or it is suppressed. Therefore, such a decision was made - instead of suppressed lines of documentation, draw a section in which instead of all the text only a badge is placed. The badge takes up less space than one line, it makes it possible to immediately understand that the documentation is there. And it makes it easy to add a tooltip for the section with the full text of the documentation. For instance:
Function with a suppressed documentation string
In cases where the documentation is voluminous and the initial familiarization with the module has already been completed, the docstring suppression functionality seems quite convenient.
Comment Suppression
Comments can also take up significant space on the chart. Sometimes there is a desire to suppress them. Do not draw comments at all seems unreasonable for the same reasons as for documentation lines, so it was decided to replace them with a small primitive that supports a tooltip with the full comment text.
Comments are shown
Comments suppressed
Suppression of except blocks
Typically, code located in except blocks does not lie on the main path of program execution, but serves to handle rare exceptions. Therefore, when analyzing the main purpose of the program, it seems convenient to be able to suppress the except blocks and present them as a small primitive, which makes it clear that they even exist.
Two except blocks shown
Suppressed except blocks
The tooltip for the primitive shows as many lines as there were except blocks indicating what is intercepted in each.
Smart scaling
Codimension in its current implementation supports two types of chart scaling. Normal, or just scaling, which is based on the size of the selected font. With this scaling, the font size increases or decreases, and then all sizes are recalculated. Smart scaling involves several fixed levels, each of which changes the contents of the chart. Each next level hides more and more elements in the diagram.
It was not possible to come up with reasonable short names for each of the levels, so they are indicated by numbers. To control the level switching on the toolbar, an indicator of the current level and buttons for moving up and down are added.
Managing smart scaling
A tooltip has been added for the level indicator that describes what changes have been made to the chart.
Smart scaling works in conjunction with element suppression settings. That is, if a level involves showing, for example, except blocks, then they can be independently suppressed by pressing the corresponding button. Using the examples below, it is assumed that no comments, no documentation lines, or except blocks are suppressed.
Level 0
This is the level of maximum detail. All graphic elements are shown.
Level 1
At this level, comments and documentation lines disappear from the chart. The remaining graphic primitives remain on the diagram, but the text in them is suppressed, with the exception of class and function headers.
Smart scaling: level 1
This representation allows you to quickly get an idea of the amount of code and its complexity: the higher the concentration of “boxes” on the diagram, the more complex the code. A tooltip has been added for each graphic at this level, in which suppressed text is shown.
Level 2
At this level, the diagram remains primitives that use the scope (with the exception of decorators) and conditional structures to display. This presentation is also intended for a quick assessment of complexity. In fact, nesting and branching remain on the diagram.
Smart Zoom: Level 2
Level 3
At this level, only classes and functions remain on the diagram. This display was conceived as a “bird's-eye view”, only large objects in the module remain visible.
Smart Zoom: Level 3
More levels
At the moment, the experience of practical application of the levels of smart scaling is very small. Therefore, it is difficult to draw firm conclusions about the usefulness of any of the levels, as well as the idea of smart scaling in general. The plans, however, include the introduction of the next level with number 4. It is supposed to show a dependency diagram of the module on it, a kind of look at the file system. In the center, the primitive of the current module can be located, and around it the primitives of the imported modules. Thus, it will be possible to visually evaluate the current dependencies.
Interestingly, the implementation of smart scaling did not require much effort. On the graphical representation, it turned out to be quite simple to manipulate the content. Therefore, the introduction of some new intermediate levels for experimental purposes can be performed very quickly.
Groups
The basic idea of the functionality is simple - to give the opportunity to combine several graphic elements in a group, followed by folding and expanding the group into one graphic element with a new signature. A use case may be, for example, such. The developer begins to work with a new code for him written by another person. In the course of the analysis, understanding gradually comes, which makes a piece of code consisting, say, of ten blocks. Then the developer combines these blocks into a group and supplies it with suitable text. Upon further analysis, the group remains minimized on the screen, that is, on the diagram, instead of 10 elements, there is only one, which should facilitate subsequent analysis. Of course, groups can be nested and there are no restrictions on the number of groups in the module.
No one forbids the use of groups at the time of initial development. The programmer can create groups, which will facilitate code analysis for those who will work with him later.
Since groups support unlimited nesting, they can be considered as introducing a similarity of the third dimension to the code. The first two are a diagram, and the third is a collapsed group, inside which you can look.
CML for groups
To draw a group on a chart, Codimension needs to know where it starts and where it ends. To leave appropriate marks in the code, the micro markup language is used in the comments of the Codimension Markup Language.
# cml 1 gb id=101 title="Module imports"
import sys
import logging
import json
# cml 1 ge id=101
The first comment has gb code (from the words group begin) and is located before the first element of the group. The second comment has the ge code (from the words group end) and is located after the last element of the group. Codimension understands that comments are paired by the value of the required id attribute, which can be an arbitrary string.
Obviously, there are a number of limitations to the formation of a group. A group must have one input and one output. When selecting elements for a group on the chart, all of them should be selected without gaps. If the if condition is selected, then all branches must also be selected. For loops, a similar requirement is associated with the else construct: if it is present, then both parts must be selected, and the loop itself and its else block. Similarly, you cannot tear apart the components of try-except-else-finally. All these restrictions are taken into account when the user interface allows grouping.
In addition, as part of the work of the groups, a change was needed in the python parser. With the initial design of the technology, it was decided not to support the final comments. However, for groups, such a concluding comment is absolutely necessary. For example, a group can combine all the elements in one of the if branches:
item = self.__processes[index]
if item.widget:
# cml 1 gb id=102 title="Script started"
msg = item.widget.appendIDEMessage('Script started')
item.procWrapper.startTime = msg.timestamp
# cml 1 ge id=102
Here, the comment ending the group should belong to the if branch, which was not provided earlier.
For groups also added the ability to change the default colors. To do this, in the opening cml comment, optional attributes are supported for setting the background, ink and stroke colors.
Graphics for groups
To render groups, three types of primitives are required: a collapsed group, an expanded group, and an empty group.
For a folded group, such a primitive is desirable that it is immediately clear that several elements are hidden behind it and that it can be revealed. It is also desirable that the rendering of the primitive be simple regardless of whether it is selected or not. As a result, a rectangle with a double stroke was selected, inside which the group header is placed.
The minimized group
There is currently no complete satisfaction from the selected primitive, so there is a chance that it will be replaced with something else in the future.
The disclosed group should indicate its boundaries, plus there should be the ability to quickly see the group heading. To make the borders look “easier”, the rectangle of the opened group is drawn with a dashed line. There is no particular sense in making the fields of the open group wide, so the dotted line is drawn closer to the primitives than if it were another element.
Open group
In the upper left corner of the open group its “sensitive point” is drawn. When the mouse cursor is over it, the outline of the group is drawn with a solid line, and its title is shown above the group.
“Sensitive point” of the opened group
If necessary, in addition to the group title, you can add some control buttons. This will allow you to perform actions with a minimal number of mouse clicks.
There is nothing in the empty group, but it can, for example, contain a heading. In the future, when the editing functions on the graphical representation appear, new elements can be added to the empty group. Therefore, you still need to draw an empty group. A rectangle with a stroke is selected for it.
Empty group
The similarities between the primitives of the empty and collapsed groups are obvious. Therefore, if a primitive is changed for a collapsed group, then corresponding changes must be made for an empty group.
Group management is carried out from the context menu. To create a group, the necessary primitives are selected, and then the Group item is selected from the context menu ... After entering the group header, the necessary CML comments are added to the code and the chart is redrawn immediately. In this case, the newly created group is drawn in minimized form.
All other manipulations with groups - opening, hiding, deleting, editing a title, changing colors - are performed by choice from the group context menu.
How to try
You can try the new functionality by installing the Codimension IDE. Codimension supports Linux and Mac. Installation instructions are available at codimension.org