
Custom instruments: when signpost is not enough
Instruments for Apple's Xcode is a tool for analyzing the performance of an iOS application. They are used to collect and display data that is needed in code debugging. Last year, Apple introduced Custom Instruments. This is an opportunity to expand the standard set of tools for profiling applications. When existing tools are not enough, you can create new ones yourself - they will collect, analyze and display the data as you need.
A year has passed, and there are almost no new public tools and information on their creation on the network. So we decided to rectify the situation and share how we created our own Custom Instrument, which determines the reason for the weak isolation of unit tests. It is based on signpost technology (we wrote about itin the previous article ) and allows you to quickly and accurately determine the location of the blinking test.

To create a new tool for Xcode, you need an understanding of two theoretical blocks. For those who want to figure it out on their own, we will immediately give the necessary links:
For the rest - below is a brief summary of the necessary topics.
First select File -> New -> Project -> macOS Category -> Instruments package. The created project includes a file with the extension .instrpkg, in which a new tool is declared declaratively in xml format. Let's get acquainted with the markup elements:
If we want to supplement the logic of the new tool, then create a .clp file with CLIPS code. Basic language entities:
Which rules and in what sequence will be activated are determined by the CLIPS itself based on incoming facts, the priorities of the rules and the conflict resolution mechanism.
The language supports the creation of data types based on primitives, the use of arithmetic, logical operations and functions. As well as full-fledged object-oriented programming (OOP) with the definition of classes, sending messages, multiple inheritance.
Consider the basic syntax of a language that allows you to create logic for custom tools.
1. To create
Thus, we get an entry
To delete a fact, use the command
2. To create
3. To create and use variables, the following syntax is used (before the variable name there is an obligatory sign "?"):
4. You can create new data types using:
So, we defined a structure with the name prospect and three attributes name, assets and age of the corresponding type and a default value.
5. Arithmetic and logical operations have prefix syntax. That is, to add 2 and 3, you must use the following construction:
Or to compare two variables x and y:
In our project, we use the OCMock library to create stub objects. However, there are situations when a mok lives longer than the test for which it was created, and affects the isolation of other tests. As a result, this leads to the “blinking” (instability) of unit tests. In order to track the lifetime of tests and mocks, we will create our own tool. The following is an algorithm of actions.
To detect problematic moxes, two categories of interval events are needed - the time of creation and destruction of the moxa, the start and end time of the test. To get these events, go to the library


Next, go to the project under study, make markup in unit tests, methods


First, we determine the data type of the input. To do this, we

Next, we determine the data type of the output. In this example, we will output simultaneous events. Each event will have a time and description. To do this, declare the scheme:

We create a separate file with the extension

In this block, using the attribute,

It

Let's move on to the file
1. Define the mock and unitTest structures with fields - the time of the event, the event identifier, the name of the test and the class of the mok.

2. Define the rules that will establish the facts of the types

Read these rules are as follows: if the input we get a fact-type os- signpost with the desired

The values of the variables from the signpost events are transmitted as follows:

3. We define the rules that release the completed events (they are redundant, since they do not affect the result).

You can read the rule like this.
If
1) there is unitTest and mock;
2) in this case, the beginning of the test occurs later than the existing moka;
3) there is a table for storing results with the detected-mocks-narrative schema;
then
4) create a new record;
5) fill in with time;
6) ... and a description.

As a result, we see the following picture when using the new instrument:

The source code for custom instrument and an example project for using the instrument can be viewed on GitHub .
Debugger is used to debug custom tools.

It allows you to
: 1. See compiled code based on the description in instrpkg.
2. See detailed information about what happens to the tool at runtime.

3. Display a complete list and description of system data schemes that can be used as input in new tools.

4. Execute arbitrary commands in the console. For example, display a list of rules with the list-defrules command or facts with the facts command

You can run tools from the command line - to profile the application during the execution of unit- or UI-tests on the CI-server. This will allow, for example, catching a memory leak as early as possible. To profile tests in the pipeline, we use the following commands:
1. Launch tools with attributes:
2. Running tests on the same simulator with the command:
As a result, a report with the extension .trace will be created in the working directory, which can be opened by the Instruments application or by right-clicking on the file and selecting Show Package Contents.
We examined an example of upgrading signpost to a full-fledged tool and told how to automatically apply it on the “runs” of the CI server and use it in solving the problem of “blinking” (unstable) tests.
As you dive into the possibilities of custom instruments, you will better understand in what other cases you can use the instruments. For example, they also help us to understand the problems of multithreading - where and when to use thread-safe data access.
Creating a new tool was quite simple. But the main thing is that after spending several days studying the mechanics and documentation to create it today, you will be able to avoid several sleepless nights in attempts to fix bugs.
The article was written with @regno , Anton Vlasov, an iOS developer.
A year has passed, and there are almost no new public tools and information on their creation on the network. So we decided to rectify the situation and share how we created our own Custom Instrument, which determines the reason for the weak isolation of unit tests. It is based on signpost technology (we wrote about itin the previous article ) and allows you to quickly and accurately determine the location of the blinking test.

Theoretical minimum
To create a new tool for Xcode, you need an understanding of two theoretical blocks. For those who want to figure it out on their own, we will immediately give the necessary links:
- on the structure of the .instrpkg tool markup file
- CLIPS for programming tool logic
For the rest - below is a brief summary of the necessary topics.
First select File -> New -> Project -> macOS Category -> Instruments package. The created project includes a file with the extension .instrpkg, in which a new tool is declared declaratively in xml format. Let's get acquainted with the markup elements:
what | Attributes | Description |
Data schemas | interval-schema, point-schema, etc. | Describes the data structure as a table like sql schemes. Schemes are used in other markup elements to determine the type of data at the input and output of the model, for example, when describing a mapping (UI). |
Import data schemas | import-schema | Import ready-made schemes. It allows you to use data structures that are defined by Apple. |
Tool model | modeler | Associates the tool with a .clp file, in which the logic of the tool is defined, and announces the expected data scheme at the input and output of the model. |
Tool description | instrument | Describes the data model and determines how events will be displayed in the UI. The data model is described using the attributes create-table, create-parameter, etc. Tool charts are defined by graph attributes, and the parts table is defined by list, narrative, etc. |
If we want to supplement the logic of the new tool, then create a .clp file with CLIPS code. Basic language entities:
- “Fact” is a certain event registered in the system using the assert command;
- “Rule” is an if-block with specific syntax that contains a condition under which a set of actions is performed.
Which rules and in what sequence will be activated are determined by the CLIPS itself based on incoming facts, the priorities of the rules and the conflict resolution mechanism.
The language supports the creation of data types based on primitives, the use of arithmetic, logical operations and functions. As well as full-fledged object-oriented programming (OOP) with the definition of classes, sending messages, multiple inheritance.
Consider the basic syntax of a language that allows you to create logic for custom tools.
1. To create
fact
, we use the construction assert
:CLIPS> (assert (duck))
Thus, we get an entry
duck
in the fact table, which can be viewed using the command facts
:CLIPS> (facts)
To delete a fact, use the command
retract
: (retract duck)
2. To create
rule
, use the construct defrule
:CLIPS> (defrule duck) — создание правила с названием duck
(animal-is duck) — если animal-is duck присутствует в таблице фактов
=>
(assert (sound-is quack))) — то создается новый факт sound-is quack
3. To create and use variables, the following syntax is used (before the variable name there is an obligatory sign "?"):
?
4. You can create new data types using:
CLIPS>
(deftemplate prospect
(slot name (type STRING) (default ?DERIVE))
(slot assets (type SYMBOL) (default rich))
(slot age (type NUMBER) (default 80)))
So, we defined a structure with the name prospect and three attributes name, assets and age of the corresponding type and a default value.
5. Arithmetic and logical operations have prefix syntax. That is, to add 2 and 3, you must use the following construction:
CLIPS> (+ 2 3)
Or to compare two variables x and y:
CLIPS> (> ?x ?y)
Practical example
In our project, we use the OCMock library to create stub objects. However, there are situations when a mok lives longer than the test for which it was created, and affects the isolation of other tests. As a result, this leads to the “blinking” (instability) of unit tests. In order to track the lifetime of tests and mocks, we will create our own tool. The following is an algorithm of actions.
Step 1. Making markup for signpost events
To detect problematic moxes, two categories of interval events are needed - the time of creation and destruction of the moxa, the start and end time of the test. To get these events, go to the library
OCMock
and mark them using signpost
methods init
and the stopMocking
class OCClassMockObject
. 

Next, go to the project under study, make markup in unit tests, methods
setUp
and tearDown
:
Step 2. Create a new tool from the Instrument Package template

First, we determine the data type of the input. To do this, we
.instrpkg
import the scheme in the file signpost
. Now the events created signpost
will fall into the tool: 
Next, we determine the data type of the output. In this example, we will output simultaneous events. Each event will have a time and description. To do this, declare the scheme:

Step 3. We describe the logic of the tool
We create a separate file with the extension
.clp
in which we set the rules using the CLIPS language. To let the new tool know in which file the logic is defined, add a block modeler
: 
In this block, using the attribute,
production-system
specify the relative path to the file with the logic. In the attributes output
and required-input
determine the data schema at the input and output, respectively.
Step 4. We describe the specifics of the presentation of the tool (UI)
It
.instrpkg
remains to describe the tool itself in the file , that is, the display of the results. Create a table for the data in the attribute create-table
using the previously declared schema detected-mocks-narrative
in the attribute schema-ref
. And set up the type of information output - narrative (descriptive):
Step 5. We write the logic code
Let's move on to the file
.clp
in which the logic of the expert system is defined. The logic will be as follows: if the start time of the test intersects with the interval of the moka’s life, then we believe that this mok “came” from another test - which violates the isolation of the current unit-test. In order to eventually create an event with information of interest, you need to do the following steps: 1. Define the mock and unitTest structures with fields - the time of the event, the event identifier, the name of the test and the class of the mok.

2. Define the rules that will establish the facts of the types
mock
and unitTest
based on incoming events signpost
: 
Read these rules are as follows: if the input we get a fact-type os- signpost with the desired
subsystem
, category
, name
andevent-type
, then create a new fact with the type that was defined above (unitTest or mock), and fill it with values. It is important to remember here - CLIPS is a case-sensitive language and the values of subsystem, category, name and event-type must match what was used in the code of the project under study. 
The values of the variables from the signpost events are transmitted as follows:

3. We define the rules that release the completed events (they are redundant, since they do not affect the result).

Step 6. Define the rule that will generate the results.
You can read the rule like this.
If
1) there is unitTest and mock;
2) in this case, the beginning of the test occurs later than the existing moka;
3) there is a table for storing results with the detected-mocks-narrative schema;
then
4) create a new record;
5) fill in with time;
6) ... and a description.

As a result, we see the following picture when using the new instrument:

The source code for custom instrument and an example project for using the instrument can be viewed on GitHub .
Tool debugging
Debugger is used to debug custom tools.

It allows you to
: 1. See compiled code based on the description in instrpkg.
2. See detailed information about what happens to the tool at runtime.

3. Display a complete list and description of system data schemes that can be used as input in new tools.

4. Execute arbitrary commands in the console. For example, display a list of rules with the list-defrules command or facts with the facts command

Setup on CI server
You can run tools from the command line - to profile the application during the execution of unit- or UI-tests on the CI-server. This will allow, for example, catching a memory leak as early as possible. To profile tests in the pipeline, we use the following commands:
1. Launch tools with attributes:
xcrun instruments -t -l -w
- where
template_name
is the path to the template with tools or the name of the template. You can get the teamxcrun instruments -s
; average_duration_ms
- the recording time in milliseconds should be greater than or equal to the test execution time;device_udid
- simulator identifier. You can get the command xcrun instruments -s. Must match the identifier of the simulator on which the tests will be run.
2. Running tests on the same simulator with the command:
xcodebuild -workspace -scheme -destination
test-without-building
- where
path_to_workspace
is the path to the Xcode workspace; scheme_with_tests
- scheme with tests;device
- simulator identifier.
As a result, a report with the extension .trace will be created in the working directory, which can be opened by the Instruments application or by right-clicking on the file and selecting Show Package Contents.
conclusions
We examined an example of upgrading signpost to a full-fledged tool and told how to automatically apply it on the “runs” of the CI server and use it in solving the problem of “blinking” (unstable) tests.
As you dive into the possibilities of custom instruments, you will better understand in what other cases you can use the instruments. For example, they also help us to understand the problems of multithreading - where and when to use thread-safe data access.
Creating a new tool was quite simple. But the main thing is that after spending several days studying the mechanics and documentation to create it today, you will be able to avoid several sleepless nights in attempts to fix bugs.
Sources
- https://developer.apple.com/videos/play/wwdc2018/410/
- https://developer.apple.com/videos/play/wwdc2018/405/
- https://help.apple.com/instruments/developer/mac/current/
- https://help.apple.com/instruments/mac/current/#/devb14 ff aa5
- http://www.clipsrules.net/Documentation.html
- https://medium.com/appspector/building-custom-instruments-package-9d84fd9339b6
- http://desappstre.com/how-to-custom-instruments-xcode/
The article was written with @regno , Anton Vlasov, an iOS developer.