March 10, 2015 at 11:10

Auto-registration of tests on With means of language

    Testing in CA relatively recent article was “Semi-automatic registration of unit tests in pure C” , in which the author demonstrated a solution to the problem using counters from Boost. Following the same principle, a (successful) attempt was made to repeat this experience already without using Boost for reasons of the inconsistency of having a Boost dependency in a C project, and even in such a small amount. At the same time, tests included auxiliary preprocessor directives in large numbers. And everything would have remained, but almost at the final stage an alternative registration method was found that allows you to completely get rid of additional actions. This is a C89 test registration solution and a slightly more demanding build solution for registering test suites.

    The motivation for all this is simple and straightforward, but for completeness it is worth briefly outlining it. In the absence of auto-registration, one has to deal with either typing / inserting repeating code, or with generators external to the compiler. The first is reluctant to do, plus this activity itself is error prone, the second adds unnecessary dependencies and complicates the assembly process. The idea to use C ++ in tests only for the sake of this opportunity, when everything else is written in C, evokes the feeling of shooting from a cannon at sparrows. To all this, in principle, it is interesting to solve the problem at the same level at which it arose.

    We define the final goal as something similar to the code below with the additional condition that the test names are not repeated anywhere except where they were defined, i.e. they are dialed once and only once and then are not copied by any oscillator.

    TEST(test1) { /* Do the test. */ }
TEST(test2) { /* Do the test. */ }

    After a brief digression to clarify the terminology, you can begin to search for a solution.

    Terminology and proposed test structure


    Different test frameworks use words inconsistently to refer to individual tests or groups of tests. Therefore, we will define some words explicitly, and at the same time we will show their meaning on the example of a fairly common test structure.

    A collection of tests (“suite”) will mean a group of test suites (“fixture”). This is the largest structural unit of the hierarchy. Sets in turn group tests within a collection. Tests are already on their own. The number of elements of each type is arbitrary.

    It’s the same graphically:



    Each higher level combines the elements of smaller ones and optionally adds the procedures for preparation (“setup”) and completion (“teardown”) of tests.

    Registration of tests in sets

    Never let your sense of morals prevent you from doing what is right.
    - ISAAC ASIMOV Foundation

    Separate tests are added more often than whole sets, therefore, auto-registration is more relevant for them. Also, all of them are located within one translation unit, which simplifies the solution of the problem.

    So, it is necessary to organize the repository of the list of tests by means of the language, without using the preprocessor as the main control element. Refusing the preprocessor means that we are left with no explicit counters. But the presence of a counter is almost necessary if it is necessary to uniquely identify the tests and, in general, somehow access them, and not just announce them. At the same time, there is always a built-in macro at hand __LINE__, but you still need to figure out how it can be applied in this situation. There is another limitation: some explicit assignments to the elements of the global array like

    test_type tests[];
static void test(void) { /* Do the test. */ }
tests[__LINE__] = &test;

    they are not suitable, because outside the functions such operations are simply not supported at the language level. The initial situation does not look very rosy:

    1. There is no way to store either intermediate or final state.
    2. There is no way to identify disconnected elements, and then put them together.
    3. As a result, there is no way to define a connected structure (basically an array, but the list would also do, there would be a way), due to the inability to refer to the previous entity.

    But not everything is as hopeless as it might seem. Imagine the perfect option, as if we have something that is missing. In this case, the code after expanding the auxiliary macros could look something like this:

    MagicDataStructure MDS;
static void test1(void) { /* Do the test. */ }
MDS[__LINE__] = &test1;
static void test2(void) { /* Do the test. */ }
MDS[__LINE__] = &test2;
static void fixture(void)
{
    int i;
    for (i = 0; i < MDS.length; ++i) {
        MDS.func[i]();
    }
}

    The thing is small: to implement a "magic" structure, which, by the way, is suspiciously similar to an array of a predetermined size. It makes sense to think about how we would work if this array were actually:

    1. We would define an array by initializing all the elements NULL.
    2. Assign values ​​to individual elements.
    3. We would go around the entire array and call each non- NULLelement.

    This set of operations is all that we need and does not look too unrealistic, perhaps arrays really come in handy here. By definition, an array is a collection of elements of the same type. Usually this is some one entity with support for the indexing operation, but it makes sense to consider the same array as a group of separate elements. Let's say whether this

    int arr[4];

    either

    int arr0, arr1, arr2, arr3;

    At the moment, and in the light of the mention of the macro __LINE__above, it should already be clear where the author is driving. It remains to understand how a pseudo-array with support for assignment at the compilation stage can be implemented. This seems like a fun exercise, so it’s worth a little more time to demonstrate the ready-made solution and ask the following questions:

    1. What entity in C can appear more than once and not cause a compilation error?
    2. What can be interpreted by the compiler differently depending on the context?

    Think of header files. After all, what is in them is usually present somewhere else in the code. For instance:

    /* file.h */
int a;
/* file.c */
#include "file.h"
int a = 4;
/* ... */

    In this case, everything works fine. Here is an example closer to the task:

    static void run(void);
int main(int argc, char *argv[])
{
    run();
    return 0;
}
static void run(void) { /* ... */ }

    It’s quite an ordinary code, which can be slightly expanded to obtain the desired functionality:

    #include 
static void (*run_func)(void);
int main(int argc, char *argv[])
{
    if (run_func) run_func();
    return 0;
}
static void run(void) { puts("Run!"); }
static void (*run_func)(void) = &run;

    The reader is invited to independently verify that changing the order or commenting on the last mention run_funcis consistent with expectations, i.e. if run_funcnot reassigned, then the only element of the "one-element array" ( run_func) is equal NULL, otherwise it points to a function run(). The absence of dependence on the order is an important property that allows you to hide all the "magic" in the header file.

    From the example above, it's easy to make a macro for auto-registration, which declares a function and stores a pointer to it in a variable numbered using a macro value__LINE__. In addition to the macro itself, you must list all the possible names of the pointer variables and call them one at a time. Here is an almost complete solution, not counting the presence of "extra" code, which should be hidden in the header file, but these are the details:

    /* test.h */
#define CAT(X, Y) CAT_(X, Y)
#define CAT_(X, Y) X##Y
typedef void test_func_type(void);
#define TEST(name) \
    static test_func_type CAT(name, __LINE__); \
    static test_func_type *CAT(test_at_, __LINE__) = &CAT(name, __LINE__); \
    static void CAT(name, __LINE__)(void)
/* test.c */
#include "test.h"
#include 
TEST(A) { puts("Test1"); }
TEST(B) { puts("Test2"); }
TEST(C) { puts("Test3"); }
typedef test_func_type *test_func_pointer;
static test_func_pointer test_at_1, test_at_2, test_at_3, test_at_4, test_at_5, test_at_6;
int main(int argc, char *argv[])
{
    /* Это упрошённая версия для наглядности, на самом деле указатели стоит
     * поместить в массив. */
    if (test_at_1) test_at_1();
    if (test_at_2) test_at_2();
    if (test_at_3) test_at_3();
    if (test_at_4) test_at_4();
    if (test_at_5) test_at_5();
    if (test_at_6) test_at_6();
    return 0;
}

    For clarity, it may be useful to look at the result of macro substitution, which implies the fact that it is impossible to place more than one test in a row, which, however, is more than acceptable.

    static test_func_type A4; static test_func_type *test_at_4 = &A4; static void A4(void) { puts("Test1"); }
static test_func_type B5; static test_func_type *test_at_5 = &B5; static void B5(void) { puts("Test2"); }
static test_func_type C6; static test_func_type *test_at_6 = &C6; static void C6(void) { puts("Test3"); }

    A link to the full implementation will be given below.

    Why does it work


    Now it's time to figure out what is happening here, in more detail, and answer the question of why this works.

    If we recall the example with the headers, we can distinguish several possible options for how data members can be represented in the code:

    int data = 0;    /* (1) */
extern int data; /* (2) */
int data;        /* (3) */

    (1)It is definitely a definition (and therefore an announcement, too) due to the presence of an initializer.

    (2)is an exclusive ad.

    (3)(our case) is a declaration and возможно,, definition. The absence of a keyword externand an initializer leaves the compiler no choice but to postpone a decision on what this statement is (“statement”). It is this “oscillation” of the compiler that is used to emulate auto-registration.

    Just in case, a few examples with comments to finally clarify the situation:

    int data1; /* Определение, так как больше нигде не встречается. */
int data2 = 1; /* Определение, из-за инициализатора. */
int data2;     /* Объявление, так как определение уже было. */
int data3;     /* Изначально, неизвестно, но после обработки следующей строки
                * становится понятно, что объявление. */
int data3 = 1; /* Определение, из-за инициализатора. */
/* Ключевое слово static ничего в этом плане не меняет. */
static int data4;     /* Изначально, неизвестно, но после обработки следующей
                       * строки становится понятно, что объявление. */
static int data4 = 1; /* Определение, из-за инициализатора. */
static int data4;     /* Объявление, так как определение уже было. */
int data5; /* Неизвестно, но в отсутствии определений считается определением. */
int data5; /* Аналогично, эти два "неизвестно" считаются за одно. */
int data6 = 0; /* Определение, из-за инициализатора. */
int data6 = 0; /* Ошибка, повторное определение. */

    Two cases are important for us:

    • There are only ads. In this case, the variable is initialized to zeros, which can be used to determine the absence of the test in the corresponding line.
    • There is at least one ad and exactly one definition. The address of the function with the test is entered in the corresponding variable.

    That, in fact, is all that is needed to implement the required operations and obtain a working automatic registration. This duality of some operators in the text allows you to expand the array element-wise and "assign" values ​​to part of the array.

    Features and disadvantages


    It is clear that if we do not want to insert a macro at the end of each test file, which would serve as the marker of the last line, then we must initially lay on some maximum number of lines. Not the best option, but not the worst. Say, one test file is unlikely to contain more than a thousand lines, and you can opt for this upper bound. There is one not very pleasant moment: if in this case the tests are defined on a line with a number greater than 1000, then they will be dead weight and will never be called. Fortunately, there is a simple “solution” option: it is enough to compile tests with a flag -Werror(a less stringent option: c -Werror=unused-function) and similar files will not be compiled. ( UPD2: prompted in the commentshow to solve this issue easier and with automatic interruption of compilation using STATIC_ASSERT. It is enough to TESTinsert a check for an acceptable value in each macro __LINE__.) The

    sufficiency of the approach with a fixed array is generally not the only reason why it is better to fix the maximum number of lines in advance. If this is not done, then the corresponding declarations (at the place where the tests were called) must be generated during compilation, which can significantly slow it down (this is not a hunch, but the result of attempts). It’s easier not to complicate things here, the benefit of being able to compile files of arbitrary size seems not worth it.

    In the macro exampleTEST()above you can see the use of a function pointer, this is just one test entry, but most likely you will want to add more. Wrong way to do this: add parallel pseudo-arrays. This will only increase compilation time. The correct way: to use the structure, in this case adding new fields is almost free.

    For real processing (not copying the code) of the elements of the pseudo-array, it is necessary to form a real array. Not the best solution would be to put the values ​​of the same function pointers into this array (or copy structures with information about the tests), since this will make the initializer not constant. But placing pointers to pointers will make the array static, which will free the compiler from having to generate code to assign values ​​on the stack at runtime, as well as reduce compilation time.

    Initially, this solution was born to implement transparent registration setup()/teardown()functions and only then it was applied to the tests themselves. In principle, this is suitable for any functionality that can be redefined. It is enough to insert a pointer declaration and provide a macro to redefine it, if the macro was not used, the pointer will be zero, otherwise - a user-defined value.

    Compiler messages about top-level errors in tests may surprise you with their volume, but this will happen in rather rare cases when there is no trailing semicolon and similar syntax errors.

    Finally, you can evaluate the result of the efforts:
    Test suite to:
    static void
teardown(void)
{
    /* ... */
}
static void
test_adds_fine_first_time(void)
{
    /* ... */
}
static void
test_errors_on_second_addition(void)
{
    /* ... */
}
void
addition_tests(void)
{
    test_fixture_start();
    fixture_teardown(teardown);
    run_test(test_adds_fine_first_time);
    run_test(test_errors_on_second_addition);
    test_fixture_end();
}
    		Test suite after:
    TEARDOWN()
{
    /* ... */
}
TEST(adds_fine_first_time)
{
    /* ... */
}
TEST(errors_on_second_addition)
{
    /* ... */
}

    Register test suites in collections

    A trick is a clever idea that can be used once, while a technique is a trick that can be used at least twice.
    - D. KNUTH, The Art Of Computer Programming 4A

    Something close to the previous task, but there are a couple of significant differences:

    1. Interesting characters (functions / data) are defined in different compilation units.
    2. And, as a result, there is no similar counter __LINE__.

    By virtue of the first paragraph, the trick from the previous section in its pure form will not work here, but the main idea will remain the same, while the means of its implementation will change slightly.

    As mentioned at the beginning, this part puts forward some additional requirements for the environment, namely, the build system, which should be able to assign files identifiers in the range [0, N)where Nrepresents the maximum number of test suites. Again, the border is on top, but, say, a hundred sets in each collection of tests should be enough for many.

    If last time the compiler did all the "dirty work" for us, then this time it was the turn to work for the linker (aka "linker"). In each translation unit, it is necessary to determine the entry point using the same file identifier, and in the main file of the test collection, check the characters for presence and call them.

    One option is to use “weak characters” . In this case, functions are defined almost everywhere as usual, but in the main file they are marked with an attribute weak(something like this:) __attribute__((weak)). An obvious drawback is the requirement for weak characters to be supported by the compiler and linker.

    If you think a little about the structure of weak characters, you will notice their similarity with pointers to functions: undefined weak characters are equal to zero. It turns out that you can completely do without them: it is enough to define pointers to functions as before, but without a keyword static. The use of pointers in explicit form also brings additional benefits in the absence of an automatically generated name in the list of stack frames.

    On this, the first difference from test suites can be considered reduced to an already known solution. It remains to determine the relationship between the units of translation. There is not enough information in the file to complete this task, therefore information from outside is needed. Here, for each build system, there will be implementation details, below is an example for GNU / Make.

    Determining the order itself is trivial enough, let it be the position of the file name in the sorted list of all files that make up the test collection. Do not worry about auxiliary files without tests, they will not interfere, as a maximum, they will create omissions in the numbering, which is insignificant. This information will be transmitted through the macro definition using the compiler flag ( -Din this case).

    Actually, the identifier definition function:

    pos = $(strip $(eval T := ) \
              $(eval i := 0) \
              $(foreach elem, $1, \
                        $(if $(filter $2,$(elem)), \
                             $(eval i := $(words $T)), \
                             $(eval T := $T $(elem)))) \
              $i)

    The first argument is a list of all file names, and the second is the name of the current file. Returns the index. The function is not the most trivial in appearance, but it does its job properly.

    Adding an identifier TESTID( $(OBJ)stores a list of object files here):

    %.o: %.c
	$(CC) -DTESTID=$(call pos, $(OBJ), $@) -c -o $@ $<

    On this, almost all difficulties have been overcome and all that remains is to use the identifier in the code, for example, like this:

    #define FIXTURE() \
    static void fixture_body(void); \
    void (*CAT(fixture_number_, TESTID))(void) = &fixture_body; \
    static void fixture_body(void)

    In the main file of the test collection, there should be corresponding declarations and their crawl.

    Remaining difficulties


    If the number of files increases above the set limit, some of them may “fall out” of our field of vision as this could happen with the tests. This time, the solution will require additional verification of compilation time. With the known number of files in the collection, it is easy to check whether they will be redundant. In fact, it’s enough to provide each broadcast unit with access to this information with the help of another macro:

        ... -DMAXTESTID=$(words $(OBJ)) ...

    All that remains is to add a check for the presence of a sufficient number of ads using something like:

    #define STATIC_ASSERT(msg, cond) \
    typedef int msg[(cond) ? 1 : -1]; \
    /* Fake use to suppress "Unused local variable" warning. */ \
    enum { CAT(msg, _use) = (size_t)(msg *)0 }

    There is a slightly less obvious problem of the conflict (double definition) of functions when adding / removing files of test suites. Such changes cause index displacement and require recompilation of all files that have been affected by this. Here it is worth recalling checking the dates of file modification by assembly systems and updating the catalog date when its composition changes, i.e. in fact, each compiled file needs to add a dependency on the directory in which it is located.

    As a result, the rule for compiling a file with tests takes a similar form:

    %.o: %.c $(dir %.c)/.
	$(CC) -DTESTID=$(call pos, $(OBJ), $@) -DMAXTESTID=$(words $(OBJ)) -c -o $@ $<

    Putting it all together, you can observe the following transformation of the definition of a collection of tests:
    Collection of tests before:
    void addition_tests(void);
void deletion_tests(void);
void expansion_tests(void);
static void
setup(void)
{
    /* ... */
}
static void
all_tests(void)
{
    addition_tests();
    deletion_tests();
    expansion_tests();
}
int
main(int argc, char *argv[])
{
    suite_setup(setup);
    return run_tests(all_tests) == 0;
}
    		Collection of tests after:
    DEFINE_SUITE();
SETUP()
{
    /* ... */
}

    Additional optimizations


    The need for periodic recompilation and some slowdown in the processing of each file make us think about ways to compensate for these costs. Recall some of the available features.

    Precompiled header. Since complex code is processed by the compiler for a long time, it will be logical to prepare the result of processing once and reuse it.

    Using ccache to speed up recompilation. A good idea in itself, for example, allows you to switch between repository branches an unlimited number of times and not wait for a full recompilation: the total time will be determined primarily by the speed of pulling data from the cache.

    -pipe compiler flag(if supported). It will reduce the number of file operations due to the use of additional RAM.

    Turn off optimization and exclude debugging information. In a normal situation, this should not affect the operation of tests in any way, except for some acceleration of the compilation process.

    Why is it all here? A possible deterioration in compilation performance was mentioned several times above, and I want to provide a means to combat this, as well as somewhat smooth out the effect with a couple of comments:

    • The drop in performance is primarily noticeable with a complete reassembly of tests and in a normal situation is not so critical.
    • Before applying the approach to tests described above, the time of complete reassembly of tests (with subsequent launch) in the case of the author was 6.5 seconds. After - it increased to 13 seconds, but the optimization of both the test announcement code and the assembly process corrected the situation, improving the indicator to 5.5 seconds. Speeding up the build process of the previous version of the tests improved the time to 5.7 seconds, which (surprisingly) is even slightly longer than the compilation time of the current version.

    References


    Initially, seatest was used to write tests , which suited almost everything, but lacked auto-registration. According to the results of the above activities on the basis seatest was made stic (it uses little the C99, but it is not mandatory in the general case), adds the missing from the point of view of the author. It is there that you can see the implementation details omitted here, namely in the stic.h header file . Selected intermediate sketches are available in a separate repository . An example of integration can be found in this Makefile (syntax is required to understand it).

    Summary


    Judging by the list on Wikipedia , stic may be the first successful attempt to implement auto-registration using C tools (naturally, with an eye on the described limitations). All proven alternatives include external test list generators ( UPD: in the comments they suggest a way to register tests close to the implementation of calling static constructors in C ++, which, however, requires the appropriate support from the compiler and linker, but the approach itself definitely deserves attention). The advantage of this method is not only the absence of additional dependencies, but also versatility (the compiler will not make a mistake because#ifdef, unlike a third-party script) and the relative ease of collecting additional test data. For example, it was quite simple to add a test run predicate in the form:

    TEST(os_independent)
{
    /* ... */
}
TEST(unix_only, IF(not_windows))
{
    /* ... */
}

    Let everyone decide for himself, but the author definitely liked the method, process and result, which seatest has now replaced, simplified the process of adding tests and reduced the volume of tests by as much as 3911 lines, which is about 16% of their previous size.
    Tags:

    Also popular now: