TextTest is a cross-platform python framework for testing GUIs and more. Part 2

Continuation of the story about the wonderful cross-platform TextTest functional-testing framework. The first part of the article.

GUI Testing Methods

Now the vast majority of GUI testing tools work in one of two ways:

1. Take screenshots after each screen change followed by a comparison of the changes.
2. Provide functions that allow you to read widget parameters through the operating system API

The problems of the first ones are instability of tests, it is worth changing the relative position of the buttons on the form, even by a couple of pixels, a bunch of tests will fall, the screen resolution will change - all the tests will drop, they will switch to a new version of the GUI framework, in which, let's say, they improved font rendering - again all tests will fall. For each operating system, you need to do your own set of tests, you may have to redo everything even after installing the next service pack, if it has affected the graphics subsystem.
I do not argue the appearance of testing is useful, but it should only be tests for design, to test through them the logic of the GUI is a mockery of a person.

The second approach has other problems - tests are written for a long time and difficult, and in addition they are often completely incomprehensible to others, especially non-programmers. Foot wrappers like this:

ActivateWindow("Unsaved Document 1 - gedit")
SetEdit(5, "5")
ClickButton("Save")
Wait(10)
VerifyLabel(2, 10)

do not give an idea of ​​what is happening and what is actually being tested. The programmer needs to explicitly write assertes, so that after pressing this button, this label has changed its value to this, but this one has not changed. Also, the operating system API may not provide full access to the widget internals, and even if it is a non-standard widget, then it becomes almost impossible to consider its properties.

But here it is already easier to test the logic of the GUI, we are not tied to the graphical representation of the elements, and having a good tool you can even try to write cross-platform tests.

Finally, both approaches have a common problem with testing non-synchronous events. Suppose you enter a URL in your browser, and when to check that the page has loaded?

You can make a timeout, but the problem is that if the page loads on average in 5 seconds, then you need to set the timeout 5 times “for fidelity”, and even that will not work. This leads to a slowdown of the tests and to the probabilistic nature of their execution.

The second option is to be “sharpened” on some secret widgets, and as soon as they take the necessary state, consider that the action has been completed. This adds work to the programmer and makes the tests even more unreadable and incomprehensible.

What alternatives does TextTest offer?

He suggests using the StoryText library to solve the problems described above. The latter, before starting your application, “wraps” the GUI library access interfaces, slipping the program its interface options. And this is done completely transparently. In most cases, you will not have to change a single line of code in the application under test. This makes it possible to save user actions and the reaction of the program to these actions when recording the test. And when testing - to repeat user actions and compare the reaction of the program with the reference.

So, what does this give us:

• We abstract from the location of the elements on the form and their appearance, it doesn’t matter to us that buttons on different OS have a different look, we only test the logic. If we need to verify that label values ​​have changed after clicking on the button, then no design changes to the form will break this test.
  
  
• We get full access to the widget, no WinApi will allow you to get as much information about the widget as the native library that creates this widget.
  
  
• For asynchronous events, a concept such as Application Events is introduced , which, although it forces the code to be slightly modified, it is only one line for each asynchronous event. In short, the essence of the latter is that upon completion of the asynchronous event, we call the code notifying StoryText that the operation has completed. StoryText remembers this fact and during playback it will honestly wait for this event from the application and only then will continue the test. All! No timeouts, no hidden widgets and other perversions, everything is simple and fast
  
  
• The ease of writing tests allows you to create them even for a person who is poorly familiar with programming. You don’t have to search for the desired button and click on it, then you don’t need to assert the values ​​of the changed elements, StoryText records all the events that the user has executed and saves all the changes in the GUI, and, as you will see later, does it in a rather readable form.
  
  
• Ease of understanding tests. After recording the first test, StoryText offers to enter aliases for all events that have occurred, i.e. an automatically generated test will not look like this:
  

  entry_in = FindEditByName("entry_in")
  SetValueForEdit(entry_in, 5)
  calc_async = FindButtonByName("calc_async")
  SendEvent(calc_async, )
  SendEvent(calc_async, )
  SendEvent(calc_async, )
  WaitEvent("data to be loaded")
  exit = FindButtonByName("exit")
  SendEvent(exit, )
  SendEvent(exit, )
  SendEvent(exit, )
  

  
  that agree a little nice, but like this:
  

  enter_data 5
  run_calc_async
  wait for data to be loaded
  exit_from_form
  

  
  Where enter_data, run_calc_async and exit_from_form are the user names that he entered after the test was written (alias values ​​- you can always see the special settings file)

Agree ideology - wonderful, tests are created quickly, conveniently, readably. Of course, there are problems, not with ideology, but with implementation. The author, it seems, the main library for the GUI is gtk, the rest are added in some experimental mode and are not fully implemented, however, I will show below how difficult it is to add new functionality.

Fourth example. Testing a synchronous GUI on Tkinter

Testing will be the TestGUI class. The class creates a very simple form on which it is proposed to enter a value, and press one of the OnCalcSync or OnCalcAsync buttons, the first will display the value multiplied by two at once, the second will wait 10 seconds and display the same thing. Yes, and I apologize in advance for the terrible appearance of the form, but this is just an example.

I want to note that when starting the test, the tkinter_ex module is imported (you can download it from here and put it next to test.py). He is needed because support for tkinter in the library is still “Experimental and rather basic support for Tkinter,” as the author himself writes. In particular, the standard module for tkinter is completely unable to track text changes in the Label class, but fortunately it’s not difficult to fix it. To do this, we replace the standard Tkinter.Label with our own, preserving the original name and functionality, and in functions that could change the label text - “configure” and “__setitem__”, we add logging like this: “Updated Text for label '% s' ( set to% s). " Thanks to the dynamism of Python, the code is not complicated

So, let's go: Add a new test-suite “Suite_GUI” and a test for it “Test_GUI_Sync” with the parameter “gui”. in simpletests \ config.cfg we add settings indicating that we will test the GUI based on tkinter

# Mode for Use-case recording (GUI, console or disabled)
use_case_record_mode:GUI
# How long in seconds to wait between each GUI action
slow_motion_replay_speed:3.0
# Which Use-case recorder is being used
use_case_recorder:storytext
# Single program to use as interpreter for the SUT
interpreter:storytext -i tkinter
virtual_display_count:0

I will not decrypt all the settings, you can read about them here . I’ll pay attention to only one thing: “virtual_display_count”, it should only make sense on UNIX systems and allows you to run tests on virtual displays through Xvfb. But due to an implementation error, if this parameter is not set, StoryText tries to create virtual displays on Windows, where Xvfb is absent. Therefore, the setting must be explicitly added and set to 0.

Do not forget to restart the IDE after making the settings. After that, in the menu "Test_GUI_Sync" the item "Record Use-Case" will become available, run, do not change anything in the window that appears. Our test unpretentious form appears:

in it you need to perform the actions that we want to test, enter the value in the "Input" field, click "OnCalcSync" and then "Exit". After that, StoryText will offer us to give human names for the actions performed, fill them, for example like this:

As usual, we save and try to run the test again, everything works fine.
To understand what happened, you can look at what appeared on the disk: simpletests \ storytext_files \ ui_map.conf, which describes the aliases that we just entered. It is also worth looking in the folder with the test (Test_GUI_Sync) on the file usecase.cfg, in which the resulting actions are described in a completely human language:

enter_data 5
run_calc_sync
exit_from_form

in stdout.cfg, you can see all the changes in the state of the form that occurred after user actions. There we will see a line generated by our class in response to a change in the Label value “Updated Text for label 'label_out' (set to 10)”. So everything works. Great move on to the next example

The fifth example. Testing the asynchronous GUI on Tkinter

Add a new test "Test_GUI_Async" to "Suite_GUI" and write it, as last time, only instead of "OnCalcSync" click on the button "OnCalcAsync" and this time you will have to wait for the calculation results for 10 seconds. Upon completion of the work, you will need to somehow call the action of clicking on the new button, well, for example, the path will be “run_calc_async”. We called all other actions last time and StoryText remembered them.

This test is interesting because actions on the form now occur asynchronously and after clicking the button ten seconds elapse before “TestGUI._operation_finish” is called from the test.py module. In this function, in addition to calculating the result, the line is added:

storytext.applicationEvent('data to be loaded')

which tells StoryText that a certain asynchronous operation has completed its work, and this needs to be reflected in the test, and when executed automatically, you should stop at the same place and wait for the event to occur.
If you look at usecase.cfg in the new test, you will see:

enter_data 5
run_calc_async
wait for data to be loaded
exit_from_form

those. the difference from the previous example is only that after pressing the button we expect the event “data to be loaded” and only after it we close the form. We save the results, run it again, we see that StoryText patiently waits for the set 10 seconds before completing the test.
As you can see, testing asynchronous events is not at all difficult.

Batch execution

Everyone knows that if running tests takes a lot of time and effort, no one will do it, so tests should be launched with one button or even run on a separate server in fully automatic mode. TextTest for this provides functions of batch execution with the subsequent generation of a report and either uploading it to a format (html, JUnit, Jenkins), or sending it by e-mail.

We will unload everything in html. You only need to configure the path to the folder for storing test results and the path for storing html reports. To do this, add the following lines to the end of the file in the main config.cfg:

[batch_result_repository]
default:batchresult
[historical_report_location]
default:historicalreport

There you can specify quite a few additional parameters, I will not dwell on them, here is a link to the documentation on batch mode.
To run the tests automatically, you need to start the texttest.py module with the "-b nightjob" parameters through python, something like this:

python c:\TextTest\texttest-3.24\source\bin\texttest.py -b nightjob

Again, for the command line, there are also a bunch of parameters, here you can read more.
After execution, next to the simpletests folder, a new batchresult will appear with the results, from which you can already generate a report in the form of html by adding the "-coll web" parameter:

python c:\TextTest\texttest-3.24\source\bin\texttest.py -b nightjob -coll web

Now another historicalreport folder will appear in which the html page with the results lies. After several days of work, it may look like this:

Well, or you can look at the test results of TextTest itself . So, by the way, quite interesting statistics - about 4,000 tests are run daily, and the test coverage of almost all modules is close to 100% (although I certainly understand that the percentage of coverage itself says little about it).

To summarize

pros

Let's try to summarize the positive aspects of the framework, mainly from the point of view of GUI testing:

• Cross-platform, cross-library (referring to the GUI libraries PyGTK, Tkinter, etc.), partially even there is cross-language support (the author often mentions working with Java, though I don’t know how convenient it is)
  
  
• Free and open source.
  
  
• The author’s activity in finalizing the framework, which is amazing for a 10-year project.
  
  
• After the initial setup - ease of use.
  
  
• A lot of work has been done to enable non-programmers to use the product, and make the tests themselves human-readable.
  
  
• For testing the GUI, it’s not the operating system APIs that are used, but the toolkit API itself (PyGTK, Tkinter, etc.), which makes integration as complete and simple as possible.
  
  
• Allows you to test the logic of work, abstracting from the appearance.
  
  
• It is quite flexible, faced with a problem, you can almost always climb the documentation and find how to fix everything by setting.
  
  
• It allows you to integrate with popular bugtrackers (Bugzilla, Jira, Trac), version control systems (CVS, Bazaar, Mercurial), and upload the results in formats (html, JUnit, Jenkins) or send by mail. Moreover, adding integration with other systems seems not very difficult. Although I admit, I have not tried how well he knows how to work with all these systems.

Minuses

It would be wrong not to mention the cons, especially since they are:

• The first is the GUI, it is not very functional, the settings are mainly done not through it, but through the documentation + manual editing of the configs. In fairness, I must say that the setup is done once, and then only rivet type tests. But at the initial stage of the study, there is really not enough opportunity to quickly tick off and start working in order to understand what's what.
  
  
• The stability of the GUI, sometimes it crashes, I don’t even know what to attribute it to, either PyGTK's work, or weak testing under Windows, but the fact remains. But again, the GUI is used only when creating a test, and then it is run many times and I did not observe any problems with the latter.
  
  
• Sometimes there are errors and you have to go into the code to understand what happened (I gave an example with the virtual_display_count parameter above.) However, everything that I came across is solved in one way or another, the more the source code is open and you can always see what needs to be corrected. Then again, I sin on Windows, it seems the author is more interested in working under UNIX systems, I hope there are no such problems.
  
  
• Documentation. Maybe it’s me, but I have to read the documentation very, very thoughtfully, several times to understand how the next chip works. Some understandable things are painted very well, but in complex ones, nuances are missed. The available documentation is quite convenient to use as a reference, but only after you understand how this or that thing works.
  
  
• Community Here things are very bad, on the Russian Internet in general, everyone is silent about the framework, they mention the maximum that it exists. In English, all I found was stories about ideology and how good it was. Some sensible descriptions, manuals, articles, etc. I did not find, everything is only on the author’s site. I can’t explain this fact in any way, the framework looks very mature, interesting and “tasty”, in fact there are no alternatives, what’s wrong with it, I don’t understand.
  
  
• He hosts the launchpad, and not the now popular GitHub, I am afraid that this fact will alienate many who have the desire and ability to help the author in writing code.

Alternatives

The description would not be complete if you did not specify alternatives for testing the GUI in python, here's what you could find:

robotframework is difficult to say something specific about it, it is very extensive and you need to read the documentation and try it for a long time thoughtfully. Apparently, this is a cross-platform framework for testing the TextTest level and it is definitely worth a look at it for everyone who chooses a tool.

ldtp is the next GUI testing tool. It has three implementations of
LDTP / Cobra / PyATOM for Linux / Windows / OS X, respectively. It supports a bunch of languages ​​(Java / Ruby / C #, etc.) including the one we need Python. I liked the documentation. Actively developing.
I did not like only the principle, judging by the description of the tests, they are written somehow like this:

selectmenuitem('frmUnsavedDocument1-gedit', 'mnuFile;mnuOpen')
settextvalue('frmUnsavedDocument1-gedit', 'txt0', 'Testing editing')
setcontext('Unsaved Document 1 - gedit', '*Unsaved Document 1 - gedit')
verifytoggled('dlgOpenFile...', 'tbtnTypeafilename')

those. an element with the desired heading is searched for and the desired action is either taken or the state is read. That after TextTest seems like a step backward, but if something still doesn’t work out with the latter, then ldtp will be one of the first candidates for the transition.

pywinauto about it and on a hub you can find a few words . The principle is the same as in ldtp - we find the element we need and do something with it. Yes, and it works only under Windows.

Dogtailjudging by the description, it is very strongly tied to Unix, there are even separate versions for “GNOME 3, KDE4.8” and “Gnome 2”, so it seems that it uses graphical shell APIs for testing, which means it is obviously not cross-platform. However, it is still developing, so if something needs to be tested under Unix, it might be worth looking at its side for the

guitest library for testing GUIs in python applications, mainly for pyGTK. Last release date is 11/13/2005. I’m afraid that after so many years without development, she won’t even be able to test the applications declared by pyGTK, not to mention the required Tkinter. I didn’t even look.

pyAA is another abandoned project, and abandoned since 2005, it’s scary to get involved in such projects, and it only works with Windows ...

pyGUIUnitthe link states that the library can test PyQt applications. If we go to the documentation, we see that the whole framework consists of one class and two functions, a quick look at which - you can understand that nothing good can be expected.

Plus, there is still a fairly large number of general testing frameworks that allow you to test any application through screenshots, I described above why they are inconvenient and therefore did not even consider, moreover, most of them work only on Windows.

Total: cross-platform frameworks for testing the logic of the GUI on python with Tkinter support are surprisingly few, most of them are abandoned and forgotten. Those 2-3 that are worth it to look at them are not the fact that they will do. TextTest is, in my opinion, the perfect choice so far, let's see what happens next.