Automated functional testing of Windows applications using Ranorex

Last year, not only the Web, but also the Windows client appeared in our company. Since the test automation tools that we used for the Web (for example, Selenium ) in this situation, of course, we could not use, we faced the need to search for automated functional testing technology for Windows applications.

Our search criteria were as follows:

• The functional test development environment should support both automatic recording of test cases (for use by testers with minimal programming experience) and the ability to write individual parts of tests manually
• Since the client is developed in C #, I would like to write functional tests either in C # or in one of the .NET languages
• Must have Data-Driven Testing support
• XPath should preferably be used to identify user interface elements.
• The code and all testing artifacts should be stored in the same version control system in which our main code is stored (in our case, Git)
• This automated functional testing technology should be easily integrated into our continuous integration system ( Jenkins )

After preliminary market research, we settled on 2 products :

1. HP QuickTest Professional , which is one of the most used products on the market, but at the same time has an extremely high price
2. Ranorex is a newer product (although I first heard about it back in 2007, but then it was still pretty raw), but it is already often mentioned as an alternative in professional forums

Over the course of a month, I tested both products (HP QTP 11 and Ranorex 4) and now I would like to share my conclusions about the capabilities of Ranorex, its comparison with HP QuickTest Professional, and also consider the integration of Ranorex with the Jenkins CI server (but integration with other servers, e.g. TeamCity will look similar).

Features of Ranorex

• What types of applications can be tested : here the Ranorex suite is comparable to HP QTP: Windows native (WinForms, WPF, Win32), Java, Qt, Delphi, Flex + of course HTML (browsers - IE, Firefox, Chrome, Safari). The latest version also added support for testing mobile clients on Android and iOS, but so far I have not used this opportunity
• Recording actions and element repositories : Ranorex supports both recording actions using the built-in recorder (similar to HP QTP) and identifying user interface elements using the Ranorex Spy component .
  
  All identified elements are stored in an XML repository, where each element is written using XPath notation. For example, a button on a toolbar might be described as:
  

  /form[@ title~'^MyApp']//toolbar[@ controlname='mainToolbar']/button[@ controlname='addUser']
  

  Based on the recording, the code is automatically generated (in C # or VB.NET). Any step can also be written manually (for example repo.MyApp.MainPain.BtnOk.Click();)
• Identification of user interface elements : Using XPath (or, as it is called here, RanoreXPath) makes finding the necessary elements extremely easy in most cases. For example, regular expressions ( form[@ title~'^MyApp.*']), search for all descendants ( .//button), logical operators ( @ text=’OK’ OR @ controlname='buttonOK'), built-in functions ( table/row/cell[first()='True']), and much more are supported . Which is very convenient, each item in the repository has a search-timeout property. For example, if we work with the button from the example above and the form[@ title~'^MyApp']search (time ) window is 30 seconds, then Ranorex will wait 30 seconds for this window to appear (for example, when the application starts), after which it will proceed with the actions with the button. This allows you to almost completely abandon the use of functions in testswait()
• The elements in the repository : in Ranorex we work with different types of elements (eg form, toolbar, container, tree, list,, etc.). Since we work with elements not directly, but with element adapters, the element is button in the case of a Windows native application, in the case of Java sentences and in the case of Flex, etc. will have a similar set of properties and methods (e.g. text, pressed, click())
  
• IDE : Ranorex Studio is based on the free SharpDevelop . As a result, the organization of the code and the test cases resembles MS Visual Studio as much as possible: All test cases are organized into projects. Projects that are used together can be combined into solutions. At the output after compilation, we get one EXEfile per solution and one DLLper project. It is enough to copy them to a test machine and run as a regular Windows application
  
• Organization of test cases : For each solution (solution - see above), there is one test suite. In this test suite we add the test cases we need and drag recordings from our projects. Recordings can either be recorded automatically, or written in one of the supported languages ​​manually. Here in the manual modules we can use any C # / VB.NET code we need, for example code to access the Windows registry, upload files via FTP, etc.
• Data-Driven Testing : any test case can be connected to data from a CSV / Excel file, or from a database. Each module (record) can have variables associated with the data or filled in the module itself
  
• Programming language : Ranorex uses either C # or VB.NET to write tests. In the case of HP QTP, VB Script is used. Probably, for people without programming experience, VB.NET or, especially, C # are more difficult to learn than VB Script. Scripting languages ​​are usually more suitable for testing automation, but I personally, choosing between C # - a full-fledged OO language with full .NET support and scripted and rather primitive VB Script, of course I prefer C #. For some, the Ranorex support for VB.NET will be the best choice.
  
• Storage and versioning of the code : HP QTP stores the code and repositories partially in binary format, which makes versioning it with third-party tools like SVN or Git quite problematic (and my own versioning system based on HP QC seemed extremely primitive to me, for example, I did not find the option committing several resources at once, tagging, etc.). In contrast, Ranorex stores all the code (including that generated by auto-generation) and all repositories in text form (for example, the XML format is used for repositories) - this gave us the opportunity to version and collaborate on using the familiar Git with all its features like a local repository, tagging, etc.

Using Ranorex with Continuous Integration

Unlike HP QTP, Ranorex does not have a system that allows you to run tests on a schedule or on an external trigger on remote computers. On the one hand, this is a minus, on the other, we were able to seamlessly integrate Ranorex tests with our Jenkins server (but any server, for example, the same TeamCity , is suitable ):

1. Compilation of Ranorex tests: On Jenkins, we have the Build_Ranorex_Tests job. As soon as one of the testers makes push code to our Git repository, one of the Git hooks starts this job. It consists of two phases: compilation of all solutions (test suites) using MS Build ( /t:Clean,Build /p:Configuration=Release). At the end of the MS Build phase, all the files necessary for testing (namely EXE (one per solution), DLL (one per project), CSV / Excel and others) are archived (step - Archive the artifacts)
2. Test execution: Next, the next “matrix" job is launched on all test machines (we test under Windows XP, Vista and Windows 7) on which Jenkins Slaves are installed . This job first installs the latest version of our application (using MSI), then sequentially launches the EXE files of the test suites that were compiled by the previous job (see above). After their execution, we get one ZIP file per test suite, each of which contains 1 XML file with the results and a directory with screenshots
3. Presentation of the results: After this, the script (it can be made a separate small suite in Ranorex, which will be run last) converts the Ranorex XML format to xUnit format. Thanks to this, for each client, we have both a report in Ranorex format and in a format that Jenkins understands, so Jenkins can draw a graph of test results. Jenkins also mails test results in Ranorex format to all interested parties.
   

Price

Unfortunately, unlike the same Selenium, Ranorex is not a free open source product. But comparing the automatic testing system for Windows is not necessary with Selenium, but with the same HP QTP or IBM Rational Functional Tester. In this case, Ranorex ne seems so expensive. 1 license with reference to the workplace costs 1,480 euros per purchase + 290 euros per year (starting from the second) for further updates and support. It’s several times cheaper than HP QTP and IBM Rational Functional Tester

conclusions

Based on our comparative testing of Ranorex and HP QTP, we preferred to use Ranorex in the future. The main advantages for us were: a convenient and understandable development environment, full support for .NET development and C # and VB.NET languages, the use of XPath to identify elements in the repository, the ability to store tests in our standard versioning system (Git), ease of integration with the CI system (Jenkins).