July 30, 2013 at 12:59CodedUI or Ranorex? Automation of functional testing of .NET applications
The author of the article is Tatyana Kurnosova, but she went to conquer the mountains of Kyrgyzstan. Therefore, the honor to publish this post fell to me.
In addition to the well-known Desktop , Mobile and Online applications, 2GIS company develops many internal Enterprise products. These products are hidden from the eyes of users, however, it is with their help that colossal work is being done to provide the entire 2GIS infrastructure with cartographic and reference data. The processing of this data is labor-intensive and requires error-free calculations, therefore, before “hiring” all products are thoroughly tested.
In this article I will tell you how we chose tools, how we analyzed their technical capabilities, so that they fit into our infrastructure, and what I stopped at.
At one time, we faced the task of choosing a package suitable for testing the user interface of the products used. Everything that lends itself to automation is covered by automated tests, and the GUI is no exception. For this, as you know, there are tools to simulate user actions.
Given that the products under test are implemented in .NET, and their testers are familiar with the C # programming language, it was decided to develop tests also in C #. From a large set of tools on the market, we settled on two. The first is CodedUI, which comes with Microsoft Visual Studio, which we are actively using. The second is Ranorex, since it is compatible with Visual Studio, it has a convenient interface and positive feedback from testers.
Coded uiis one of the solutions that come with Microsoft Visual Studio Premium / Ultimate, provides access to libraries for test development.
Ranorex Automation Tools is a complete development environment, as well as a set of tools and libraries for writing tests.
Next, we had to make a choice in favor of one of these products.
We formulated the main criteria by which the packages were compared:
To compare the features, a test application with a GUI based on the Windows Presentation Foundation (WPF) was developed. It contained all the key elements of the GUI, which in the future may participate in testing real products.
CodedUI
CodedUI provides an object of the UIMap class that implements access to all UI elements of the application. You can dynamically obtain any interface element, knowing its name, or by going through a chain of nested elements, for example:
“The first text field on the first tab”
However, such a record can only be formed directly in the test code.
Ranorex
In Ranorex, the GUI Object Repository is an analogue of UIMap. XPath language is used to access the elements, so you can select an element by generating a corresponding request using the built-in recorder.
“The first text box on the first tab”
CodedUI
Ranorex
According to this criterion, Ranorex looks preferable, since it has a simpler (easily mastered) system for working with controls.
CodedUI
CodedUI by default does not provide work with XPath for searching controls, but you can connect a special library .
Ranorex
In Ranorex XPath, expressions are called RanoreXPath. If an element does not have a unique name, then you can access it in one action, knowing the XPath expression for it (see the previous paragraph).
CodedUI
Ranorex
Extra point for Ranorex for XPath support by default.
Data Driven Testing is a test automation approach based on the use of test data sets located in a repository separate from the source code of the tests.
CodedUI
When developing CodedUI tests, the MSTest framework is used to organize the test structure and run the test, which contains functionality for creating Data Driven tests. Any test can be bound to data sets from a CSV, XML file or database selection.
Ranorex
Ranorex also has the ability to develop Data Driven tests, but excluding the XML format for data. Apparently, the developers of Ranorex found it difficult enough for the tester.
CodedUI
Ranorex
According to this criterion, the advantage is on the side of CodedUI, since the XML format in our products is very common, and in many cases it is more convenient for the tester to work with it.
Experience in test automation shows that sooner or later, some basic functionality is allocated in any test project, which makes it possible to reduce the test code. This functionality then grows into a small test framework. At this stage, tests can already be constructed from various methods of this framework.
In the future, we are going to use the tool we have chosen within the framework of such a framework. The framework encapsulates the work with CodedUI or Ranorex, necessary for accessing user interface controls, and provides a set of methods that can be associated with user actions, for example, “Open a card”, “Fill out a form”. Any UI test consists of a set of calls to such methods and checks for the correct processing of the data of the application under test.
The purpose of this criterion is to understand how convenient it is by choosing a tool, you can develop a test framework.
CodedUI
When used, CodedUI tests are developed in Visual Studio in C #. Tests can be expanded, modified, refactored to infinity in the traditional way, attract developers.
Ranorex
Ranorex provides its own IDE - Ranorex Studio, which is based on SharpDevelop and therefore has a very similar interface and principle of operation to Visual Studio. Ranorex has a convenient inspector of elements and the ability to create tests almost exclusively with mouse clicks. In addition, it is possible to develop additional software modules, which will then be used to create more complex test scenarios.
However, if the module is developed not by a tester, but by a programmer or automation specialist (as in our case), it is more convenient for him to use Visual Studio, since it has a more adapted debugging system, ReSharper, etc.
CodedUI
Ranorex Framework
development is, in general, not the task of the tester, therefore the tool is needed more suitable for development. In general, Ranorex Studio is inferior to CodedUI in this respect.
The tested applications are deployed on the Continuous Integration server (Build server), in which TeamCity is selected. According to team agreements, all tests are committed to the repository of the product under test. This allows us to use shared code, while updating and compiling application and test code on a Build server. For GUI tests, the same scheme should remain.
CodedUI
TeamCity allows you to easily connect tests developed using the MSTest, NUnit frameworks.
Ranorex
Ranorex, in turn, can compile Test Suits into an executable file, which can then be simply launched as a separate TeamCity build task.
CodedUI
Ranorex
By this criterion, the tools are conveniently used equally.
CodedUI
MSTest prints results in TRX format. This is an XML document, but it is quite difficult to read with the naked eye, and it is not necessary, because TeamCity after running the tests provides the same report in CSV format.
Ranorex
IDE Ranorex provides excellent reports with illustrations and explanations. However, for this, tests must be run from Ranorex Studio itself. Compiled tests after the run return a Ranorex XML report, which can then be converted to xUnit.
CodedUI
Ranorex
Both cases do not give the final desired view of the test report. However, we have already implemented a utility that can create test runs in our test management system (TMS) and import the results of passed automated tests. Moreover, the original report format can be either xUnit or CSV. In TMS, the report looks clear, human-readable and demonstrates the overall picture of testing at the moment.
In addition to importing test results into the test management system, we also import test cases themselves.
How this happens: The
tester, developing an automated test, writes a step-by-step description directly in the code of the test class, using special attributes. Also, with the help of attributes, indicates which suite and section this case belongs to. Then, using a special utility, imports all the cases of the test project into the corresponding TMS project. Import can be done manually or as part of the TeamCity build task. The utility file for the test project is fed to this utility.
We would like to use this opportunity when developing GUI tests. So, you need the ability to connect your libraries and edit tests.
Codedui
In Visual Studio, you can view and edit the code of the CodedUI test, even if it is recorded using the built-in recorder. This makes it possible to set the necessary attributes for synchronization with TMS. By default, the test project is assembled into a DLL library, which can then be used to import cases.
Ranorex
In Ranorex Studio, a test project is assembled into an exe file. That is, with the current implementation of our utility, it is not possible to use it as a source of test cases.
CodedUI
Ranorex
Thus, using CodedUI, it will be easier for us to bind our integration system to new tests.
CodedUI
1. After creating several examples on our test application, we immediately found problems when recording with the built-in recorder the actions associated with the consciousness of dynamic controls. When the test was restarted, the name of the control changed and the test failed. To avoid this and make the tests more universal, I had to edit the test code manually.
2. Testing using CodedUI is only possible in Visual Studio. But VS for a non-programmer is a very complex and cumbersome tool.
Ranorex
Ranorex is specifically designed as a tester environment. With its help it is convenient to create, run and view test results. In addition, after we developed several examples for a test application, there were never any problems with finding dynamic elements. Ranorex also successfully worked with .NET WinForms applications, for which CodedUI we were unable to launch.
CodedUI
Ranorex
Ranorex has a tester-friendly interface and system for working with elements.
As a conclusion, I will give a summary table according to the criteria described above: the
opportunity is realized the
opportunity is not realized the
opportunity is realized and is convenient
According to our comparison criteria, Ranorex was rated higher, mainly due to convenience.
In the process of analysis, we involved testers in the development of small test scripts and seemed to work with Ranorex much more convenient and intuitive.
If you are a tester and you need to test a Windows application from scratch and you choose between purchasing a license to use Visual Studio with CodedUI or Ranorex support, then choosing Ranorex would be more logical.
As for our projects.
Starting to test the project, we conditionally break the product into independent parts that can be tested separately. For example, the application has an interface that processes database data through the API, integration services synchronize data with other products, and stored procedures at the database level are processed according to a schedule and calculate the values of some fields. And for each such part, its own test suite is being developed as a separate application (test project = test framework adapted for the needs of product testing + many tests combined into test suites). All test projects are part of the solution for the entire product under test.
We needed to add GUI testing to this structure.
Since Ranorex and CodedUI did not differ much in terms of critical parameters for us, we thought about the prospect of developing tests.
In the end, the choice was between a tool in which testers would write GUI tests separately from the product and developers, and the time spent on developing everything necessary (framework, documentation) for quick and convenient test development using CodedUI.
Since in the second case such a system can be made more universal, we chose CodedUI. Yes, this is more than hours of development, but at the same time we had some universal solution for automating functional testing of the GUI of all Enterprise products of the company.
In addition to the well-known Desktop , Mobile and Online applications, 2GIS company develops many internal Enterprise products. These products are hidden from the eyes of users, however, it is with their help that colossal work is being done to provide the entire 2GIS infrastructure with cartographic and reference data. The processing of this data is labor-intensive and requires error-free calculations, therefore, before “hiring” all products are thoroughly tested.
In this article I will tell you how we chose tools, how we analyzed their technical capabilities, so that they fit into our infrastructure, and what I stopped at.
At one time, we faced the task of choosing a package suitable for testing the user interface of the products used. Everything that lends itself to automation is covered by automated tests, and the GUI is no exception. For this, as you know, there are tools to simulate user actions.
Given that the products under test are implemented in .NET, and their testers are familiar with the C # programming language, it was decided to develop tests also in C #. From a large set of tools on the market, we settled on two. The first is CodedUI, which comes with Microsoft Visual Studio, which we are actively using. The second is Ranorex, since it is compatible with Visual Studio, it has a convenient interface and positive feedback from testers.
Coded uiis one of the solutions that come with Microsoft Visual Studio Premium / Ultimate, provides access to libraries for test development.
Ranorex Automation Tools is a complete development environment, as well as a set of tools and libraries for writing tests.
Next, we had to make a choice in favor of one of these products.
Comparison criteria
We formulated the main criteria by which the packages were compared:
- Support for dynamically generated graphical controls (controls)
- Custom Search Control System
- Simple support for data-driven tests (Data Driven Testing)
- The ability to develop your own modules (frameworks) and use them when developing tests in C #
- Support for running tests on the Continuous Integration server (TeamCity)
- Generation of informative reports on the results of the test run
- Ability to integrate tests with test cases of the test management system (TMS)
- Easy to learn and use by testers
Test layout
To compare the features, a test application with a GUI based on the Windows Presentation Foundation (WPF) was developed. It contained all the key elements of the GUI, which in the future may participate in testing real products.
Product comparison by criteria
1. Support for dynamically generated controls
CodedUI
CodedUI provides an object of the UIMap class that implements access to all UI elements of the application. You can dynamically obtain any interface element, knowing its name, or by going through a chain of nested elements, for example:
“The first text field on the first tab”
this.UIViewmodeltitleWindow.UIItemTabList.Tabs[0].GetChildren()[1].GetChildren()[1]
However, such a record can only be formed directly in the test code.
Ranorex
In Ranorex, the GUI Object Repository is an analogue of UIMap. XPath language is used to access the elements, so you can select an element by generating a corresponding request using the built-in recorder.
“The first text box on the first tab”
/form[1]/tabpagelist[1]/tabpage[1]/element[@classname='CardEditView']/text[4]
CodedUI
Ranorex
According to this criterion, Ranorex looks preferable, since it has a simpler (easily mastered) system for working with controls.
2. Flexible and customizable control search system
CodedUI
CodedUI by default does not provide work with XPath for searching controls, but you can connect a special library .
Ranorex
In Ranorex XPath, expressions are called RanoreXPath. If an element does not have a unique name, then you can access it in one action, knowing the XPath expression for it (see the previous paragraph).
CodedUI
Ranorex
Extra point for Ranorex for XPath support by default.
3. Simple Data Driven Testing Support
Data Driven Testing is a test automation approach based on the use of test data sets located in a repository separate from the source code of the tests.
CodedUI
When developing CodedUI tests, the MSTest framework is used to organize the test structure and run the test, which contains functionality for creating Data Driven tests. Any test can be bound to data sets from a CSV, XML file or database selection.
Ranorex
Ranorex also has the ability to develop Data Driven tests, but excluding the XML format for data. Apparently, the developers of Ranorex found it difficult enough for the tester.
CodedUI
Ranorex
According to this criterion, the advantage is on the side of CodedUI, since the XML format in our products is very common, and in many cases it is more convenient for the tester to work with it.
4. Ability to develop test frameworks
Experience in test automation shows that sooner or later, some basic functionality is allocated in any test project, which makes it possible to reduce the test code. This functionality then grows into a small test framework. At this stage, tests can already be constructed from various methods of this framework.
In the future, we are going to use the tool we have chosen within the framework of such a framework. The framework encapsulates the work with CodedUI or Ranorex, necessary for accessing user interface controls, and provides a set of methods that can be associated with user actions, for example, “Open a card”, “Fill out a form”. Any UI test consists of a set of calls to such methods and checks for the correct processing of the data of the application under test.
The purpose of this criterion is to understand how convenient it is by choosing a tool, you can develop a test framework.
CodedUI
When used, CodedUI tests are developed in Visual Studio in C #. Tests can be expanded, modified, refactored to infinity in the traditional way, attract developers.
Ranorex
Ranorex provides its own IDE - Ranorex Studio, which is based on SharpDevelop and therefore has a very similar interface and principle of operation to Visual Studio. Ranorex has a convenient inspector of elements and the ability to create tests almost exclusively with mouse clicks. In addition, it is possible to develop additional software modules, which will then be used to create more complex test scenarios.
However, if the module is developed not by a tester, but by a programmer or automation specialist (as in our case), it is more convenient for him to use Visual Studio, since it has a more adapted debugging system, ReSharper, etc.
CodedUI
Ranorex Framework
development is, in general, not the task of the tester, therefore the tool is needed more suitable for development. In general, Ranorex Studio is inferior to CodedUI in this respect.
5. Support for running tests on the CI server
The tested applications are deployed on the Continuous Integration server (Build server), in which TeamCity is selected. According to team agreements, all tests are committed to the repository of the product under test. This allows us to use shared code, while updating and compiling application and test code on a Build server. For GUI tests, the same scheme should remain.
CodedUI
TeamCity allows you to easily connect tests developed using the MSTest, NUnit frameworks.
Ranorex
Ranorex, in turn, can compile Test Suits into an executable file, which can then be simply launched as a separate TeamCity build task.
CodedUI
Ranorex
By this criterion, the tools are conveniently used equally.
6. Generation of informative reports on the result of the run
CodedUI
MSTest prints results in TRX format. This is an XML document, but it is quite difficult to read with the naked eye, and it is not necessary, because TeamCity after running the tests provides the same report in CSV format.
Ranorex
IDE Ranorex provides excellent reports with illustrations and explanations. However, for this, tests must be run from Ranorex Studio itself. Compiled tests after the run return a Ranorex XML report, which can then be converted to xUnit.
CodedUI
Ranorex
Both cases do not give the final desired view of the test report. However, we have already implemented a utility that can create test runs in our test management system (TMS) and import the results of passed automated tests. Moreover, the original report format can be either xUnit or CSV. In TMS, the report looks clear, human-readable and demonstrates the overall picture of testing at the moment.
7. The ability to integrate tests with TMS test cases
In addition to importing test results into the test management system, we also import test cases themselves.
How this happens: The
tester, developing an automated test, writes a step-by-step description directly in the code of the test class, using special attributes. Also, with the help of attributes, indicates which suite and section this case belongs to. Then, using a special utility, imports all the cases of the test project into the corresponding TMS project. Import can be done manually or as part of the TeamCity build task. The utility file for the test project is fed to this utility.
We would like to use this opportunity when developing GUI tests. So, you need the ability to connect your libraries and edit tests.
Codedui
In Visual Studio, you can view and edit the code of the CodedUI test, even if it is recorded using the built-in recorder. This makes it possible to set the necessary attributes for synchronization with TMS. By default, the test project is assembled into a DLL library, which can then be used to import cases.
Ranorex
In Ranorex Studio, a test project is assembled into an exe file. That is, with the current implementation of our utility, it is not possible to use it as a source of test cases.
CodedUI
Ranorex
Thus, using CodedUI, it will be easier for us to bind our integration system to new tests.
8. Easy to learn and use by testers
CodedUI
1. After creating several examples on our test application, we immediately found problems when recording with the built-in recorder the actions associated with the consciousness of dynamic controls. When the test was restarted, the name of the control changed and the test failed. To avoid this and make the tests more universal, I had to edit the test code manually.
2. Testing using CodedUI is only possible in Visual Studio. But VS for a non-programmer is a very complex and cumbersome tool.
Ranorex
Ranorex is specifically designed as a tester environment. With its help it is convenient to create, run and view test results. In addition, after we developed several examples for a test application, there were never any problems with finding dynamic elements. Ranorex also successfully worked with .NET WinForms applications, for which CodedUI we were unable to launch.
CodedUI
Ranorex
Ranorex has a tester-friendly interface and system for working with elements.
Output
As a conclusion, I will give a summary table according to the criteria described above: the
opportunity is realized the opportunity is not realized the opportunity is realized and is convenient| Criterion | Codedui | Ranorex |
|---|---|---|
| Support for dynamically generated controls | | |
| Custom Search Control System | | |
| Simple Data Driven Testing Support | | |
| Ability to develop your own modules | | |
| Support for running tests in TeamCity | | |
| Generation of informative reports on the results of the test run | | |
| Ability to integrate tests with TMS test cases | | |
| Easy to learn and use by testers | | |
According to our comparison criteria, Ranorex was rated higher, mainly due to convenience.
In the process of analysis, we involved testers in the development of small test scripts and seemed to work with Ranorex much more convenient and intuitive.
If you are a tester and you need to test a Windows application from scratch and you choose between purchasing a license to use Visual Studio with CodedUI or Ranorex support, then choosing Ranorex would be more logical.
As for our projects.
Starting to test the project, we conditionally break the product into independent parts that can be tested separately. For example, the application has an interface that processes database data through the API, integration services synchronize data with other products, and stored procedures at the database level are processed according to a schedule and calculate the values of some fields. And for each such part, its own test suite is being developed as a separate application (test project = test framework adapted for the needs of product testing + many tests combined into test suites). All test projects are part of the solution for the entire product under test.
We needed to add GUI testing to this structure.
Since Ranorex and CodedUI did not differ much in terms of critical parameters for us, we thought about the prospect of developing tests.
In the end, the choice was between a tool in which testers would write GUI tests separately from the product and developers, and the time spent on developing everything necessary (framework, documentation) for quick and convenient test development using CodedUI.
Since in the second case such a system can be made more universal, we chose CodedUI. Yes, this is more than hours of development, but at the same time we had some universal solution for automating functional testing of the GUI of all Enterprise products of the company.