July 16, 2010 at 13:48Implement automated testing
When using automatic tests to control the quality of numerous assemblies of our software solutions, we, without a doubt, received the initial gain in comparison with manual testing. At the same time, however, the convention, or incompleteness of automation, associated with a large amount of manual operations to start, configure and process the results, significantly reduced the efficiency of the process, forming a kind of semi-automatic hybrid in testing.
What kind of problems took our time and energy, as well as in any other company operating at the same level of “semi-automatic”?
For automatic testing, at least three types of scripts were used. Of course, this was due to the specifics of the tests, but this made it no easier for us. Each script required its own special launch procedure, performed manually between starts of the next automatic test.
Setting the order of test execution, also performed manually, took considerable time and required considerable effort.
Each automatic test required its original setting of the environment - moving the necessary files from different places, requests, etc. All these operations, carried out again by hand, gave a very clear idea of the first part in the definition of "semi-automatic".
Each type of script gave a report in its form; two of them were absolutely unreadable for the manager, allowing you to get the information you need only at the cost of tremendous effort and stress, not to mention the effort of switching from one form to another.
Faced with the above problems, I began to look for ways to solve them. We agreed with the boss that you can write a complex for managing automated tests (the developers had free time at that time). And here is what we got after 3 months of development (1 developer + 1 tester, total 6 man-months).
The automatic test management system, which covers the entire software development cycle, from assembling the next version of the product to the final generation of the test report, is based on Visual Studio 2008 & Hyper-V. The created system provides an opportunity to switch from the “Developer - Tester - Developer” scheme to the actual operating scheme “Developer - Automated System - Developer” with a sharp increase in the quality and level of testing. Due to its versatility, functional power and flexibility, it allows you to perform any prepared tests on any software products.
The system, as a single tool for performing a huge number of a wide variety of tests (Visual Studio MS tests, HP QTP, etc.), while covering the entire software development cycle, allows you to automatically fix detected defects on TFS, controlling each stage of the cycle; expand and parameterize the automatic testing system itself, based on the amount of available resources, the quantity, type and purpose of tests and products being tested.
A friendly user interface allows you to easily perform many operations to manage and control the actual status of expected / executed / completed tests, to schedule tests on different versions of the product with setting priorities, parameterizing the system depending on the environment and operating conditions, and much more.
A logging system that flexibly uses various technologies for different stages of testing allows you to obtain accurate and detailed information about the execution of the test sequence and problems encountered.
Automatic registration of defects on TFS without the participation of a tester, which includes both opening and closing records with the ability to parameterize this function, frees up a significant reserve of company time resources.
The expandable data architecture system allows using modern technologies for storing test information on various resources - a file server, Source Control Visual Studio 2008, iSCSI.
These are just the main features of the automatic test management system; Being in the process of constant development and improvement, this unique software product is steadily gaining new functional capabilities and special reliability, making it an indispensable tool for any company producing or testing software.
At the moment we are working on creating videos that will show how the system works in real conditions. As soon as we do, we’ll post it on Yuotub.
If someone is interested in our system - write, I will always be happy to answer your questions.
UPD : at the request of workers combined in one topic. Soon I will add dry technical data. Let it be for now:
Octopus works with TFS 2008 as a build machine + BTS, HyperV as a virtualization system.
What kind of problems took our time and energy, as well as in any other company operating at the same level of “semi-automatic”?
Three types of scripts
For automatic testing, at least three types of scripts were used. Of course, this was due to the specifics of the tests, but this made it no easier for us. Each script required its own special launch procedure, performed manually between starts of the next automatic test.
Priority setting
Setting the order of test execution, also performed manually, took considerable time and required considerable effort.
Environment setting
Each automatic test required its original setting of the environment - moving the necessary files from different places, requests, etc. All these operations, carried out again by hand, gave a very clear idea of the first part in the definition of "semi-automatic".
Receiving reports
Each type of script gave a report in its form; two of them were absolutely unreadable for the manager, allowing you to get the information you need only at the cost of tremendous effort and stress, not to mention the effort of switching from one form to another.
Faced with the above problems, I began to look for ways to solve them. We agreed with the boss that you can write a complex for managing automated tests (the developers had free time at that time). And here is what we got after 3 months of development (1 developer + 1 tester, total 6 man-months).
Standalone Testing Process
The automatic test management system, which covers the entire software development cycle, from assembling the next version of the product to the final generation of the test report, is based on Visual Studio 2008 & Hyper-V. The created system provides an opportunity to switch from the “Developer - Tester - Developer” scheme to the actual operating scheme “Developer - Automated System - Developer” with a sharp increase in the quality and level of testing. Due to its versatility, functional power and flexibility, it allows you to perform any prepared tests on any software products.
Any tests on any products
The system, as a single tool for performing a huge number of a wide variety of tests (Visual Studio MS tests, HP QTP, etc.), while covering the entire software development cycle, allows you to automatically fix detected defects on TFS, controlling each stage of the cycle; expand and parameterize the automatic testing system itself, based on the amount of available resources, the quantity, type and purpose of tests and products being tested.
A friendly user interface allows you to easily perform many operations to manage and control the actual status of expected / executed / completed tests, to schedule tests on different versions of the product with setting priorities, parameterizing the system depending on the environment and operating conditions, and much more.
A logging system that flexibly uses various technologies for different stages of testing allows you to obtain accurate and detailed information about the execution of the test sequence and problems encountered.
Automatic registration of defects on TFS without the participation of a tester, which includes both opening and closing records with the ability to parameterize this function, frees up a significant reserve of company time resources.
The expandable data architecture system allows using modern technologies for storing test information on various resources - a file server, Source Control Visual Studio 2008, iSCSI.
These are just the main features of the automatic test management system; Being in the process of constant development and improvement, this unique software product is steadily gaining new functional capabilities and special reliability, making it an indispensable tool for any company producing or testing software.
At the moment we are working on creating videos that will show how the system works in real conditions. As soon as we do, we’ll post it on Yuotub.
If someone is interested in our system - write, I will always be happy to answer your questions.
UPD : at the request of workers combined in one topic. Soon I will add dry technical data. Let it be for now:
Octopus works with TFS 2008 as a build machine + BTS, HyperV as a virtualization system.