It’s difficult to read a long article ... Testing a complex product is easy
- Tutorial
In the modern IT industry, many are still surprised: “Why do we need a separate test department?”
What is Petersburg autumn? Rains, Rosenbaum, melancholy ... Is that all ?! No, this is still a new academic year, which in many regions of our country began precisely with Diary.ru. Over the summer, we managed to achieve certain successes: the number of users of the project exceeded the mark of 7,000,000, we became the only Russian company included in the list of the World Economic Forum Technology Pioneers 2014 , and the Republic of North Ossetia-Alania completely switched to electronic journals from the Diary. ru . On this lyrical introduction we will end and move on to the essence of this post.
In the modern IT industry, many are still surprised: “Why do we need a separate test department?” As one of the participants in the test team of our project, I declare responsibly: this is an erroneous opinion. Organizing a QA department is not easy, but it's worth it.
In this case, one must clearly understand for oneself:
- what is considered a quality criterion?
- which metrics to use?
- with what tools to test?
- how to build interaction with developers / analysts, etc.
By and large, it’s not worth reinventing the wheel - these practices are described on millions of web pages and are successfully used in many corporations whose experience can be adopted. However, one must not forget the simple truth - everywhere there is a specificity. For example, in Diary.ru these are strongly-interconnected components and CI (Continuous Integration), and simply a high level of integration of related areas. What is the best test in this case?
Let's start with a simple one: any information can never 100% automatically “pop up” in your (undeniably, very smart) head. First of all, QA-shnikov need to memorize and run test scenarios, which are checked from release to release - “but has something broken?”. As we all know - regression testing. But this is not about that, about species - welcome to protesting.ru (a good resource, by the way).
So, for writing and storing test cases (“A test case is an artifact that describes a set of steps, specific conditions and parameters necessary to verify the implementation of the tested function or its part”), you can use a lot of tools: Excel, notepad, google-docs , sheets of paper, after all. But is it effective and convenient? For one person - maybe. For a team where the project status “you should have known yesterday” is definitely not.
For a long time already there are many heavy Test Management programs that allow you to fully manage the test process: work with test plans, test cycles, requisites, test cases, etc. I worked with different utilities - paid, free, self-written-written ... For many years I was sure that it’s better not to skimp on this matter: either purchase a high-quality tool, or hire a team and write “an ideal tool for the company, with those requirements that You need".
So, firstly: it became clear that the existing checklists could not cope with the success of testing. They are not enough, they have little information, they are not transparent, it is difficult to present results on them, they are difficult to maintain, there is not enough time to write new ones. Secondly: in this situation, we needed a tool that was easy enough to understand, which would allow us to effectively manage the test management scenario: "requirements -> test cases -> review -> test suite -> test cycle -> results."
- convenient user interface
- ease of installation, implementation and support
- convenient writing and storage of test scripts
- communication with JIRA
- effective management of the following scenario: "requirements -> test cases -> review -> test suite -> test cycle -> results"
- convenient selection of test cases from different areas to run in one test run
- ease of receiving transparent reports on the launch of test suites
- the ability to connect the tool in the future with auto-testing and running written auto-tests
- Microsoft Test Lab ( how to in Russian )
- Zephyr
- Test Link
- Testia Tarantula
- Test Rail
PS: warning questions - we did not consider the HP Quality Center , because 1) it’s unreasonably expensive 2) it’s not very “friendly”, especially to those who have never worked with test tools, and to write your own - resource costs and lack of time ...
I will try to meet a couple of main points:
“Very-very-cool-thing-in-everything”, isn't it? However, if you have never encountered its installation and implementation, optimistic forecasts are not necessary. To put it mildly, if you have at your disposal a couple of free months - welcome. Install, configure, there is also a nice bonus - complete how to in Russian. For us, the main advantages were the license and easy setup of communication with auto-tests.
Total: there was no time to implement and learn to use it.
A good, modern, colorful tool integrated with JIRA. And yet - have you seen how to start test cycles and write test cases? For editing - a separate window, adding steps and results - only line by line and numbered. A lot of unnecessary actions to "create-save" the script.
Total: If we have one project for half a year, carefully and slowly drive test cases, as a result, everything will be quite beautiful and transparent. The option also did not fit for us.
A very simple tool that allows you to do the most basic thing in the work of the tester - test cases + test cycles. And that’s it. Design - in the C grade, convenience - too. It seems to me that Test Link can be installed at home in order to just learn how to enter test cases and pass test runs with setting Passed / Blocked / Failed. The option, of course, is not bad, all the more free. Cons: non-obvious with auto tests, imperfect writing test cases.
Total: Okay, let’s leave it to review.
Free Finnish toolkit that is easy to implement in the QA team. Pros - it’s not bad enough to control the test process: test cases, test wounds, reports. Correct structure: test case -> test suite. There are quite amusing moments, such as “tags” in test cases, by which it will be possible to find the set you need (for example, we need negative test cases only for a certain functionality that affects this-and-this -> we initially set in all scenarios the correct set of tags, this makes it easy to select a separate test run). Convenience is not bad, you can configure it for auto-tests ... Cons: reporting is lame, customization is problematic, it is not connected with JIRA.
Total: Let’s leave as one of the options for consideration, why not?
In my opinion - one of the easiest, but at the same time very large-scale existing tools. The main plus - customization is possible in almost everything. Starting from design, ending with logic. Direct integration with JIRA. Effectively enter test cases and break them down into the right suites. As one of the minuses - until recently, Test Rail had a poorly implemented reporting system: poor results on test runs, lack of customization of reports, etc. However, Gurock & Co took up this matter and by the 3rd version of the tool the guys did the work on the bugs. Reporting has begun to play in new colors: if you want - a report on a milestone, if you want - only on defects, if you want - we will set up a regular e-mail with a report on testing strictly defined functionality, etc. Well done! Bottom line - this tool met all (!) My requirements,
• This is an effective management of test cases, plans and test cycles (creating / storing / editing test cases, managing test plans, launching test cycles, recording test results)
• This is a significant improvement in the quality of testing (clear description of test cases, their review, correlation with requirements, dividing into areas - all this allows us to evaluate how complete the coverage of functional tests is, and is a necessary material for the entire project team)
• This is the receipt of test results in real-time (reporting on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components / milestones, etc.)
• This is an easy organization and convenient tracking of the testing department’s workload (opportunities for complete customization of the “working dashboard”, as well as convenient obtaining the status of the QA department)
1. We installed TestRail on a Windows Server 2008 R2
virtual machine with MSSQL base 2. Configured AD authentication
3. Configured a full-fledged connection with JIRA
First, each logged-in user TestRail needs to configure direct access to JIRA. To do this, we need a couple of custom variables jira_user + jira_password (welcome to: docs.gurock.com/testrail-integration/defects-plugins-variables ). After creating them in the JIRA_Rest plugin (Administration -> Integration -> JIRA_Rest), set the following:
[connection]
address = http: // jira / ***
user =% jira_user%
password =% jira_password%
Ur. Now each user has the opportunity to set the login / password from JIRA in My Settings.
Next, in JIRA_Rest, we register both the connection directly with the established JIRA, and subsequently the links of all the existing JIRA fields: whether it is the main field or custom (welcome to: docs.gurock.com/testrail-integration/tools-jira-rest ).
Direct connection with JIRA (example):
Reference View Url: your-server / jira / browse /% id%
Reference Add Url: your-server / jira / secure / CreateIssue ! Default.jspa
Links to the main fields (example):
[push .fields]
summary = on
project = on
issuetype = on
component = on
assignee = on
priority = on
affects_version = on
fix_version = off
estimate = off
labels = off
environment = off
description = on
Relations with custom fields (example):
[push.fields]
customfield_XXXXX = on
[push.field.customfield_XXXXX]
label = Customer
size = compact
type = dropdown
required = true
As a result, you can safely use the bug Test Rail tools, and at the same time do not waste time moving to the next window with JIRA. Is it a plus? You know, when running large test cycles and setting test results, this is a definite plus, since those very tens of seconds finally add up to precious minutes and simply the convenience of work, which is important.
And one more thing: When generating a test report, we usually display after the results a list of known defects that block the passage of the test case. So, imagine such a list, and when you hover over a defect in this report, a popup pops up with JIRA information about the ticket - Summary, Description, etc. Just brilliant! Everything is done for you.
4. After the main tech. of settings, you can safely create projects tied to milestones and test suites containing various sections with test cases, as well as configure dashboard solely according to your desires and criteria
We all know that when creating test cases for any functionality, for starters, it is advisable to draw up a test plan where equivalence classes, sections, and dividing at least into positive-negative scenarios will be clearly described. TestRail allows you to most effectively draw up such a plan and subsequently expand it to full-fledged scenarios.
A little more in detail: we create a test suite. Then inside it we have the opportunity to add both sections and test cases inside these sections. We have the opportunity, without going to the internal essence of the case, to compose templates (= checklist / test plan) by entering only Summary and pressing Enter after entering. It looks like this:
Type the name (test / group / equivalence class), press Enter -> and you are already on the next case with the cursor in the input field. Thus, as quickly as possible, you can draw up the necessary plan, which was often written before in Word, Excel or even not created at all. These developments should go through the teams of analysts / developers -> and according to the results of the review it will be absolutely clear whether something needs to be added or not. Do not forget that this is just a plan, and in order not to get confused with test cases that are already completely filled, we introduced additional ones. Only Title field, which defaults to Yes.
This field can be corrected to No, only by writing the test case itself in the editing form (you just need to tick “Only Title” = No).
Any custom fields that you enter can also be put on the dashboard like all the others by selecting them in the displayed columns (clicking on the “tab” on the main page for writing test cases in the line listing the field names: ID, Title an so on).
5. All internal fields of the test case are also customizable.
In the initial configuration of Test Rail, a classic and unconditionally necessary set of internal fields of the test scenario is presented: ID, Priority, Initial Conditions, View, Steps, Expected Result, Link to requirements or implementation task in JIRA, etc. .d. You have the right to add any field and configure it as you want: whether it is string, checkbox, dropdown - it does not matter. Everything is in your hands and heads! What we did:
- Added a Data for test-case field, in which for a test from one equivalence class we simply list any data with which the test should be performed. It can be anything: from the role of the executor to the listing of input lines.
- Added ID Auto and Auto fields . ID Auto - a unique ID for auto test. Auto - the Yes / No checkbox, which shows whether the test case is automated or not.
This is done so that in the future, when setting up Test Rail for automation, you can immediately use these parameters to run the necessary auto-tests and generate the necessary report on them (http://www.gurock.com/testrail/tour/5/ - Automated tests and API).
6. For any test run, you can choose absolutely any set of test cases
Test Rail allows you to create test cycles by any criteria. This is a very important feature of the tool, since it is often necessary to run strictly defined scenarios (selected by priority, by milestones, by name, by part of the name, but in any way actually - it all depends on the goals). So, when assigning Test Run, we have the opportunity to select "your" suite by filter. The general view is as follows:
But, let’s say, we select by name that “is equal to or contains something”:
With all this, the test suite can be selected for full run.
7. Customizable reporting system
Any test run can be presented in the form of a clear report: “what-where-when-why”. It is possible to select any criteria, just like when creating a test cycle. Among other things, it makes sense to configure the distribution of all kinds of reports for the necessary time and interested recipients.
Checklists, ad-hoc testing are good, however, without clear procedures, often the product has weak points that increase over time and pull the company down. Having both a set of test cases and a coverage matrix (at least component-wise) - this can be avoided.
I have listed far from all the features of TestRail and gave just a few examples of its use. Nevertheless, it is obvious that such flexible customization will allow many to customize this tool for their development process. Among other things, the tool allows you to completely transparently monitor the QA department, which, firstly, saves everyone from the understatement of “what are they testing,” and, secondly, it establishes a clear interaction between the teams and gives an understanding of “why and how to test” .
Choose your tool, test, control the process! Good luck in all your endeavors!
Article author: Vasily Petukhov, head of the analysis and quality management department Diary.ru
Lyrical introduction
What is Petersburg autumn? Rains, Rosenbaum, melancholy ... Is that all ?! No, this is still a new academic year, which in many regions of our country began precisely with Diary.ru. Over the summer, we managed to achieve certain successes: the number of users of the project exceeded the mark of 7,000,000, we became the only Russian company included in the list of the World Economic Forum Technology Pioneers 2014 , and the Republic of North Ossetia-Alania completely switched to electronic journals from the Diary. ru . On this lyrical introduction we will end and move on to the essence of this post.
Introduction to the topic
In the modern IT industry, many are still surprised: “Why do we need a separate test department?” As one of the participants in the test team of our project, I declare responsibly: this is an erroneous opinion. Organizing a QA department is not easy, but it's worth it.
In this case, one must clearly understand for oneself:
- what is considered a quality criterion?
- which metrics to use?
- with what tools to test?
- how to build interaction with developers / analysts, etc.
By and large, it’s not worth reinventing the wheel - these practices are described on millions of web pages and are successfully used in many corporations whose experience can be adopted. However, one must not forget the simple truth - everywhere there is a specificity. For example, in Diary.ru these are strongly-interconnected components and CI (Continuous Integration), and simply a high level of integration of related areas. What is the best test in this case?
Let's start with a simple one: any information can never 100% automatically “pop up” in your (undeniably, very smart) head. First of all, QA-shnikov need to memorize and run test scenarios, which are checked from release to release - “but has something broken?”. As we all know - regression testing. But this is not about that, about species - welcome to protesting.ru (a good resource, by the way).
So, for writing and storing test cases (“A test case is an artifact that describes a set of steps, specific conditions and parameters necessary to verify the implementation of the tested function or its part”), you can use a lot of tools: Excel, notepad, google-docs , sheets of paper, after all. But is it effective and convenient? For one person - maybe. For a team where the project status “you should have known yesterday” is definitely not.
Preparatory work
For a long time already there are many heavy Test Management programs that allow you to fully manage the test process: work with test plans, test cycles, requisites, test cases, etc. I worked with different utilities - paid, free, self-written-written ... For many years I was sure that it’s better not to skimp on this matter: either purchase a high-quality tool, or hire a team and write “an ideal tool for the company, with those requirements that You need".
So, firstly: it became clear that the existing checklists could not cope with the success of testing. They are not enough, they have little information, they are not transparent, it is difficult to present results on them, they are difficult to maintain, there is not enough time to write new ones. Secondly: in this situation, we needed a tool that was easy enough to understand, which would allow us to effectively manage the test management scenario: "requirements -> test cases -> review -> test suite -> test cycle -> results."
Summarizing all of the above, the requirements for the tool were as follows:
- convenient user interface
- ease of installation, implementation and support
- convenient writing and storage of test scripts
- communication with JIRA
- effective management of the following scenario: "requirements -> test cases -> review -> test suite -> test cycle -> results"
- convenient selection of test cases from different areas to run in one test run
- ease of receiving transparent reports on the launch of test suites
- the ability to connect the tool in the future with auto-testing and running written auto-tests
So, what was considered:
- Microsoft Test Lab ( how to in Russian )
- Zephyr
- Test Link
- Testia Tarantula
- Test Rail
PS: warning questions - we did not consider the HP Quality Center , because 1) it’s unreasonably expensive 2) it’s not very “friendly”, especially to those who have never worked with test tools, and to write your own - resource costs and lack of time ...
I will try to meet a couple of main points:
Microsoft Test Lab
“Very-very-cool-thing-in-everything”, isn't it? However, if you have never encountered its installation and implementation, optimistic forecasts are not necessary. To put it mildly, if you have at your disposal a couple of free months - welcome. Install, configure, there is also a nice bonus - complete how to in Russian. For us, the main advantages were the license and easy setup of communication with auto-tests.
Total: there was no time to implement and learn to use it.
Zephyr
A good, modern, colorful tool integrated with JIRA. And yet - have you seen how to start test cycles and write test cases? For editing - a separate window, adding steps and results - only line by line and numbered. A lot of unnecessary actions to "create-save" the script.
Total: If we have one project for half a year, carefully and slowly drive test cases, as a result, everything will be quite beautiful and transparent. The option also did not fit for us.
Test link
A very simple tool that allows you to do the most basic thing in the work of the tester - test cases + test cycles. And that’s it. Design - in the C grade, convenience - too. It seems to me that Test Link can be installed at home in order to just learn how to enter test cases and pass test runs with setting Passed / Blocked / Failed. The option, of course, is not bad, all the more free. Cons: non-obvious with auto tests, imperfect writing test cases.
Total: Okay, let’s leave it to review.
Testia tarantula
Free Finnish toolkit that is easy to implement in the QA team. Pros - it’s not bad enough to control the test process: test cases, test wounds, reports. Correct structure: test case -> test suite. There are quite amusing moments, such as “tags” in test cases, by which it will be possible to find the set you need (for example, we need negative test cases only for a certain functionality that affects this-and-this -> we initially set in all scenarios the correct set of tags, this makes it easy to select a separate test run). Convenience is not bad, you can configure it for auto-tests ... Cons: reporting is lame, customization is problematic, it is not connected with JIRA.
Total: Let’s leave as one of the options for consideration, why not?
Test rail
In my opinion - one of the easiest, but at the same time very large-scale existing tools. The main plus - customization is possible in almost everything. Starting from design, ending with logic. Direct integration with JIRA. Effectively enter test cases and break them down into the right suites. As one of the minuses - until recently, Test Rail had a poorly implemented reporting system: poor results on test runs, lack of customization of reports, etc. However, Gurock & Co took up this matter and by the 3rd version of the tool the guys did the work on the bugs. Reporting has begun to play in new colors: if you want - a report on a milestone, if you want - only on defects, if you want - we will set up a regular e-mail with a report on testing strictly defined functionality, etc. Well done! Bottom line - this tool met all (!) My requirements,
Total: our choice is the introduction of Test Rail. In fact, what is Test Rail?
• This is an effective management of test cases, plans and test cycles (creating / storing / editing test cases, managing test plans, launching test cycles, recording test results)
• This is a significant improvement in the quality of testing (clear description of test cases, their review, correlation with requirements, dividing into areas - all this allows us to evaluate how complete the coverage of functional tests is, and is a necessary material for the entire project team)
• This is the receipt of test results in real-time (reporting on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components / milestones, etc.)
• This is an easy organization and convenient tracking of the testing department’s workload (opportunities for complete customization of the “working dashboard”, as well as convenient obtaining the status of the QA department)
Implementation
1. We installed TestRail on a Windows Server 2008 R2
virtual machine with MSSQL base 2. Configured AD authentication
3. Configured a full-fledged connection with JIRA
First, each logged-in user TestRail needs to configure direct access to JIRA. To do this, we need a couple of custom variables jira_user + jira_password (welcome to: docs.gurock.com/testrail-integration/defects-plugins-variables ). After creating them in the JIRA_Rest plugin (Administration -> Integration -> JIRA_Rest), set the following:
[connection]
address = http: // jira / ***
user =% jira_user%
password =% jira_password%
Ur. Now each user has the opportunity to set the login / password from JIRA in My Settings.
Next, in JIRA_Rest, we register both the connection directly with the established JIRA, and subsequently the links of all the existing JIRA fields: whether it is the main field or custom (welcome to: docs.gurock.com/testrail-integration/tools-jira-rest ).
Direct connection with JIRA (example):
Reference View Url: your-server / jira / browse /% id%
Reference Add Url: your-server / jira / secure / CreateIssue ! Default.jspa
Links to the main fields (example):
[push .fields]
summary = on
project = on
issuetype = on
component = on
assignee = on
priority = on
affects_version = on
fix_version = off
estimate = off
labels = off
environment = off
description = on
Relations with custom fields (example):
[push.fields]
customfield_XXXXX = on
[push.field.customfield_XXXXX]
label = Customer
size = compact
type = dropdown
required = true
As a result, you can safely use the bug Test Rail tools, and at the same time do not waste time moving to the next window with JIRA. Is it a plus? You know, when running large test cycles and setting test results, this is a definite plus, since those very tens of seconds finally add up to precious minutes and simply the convenience of work, which is important.
And one more thing: When generating a test report, we usually display after the results a list of known defects that block the passage of the test case. So, imagine such a list, and when you hover over a defect in this report, a popup pops up with JIRA information about the ticket - Summary, Description, etc. Just brilliant! Everything is done for you.
4. After the main tech. of settings, you can safely create projects tied to milestones and test suites containing various sections with test cases, as well as configure dashboard solely according to your desires and criteria
We all know that when creating test cases for any functionality, for starters, it is advisable to draw up a test plan where equivalence classes, sections, and dividing at least into positive-negative scenarios will be clearly described. TestRail allows you to most effectively draw up such a plan and subsequently expand it to full-fledged scenarios.
A little more in detail: we create a test suite. Then inside it we have the opportunity to add both sections and test cases inside these sections. We have the opportunity, without going to the internal essence of the case, to compose templates (= checklist / test plan) by entering only Summary and pressing Enter after entering. It looks like this:
Type the name (test / group / equivalence class), press Enter -> and you are already on the next case with the cursor in the input field. Thus, as quickly as possible, you can draw up the necessary plan, which was often written before in Word, Excel or even not created at all. These developments should go through the teams of analysts / developers -> and according to the results of the review it will be absolutely clear whether something needs to be added or not. Do not forget that this is just a plan, and in order not to get confused with test cases that are already completely filled, we introduced additional ones. Only Title field, which defaults to Yes.
This field can be corrected to No, only by writing the test case itself in the editing form (you just need to tick “Only Title” = No).
Any custom fields that you enter can also be put on the dashboard like all the others by selecting them in the displayed columns (clicking on the “tab” on the main page for writing test cases in the line listing the field names: ID, Title an so on).
5. All internal fields of the test case are also customizable.
In the initial configuration of Test Rail, a classic and unconditionally necessary set of internal fields of the test scenario is presented: ID, Priority, Initial Conditions, View, Steps, Expected Result, Link to requirements or implementation task in JIRA, etc. .d. You have the right to add any field and configure it as you want: whether it is string, checkbox, dropdown - it does not matter. Everything is in your hands and heads! What we did:
- Added a Data for test-case field, in which for a test from one equivalence class we simply list any data with which the test should be performed. It can be anything: from the role of the executor to the listing of input lines.
- Added ID Auto and Auto fields . ID Auto - a unique ID for auto test. Auto - the Yes / No checkbox, which shows whether the test case is automated or not.
This is done so that in the future, when setting up Test Rail for automation, you can immediately use these parameters to run the necessary auto-tests and generate the necessary report on them (http://www.gurock.com/testrail/tour/5/ - Automated tests and API).
6. For any test run, you can choose absolutely any set of test cases
Test Rail allows you to create test cycles by any criteria. This is a very important feature of the tool, since it is often necessary to run strictly defined scenarios (selected by priority, by milestones, by name, by part of the name, but in any way actually - it all depends on the goals). So, when assigning Test Run, we have the opportunity to select "your" suite by filter. The general view is as follows:
But, let’s say, we select by name that “is equal to or contains something”:
With all this, the test suite can be selected for full run.
7. Customizable reporting system
Any test run can be presented in the form of a clear report: “what-where-when-why”. It is possible to select any criteria, just like when creating a test cycle. Among other things, it makes sense to configure the distribution of all kinds of reports for the necessary time and interested recipients.
Conclusion
Checklists, ad-hoc testing are good, however, without clear procedures, often the product has weak points that increase over time and pull the company down. Having both a set of test cases and a coverage matrix (at least component-wise) - this can be avoided.
I have listed far from all the features of TestRail and gave just a few examples of its use. Nevertheless, it is obvious that such flexible customization will allow many to customize this tool for their development process. Among other things, the tool allows you to completely transparently monitor the QA department, which, firstly, saves everyone from the understatement of “what are they testing,” and, secondly, it establishes a clear interaction between the teams and gives an understanding of “why and how to test” .
Choose your tool, test, control the process! Good luck in all your endeavors!
Article author: Vasily Petukhov, head of the analysis and quality management department Diary.ru