Kitchen accessories in testing: or what to choose from online resources for testing products?
The organization of the process is very important in software testing: who is doing what? Who, what, when tested? On which assembly did this or that error first appear? Who tested this? What is our requirements status?
Trying to answer all these questions, we organized the storage of requirements and checklists in the form of google tables, but eventually realized: it is time to look for a specialized tool. The results of the search and comparison of options I want to share with you in this article.
In connection with the remote work team and in general, due to habit, we considered only web options. Having rejected completely inappropriate solutions for the initial analysis, there were three candidates to choose from:
TestLink
TestRail
Sitechco
We get as a result: Triple Threat Match. Who will see whom in fair fight.
To begin with, we in the team determined the selection criteria:
According to these criteria, we will analyze each of the candidates.
Let's go in descending order:
With regard to compatibility, all browsers are declared to support TestRails and TestLink, but IE is not supported by Sitechco:
Given our set of environments, this is not critical, but for many users (especially those testing some banking software in truncated environments) it may be.
The most important criterion in choosing a system for us, of course, is its functionality. We jointly worked out a list of necessary “features”:
Of the desirable, but not required, the list includes:
In TestRail, the creation of tests is implemented using the Test Suite, in which the Tests are placed.
Adding is done quickly, according to the principle of creating a Test Case and a separate description, with the possibility of dividing into sections and attaching, if necessary, files. Inside the test, you can write down the steps in detail, but in most cases we limited ourselves to clear headers.
Sitechco implemented test suites in the form of Check and Cheat Lists. The checklists themselves are simple, nothing superfluous or abstruse: test, check, priority, well, various additions: attachment of requirements, labor costs, hierarchy in the form of groups and master groups.
An important, as for me, role is played by cheat lists, with an existing set of tests. It will save time for creating your own and will be useful for the future. These elements play the role of a designer: write standard cheat lists and insert them into check lists as necessary. It is convenient when there are many standard checks, for example, we need to check different fields for usability guidelines and for XSS- / SQL injections. Data is repeated, copy-paste is evil, but you need the status of the passage in different contexts and for different fields.
Creating tests in TestLink is somewhat similar to creating forms in TestRail. By analogy, test suites are created in Test Suite, in which tests are embedded.
It looks rather confusing and at the same time primitive:
The same tests can be added to different scenarios, which in our case performs a similar function, like cheat lists in Sievechko. But in order for the tests to be atomic and small (with a visual result for the report), too many gestures appeared in TestLink. Well, a LOT .
The developers of TestRail offered their customers several reporting options:
What allows you to collect information about testing on bugs, results, launches and users. But! Each report needs to be processed, which is very inconvenient and makes you make unnecessary movements to obtain information, plus it takes time to generate a report:
Although this has its own plus, there is the possibility of sending reports to several e-mails. As for the type of report itself, a graphical representation of the results and a list of all tests with the corresponding value are displayed.
Sitechco allows you to generate reports automatically, without performing unnecessary actions to obtain it. 4 main types of reports give complete information about testing a project.
The launch report displays the entire list of Checklists for which testing was performed, there is the ability to view general and detailed information. According to the results of the passage, it is possible to see a report on the last 5 starts of the Checklist, plus, it allows you to customize the report for yourself. The group report form shows in graphical form the presentation of the result:
TestLink made me sweat and torment myself with running tests for passing. The system did not give in for a long time, which I did not like anymore. After the joint proceedings, we figured out how to do it. The developers proposed a report option in a form that seemed to us not sufficiently informative:
Although there are a lot of report options, they are all of the same type and in an unacceptable format, at least for us.
TestRail can export in 3 formats: xml, CSV and Excel. But strangely enough, when I tried to get a report in Excel I was given in CSV format.
In import, everything looks simpler and is presented in the form of selecting a file from a PC in xml format. There are also opportunities to add to an existing list of tests and create a new one.
Sitechco does not lose its ambition and introduced its users to export data in xls format. But, here, too, has its own catch, opening a file is only possible in Microsoft Office.
But the import was very interesting. I didn’t have to save anything to a separate file - the insertion from a file on a PC or Google Docs was successful and without unnecessary gestures, with a simple copy-paste of the necessary rows of the table.
Export to TestLink is presented in only one format - xml. It is possible to download both a complete list of test cases and individual folders.
Import is supported in a similar format - xml. It is presented in the form of downloading a separate file from a PC for the general list and for the folder separately.
So, let's start in order. TestRail offers us a choice of 4 roles:
Roles are easily edited by setting check boxes to allow actions:
The function of creating separate user groups was also pleased, which allows you to break users into separate lists:
In the “kitchen” friend, I found 5 user roles:
Long I tried to understand who and what could, but the answer was “on the surface” - the help contains a detailed description, which simplifies the task. The only thing I would like is to create your own lists, but apparently this is not provided for in the web version.
TestLink offers its users to use the already built-in options:
There is also the opportunity to create your own user lists, with setting rights:
I have the opinion that TestLink and TestRail have, at some points, an identical modification in the settings. But here at TestRail they worked on usability, and TestLink is not comfortable.
Our company uses Redmine and it is logical that we were looking for a system that supports integration with this bug tracker. All three applicants fall under the selection list. I will try to describe in more detail for each.
Testrail. Integration, as other users said earlier: “simple and ingenious” at the same time:
This setting is suitable for most online bug trackers. When I started a bug, the form, it seemed to me, was not thought out to the end and therefore contained a minimum number of fields.
Perhaps the developers did not want to complicate their lives, but maybe it was done in haste. But personally, I didn’t really like it, we fill in more fields when making mistakes, and it’s inconvenient to enter and edit it in Redmine after making mistakes from TMS. The link to the bug itself is displayed next to the test data, which does not complicate the perception, and when you click it you can see the information in more detail - this is very convenient:
But the main drawback of integration through TestRail is the authorization of only one user, which is very inconvenient for the programmer to parse the bugs. There are unnecessary questions and excessive fuss.
Sitechco integration settings are fairly simple and step-by-step. It is difficult to get confused. Initially, a project is selected, a link to the bug tracker itself and administrator access to it.
At first everything is normal, but further on - a clear plus: the ability to customize the fields that will be displayed in the form of a defect. Undoubtedly the right thing!
All my settings had an effect on the forms to fill out. You will not say anything, soundly thought out and for people.
There is nothing supernatural in the reports either, a list of bugs separated by commas, the data is referenced and displays the status of the bug at the moment.
Testlink Integration is similar to TestRail, so I won’t spray too much, because I did not like the integration itself initially.
It is very inconvenient to start bugs in this format, living in the world of developing technologies.
The next important point for us is the connection with design requirements. At our project, they are not being conducted in detail, but the status on them should be transparent, timely, and before we used a google-tablet in which we forgot to update something all the time.
After clicking on all three of our candidates, I did not notice the obvious in TestRail, so on this point he is out of the game.
In Sitechco, the “Requirements” item is in the main menu, but it made me snort before I could make it active (through the project settings, as it turned out).
Then we get to the list of requirements that we can create or import from tables.
The creation window does not contain anything superfluous: the name and description. All that is missing is the addition of attached files - you may have to use links to third-party file hosting services.
I was also pleased with the import of requirements, apparently created by analogy with the import of tests, also nothing complicated and abstruse. It is presented in the form of a window, with the choice of the directory for import and the field itself. There is also help:
By creating / importing requirements, we can make a connection between each test (checking in the checklist, and not the whole checklist) and the requirement, this connection works on a many-to-many basis:
Let's see how things are in TestLink. The manufacturer offers a set of functions for working with requirements: create, import, export, create test suites and copy requirements.
The requirements creation operations are quite simple, they have the ability to attach files, which more than covered all my desires for the type of requirements. But using it was not very inconvenient - just like almost everything in TestLink.
Import itself as well as export is supported in CSV \ XML \ DocBook format. Apparently the entire Testlink project is focused on the CSV format.
Requirements are attached to the Test Case and help in the testing process, in order to verify the completeness of the coverage of requirements with checks and to prevent gaps during testing.
I said a little about the simplicity of the work at the beginning, I would like to continue the conversation on this topic and reveal the following features. The ability to use hotkeys to create tests and groups in checklists is implemented only in Sitechco.
The same points as the description of user groups and ease of use are noted in all three products. But against the background of all standard requirements, I really liked the fact that TestRails allows you to create the project address yourself, and testlink took it even further and offered its own data storage server. Sitechco, however, was conservative in this regard and did not indulge in additional twists, apparently relying on the broadest functionality and ease of use.
I could not find something supernatural. The only thing that struck me was the Scheduler task scheduler, which reminds me of Todoist , due to the display of the number of tasks on me and the ability to plan for the future.
Most importantly, as our test manager said: “I can now control you” and smiled maliciously. It turns out that this tab also allows you to view which of the checklists is currently running, which has been completed. “Sandy” - I thought and tried to look for this function in TestRail and TestLink. Alas! This functionality is supported only in Sitechco, and consider it a plus or minus - we and the authorities adhere to different points of view.
Due to the development of all kinds of mobile chips, I also expected that I would dig out such things (mobile versions, mobile applications), but, unfortunately, none of the rivals had this functionality in their armament. Testing with a bunch of tabs is inconvenient, agree. Therefore, it would be “to the cash register” to make notes on the results on the tablet and at the same time not to clutter up the space on the table.
According to one of the TV presenters of the Ukrainian channel, Olga Freimut, in her inspections: “Well, what? Let's talk about?" (Well? Talk?)
Based on your preferences, I tried to objectively compare these three fighters, but so far I have not decided on 2 winners: kitchen utensils or a business class thing. Both candidates have their pros and cons, which in their own way attract. In any case, the issue will be decided by the fin. Director based on the allocated budget, but I can only wait.
Trying to answer all these questions, we organized the storage of requirements and checklists in the form of google tables, but eventually realized: it is time to look for a specialized tool. The results of the search and comparison of options I want to share with you in this article.
In connection with the remote work team and in general, due to habit, we considered only web options. Having rejected completely inappropriate solutions for the initial analysis, there were three candidates to choose from:
TestLink
TestRail
Sitechco
We get as a result: Triple Threat Match. Who will see whom in fair fight.
To begin with, we in the team determined the selection criteria:
- License cost : preferably free, or at least inexpensive;
- Supported environments : the team uses Google Chrome, Mozilla Firefox, Safari - no one is ready to refuse or change browsers;
- Functionality of test management : here, the most important thing for us was the convenient creation of test sets, maintaining statistics of progress and visual reporting;
- The presence of elements of test management : I really want the tool to plan our tasks, control plans and deadlines;
- Usability : if someone in the team doesn’t like something, we’ll hardly spend our time and “delve into” it. Everything should be clear right away.
According to these criteria, we will analyze each of the candidates.
Cost
Let's go in descending order:
- TestRails offers us exclusively paid licenses, depending on their conditions it will cost from $ 69 to $ 8999, a serious fee for a serious product. In order not to buy a cat in a poke, we are offered to use the test version for 30 days, which will be free - we used it to analyze it. In our case (up to 15 users) TestRails will cost 1399 € or 2149 $.
- Sitechсo for local installation costs 12,000 rubles, SaaS use is free. Given the work on the https protocol and not too strict, in terms of security, customers, the web version is quite suitable for us.
- TestLink is free provided it is installed on your server. I had to tinker, but it started. For the price a clear leader.
Supported Environments
With regard to compatibility, all browsers are declared to support TestRails and TestLink, but IE is not supported by Sitechco:
Given our set of environments, this is not critical, but for many users (especially those testing some banking software in truncated environments) it may be.
Test management functionality
The most important criterion in choosing a system for us, of course, is its functionality. We jointly worked out a list of necessary “features”:
- Creation of tests and checklists;
- Testing control;
- Import and export of tests (the desired format is tables);
- Testing and software quality assessment reports;
- Division of user rights;
Of the desirable, but not required, the list includes:
- Integration with Redmine;
- Maintaining requirements with test coverage analysis;
- External API for auto tests;
Creating tests and checklists
In TestRail, the creation of tests is implemented using the Test Suite, in which the Tests are placed.
Adding is done quickly, according to the principle of creating a Test Case and a separate description, with the possibility of dividing into sections and attaching, if necessary, files. Inside the test, you can write down the steps in detail, but in most cases we limited ourselves to clear headers.
Sitechco implemented test suites in the form of Check and Cheat Lists. The checklists themselves are simple, nothing superfluous or abstruse: test, check, priority, well, various additions: attachment of requirements, labor costs, hierarchy in the form of groups and master groups.
An important, as for me, role is played by cheat lists, with an existing set of tests. It will save time for creating your own and will be useful for the future. These elements play the role of a designer: write standard cheat lists and insert them into check lists as necessary. It is convenient when there are many standard checks, for example, we need to check different fields for usability guidelines and for XSS- / SQL injections. Data is repeated, copy-paste is evil, but you need the status of the passage in different contexts and for different fields.
Creating tests in TestLink is somewhat similar to creating forms in TestRail. By analogy, test suites are created in Test Suite, in which tests are embedded.
It looks rather confusing and at the same time primitive:
The same tests can be added to different scenarios, which in our case performs a similar function, like cheat lists in Sievechko. But in order for the tests to be atomic and small (with a visual result for the report), too many gestures appeared in TestLink. Well, a LOT .
Monitoring Results
The developers of TestRail offered their customers several reporting options:
What allows you to collect information about testing on bugs, results, launches and users. But! Each report needs to be processed, which is very inconvenient and makes you make unnecessary movements to obtain information, plus it takes time to generate a report:
Although this has its own plus, there is the possibility of sending reports to several e-mails. As for the type of report itself, a graphical representation of the results and a list of all tests with the corresponding value are displayed.
Sitechco allows you to generate reports automatically, without performing unnecessary actions to obtain it. 4 main types of reports give complete information about testing a project.
The launch report displays the entire list of Checklists for which testing was performed, there is the ability to view general and detailed information. According to the results of the passage, it is possible to see a report on the last 5 starts of the Checklist, plus, it allows you to customize the report for yourself. The group report form shows in graphical form the presentation of the result:
TestLink made me sweat and torment myself with running tests for passing. The system did not give in for a long time, which I did not like anymore. After the joint proceedings, we figured out how to do it. The developers proposed a report option in a form that seemed to us not sufficiently informative:
Although there are a lot of report options, they are all of the same type and in an unacceptable format, at least for us.
Import and export tests (preferred format - tables)
TestRail can export in 3 formats: xml, CSV and Excel. But strangely enough, when I tried to get a report in Excel I was given in CSV format.
In import, everything looks simpler and is presented in the form of selecting a file from a PC in xml format. There are also opportunities to add to an existing list of tests and create a new one.
Sitechco does not lose its ambition and introduced its users to export data in xls format. But, here, too, has its own catch, opening a file is only possible in Microsoft Office.
But the import was very interesting. I didn’t have to save anything to a separate file - the insertion from a file on a PC or Google Docs was successful and without unnecessary gestures, with a simple copy-paste of the necessary rows of the table.
Export to TestLink is presented in only one format - xml. It is possible to download both a complete list of test cases and individual folders.
Import is supported in a similar format - xml. It is presented in the form of downloading a separate file from a PC for the general list and for the folder separately.
User rights sharing
So, let's start in order. TestRail offers us a choice of 4 roles:
Roles are easily edited by setting check boxes to allow actions:
The function of creating separate user groups was also pleased, which allows you to break users into separate lists:
In the “kitchen” friend, I found 5 user roles:
Long I tried to understand who and what could, but the answer was “on the surface” - the help contains a detailed description, which simplifies the task. The only thing I would like is to create your own lists, but apparently this is not provided for in the web version.
TestLink offers its users to use the already built-in options:
There is also the opportunity to create your own user lists, with setting rights:
I have the opinion that TestLink and TestRail have, at some points, an identical modification in the settings. But here at TestRail they worked on usability, and TestLink is not comfortable.
Redmine Integration
Our company uses Redmine and it is logical that we were looking for a system that supports integration with this bug tracker. All three applicants fall under the selection list. I will try to describe in more detail for each.
Testrail. Integration, as other users said earlier: “simple and ingenious” at the same time:
This setting is suitable for most online bug trackers. When I started a bug, the form, it seemed to me, was not thought out to the end and therefore contained a minimum number of fields.
Perhaps the developers did not want to complicate their lives, but maybe it was done in haste. But personally, I didn’t really like it, we fill in more fields when making mistakes, and it’s inconvenient to enter and edit it in Redmine after making mistakes from TMS. The link to the bug itself is displayed next to the test data, which does not complicate the perception, and when you click it you can see the information in more detail - this is very convenient:
But the main drawback of integration through TestRail is the authorization of only one user, which is very inconvenient for the programmer to parse the bugs. There are unnecessary questions and excessive fuss.
Sitechco integration settings are fairly simple and step-by-step. It is difficult to get confused. Initially, a project is selected, a link to the bug tracker itself and administrator access to it.
At first everything is normal, but further on - a clear plus: the ability to customize the fields that will be displayed in the form of a defect. Undoubtedly the right thing!
All my settings had an effect on the forms to fill out. You will not say anything, soundly thought out and for people.
There is nothing supernatural in the reports either, a list of bugs separated by commas, the data is referenced and displays the status of the bug at the moment.
Testlink Integration is similar to TestRail, so I won’t spray too much, because I did not like the integration itself initially.
It is very inconvenient to start bugs in this format, living in the world of developing technologies.
Maintain requirements with test coverage analysis
The next important point for us is the connection with design requirements. At our project, they are not being conducted in detail, but the status on them should be transparent, timely, and before we used a google-tablet in which we forgot to update something all the time.
After clicking on all three of our candidates, I did not notice the obvious in TestRail, so on this point he is out of the game.
In Sitechco, the “Requirements” item is in the main menu, but it made me snort before I could make it active (through the project settings, as it turned out).
Then we get to the list of requirements that we can create or import from tables.
The creation window does not contain anything superfluous: the name and description. All that is missing is the addition of attached files - you may have to use links to third-party file hosting services.
I was also pleased with the import of requirements, apparently created by analogy with the import of tests, also nothing complicated and abstruse. It is presented in the form of a window, with the choice of the directory for import and the field itself. There is also help:
By creating / importing requirements, we can make a connection between each test (checking in the checklist, and not the whole checklist) and the requirement, this connection works on a many-to-many basis:
Let's see how things are in TestLink. The manufacturer offers a set of functions for working with requirements: create, import, export, create test suites and copy requirements.
The requirements creation operations are quite simple, they have the ability to attach files, which more than covered all my desires for the type of requirements. But using it was not very inconvenient - just like almost everything in TestLink.
Import itself as well as export is supported in CSV \ XML \ DocBook format. Apparently the entire Testlink project is focused on the CSV format.
Requirements are attached to the Test Case and help in the testing process, in order to verify the completeness of the coverage of requirements with checks and to prevent gaps during testing.
Usability:
I said a little about the simplicity of the work at the beginning, I would like to continue the conversation on this topic and reveal the following features. The ability to use hotkeys to create tests and groups in checklists is implemented only in Sitechco.
The same points as the description of user groups and ease of use are noted in all three products. But against the background of all standard requirements, I really liked the fact that TestRails allows you to create the project address yourself, and testlink took it even further and offered its own data storage server. Sitechco, however, was conservative in this regard and did not indulge in additional twists, apparently relying on the broadest functionality and ease of use.
Special offers:
I could not find something supernatural. The only thing that struck me was the Scheduler task scheduler, which reminds me of Todoist , due to the display of the number of tasks on me and the ability to plan for the future.
Most importantly, as our test manager said: “I can now control you” and smiled maliciously. It turns out that this tab also allows you to view which of the checklists is currently running, which has been completed. “Sandy” - I thought and tried to look for this function in TestRail and TestLink. Alas! This functionality is supported only in Sitechco, and consider it a plus or minus - we and the authorities adhere to different points of view.
Due to the development of all kinds of mobile chips, I also expected that I would dig out such things (mobile versions, mobile applications), but, unfortunately, none of the rivals had this functionality in their armament. Testing with a bunch of tabs is inconvenient, agree. Therefore, it would be “to the cash register” to make notes on the results on the tablet and at the same time not to clutter up the space on the table.
According to one of the TV presenters of the Ukrainian channel, Olga Freimut, in her inspections: “Well, what? Let's talk about?" (Well? Talk?)
- Testlink Actually what did I expect from a free service? All guesses were confirmed. Check the box that is free, there is a basic set of functions - that's all. Sorry, buddy, you are in a knockout, everything is somehow sophisticated and did not see the highlight.
- TestRails. Да, ничего не скажешь, штука мощная, во своими плюсами и своими минусами. Но в силу отсутствия мультиязычности и не отличающийся такой уж и “дешевизной”, я так и не смог себе это позволить. Не один я ведь буду работать, потому эгоизм свой пришлось “удалить из настроек”.
- Sitechco . Просто и со вкусом, ничего заумного или сложного для понимания. Мне, возможно показалось, но компания только недавно начала появляться на арене IT и недооценивает свои возможности, либо специальная цена продукта в силу экономического кризиса стран.
| Критерии сравнения | Sitechco | TestLink | TestRails |
|---|---|---|---|
| Основные | |||
| стоимость лицензии | $0-$500 | - | $69-$8,999 |
| браузеры | кроме iE | все | все |
| функционал | |||
| автосохранение/контроль версий | да | да | да |
| шаблоны | да | нет | нет |
| печать/ссылка на отчет | да | да | да |
| импорт/экспорт | да | да/нет | да/да |
| мобильная версия | нет | нет | нет |
| окружения/сборки/требования | да/да/да | нет/да/да | нет/да/нет |
| подсказки | да | нет | да |
| Элементы тест-менеджмента: | |||
| тест план | да | да | нет |
| тест кейсы | да | да | да |
| поддержка автотестов | нет | нет | нет |
| чек-листы | да | да | нет |
| Разграничение ролей пользователей | есть | есть | есть |
| поддержка аттача файлов(файловое хранилище) | нет | нет | нет |
| е-мейл нотификации | есть | нет | есть |
| Работа с Чек-листами | |||
| настройка полей в ЧЛ (разные типы ЧЛ, отдельно настройка тестового окружения, например) | есть | нет | нет |
| удобство редактирования (субъективная оценка) | удобно | не удобно | не особо удобно |
| связь с требованиями | да | да | нет |
| Отчётность | |||
| по сборкам | да | да | нет |
| по окружениям | да | нет | нет |
| по тестерам | да | нет | нет |
| по дате выполнения | да | нет | да |
| по результату | да | да | да |
| какое-либо отображение статуса тестируемого продукта | есть | есть | есть |
| отчётность по разным ЧЛ сразу | да | да | да |
| отчётность по покрытию требований | да | да | нет |
| отчётность по результатам тестирования требований | да | да | нет |
| Юзабилити | |||
| простота | да | да | да |
| хоткеи | да | нет | нет |
| описание групп пользователей | да | да | да |
Based on your preferences, I tried to objectively compare these three fighters, but so far I have not decided on 2 winners: kitchen utensils or a business class thing. Both candidates have their pros and cons, which in their own way attract. In any case, the issue will be decided by the fin. Director based on the allocated budget, but I can only wait.