Introduction

The third part of the review will be devoted to the Ubertesters service (the first part of the review , the second part of the review ).

This is a fairly “young” service, with ambitions to enter the list of leaders (it will not be easy to do this, given the recent acquisitions of competitors by the largest IT players). A big difference and advantage of Ubertesters is the ability to attract external testers on a paid basis (anyone who wants it can register as a tester himself and get paid for his work).

Separately, I want to note that Ubertesters was the only company from the review that followed its publications, provided additional information and shared plans to update the service in order to improve the quality of the review.

Evaluation system: services are evaluated on a 10-point scale for each of the sections (Registration and Integration, Basic Functionality, Additional Functionality, Continuous Integration). The total score will determine the winner (the final conclusion will be included in the last part of the review).

Ubersters

Registration and integration

Registration in the service is simple, in addition to paid subscription options, it is possible to use the main functionality for free with restrictions on additional functions and the number of team members, projects, and so on ( more ).

Note from the category of funny things: after registration, check your profile - in my case, the country was indicated correctly, but the subject of the federation was selected first from the list and it turned out to be the Republic of Adygea!

Integration with the SDK is simple ( http://ubertesters.com/download-sdk/ ):


Rating: 7/10.

Main functionality

Note (updated June 15, 2015): when talking about the distribution of iOS apps, you need to remember that at the moment, all services except Apple’s TestFlight should still consider the number of devices available when using AdHoc provision profiles. Therefore, the maximum number of devices used is 100 per device family (this condition was changed on June 8, 2015, after the announcement at WWDC that all developers programs were merged into one), which includes devices used directly by developers. The biggest limitation is that the list of registered devices can be “reset” only once a year, after renewing the subscription to the iOS Developer Program (possibly in the future this condition will be changed).

Immediately after completing the registration procedure, the user has the opportunity to choose one of four options for further actions:

After selecting the first option - Start a new project - a brief description of the upcoming steps will be presented:

Note: the window that seems to be a step-by-step wizard is actually only informative after completion of the first step, the user gets to a regular page (without indicating the current progress), where he has the opportunity to choose one of the two proposed options (or use other functions with ayta, through the navigation menu):

Another remark about the second option - Invite members to the team - its icon may suggest the use of social networks as a tool to attract team members, but in fact sending invitations is carried out by e-mail.

When creating a new project in the first step, the easiest way is to use the auto-fill form when loading the application distribution kit:

It’s a little inconvenient, in my opinion, that the downloaded distribution package is not used by the service for its intended purpose, since the next step is to separately download the distribution package for the first revision applications (revision - a term used by a service to refer to versions / assemblies).

The Overview page contains basic information about the project, as well as summary statistics:

Note: Notifications in the system are by default set to Watching mode for users with the Admin or Manager role and Not Watching for users with the QA or Developer role , which assumes notification of only those events in which they are a member. Ignoring mode is also available when the system will not send any notifications at all:

For a description of the above modes, see the frequently asked questions page “What is notification status? In what case I will be notified? ”.

When adding a new revision in the first step, you need to download the application distribution package (* .ipa):

In the next step, the basic information about the revision is set:

After downloading, the revision is inactive, that is, it is not available for installation on test devices. In order to start testing, you need to click the Start button :

And on the next screen, indicate which of the team members this revision will be available (at the same time it is possible to send a notification to all participants):

After that, the revision receives the status In progressand can be installed on devices.

Note: if there is a default user group in the project, then the team members included in it will be immediately selected (switch position in the first column On), if there is no such group, then users will need to be selected manually. Access to each revision can be stopped either for everyone or for some users, which allows you to effectively solve organizational issues that arise during the work on the project. Also, when initializing Ubertesters, it is possible to allow access to the application, but disable the ability to use the service’s functionality: UbertestersLockingModeDisableUbertesters.

The February service update (in more detail ) added, including the ability to edit the description for the revision:

Invitations to the team, as I said, are sent by e-mail (Administration - Team - Add members), and this task can be facilitated by using the import function from the CSV file:

Users can define one of three roles, as well as add to groups (used for distributing the application to a limited number of users).

Note. In general, within the service, the roles are hierarchically distributed as follows: at the organization level, there are two roles, Admin and Member , at the project level, Member can be QA , Developer and Manager . An important nuance: the first revision in each project must be loaded by the user with rolesAdmin or Manager . Further revisions can be downloaded by any team member (including QA ). The service is designed in such a way as to take into account all possible options for building business processes in a particular company. More details about the roles can be found on the frequently asked questions page .

When registering a new device, two applications are installed on it: one native (icon with a white background), the other web (icon with a black background). All the main functionality is in the native application, so the web application can be deleted after the device has been registered in order to avoid further confusion:




Note: in the native application there is no way to update the contents of the open tab, you have to switch between the tabs to see the new data (new revision of the application, new application for testing, etc.).

You can add a new testing device in several ways:

Rating: 8/10.

Additional functionality

I emphasize once again that when testing software it is very important to get the most complete and reliable information. Unfortunately, at the moment, Ubertesters lacks functionality for symbolizing crash reports (in this it loses in comparison with other services), but it has in its baggage another, often more useful, functionality: the ability to create tickets directly from the application under test, and also provide them with screenshots.

This functionality is available to the user after pressing the round button, which by default is placed in the upper right corner of the screen, but can be freely moved to any other convenient place:



The system for tracking errors in a service may not be a complete alternative to specialized services, but in combination with the ability to accompany each ticket with screenshots, it copes with its tasks perfectly:



Each screenshot can be accompanied by notes (among the editing tools: pencil, marker, text labels, arrows, and the ability to cut out part of the screenshot). The screenshots taken remain on the device and can be reused. It is also possible to add screenshots from the standard device photo library.

Note: a letter with a notification about a new ticket does not contain information about whether screenshots were uploaded during creation. In my opinion, it would be convenient to have such data, at least the number of attached screenshots.

On the service website, information on all tickets is presented very clearly, you can view general statistics for a single revision, and for all revisions in total:

In addition to status statistics, information is also presented by type (Bug, Feature, Improvement, Task, Crash), priority and error playback frequency.

An important advantage is that each ticket is tied to a specific revision, which makes it easier to work with feedback from testers and increases the value of the information collected.

Note: if in some case the crash reporting svolization plays a critical role, it is possible to use another service SDK, and after the initialization of the Ubertesters SDK it is necessary to disable the crash processing:

[[Ubertesters shared] disableCrashHandler];

Important! In this case, no information about the falls will not go to the Ubertesters website.

The service provides the ability to integrate with external trackers (Jira, Mantis, Redmine, YouTrack, Unfuddle, HPQC). Integration can be carried out only with one of the available services at a time. The interface for managing this functionality is located in the Administration - Issue Tracker section (settings are made for each project individually).

Example of a JIRA-related ticket: The

work of testers can be more organized when using the Tests functionality. In a special section of the service, you can view statistics on existing tests, as well as create new ones:


Note: the ID field is for a unique identifier, it can be numeric or string.

Adding new tests is also possible by importing XLSX files:

On the test device, the user has the opportunity to view the list of available tests, select one of them, test and report the result:



Note: in case of unsuccessful test execution, it is possible to create a ticket (issue), which will later be tied to this specific test.

In February, the functionality related to monitoring and statistics of testers was significantly updated. The new Activity section provides extensive information about the current testing session:

The Summary tab displays only the main statuses (Ative, Suspended, Crashed, etc.), and when you select a specific test session - the Activity Stream block - it is possible to gradually monitor the tester’s work:

Key events from the session history are interactive: in this way it is possible to immediately view the test description, the ticket sent, or get basic information about the application crash.

Note: the date of sessions in the Activity Stream is indicated at the beginning of the session, if the session lasted several days, then this should be taken into account if you need to view the chronology of user actions (that is, what event can occur on the second day, but you will need to look for this one in the list of sessions day, and the day the session begins - that is, the previous one).

Official service guide: http://ubertesters.com/step-by-step-instruction/

Rating: 8/10.

Continuous integration

Note: The nuances of using the services described below as part of continuous integration in this review imply that it will be based on the solution offered by Apple. That is, with the help of Mac OS X Server and Xcode bots - I will not dwell on the organization of such integration in detail, perhaps this will be the topic for a separate article. Those who wish can familiarize themselves with the topic on their own, for example, referring to the official guide from Apple .

Access to the API for automating downloads is enabled separately, in the profile settings (the API description can be found at http://ubertesters.com/upload-api/ ).

Using the API is possible with curl:

curl -X POST http://beta.ubertesters.com/api/client/upload_build.json -H "X-UbertestersApiKey:PERSONAL_API_KEY" -F "file=@upload.ipa" -F "title=build title" -F "notes=build notes" -F "status=in_progress" -F 

Parameters:

    X-UbertestersApiKey- required, your personal 'API access KEY'. Please see item 1
    file - required, path to your build file (.IPA or .APK)
    title - optional, your build title
    notes - optional, build description, plain text
    status - optional
        - pending - default, create a new revision only
        - in_progress - create and start revision
    stop_previous - optional
        - true - stop all previous revisions ( if any)
        - false - keep current statuses of previous revisions

Note: in order to download and publish the new version in a completely automatic mode, in the project settings ( Administration - Distribution Groups ) you need to create a group of testers by default, in this case, when the “status = in_progress” key is added, the new version will immediately become available to all members of this group. Several groups can be set as the default group at once.

Important: before publishing the application in the AppStore, you must at least remove the initialization of Ubertesters from the code, if the application size is also important, it is recommended to completely remove the framework from the project (the difference in the size of the distribution package is just over 1 megabyte).

Rating: 9/10.

Total total score for all sections: 32 points.

PS Despite the lack of (so far!) The functionality of symbolizing crash reports, Ubertesters can be used much more effectively than its competitors in projects where the importance of obtaining feedback from testers and other team members (customers, managers, designers and other). This is especially important for projects with a distributed team.

The developers of Ubertesters promised to take into account many of the comments and suggestions outlined in the article, but I, in turn, will try to update the material as these promises are realized.