Introduction

With each passing year, it is becoming increasingly clear that specialized services are needed to distribute test versions of iOS applications. This is true not only for companies involved in application development on an “industrial scale”, but also for indie developers. Reasons: an increase in the number of devices with “non-standard” / “non-multiple” screen sizes, differences in the architecture of the central and GPUs - everything that used to be a daily routine for developing for Android is now becoming true for iOS.

Historically, the most popular distribution service was TestFlight - almost ideal, both technically and in usability. However, after Apple acquired the company in early 2014, the service underwent a number of controversial changes that, on the one hand, negatively affected its leadership position, and on the other, gave alternative services a chance to compete for first place and offer at least an equivalent replacement, and maybe something more comfortable and functional.

In my series of articles, I am going to share the results of comparing several distribution services and, as a result, help others who are faced with the same issue of choice.

The following services participate in the review (in this order, parts of the review will be published):

• Apple TestFlight
• Hockeyapp
• Ubersters
• Crashlytics (as part of Fabric)

TestFlight of the old version will not be reviewed, because although it still works and is available at www.testflightapp.com , using it in the long term does not make sense for several reasons:

• there is no guarantee that Apple will not suspend or stop its operation at any time
• The service SDK is not available for download, moreover, it is impossible to use the service if the application uses the SDK
• the service is no longer cross-platform, only iOS application distribution is supported ( more )

Important! Just before the publication of the article, the TestFlight team (old version) sent a notification informing that the old service would stop its work on February 26, 2015 ( details ).

Evaluation system: services will be evaluated on a 10-point scale for each of the sections (Registration and Integration, Basic Functionality, Additional Functionality, Continuous Integration). The total score will determine the winner (the final conclusion will be included in the last part of the review).

Apple TestFlight

Registration and integration

To use the service, in addition to the developer account (iOS Developer Program), you must access iTunesConnect. The functionality depends on the role (group) to which the iTunesConnect account belongs. Users from the Technical group can:

• enable / disable TestFlight Beta Testing mode for applications (Prerelease tab)
• select testers from the list of available Internal Testers, but are not able to form this list on the page iTunes Connect -> Users and Roles -> TestFlight Beta Testers -> Internal
• create a list of external testers My Apps -> APP -> Prerelease -> External Testers (but in order to distribute versions for external testers the application must undergo a review in Apple)

Users from the Admin group have the same rights as users of the Technical group, and can also add new users to the Internal Testers list.

At the moment, the installation or use of any specialized SDK is not required.





Rating: 8/10.

Main functionality

If we compare the new version of TestFlight with what was previously available to developers in the framework of standard Apple services, then it, in some ways, is undoubtedly a step forward. So, for example, the number of testers increased from the conditional 100 to 1125 (of which 100 are devices that were previously available for AdHoc distribution, 25 are internal testers (Internal Testers), and 1000 are external (External Testers)), and if the binding is earlier was carried out to the device, now Apple ID is a unique identifier, which means that each tester can check the application on all devices available to him (and, of course, supported by the application).

Note (dated June 15, 2015): the number of devices available when using AdHoc provision profiles from June 8, 2015 was changed from 100 in total, to 100 per device family (the change occurred after the announcement on WWDC that all developers programs were merged into one) .

The difference between internal and external testers is an additional step, which is added in the second case: the test version of the application must be sent for review and approved by Apple specialists (Beta App Review). Moreover, the initial review will be carried out more carefully, and all subsequent ones may be faster (but not necessary). This, of course, imposes certain restrictions on the development process and requires additional planning of the application testing phases, taking into account the uncertainty of the review period (as in the case of the usual review of applications before publication in the AppStore, in case of refusal to publish, the repeated application is considered again in order )

But there are other features that distinguish Apple from alternative solutions, alas, not for the better:

• The application client service can be installed only on iOS 8.0 and higher, respectively, there is no way to test applications on devices with iOS previous versions.
• To distribute test versions, you must create an AppStore Distribution provisioning profile and configure the application profile in iTunes Connect.
• Since the application distribution package is loaded through the standard Xcode -> Archive -> Export wizard, it is necessary that the application project contains 3 main icons.

• The number of applications simultaneously available for testing is limited to 10.
• The service works very slowly, this concerns both the speed of updating the list of available applications and their versions, and the speed of downloading / installing applications on test devices.
• Sending feedback from the test device is carried out using the client application, but since this is a regular letter, there is no way to attach a screenshot (as a plus, it can be noted that a block with a description of the characteristics of the test device is automatically inserted into the letter).
• The ability to send feedback appears only if iTunes Connect indicates in the version settings what it is necessary to test (and if the email address for receiving feedback is not specified, then the “To” field in the letter will be blank).
• Only one version of the application can be available for testing (the tester does not have the ability to select, accessibility settings are made in the application profile in iTunesConnect).
• The field “Feedback Email” in the application version settings in iTunesConnect has to be filled in every time (it would be more logical to remember, leaving the possibility for editing).
• It is not surprising, but TestFlight in the version from Apple is not a 24x7 service, for example, at the end of December, when trying to enter iTunesConnect, you could see such a message: iTunes Connect is unavailable until December 29. (in fairness, it is worth noting that Apple arranges such "holidays" annually and sends out a notification a week before their onset).

However, there are some unique features: for example, the service client (TestFlight application) is the only one who probably knows the installation status of the test version, alternative solutions simply do not have the ability to use system functions that allow you to implement this.

Note: installation status is a change in the installation process indication, familiar to AppStore users. When the status changes Get - Install - Installing - Open. In other distribution services, this chain looks like this: Install - Installing - Install - applications simply do not know the result of whether a test distribution was installed or not. In the case of Apple TestFlight, the chain has the usual look: Install - Installing - Open.



Icon of the updated TestFlight application (it would be more correct to say that this is a completely new native application developed by Apple). Below is an example of the icon of the test version of the application - an orange “dot” allows you to determine that this is not a version from the AppStore.



Apple “spam” insurance.



Apple does not allow distributions to be downloaded without changing the version or build number.



A fairly concise and clear page for administering application versions for testing.



An invitation letter to participate in testing, HTML formatting is not too optimized for mobile platforms (in fairness, I want to note that the Mail application displays such letters correctly, but letters from other services are displayed normally in OWA).



TestFlight Application: Information about the application available for testing.




The TestFlight application is the only one that can determine the result of installing a test version (the Install button has changed to Open) - the advantage gained by switching to Apple.



A letter with feedback sent from the TestFlight application. In addition to poor formatting and the inability to add attachments, one more drawback can be noted: sent reviews go only to the mailbox, in iTunes Connect this information is not saved and is not available.

In summary, we can say that the updated version of TestFlight just works. At the moment, it does not have particular competitive advantages (we are talking about functionality, and not an indisputable advantage - Apple's affiliation, which so far is more of a potential).

Rating: 6/10.

Additional functionality

At the moment, unfortunately, Apple takes only a number, not a skill. Of the obvious advantages, it can be noted that there is a much larger number of beta testers that can be used for external testing: an order of magnitude more than all other alternative solutions (the reason is clear: the company itself controls the distribution conditions of applications and has an advantage over third-party solutions).

Update (May 5, 2015): Crashes Organizer became available to developers with the release of Xcode 6.3 . The new functionality is undoubtedly a good sign that Apple continues to develop its services, but, unfortunately, the company is still lagging behind other services:

When you open the Crashes organizer, Xcode begins refreshing the crash reports for your apps. Xcode downloads the top crash reports — crash reports with the most number of occurrences on unique devices — that occurred during the past two weeks. However, there may be up to a three day delay between when you first distribute your app and when crash reports are available in Xcode.

Three days of delay and a period of archival access of two weeks looks somewhat discouraging.

Rating: 1/10.

Continuous integration

Note: the nuances of using the services described below as part of continuous integration in this review imply that it will be made based on the solution offered by Apple. That is, with the help of Mac OS X Server and Xcode bots - I will not dwell on the organization of such integration in detail, perhaps this will be the topic for a separate article. Those who wish can familiarize themselves with the topic on their own, for example, referring to the official guide from Apple.

No matter how strange it may seem, but at the moment there is no way to publish applications in Apple TestFlight in automatic mode. I hope that this big flaw will be fixed by the company in one of the next major updates of Xcode, especially since the original TestFlight had such an opportunity.

Rating: 0/10.

Total total score for all sections: 15 points.

To be continued...