Run instrumental tests in the Firebase Test Lab. Part 1: iOS project
- Tutorial
My name is Dmitry, I work as a tester at MEL Science . Most recently, I ended up dealing with a relatively fresh feature from the Firebase Test Lab - namely, with instrumental testing of iOS applications using the native XCUITest testing framework.
Before that, I had already tried the Firebase Test Lab for Android and I really liked everything, so I decided to try to put the test infrastructure of the iOS project on the same track. I had to google a lot and not everything worked right the first time, so I decided to write a tutorial article for those who still have to.
So, if you have UI tests on an iOS project, today you can try to run them on real devices kindly provided by Good Corporation. Interested - welcome to cat.
In the story, I decided to build on some source data - a private repository on GitHub and the CircleCI build system. The name of the application is AmazingApp, bundleID is com.company.amazingapp. I cite this data immediately, to reduce the subsequent confusion.
If you implemented different solutions in your project differently - share your experience in the comments.
1. The tests themselves
Create a new project branch for UI tests:
$ git checkout develop
$ git pull
$ git checkout -b “feature/add-ui-tests”
Open the project in Xcode and create a new Target with the UI tests [XCode -> File -> New -> Target -> iOS Testing Bundle], give it the talking name AmazingAppUITests.
Go to the Build Phases section of the created Target and check for the presence of Target Dependencies - AmazingApp, in Compile Sources - AmazingAppUITests.swift.
It is good practice to isolate the various assembly options into separate Schemes. We create the scheme for our UI tests [XCode -> Product -> Scheme -> New Scheme] and give it the same name: AmazingAppUITests.
Build of the created scheme should include the Target of the main application - AmazingApp and Target UI tests - AmazingAppUITests - see screenshot
Next, create a new build configuration for UI tests. In Xcode, click on the project file, go to the Info section. Click on “+” and create a new configuration, for example XCtest. We will need this in the future in order to avoid dancing with a tambourine when it comes to code signing.
Your project has at least three Target: the main application, unit tests (because they are, right?) And the Target UI tests we created.
Go to Target AmazingApp, Build Settings tab, Code Signing Identity section. To configure XCtest, select iOS Developer. In the Code Signing Style section, select Manual. We have not generated the Provisioning profile yet, but a little later we will definitely return to it.
For Target AmazingAppUITests we do the same, but in the column Product Bundle Identifier we enter com.company.amazingappuitests.
2. Setting up a project in the Apple Developer Program
We go to the Apple Developer Program page, go to the Certificates, Identifiers & Profiles section and then to the App IDs column of the Identifiers item. Create a new App ID named AmazingAppUITests and bundleID com.company.amazingappuitests.
Now we have the opportunity to sign our tests with a separate certificate, but ... The build procedure for testing involves building the application itself and building the test runner. Accordingly, we are faced with the problem of signing two bundle IDs with one provisioning profile. Fortunately, there is a simple and elegant solution - Wildcard App ID. We repeat the procedure for creating a new App ID, but instead of the Explicit App ID, select Wildcard App ID as in the screenshot.
We’ve finished working with developer.apple.com at this point, but we won’t minimize the browser window. Go toa site with Fastlane documentation and read about Match from cover to cover utility.
An attentive reader noticed that to use this utility we will need a private repository and an account that has access to both the Apple Developer Program and Github. We create (if suddenly this is not) an account of the form InfrastructureAccount@your.company.domain, come up with a powerful password, register it in developer.apple.com, and appoint it as the project administrator. Next, give your account access to the github repository of your company and create a new private repository with a name like AmazingAppMatch.
3. Configuring Fastlane and the match utility
Open the terminal, go to the project folder and initialize fastlane as indicated in the official manual . After entering the command
$ fastlane initYou will be prompted to select available usage configurations. We select the fourth item - manual project setup.
A new fastlane directory has appeared in the project, in which there are two files - Appfile and Fastfile. In a nutshell - in the Appfile we store service data, and in Fastfile we write jobs, in Fastlane terminology called lanes. I recommend reading the official documentation: one , two .
Open the Appfile in your favorite text editor and bring it to the following form:
app_identifier "com.company.amazingapp" # Bundle ID
apple_dev_portal_id "infrastructureaccount@your.company.domain" # Созданный инфраструктурный аккаунт, имеющий право на редактирование iOS проекта в Apple Developer Program.
team_id "LSDY3IFJAY9" # Your Developer Portal Team ID
We return to the terminal and according to the official manual we begin to configure match.
$ fastlane match init
$ fastlane match development
Next, enter the requested data - repository, account, password, etc.
Important: the first time you start the match utility, you will be asked to enter a password to decrypt the repository. It is very important to keep this password, at the stage of setting up the CI server it will be useful to us!
A new file appeared in the fastlane folder - Matchfile. Open in your favorite text editor and bring to the form:
git_url("https://github.com/YourCompany/AmazingAppMatch") #Созданный приватный репозиторий для хранения сертификатов и профайлов.
type("development") # The default type, can be: appstore, adhoc, enterprise or development
app_identifier("com.company.amazingapp")
username("infrastructureaccount@your.company.domain") # Your Infrastructure account Apple Developer Portal username
We fill it in this way if we want to use match to sign builds for posting in Crashlytics and / or AppStore, i.e. to sign the bundle ID of your application.
But, as we recall, to create a test build we created a special Wildcard ID. Therefore, open Fastfile and enter a new lane:
lane :testing_build_for_firebase do
match(
type: "development",
readonly: true,
app_identifier: "com.company.*",
git_branch: "uitests" # создаем отдельный бранч для development сертификата для подписи тестовой сборки.
)
end
Save, enter in the terminal
fastlane testing_build_for_firebaseand see how fastlane created a new certificate and put it in the repository. Excellent!
Open Xcode. Now we have the necessary provisioning profile of the Match Development com.company. * Type, which must be specified in the Provisioning profile section for the AmazingApp and AmazingAppUITests targets.
It remains to add lane to build the tests. We go to the repository of the plugin project for fastlane, which facilitates the configuration of export to the Firebase Test Lab and follows the instructions.
Copy from the original example so that our lane testing_build_for_firebase eventually looks like this:
lane :testing_build_for_firebase do
match(
type: "development",
readonly: true,
app_identifier: "com.company.*",
git_branch: "uitests"
)
scan(
scheme: 'AmazingAppUITests', # UI Test scheme
clean: true, # Recommended: This would ensure the build would not include unnecessary files
skip_detect_devices: true, # Required
build_for_testing: true, # Required
sdk: 'iphoneos', # Required
should_zip_build_products: true, # Must be true to set the correct format for Firebase Test Lab
)
firebase_test_lab_ios_xctest(
gcp_project: 'AmazingAppUITests', # Your Google Cloud project name (к этой строчке вернемся позже)
devices: [ # Device(s) to run tests on
{
ios_model_id: 'iphonex', # Device model ID, see gcloud command above
ios_version_id: '12.0', # iOS version ID, see gcloud command above
locale: 'en_US', # Optional: default to en_US if not set
orientation: 'portrait' # Optional: default to portrait if not set
}
]
)
end
For complete information on configuring fastlane in CircleCI, I recommend reading the official documentation one, two .
Do not forget to add our config.yml with a new task:
build-for-firebase-test-lab:
macos:
xcode: "10.1.0"
working_directory: ~/project
shell: /bin/bash --login -o pipefail
steps:
- checkout
- attach_workspace:
at: ~/project
- run: sudo bundle install # обновляем зависимости
- run:
name: install gcloud-sdk # на mac машину необходимо установить gcloud
command: |
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null ; brew install caskroom/cask/brew-cask 2> /dev/null
brew cask install google-cloud-sdk
- run:
name: build app for testing
command: fastlane testing_build_for_firebase # запускаем lane сборки и отправки в firebase
4. But what about our test bench? Configure Firebase.
We proceed, in fact, to what the article was written for.
Perhaps your application uses Firebase on a free tariff plan, perhaps it doesn’t use it at all. There is absolutely no fundamental difference, because for testing needs we can create a separate project with a year of free use (cool, right?).
Log in to our infrastructure account (or any other, no difference), and go to the Firebase console page . Create a new project called AmazingAppUITests.
Important: In the previous step in the Fastfile in lane firebase_test_lab_ios_xctest, the gcp_project parameter must match the name of the project.
The default settings are quite fine with us.
Do not close the tab, under the same account, register in Gcloud- This is a necessary measure, since communication with Firebase occurs using the gcloud console interface.
Google gives $ 300 per year, which in the context of performing auto tests is equivalent to a year of free use of the service. We enter the payment data, wait for the test charge of $ 1 and get $ 300 in the account. After a year, the project will be automatically transferred to a free tariff plan, so you should not worry about a possible loss of money.
Let's go back to the tab with the Firebase project and transfer it to the Blaze tariff plan - now we have something to pay in case the limit is exceeded.
In the gcloud interface, select our Firebase project, select the "Catalog" main menu item and add the Cloud Testing API and Cloud Tools Result API.
Then go to the menu item "IAM and Administration" -> Service accounts -> Create a service account. We give the right to edit the project.
We create the API key in the JSON
format. We will need the downloaded JSON a bit later, but for now, the setting of Test Lab will be considered finished.
5. Configure CircleCI
A reasonable question is brewing - what to do with passwords? Reliably save our passwords and other sensitive data will help us the mechanism of the environment variables of our build-machine. In the settings of the CircleCI project, select Environment Variables
and start the following variables:
- key: GOOGLE_APPLICATION_CREDENTIALS
value: json contents of the gcloud service account key file - key: MATCH_PASSWORD
value: password for decrypting the github repository with certificates - key: FASTLANE_PASSWORD
value: password for the Apple Developer Portal infrastructure account
We save the changes, create a PR and send it to review for our team lead.
Summary
As a result of performing these simple manipulations, we got a good, stable working stand with the ability to record video on the device’s screen at the time of testing. In a test case, I specified an iPhone X device model, but the farm provides a wide selection of combinations of different models and versions of iOS.
The second part will be devoted to step-by-step configuration of the Firebase Test Lab for an Android project.