WWDC19: Getting Started with Test Plan for XCTest

Hello, Habr! I present to you the translation of the article “WWDC19: Getting Started with Test Plan for XCTest” by Shashikant Jagtap.



At the last WWDC conference , Apple showed off cool new features for developers. Xcode 11 also adds some awesome features. You can read about them in release notes . One of the most significant tools announced this year is Test Plan for XCTest and Xcode UI tests. In this article, we will examine in detail how the functionality of Test Plan will work with XCTest. If you want to know more details, watch the “Testing on Xcode” session video .

Xcode Test Plan

Prior to Xcode 11, test configuration was part of Xcode schemas. If the developers wanted to configure different test suites for different conditions, they either created a new circuit or edited an existing one for specific test needs. The test configuration was closely related to Xcode schemes, so I had to create many schemes for different tasks during testing.
New functionality in Xcode 11 allows developers and QA engineers to customize tests to suit their needs. Test Plan allows you to determine:

• what tests to run in the assembly;
• how to run these tests (for example, in random or alphabetical order);
• How to manage test artifacts (for example, attachments and screenshots);
• how to use runtime tools as part of testing.

With Xcode Test Plan, you can configure test suites independently of Xcode schemas. This is a significant improvement in testing technology that can take your tests to the next level. Now let's take a look at the Xcode Test plan in action.

Scenario

Suppose we create a test plan for functional tests with the following requirements:

• Functional tests should include kits for smoke and regression tests;
• Smoke tests should be run in London (UK) and regression tests in San Francisco (USA);
• Smoke tests should be run in alphabetical order, and regression tests should be run in random order to catch problems between tests.

We will create this test plan for Xcode for various Apple platforms.

Creating a Test Plan in Xcode

Xcode 11 allows us to define these requirements in Test Plan. Xcode allows you to create a Test Plan from the current schema configuration. When you edit the circuit and proceed to the current test steps, Xcode displays an option to create a Test Plan. You can create a Test Plan with the following settings:

• Blank Test Plan : Creates an empty template and allows you to make your own plan;
• Test Plan from Scheme : Takes a circuit configuration and creates a Test Plan;
• Choose Test Plan : If you already have a Test Plan created, you can select and add it to the chart.

To familiarize yourself, create a Test Plan from the circuit configuration.



You need to name Test Plan and specify the correct test target. When a Test Plan is created, it is attached to the circuit. However, the schema can override any Test Plan created after using the flag when invoking the Xcodebuild command. After creating a Test Plan, you can attach it to a specific target. Let's add the current plan to the target with UI tests and give it a name.



You will find a new file with the extension functional.xctestplan. This is your Test Plan. It can be edited if necessary. You can make a composite Test Plan in the same target, depending on the needs of the tests. At this point, we have an available Test Plan for our target with UI tests.

Test plan file

The Test Plan file is a simple json-like configuration file that contains information on how to configure your tests to run independently of any circuit. Here are the various test configuration settings available in Test Plan. The basic Test Plan contains three key elements:

• Configurations
• Test Targets
• Settings

Each configuration in Test Plan contains a set of tests tied to it with different settings. In accordance with the requirements of our test plan, we will need two configurations: one for smoke tests, the other for regression tests.
Now let's open the functional.xctestplan file in Xcode and rename configuration1 to smoke-uk, then click on the “+” button at the top of the plan and add another regression-usa configuration.
Now we have two configurations in Test Plan and Test Plan is already tied to a target with UI tests. Add some settings to each configuration. When smoke-uk configuration is selected, we can change the location to London and make other changes to the settings. Similarly, change the configuration of regression-usa. This is what Test Plan will look like to trigger a regression:



Settings available in Test Plan:

• Arguments - allows you to override the launch arguments and launch environment for tests;
• Localization - allows you to configure tests for various geolocations, locales and languages. This is a very cool setting for testing internationalization and localization;
• UI Testing / Attachment / Test Execution / Code Coverage - specific options for UI testing and test artifacts. You can decide whether to leave artifacts or delete them as soon as possible. You can also choose the order in which the tests are run and the code coverage calculation;
• Xcode Code Diagnostic tools - several Xcode tools for diagnosing code, such as Address Sanitizer or Thread Sanitizer, are available for searching runtime errors. You can also use other options that help you look for memory leaks, etc.

Creating a Test Plan from scratch

If you do not want to create a Test Plan using the Xcode scheme, then you can always create a new Test Plan through Xcode → Product → Test Plan.



Having created Test Plan in this way, we will attach it to some test target by adding tests to it. In the example below, we created a new Test Plan and added tests to it.

Creating a Test Plan from the source

As mentioned earlier, Xcode Test Plan is just a configuration file, where we installed all the configurations we defined. In our scenario, we created functional.xctestplan, and the source code for it looks something like this:

{
  "configurations" : [
    {
      "name" : "smoke-uk",
      "options" : {
        "language" : "en-GB",
        "locationScenario" : {
          "identifier" : "London, England",
          "referenceType" : "built-in"
        },
        "region" : "GB",
        "testExecutionOrdering" : "lexical"
      }
    },
    {
      "name" : "Regression-usa",
      "options" : {
        "addressSanitizer" : {
          "enabled" : true
        },
        "language" : "en",
        "locationScenario" : {
          "identifier" : "San Francisco, CA, USA",
          "referenceType" : "built-in"
        },
        "region" : "US",
        "testExecutionOrdering" : "random"
      }
    }
  ],
  "defaultOptions" : {
    "codeCoverage" : false
  },
  "testTargets" : [
    {
      "target" : {
        "containerPath" : "container:XCTestPlan.xcodeproj",
        "identifier" : "9B0E393A22A6EB5400F666A1",
        "name" : "XCTestPlanUITests"
      }
    }
  ],
  "version" : 1
}

You can create the configuration in this way and save it to a file with the extension .xctestplan in Xcode 11.

Launch Test Plan

Now we understand the process of creating a Test Plan file in Xcode. Let's see how to run Test Plan and get reports in Xcode. You can run tests through Xcode -> Product -> Test as usual, by running the attached Test Plans on the schema. In our case, this will launch both test configurations - smoke and regression.
There are several command line options in Xcodebuild to view and run Test Plans. You can run the following command:

$ Xcodebuild -showTestPlans -scheme YOUR_SCEME

It will display all Test Plans for the current circuit. You can also run a separate test:

$ Xcodebuild -project XCTestPlan.Xcodeproj/ -scheme XCTestPlan -destination 'platform=iOS Simulator,OS=13.0,name=iPhone XR'  test -testPlan functional

This script will run the test using the Test Plan configuration.

Test Plan Reports

After running the tests for a specific Test Plan, you can see the good reports generated in Xcode. Reports are generated separately for each configuration.



In Xcode, in the test navigator section, you can filter reports by Test Plans. Xcode Test Plan reports can be configured on the Xcode server.

Demo source code available on Github: XCTestPlan

Xcode Test Plan: Application

Briefly about where the Xcode Test Plan can be used and how it changes depending on the configuration of the circuit. Here are some examples of using the Xcode Test Plan:

• Various types of testing, as was shown in the demo: smoke, regression, etc .;
• Testing the internationalization and localization of the application, for example, if you want to test your application for several languages, you can create various configurations;
• Running heavy tests, such as performance and stress tests, and separating them from functional tests. You can configure tests for work along with code diagnostic tools or other non-functional tests.

Many other scenarios are also possible where Xcode Test Plans can help expand the testing process.

Conclusion

With the Xcode Test Plan, XCTest can be taken to the next level with the ability to manage your configuration and run your tests. Xcode Test Plans gives you better control over the Apple-based testing platform.