What’s New in Testing With Xcode 12

The idea behind test-driven development is simple: Think about what you want to accomplish, break that down into use cases and write tests to verify those cases. Only then do you get to implementation, with the goal of getting your tests to pass. As a result, you avoid unintentionally matching your tests to the code you’ve already written and force yourself to think about what you want to achieve before writing a single line of code.

Xcode 12 introduces improvements to enhance the testing experience, including:

• Improved test results with the new XCTIssue type.
• More robust test error handling.
• Breadcrumbs for test failures which point you directly to the point of failure.
• Streamlined test initialization code.
• The ability to skip tests instead of having to comment them out using XCTSkip.
• Execution Time Allowance to better handle hung tests.
• XCTest improvements.

It may be a bit painful to admit, but everyone writes buggy code sometimes. :] In this article, you’ll learn how to use these Xcode 12 testing topics to enhance your app’s quality and your development velocity.

Sound like a plan? Time to get started!

Getting Started

To demonstrate the testing concepts and tools discussed in this article, you’ll work on a Scattergories client app. Like the original game, you score points by naming objects from a set of categories, all of which have to start with a randomly selected letter.

Currently, the app only supports randomly selecting a letter. This is a great time to add tests, before you get to the heavy-lifting part of implementing gameplay.

Use the Download Materials button at the top or bottom of this article to download the starter and final Scattergories app used throughout this article. If you prefer, follow along using your own app.

Adding Testing Targets and a Test Plan

First, add two new testing targets: one for unit tests and another for UI tests. If you’re using your own app and already have test targets and a test plan, feel free to skip this section.

To add a testing target:

1. Click the project file.
2. Click + in the bottom left corner.
3. Select Unit Testing Bundle.
4. Click Next.

Leave all of the pre-filled fields on the next screen as-is. But if any are missing, fill them in making sure that the Language is set to Swift. Then click Finish.

Repeat the same steps described above to create a UI Testing target by selecting the UI Testing Bundle option when needed.

Next, create a test plan by selecting the current scheme and then Edit Scheme…

To add the testing targets, select the Test menu on the left side panel and make sure that the Info tab is selected along the top. Now do the following:

1. Click Convert to use Test Plans….
2. Select Create empty Test Plan.
3. Click Convert….
4. When prompted, leave the suggested values unchanged and click Save. Click Replace if prompted to replace the existing file.

Click Close to dismiss the window. Now add your test targets to the test plan:

1. Click the Test navigator.
2. Click the current test plan, Scattergories (Default).
3. In the menu that appears, click Edit Test Plan.

Now do the following:

1. Click the + in the bottom left corner.
2. In the dialog that appears, choose both ScattergoriesTests and ScattergoriesUITests.
3. Click Add.

That’s it! Now that everything is fully wired, you’re ready to dive into Xcode 12’s testing enhancements. So, what are you waiting for?

The Testing Feedback Loop

The testing feedback loop is a simple concept. It states you should:

1. First, write tests.
2. Then, run them.
3. Finally, interpret the results.

You should rinse and repeat these three steps until you gain enough confidence in both your tests and your app code to move on. Xcode 12 provides tools to ensure you always get faster feedback and your testing loop doesn’t break due to hung tests.

Writing Your First Test

Each test should focus on performing a single action and then asserting when the action completes. The assertion message should be specific, but not too specific. People, or scripts, reviewing the results should easily recognize multiple tests failing due to the same reason.

Go to ScattergoriesUITests.swift and add a new test at the end of the class:

func testInitialLetter() {
  let app = XCUIApplication()
  app.launch()

  XCTAssertEqual(
    app.staticTexts.count, 
    1, 
    "There should only be one text view - the letter")
}

This is a simple test with the sole goal of ensuring there’s exactly one label in your app.

Use the small Play button next to the test name to run it. It’s an empty diamond until you hover your mouse over it.

You’ll see the simulator open in the background, and the test run until it passes. The status indicator next to the test will turn to a green checkmark.

Isn’t it fun to see a green test? Don’t get used to it. You’ll start seeing red tests and learning how to fix them soon. :]

Getting Feedback Faster

One pitfall of asserting is asynchronous events. XCTest has built-in re-tries, but depending on the code you’re testing, it may not be enough. You’ll add a new test to demonstrate this.

Add the following code to ScattergoriesUITests.swift:

func testGetLetter() {
  let app = XCUIApplication()
  app.launch()

  app.buttons.element.tap()
  XCTAssertEqual(
    app.alerts.count, 
    1, 
    "An alert should appear once a letter is selected")
}

In this new test, you simulate a button click, which will iterate through the letters for a randomly pre-selected number of milliseconds. When the app reaches the final letter it displays an alert. You’re testing for the existence of this alert.

Run the test by clicking the small Play button next to the test name. When the test finishes running, you’ll notice it’s failing.

Open the Report navigator by typing Command-9. Select the failing test to display the associated test report in the center panel and expand the report by clicking the disclosure indicator. You’ll see the description you provided earlier for the assert call. Expanding this line exposes an Automatic Screenshot that was captured and attached to the assert message. Open the screenshot and you’ll see that there wasn’t an alert, so the test did actually fail.

The alert isn’t showing up because the test checks for its existence right after simulating the button click instead of giving the app time to run through the letters. The app takes a few milliseconds to run through the letters before selecting the last one. At that point, the user can get the selected letter and start playing.

You could mitigate this by waiting the maximum number of milliseconds the app can take to iterate through the letters and then checking for the alert. This lets the test pass or fail deterministically and in an environment you designed. However, in most cases, the alert presents much sooner than that, so why wait?

A better solution is to use waitForExistence(timeout:). If the expectation is true before the timeout, you save waiting time.

Replace the existing XCTAssertEqual line in testGetLetter() with:

XCTAssertTrue(
  app.alerts.element.waitForExistence(timeout: 10), 
  "Letter should be selected within the provided timeframe")

Run the updated test and see it now passes. More than that, look at what’s happening in the simulator while the app runs and see the test is waiting for the app to stop iterating through the letters before checking for the alert. In the test report, you can see how long the test waited for the result.

Don’t Let Hung Tests Break the Loop

Execution Time Allowance is a new customizable feature you can opt-in to, which enforces a time limit on each test. You can use this option to guard against test hangs and ensure the rest of your tests always complete running.

By default, each test gets ten minutes. You can customize this value in the test plan if you want to apply it to all tests. Use the executionTimeAllowance API if you only need it for specific tests or cases.

The value is provided as TimeInterval but rounded to the nearest minute. Values under 60 seconds are rounded up to one minute.

When a test exceeds the provided limit, Xcode will:

• Capture a spindump and attach it to the test results to shed light on what went wrong. The spindump shows you which functions each thread is spending the most time in and can help you better diagnose app stalls and hangs.
• Kill the test that hung.
• Restart the test runner so the rest of the test suite can execute.

To experiment with execution time allowance, you need to turn this option on:

1. Go to Edit Test Plan.
2. Select the Configurations tab.
3. Find the Test Timeouts option and set it to On.

Now, go back to the first test you wrote, testInitialLetter(), and add these two lines, right before calling XCTAssertEqual:

executionTimeAllowance = 30
sleep(70)

This code sets the execution time allowance for this test to 30, which will be rounded up to one minute, then sleeps for a little over a minute to ensure the test fails. Run the test now and notice it hangs for about a minute before failing.

Once the test finishes with a failure, go to the test report and see the attached spindump. This can be a life-changer in cases of complex tests with multiple threads when you’re trying to figure out what went wrong.

There are other ways to set the execution time allowance, and Xcode has a clear precedence order for this:

1. executionTimeAllowance: You used this in the example above. It takes top priority.
2. xcodebuild option: -default-test-execution-time-allowance.
3. Test plan setting.
4. Default value (10 minutes): Lowest priority.

Speeding Things Up Even More

If you have a large test suite and are running it frequently, which you definitely should, you may find it takes a long time to complete. Xcode 12 introduces another great way to speed things up: Parallel distributed testing.

Parallel distributed testing involves running a test plan on multiple devices in parallel. Once a specific device finishes running a test, Xcode gives it a new test until there aren’t any tests left. Even by adding one additional device, you can get a 30% speed-up compared to testing on a single device.

While a deep dive into implementing Parallel distributed testing is beyond the scope of this article, there are a few principles worth stressing:

• The allocation of tests to run destinations is non-deterministic. If the logic you’re testing is device or OS specific, using this technique can cause issues such as unexpected results or skipped tests.
• Ideally, you should use the same kinds of devices and OS versions per test suite run.
• If you’re using multiple types of devices or OS versions, you should aim to run tests that are device and OS agnostic, for example, pure business logic.

Failure = Success

It’s always fun to see green tests, but great tests catch bugs. You write them once then run them many times throughout the development cycle. You should expect and plan for them to fail.

If written well, failed tests help you improve the quality of your app and the confidence you have in it. New APIs and enhanced test failure reporting UI in Xcode 12 make researching and resolving test failures more efficient.

If the line of failure in the failed test is gray then the failure happened underneath the annotated line, but not at that line itself. If you then go to the test report to explore this further, you’ll see the call stack. Clicking a specific row in the call stack will take you to the exact code location.

If the annotation is red, this is the actual point of failure.

XCTest has always recorded failures as four discrete values:

• Failure message
• File path
• Line number
• Expected flag

Xcode 12 introduces XCTIssue, a new object which encapsulates these four values and adds additional details and capabilities to improve your debugging experience. These include:

• Issue type for classifying different kinds of issues that may occur during testing.
• Detailed description that may include transient data, such as numbers, object identifiers and timestamps, to help diagnose the issue.
• An optional associated error.
• An array of attachments, such as files, images, screenshots, data blobs or ZIP files, to make it possible to assess customer diagnostics for test failures.
• sourceCodeContext, which captures and symbolicates call stacks to provide additional context for failures in complex test code.

Setting Up and Tearing Down

Xcode 11.4 provides new APIs for throwing from setUp and tearDown: setUpWithError() and tearDownWithError(). You’ll find both of these methods in the template for new tests.

These new APIs let you have a simplified and tailored flow instead of boilerplate code. Source code locations are included as part of thrown errors starting with iOS and tvOS 13.4 and macOS 10.15.4, saving you the extra handling code for attaching these details as parameters.

Use setUpWithError() to state the assumptions needed for your tests to run properly and set the app’s state to match those assumptions. Previous tests may have changed the app’s state or modified data the test relies on, so this is a good opportunity to start with a clean slate.

You can leverage launchArguments and launchEnvironments to set the needed state. For example, you might bypass 2FA during testing or bypass the main tab to start with the menu tab. This is also a good opportunity to adopt product changes to keep your tests focused.

tearDownWithError() is also a good opportunity to take advantage of the new error management. You can use it to collect additional logging during tear-down, including some analysis of the failures, while the data is still available. You can also use this as an opportunity to reset the environment from the changes made during the most recent test run.

Adding Unit Tests

You wrote a couple of UI tests. Now it’s time to add a couple of unit tests.

Go to ScattergoriesTests.swift and add these two tests:

func testCategories() throws {
  let gameData = GameData()
  XCTAssertEqual(
    gameData.categories.count, 
    numOfCategories, 
    "The game must have \(numOfCategories) categories")
}

func testPlayerData() throws {
  let gameData = GameData()

  // TODO: Implement logic to test player words 
  //   (depending on actual game implementation)
  XCTFail("Player testing logic not implemented yet")
}

Add this to the import section at the top of the file:

@testable import Scattergories

Also, be sure to add the numOfCategories constant as the first line of the ScattergoriesTests class implementation:

let numOfCategories: Int = 14

These are two simple tests to verify your basic game logic is working as expected. testCategories() verifies your game data has exactly 14 categories. This is set as a constant in case the number of categories changes in the future. In this case, you don’t want to go over all of your tests, but rather change a constant to affect all of them.

The second test focuses on the game logic. Even though you haven’t implemented the logic yet, you want to have a test, or at least a test stub, ready for it so:

• You’ll remember this needs to be properly implemented and tested, and,
• Decrease your chances of matching the tests to the code you’ve already written, but rather to the expected behavior.

Run your newly added unit tests by clicking Play for the test suite:

You’ll see the first test passes. The second one fails, which makes sense because you haven’t implemented the game logic yet, and there’s an explicit XCTFail call there.

On one hand, you want to adhere to test-driven development, or TDD, principles and write your tests early in the process. On the other hand, you don’t want to have red tests you can’t address yet. This can create blindness and prevent you from noticing real red tests.

So what can you do? XCTSkip to the rescue.

XCTSkip

Apple introduced XCTSkip in Xcode 11.4. It provides a new possible test result. Now, instead of pass or fail, tests can also have the result skip.

An annotation in the source code shows you where and why the test was skipped. The result also appears as an indication next to the test, like pass and fail results. You can even filter the report to show only the skipped tests, or passed or failed ones.

Using XCTSkip resolves the conflict of whether to implement a test, and then possibly forget about it, or have a failed test as part of the test run, which may desensitize you to red tests and prevent you from noticing real reds.

XCTSkip lets you enjoy both worlds. While you can still see the test didn’t pass, it won’t be red and therefore won’t distract from anything that may need your immediate attention. You can also document why the test isn’t running using an optional message.

You’ll usually use XCTSkip for skipping tests that aren’t relevant for the platform or OS the test is currently running on, or for cases when you haven’t fully implemented the functionality yet. Sometimes, you may have a test that started failing but you can’t fix for a while. Skipping such tests prevents you from having to keep triaging this failure over and over, while at the same time ensuring you won’t lose track of the test by disabling it.

To use XCTSkip on the test for the not-yet-implemented game logic, add the following line to testPlayerData(), right above the // TODO: line:

try XCTSkipUnless(
  !gameData.playerWords.isEmpty,
  "The player must have more than 0 words")

Now, once your player has more than 0 words, which means you got to a point where your game logic is at least partially implemented, the test will run. Until then, the test will skip.

Run your unit tests and you’ll see the test is indeed skipped, preventing you from having to deal with a red test you can’t do anything about yet.

Testing Dos and Don’ts

Here are some useful tips that can make your testing experience with Xcode better and more efficient:

• Each test should have one specific goal in mind. The test name should reflect this goal for easy triaging after a failed or successful run.
• Command-U runs all of the tests on your current test plan. Control-Option-Command-G repeats the previous test run.
• Use enums for all string values because UI elements labels change often. Doing so will make it much easier to fix tests to match changed UI.
• Factor common code into helper methods. This is especially true given Xcode 12’s enhanced error handling features, such as including call stacks for each test failure. These features will help you focus on fine-tuning your test logic.
• Always throw from shared code rather than asserting. Shared code is run from multiple tests and, in some cases, you may be purposefully testing negative cases.
• Don’t add file paths to an assert comment. Instead, add any information needed to triage or debug the issue as an attachment. It’ll be much easier to triage the failure later.
• Unwrap optionals. When there’s a failure to unwrap optionals in a continuous integration bundle, you get a message saying test crashed with signal ill.
• Parallel distributed testing isn’t the same as parallel destination testing, which simultaneously runs your tests on more OS versions and devices and has been available since 2018. Use both methods wisely.

Where to Go From Here?

Use the Download Materials button at the top or bottom of this tutorial to download the final Scattergories app.

Now that you’ve had a chance to experiment with some of Xcode 12’s new testing enhancements, you can use this new found knowledge to add tests to improve your own app. Take a look at some WWDC’20 sessions, discussing these capabilities:

• Triage test failures with XCTIssue
• Get your tests results faster
• Write tests to fail
• XCTSkip your tests

And, of course, there is also great testing-related content right here on raywenderlich.com. Specifically:

• iOS Test-Driven Development by Tutorials
• Testing in iOS video course
• What’s new in Xcode 11: Testing This screencast is useful if you want to learn more about test plans and how to use them.

I encourage you to use Xcode 12’s new testing enhancements to find new and creative ways to get your app to fail, crash, and burn so you can fix everything before it reaches real users. If you have any questions or comments, please join the forum below.