Advanced Semantics

Semantics Tree

Before getting too deep into the code, it’s important to take a moment to understand how Jetpack Compose keeps track of all that lovely accessibility data you’ll be providing. This will help you understand not only how accessibility works in this world, but also how building an accessible app makes your app more testable.

In short, alongside the view node tree, there is a semantics tree with a very similar structure but slightly different information. While the view node tree includes details such as layout information, the semantics tree contains information that accessibility services need to understand. This includes descriptions, labels, and any actions exposed to the service.

Viewing the Semantics Tree

Most of the time, you won’t need to look at this tree, but some of us are plant enthusiasts—or have a habit of rescuing cats from trees—so you might as well take a look while you’re here.

There are a couple of ways to do this. Most of the time, you’ll use the built-in tools in Android Studio to find what you’re looking for.

To start, in the Running Devices section of Android Studio, toggle on the Layout Inspector.

From there, you can navigate the tree to view the properties of different elements on the screen. In the image below, you can see that the favorite icon toggle is a Toggle with the content description Not Favorite.

Testing Semantics

As hinted earlier, building for accessibility can also be helpful for testing. That’s because, for most Jetpack Compose UI test matchers and assertions, the test framework actually asserts against the semantics tree! Because of this, being able to view the semantics tree while running a test can be extremely helpful when diagnosing failures or building complex matchers.

Note: Need a refresher on Jetpack Compose UI Testing? Check out this chapter or this codelab.

Let’s walk through an example test. Start by opening SemanticsTest.kt. It’s an empty test, ready for you to add your assertions.

Add the following test to the file before walking through each assertion step by step:

@Test
fun detailScreen_verifyToggleAndHeading() {
  composeTestRule.setContent {
    CatNapperTheme { CatNapperApp() }
  }

  // 1. Navigate to the detail screen for the first cat
  composeTestRule.onNodeWithText("Luna")
    .performClick()

  // 2. Verify that the "Favorite" icon is toggleable
  composeTestRule
    .onNode(hasContentDescription("Favorite", substring = true, ignoreCase = true))
    .assertIsToggleable()

  // 3. Verify that the "Naps:" heading is displayed
  composeTestRule.onNode(isHeading() and hasText("Naps:"))
    .assertIsDisplayed()
}

At the beginning of the test, you set the content the test will run against. In this case, it’s the full app composable content, CatNapperApp(). Next comes the meaty bits.

1. Navigate to the detail screen: By matching on the list item for the cat named “Luna”, you’re able to emulate a click on that item to navigate where you want to go. Thanks, clickable Modifier!

2. Ensure the favorite icon is toggleable: Notice two things here. First, you can match based on the content description, which makes testing your icons easier. Second, you assert that it’s toggleable to prevent accessibility regressions. There are also assertions to verify the value of the toggle to further test the behavior of your app.

3. Check for the “Naps” heading: Here, you’re looking for both the text and this isHeading() thing (you’ll learn more in a moment) to make sure it’s there.

Build and run the test.

Oh no, it fails! There’s no heading in sight. This example surprised you with some Test Driven Development, but you’ve got cat-like reflexes and will land on your feet. Time to learn about this new semantic.

Adding Heading Semantics

At the beginning of this lesson, we promised you’d learn additional semantics you can use. Now’s the time!

There’s a bunch of different extensions you can use within that .semantics {} modifier. One of these is heading(). When traversing a screen with a lot of content, it allows the user to skip across headings to find the section they’re interested in. This screen is pretty small, but you’ll add it anyway to learn how it works.

Find the Naps: text on the detail screen and add the following Modifier:

Text(
  text = stringResource(id = R.string.details_naps),
  style = MaterialTheme.typography.h6,
  modifier = Modifier.semantics { heading() }
)

You can add the semantic to the cat’s name, too, if you want to make the example more interesting.

If you run the app and navigate to the detail screen, you can now move between headings.

To use this TalkBack feature, “swipe up or down with three fingers”, or “swipe up AND down with one finger”, until you find Headings. Then, “swipe up or down with one finger” to navigate between them.

Note: If you’re using an emulator you probably wont be able to perform these gestures. If that’s the case, or if this gestures don’t seem to do it, you can see and change what gesture it uses under Settings -> Accessibility -> TalkBack -> Settings -> Customize Gestures. The action you’re looking for is Previous/Next reading control.

Run the test again, and it will pass.

Printing the Node Tree

Sometimes, you need a few extra hints about what’s happening in the semantics node tree when writing effective tests. Thankfully, there’s a handy way to do this.

Anywhere in your test function, add the following and run the test:

composeTestRule.onRoot()
  .printToLog("TESTING123")

After the test runs, look at Logcat for the device and filter for the TESTING123 tag to make it easier to find. You’ll see something that looks a lot like this:

It’s a node tree representation of the whole screen, complete with the semantics you’ve added. This can be really helpful, especially when trying to understand node children and parents, along with their properties, when writing tests.

Collection Semantics

If you haven’t already, you’ll likely come across a time where you’re displaying a lot of information on a screen. Sometimes in a list, and sometimes in a more complicated arrangement.

Most of the time you will use something like LazyColumn, which takes care of collection semantics for you. You can see an example of this on the home screen with the list of cats.

Accessibility services automatically know it’s a list and how many items it contains, allowing that information to be presented via TalkBack.

If you’re building a complicated screen, maybe a custom table view or multi-dimensional graph, it’s helpful to add this information manually. It’s a simplified example, but you can use these semantics to improve the log of naps on the detail screen.

Keep your ears perked up, because there are two steps to this part. On the detail screen, look for the Column under the naps header. Add the following semantic:

Modifier.semantics {
  collectionInfo = CollectionInfo(cat.naps.size, 1)
}

This semantic communicates that the column is a collection with a specific size and only one column.

Then, on each item, add the following semantics modifier:

Modifier.semantics {
  collectionItemInfo = CollectionItemInfo(
    rowIndex = index,
    rowSpan = 1,
    columnIndex = 0,
    columnSpan = 1
  )
}

For each item, this describes its position within the collection.

The updated Column will look something like this:

Column(modifier = Modifier.semantics {
  collectionInfo = CollectionInfo(cat.naps.size, 1)
}) {
  val formatter = DateTimeFormatter.ofPattern("h:mm a")
  cat.naps.forEachIndexed { index, nap ->
    Text(
      text = "${nap.start.format(formatter)} - ${nap.end.format(formatter)}",
      modifier = Modifier.semantics {
        collectionItemInfo = CollectionItemInfo(
          rowIndex = index,
          rowSpan = 1,
          columnIndex = 0,
          columnSpan = 1
        )
      }
    )
  }
}

Build and run the app with TalkBack and see the result.

Now, imagine if this was a complicated table of a bunch of information in this log. Navigating it is much easier now! You’re the cat’s meow.