10. Face Anchors

Written by Chris Language

This chapter continues from the point where the previous one left off. Thanks to SwiftUI, the AR Funny Face app now sports a very basic UI. In this chapter, you’ll continue to focus on that facial recognition component by using face anchors in RealityKit.

You’ll also get to build some funny scenes with these crazy props:

A humongous pair of sunglasses, a glass eyeball and one epic mustache. With these cool props at one’s disposal, the AR Funny Face app is off to a great start.

Note: Feel free to continue with your own final project from the previous chapter. If you skipped a few things, load the starter project from starter/ARFunnyFace/ARFunnyFace.xcodeproj before you continue.

What are face anchors?

Hanging an anchor from one’s face sounds extremely painful. Luckily, the type of anchor we’re referring to here is an AR Face Anchor — undeniably one of the coolest types of anchors available. Thanks to Reality Composer, creating face anchors is super easy.

By using the TrueDepth front-facing camera, face anchors provide information about the user’s facial position, orientation, topology and facial expression.

Unfortunately, you can only use face tracking if you have a device equipped with a TrueDepth front-facing camera. If the device is Face-Id capable, you’re good to go.

When the camera detects a face, an anchor gets added slightly behind the nose in the middle of the head.

Here, the cute monkey head demonstrates how a face anchor is created after a face is detected.

It’s also important to know that the anchor uses a right-handed coordinate system measured in meters.

Here’s a breakdown of each axis:

• X-Axis: The red arrow pointing right represents this axis.
• Y-Axis: The green arrow pointing up represents this axis.
• Z-Axis: The blue arrow pointing forward represents this axis.

Creating face anchors

It’s time to dive into the action and build some face anchor scenes with the provided props.

With the AR Funny Face app project open, select Experience.rcproject then select Open in Reality Composer.

The project opens in Reality Composer with the default scene selected. The scene contains a cube object.

Select the default scene and open the Properties panel. Rename the Scene to Eyes. Change the Scene Anchor Type to Face and disable the Scene Physics by setting Objects collide with to Nothing. Finally, select the default Cube within the scene and delete it.

Because you set the Anchor Type to Face, you’ll now see a white face mask in the center of the scene. The mask represents a detected face in scale. The mask serves as a visual guide to know where you can place objects in relation to the detected face.

Add an object to the scene, but select the Import option to select a custom object. Find and select starter/resources/Eyeball.usdz then select Import to complete the process.

This imports a beautiful, shiny glass eyeball into the scene, hiding behind the white mask. Rotate until you can see the eyeball behind the mask. Select it, then open the Properties panel.

Rename the object to Eye_R, which represents the right eye. Under the Transform section, set the Position to (X:-3.5cm, Y:4cm, Z:-3cm) and set the Scale to 170%.

Now, you have a good idea of how the end product will look when you view the scene. You need at least one more eye, though.

With the right eyeball still selected, press Command-C then Command-V to create a quick copy.

Rename the object to Eye_L, which represents the left eye. Under the Transform section, set the Position to (X:3.5cm, Y:4cm, Z:-3cm) and set the Scale to 170%.

The final result resembles Homer Simpson. Perfect!

Now that the eyes are done, it’s time to add the other props.

Creating multiple scenes

To add more props, you have to create multiple scenes within the Reality Composer project. Each scene will house a single facial prop. To switch between the different props, you simply need to switch the different scenes.

That sounds like a great plan, so get to it!

With the Scenes panel open, add a new scene to the project by selecting the + button at the top-left of the panel.

Each scene has its own anchor, so when you create a new scene, you’ll need to select an anchor type for it. When asked, choose Face as the anchor type and make sure to uncheck Use template content to create the scene with no content.

Rename the Scene to Glasses and set Objects collide with to Nothing.

Add an object to the scene then Import the starter/resources/Glasses.usdz.

Rotate the scene until you can see the glasses behind the face mask, then select them and open the Properties panel. Under the Transform section, set the Position to (X:0cm, Y:6cm, Z:-3cm:). To make the glasses nice and big, set the Scale to 140%.

All right Willy Wonka, you’re almost done. There’s one more prop left to add to the project.

Following the same process as before, make sure the Scenes panel is open, then add another scene to the project by clicking the + button at the top-left of the panel.

When asked, set the Anchor Type to Face. Don’t forget to make sure you’ve unchecked Use template content.

Rename the scene to Mustache and add an object to the scene, then import starter/resources/Mustache.usdz.

To wrap things up, rename the object to Mustache, set the Position to (X:0cm, Y:6.5cm, Z:3cm) and set the Scale to 130%.

That’s it, you’re all done. Your Reality Composer project now contains three different face anchor scenes, one for each prop.

Save your changes, close Reality Composer and return to Xcode.

You’ll now see your three scenes within Xcode.

Code generation

Reality Composer is tightly integrated into Xcode. When you build the project, Xcode will inspect all the associated Reality files within the project and generate Swift code.

The generated code provides strongly-typed access to all the content within the Reality file. It also provides direct access to invoke triggers for custom actions within your code.

In this case, Xcode generates an Experience.swift file with strongly-typed access to the three scenes you created within the Reality file.

Fixing the project

You’ll look at the coding side of the project next. When you recompile your project, it generates an error.

That’s because you removed the default box scene. The generated Experience.swift code no longer has any reference to Experience.loadBox().

Select ContentView.swift. You’ll add some code to it next.

Find makeUIView(context:) and replace its contents with the following block of code:

arView = ARView(frame: .zero)
return arView

This initializes arView. As an added bonus, you just got rid of the code that was causing a compiler error.

Switching to the front-facing camera

When the app starts, you need to manually switch to the front-facing camera. To do that, you’ll need a little help from ARKit.

ARKit is the technology behind RealityKit; you’ll learn more about it later in the book.

Add the following import to the top of ContentView.swift:

import ARKit

Great, now you have low-level access to some additional content.

AR session

Before moving on, you need to learn about the AR session, which you can access via ARView.session.

The AR session object is the key technology responsible for motion tracking and image processing. It’s session-based, so you have to create an AR session instance, then you have to run that session to start the AR tracking process.

AR configuration

Before starting an AR session, you have to create an AR session configuration. You use this configuration to establish the connection between the real world, where your device is, and the virtual 3D world, where your virtual content is.

There are six types of configurations:

• AROrientationTrackingConfiguration: Basic three degrees of freedom (3DOF) tracking, which uses the back camera.

• ARWorldTrackingConfiguration: Six degrees of freedom (6DOF) tracking, which also tracks people, known images and objects. It uses the back camera.

• ARBodyTrackingConfiguration: Tracks human bodies only. It uses the back camera.

• ARImageTrackingConfiguration: Tracks known images only, using the back camera.

• ARObjectScanningConfiguration: Use this to scan 3D objects you want to track. It uses the back camera.

• ARFaceTrackingConfiguration: Tracks faces and facial expressions only. This uses the front camera.

The one you’ll use now is ARFaceTrackingConfiguration. When you run this configuration, your app will switch to the front-facing camera.

Add the following block of code to updateView(_:context:):

// 1
let arConfiguration = ARFaceTrackingConfiguration()
// 2
uiView.session.run(arConfiguration, 
  options:[.resetTracking, .removeExistingAnchors])

Take a closer look at what you’re doing with this code:

1. First, you create a new instance of ARFaceTrackingConfiguration within arConfiguration. The configuration now contains the necessary information to let the AR session know that you want to start tracking faces.

2. This starts the AR session with the newly-created AR configuration along with a few additional options. resetTracking indicates that you want to restart the AR tracking process. removeExistingAnchors removes all existing anchors, if there are any.

Switching between multiple scenes

Finally, you need to switch between the three scenes when the user presses the Previous or Next buttons.

Add the following block of code to the bottom of updateView(:context:):

switch(propId) {

  case 0: // Eyes
    let arAnchor = try! Experience.loadEyes()
    uiView.scene.anchors.append(arAnchor)
    break
            
  case 1: // Glasses
    let arAnchor = try! Experience.loadGlasses()
    uiView.scene.anchors.append(arAnchor)
    break
            
  case 2: // Mustache
    let arAnchor = try! Experience.loadMustache()
    uiView.scene.anchors.append(arAnchor)
    break
  
  default:
    break
}

Every time the user presses Next or Previous, the value of propId increases or decreases by 1, which invalidates the scene and makes a call to updateView(:context:). The switch statement then inspects the value of propId to switch to the appropriate scene.

First, you initialize arAnchor by loading the corresponding anchor scene from Experience. Once loaded, you append the anchor to arView.scene.anchors, which presents that particular scene in the view.

Testing the app

Finally, you’re ready to do your very first build and run. Before you do, connect your physical device to your machine and select it in Xcode.

Note: You will experience compiler issues if you don’t connect your physical device to Xcode and select it as the build destination.

Ready to build and run? Go for it!

Excellent, the app started and you can use the Next and Previous buttons to select the different props.

Although it looks kind of cool, there’s a small problem — the props get stuck and never go away.

Manually removing anchors

Every time you switch from one scene to another, you load an anchor that appends to ARView.Scene.anchors. If you continue adding multiple anchors, you’ll end up with multiple props stacked on top of each other.

To solve that problem, you need to manually remove the previously-appended anchors before appending a new one.

Add the following line of code to the top of updateView(_:context:):

arView.scene.anchors.removeAll()

This removes all the available anchors within arView.scene.anchors.

Remember that every time the user selects the Next or Previous button, it invalidates the scene and makes a call to updateView(_:context:). So every time the user switches to a new prop, the app removes the previous prop (face anchor) before appending the new one.

Do another build and run to test the app again.

Excellent, you can now swap between props without any issues.

Note: You can find the final project at final/ARFunnyFace/ARFunnyFace.xcodeproj.

Key points

Congratulations, you’ve reached the end of this chapter and your AR Funny Face app looks great. Take some selfies of yourself and some friends and try the funny props.

Here are some of the key points you covered in this chapter:

• Face Anchors: You now have a good idea of what face anchors are and how to use them within your scenes.

• Creating Face Anchors: Thanks to Reality Creator, creating face anchors is really easy.

• Creating Multiple Scenes: A single Reality Composer can contain multiple scenes, which are really easy to add.

• Code Generation: Reality Composer is tightly integrated into Xcode. When compiling, Xcode generates code that provides strongly-typed access to the content of the scenes and objects within the Reality file.

• AR Session & Configuration: You learned about AR session and how to create your own AR configurations.

• Switching Between Scenes: Finally, you learned how easy it is to switch between multiple scenes within a RealityKit app.

In the next chapter, you’ll extend the AR Funny Face app with one more scene, where you get to control a giant robot head with your own facial expressions!