11. Facial Blend Shapes

Written by Chris Language

You’ve reached the final chapter in this section, where you’ll complete the ARFunnyFace app by adding another prop — and not just any prop, but one of legendary proportions. Prepare to come face-to-face with the mighty Green Robot Head!

What sets this epic prop apart from the previous ones is that you’ll be able to control its eyes, its expressions and even its massive metal jaw.

Like a true puppeteer, you’ll be able to animate this robot head using your own facial movements and expressions, thanks to facial blend shapes. Awesome!

What Are Facial Blend Shapes?

ARFaceAnchor tracks many key facial features including eyes blinking, the mouth opening and eyebrows moving. These key tracking points are known as blend shapes.

You can easily use these blend shapes to animate 2D or 3D characters to mimic the user’s facial expressions.

Here’s an example of a 2D smiley character that animates by tracking the user’s eyes and mouth.

Each key tracking feature is represented by a floating point number that indicates the current position of the corresponding facial feature.

These blend shape values range from 0.0, indicating a neutral position to 1.0, indicating the maximum position. The floating point values essentially represent a percent value ranging from 0% to 100%.

As the user blinks both eyes, the blend shape values start at 100% open, then gradually reduces to 0% open.

The mouth works the same way, starting at 100% open then reducing to 0% open.

You use the percentage values to animate the corresponding facial features from a 100% open position to a 0% open position — aka, closed.

You can even prerecord the blend shape data, which you can play back at a later time to animate your game characters, for example. Sweet!

Building the Robot

Next, it’s time to build the Mighty Green Robot head. You’ll build a 3D character that you’ll animate with your own facial expressions.

Open starter/ARFunnyFace/ARFunnyFace.xcodeproj in Xcode, then select Experience.rcproject and open it in Reality Composer.

Open the Scenes panel and create a new scene that uses a Face Anchor. With the Properties panel open, rename the scene to Robot.

Now, you’ll add a Basic ▸ Capsule to the Robot scene.

Under the Transform section, set Position to (X:0, Y:0, Z:0), Rotation to (X:90°, Y:0°, Z:0°) and leave Scale at 100%.

Finally, go to the Look section, choose Matte Paint for the Material and set the Material Color to Black. Set the Capsule Diameter to 18 cm and the Height to 22 cm.

Both the face mask in the scene and the user’s face should now be fully obscured.

Now you’ll create the rest of the robot head, which consists of four basic parts: a RobotEye, a RobotEyelid, a RobotJaw and a RobotSkull.

Import all of these assets into the Robot scene from starter/resources.

As always, the parts are stacked next to each other when you import them. You’ll need to put them in their rightful places.

The robot needs two eyes and two eyelids, so create a copy of the RobotEye and the RobotEyeLid.

Apply the following settings to each element using the data from the table below:

The result should look like this:

The Mighty Green Robot Head is done! Save all of your changes and close Reality Composer, then jump back to Xcode.

Adding the New Robot Scene

Now that you’ve built the Robot scene, you need to update the app so it knows about the additional prop.

Add the following variable to the top of ContentView.swift:

var robot: Experience.Robot!

This gives you quick access to the robot anchor, once you initialize it.

Now, initialize the new variable, robot, at the top of updateUIView(_:context:):

robot = nil

This makes sure that, if you haven’t loaded the anchor, robot is nil. When you inspect the value of robot and it’s nil, you know the robot head hasn’t been activated.

Next, you need to modify the Next button so, when the user selects it, the app increases the propId count to handle four props instead of just three.

Within the body variable, modify the Next button action code block to look like this:

self.propId = self.propId >= 3 ? 3 : self.propId + 1

Great, now the app can select the additional prop, giving it a total of four available props. Now, you just need to cater for this new case in the update.

Add the following case to handle the robot prop in updateUIView(_:context:):

case 3: // Robot
  // 1
  let arAnchor = try! Experience.loadRobot()
  // 2
  uiView.scene.anchors.append(arAnchor)
  // 3
  robot = arAnchor
  break

Here’s how it breaks down:

1. This loads the Robot scene from Experience.rcproject and stores it in arAnchor.
2. It then appends arAnchor to the scene anchors.
3. Finally, it stores arAnchor in robot so other parts of the code can use it to get notifications when the robot prop is active. It also provides quick access to all the elements of the robot head.

Now would be a great time to do a quick check to make sure everything still works as intended. Do a quick build and run.

Fantastic! You can select the new prop and, goodness, the Mighty Green Robot head looks so shiny.

Everything is still very static… you’ll address that next.

Using the ARSessionDelegate Protocol

To animate the robot’s eyelids and jaw, you need to update their positions and rotations as ARFaceAnchor tracks the user’s facial expressions in real time. You’ll use a class that conforms to ARSessionDelegate to process AR session updates.

With this protocol, the delegate gains access to the following information:

• Updated Frame Data: Provides a newly-captured camera image along with AR information to the delegate, provided in an ARFrame.

• Added Anchors: Informs the delegate that one or more anchors have been added to the session.

• Removed Anchors: Informs the delegate that one or more anchors have been removed from the session.

• Updated Anchors: Informs the delegate that the session has adjusted the properties of one or more anchors. This is where you can monitor any changes in the blend shapes you’re tracking. Modifying a blend shape will trigger a session update.

Adding ARDelegateHandler

For your next step, you’ll create a new class that inherits from this protocol so you can track changes to the facial blend shapes.

Add the following class to ARViewContainer:

// 1
class ARDelegateHandler: NSObject, ARSessionDelegate {
  // 2
  var arViewContainer: ARViewContainer
  // 3      
  init(_ control: ARViewContainer) {
    arViewContainer = control
    super.init()
  }
}      

Here’s a closer look at what this does:

1. This defines a new class called ARDelegateHandler that inherits ARSessionDelegate.
2. When the class instantiates, it provides ARViewContainer and stores it in arViewContainer.
3. This is the class initializer, which simply stores the provided ARViewController then initializes the super class.

From a SwiftUI perspective, you now need to create a custom instance to communicate changes from the view controller to the other parts of the SwiftUI interface. You’ll use a makeCoordinator to create this custom instance.

To do so, add the following function to ARViewContainer:

func makeCoordinator() -> ARDelegateHandler {
  ARDelegateHandler(self)
}

This defines makeCoordinator and indicates that it will provide an instance of ARDelegateHandler. It then creates an actual instance of ARDelegateHandler, providing self as the ARViewContainer.

Now that everything’s in place, you can set the session delegate for the view. Add the following line of code to makeUIView(context:), just after initializing arView:

arView.session.delegate = context.coordinator

Here, you set the view’s session delegate to the context coordinator, which now starts updating the session when it detects any changes.

Handling ARSession Updates

With the delegate class in place, you can now start tracking updates to any of the facial blend shapes.

Add the following function to ARDelegateHandler:

// 1
func session(_ session: ARSession, 
  didUpdate anchors: [ARAnchor]) {
  // 2
  guard robot != nil else { return }
  // 3
  var faceAnchor: ARFaceAnchor?
  for anchor in anchors {
    if let a = anchor as? ARFaceAnchor {
      faceAnchor = a
    }
  }
}

Here’s what’s happening above:

1. This defines session(_:didUpdate:), which triggers every time there’s an update available on the anchor.

2. You’re only interested in anchor updates while the robot scene is active. When robot is nil, you simply skip any updates.

3. This extracts the first available anchor from the received anchors that conforms to an ARFaceAnchor, then stores it in arFaceAnchor. You’ll extract all the updated blend shape information from here.

Tracking Blinking Eyes

Now that the update handling function is in place, you can inspect the actual blend shape values and use them to update the scene elements so the robot blinks its eyes when the user blinks theirs.

You’ll use the eyeBlinkLeft and eyeBlinkRight blend shapes to track the user’s eyes.

Start by adding the following block of code to the bottom of session(_:didUpdate:):

let blendShapes = faceAnchor?.blendShapes
let eyeBlinkLeft = blendShapes?[.eyeBlinkLeft]?.floatValue
let eyeBlinkRight = blendShapes?[.eyeBlinkRight]?.floatValue

Here, you access the blendShapes through the updated faceAnchor. You then inspect the specific blend shape key eyeBlinkLeft to get its current value, which is provided as a floatValue.

Then you use the same approach to get the current value for eyeBlinkRight.

Tracking Eyebrows

To make the eyes more expressive, you’ll use the user’s eyebrows to tilt the eyelids inwards or outwards around the z axis. This makes the robot look angry or sad, depending on the user’s expression.

This time, you’ll use three blend shapes to track the user’s eyebrow movements:

• browInnerUp: Tracks the inner, upward movement of both eyebrows.

• browDownLeft and browDownRight: Tracks the left- and right-side downward movement of the user’s eyebrows.

To put them into place, add the following to the bottom of session(_:didUpdate:):

let browInnerUp = blendShapes?[.browInnerUp]?.floatValue
let browLeft = blendShapes?[.browDownLeft]?.floatValue
let browRight = blendShapes?[.browDownRight]?.floatValue

Great, now you’re tracking the eyebrows. The only thing left to do is to align the orientation of the eyelids with these blend shape values. To do it, though, you’ll also need to track what the user is doing with their jaw.

Tracking the Jaw

Now, you’ll track the user’s jaw, and use it to update the orientation. You’ll use the jawOpen blend shape to track the user’s jaw movement.

Add the following code to the bottom of session(_:didUpdate:):

let jawOpen = blendShapes?[.jawOpen]?.floatValue

Now, you’re ready to use special vectors to align both the eyelids and the jaw.

Positioning with Quaternions

In the next section, you’ll update the orientations of the eyelids and jaw based on the blend shape values you’re capturing. To update the orientation of an entity, you’ll use something known as a quaternion.

A quaternion is a four-element vector used to encode any possible rotation in a 3D coordinate system. A quaternion represents two components: a rotation axis and the amount of rotation around the rotation axis.

Three vector components, x, y and z represent the axis, while a w component represents the rotation amount.

Quaternions are difficult to use. Luckily, there are a few handy functions that make working with them a breeze.

Here are two important quaternion functions you’ll use in this chapter:

• simd_quatf(angle:,axis:): Allows you to specify a single rotation by means of an angle amount along with the axis the rotation will revolve around.

• simd_mul(p:, q:): Lets you multiply two quaternions together to form a single quaternion. Use this function when you want to apply more than one rotation to an entity.

You have to specify angles in radians. To make life a little easier, you’ll use a little helper function that converts degrees into radians.

Add the following helper function to ARDelegateHandler:

func Deg2Rad(_ value: Float) -> Float {
  return value * .pi / 180
}

And that’s all you need to make the conversion easy.

Updating the Eyelids

Now that you’ve collected all the blend shape data, you need to update the eyelid orientation.

Take a look at a side view of a single eye with an eyelid.

Here’s how to use the captured blink blend shape data to update the eyelid orientation:

1. Import the eyelid with a rotation around the x-axis of 0°. This is a fully shut eye.

2. To open the eye, set the eyelid’s natural resting orientation to -120° around the x-axis. Then rotate the eyelid through a 90° ranged angle based on the value of the blink blend shape. When the user’s eye is open, the blend shape will be 0%, adding 0° to the current resting orientation.

3. As the user closes their eyes, the blink blend shape’s value will increase. At a 50% position, you’ll add 45° to the current resting -120° orientation, setting the orientation to about -75°. The eyes will appear partially shut now.

4. When the user fully closes their eyes, the blink blend shape increases toward 100%, adding the full 90° angle range to the current resting orientation. This makes the eyes appear completely closed.

At the same time, you’ll apply a rotation around the z-axis to make the eye appear angry or sad. You’ll use the same approach with the captured brow blend shapes.

Here’s what all that looks like in code. Add the following block of code to the bottom of session(_:didUpdate:):

// 1
robot.eyeLidL?.orientation = simd_mul(
  // 2
  simd_quatf(
    angle: Deg2Rad(-120 + (90 * eyeBlinkLeft!)),
    axis: [1, 0, 0]),
  // 3  
  simd_quatf(
    angle: Deg2Rad((90 * browLeft!) - (30 * browInnerUp!)),
    axis: [0, 0, 1]))
// 4            
robot.eyeLidR?.orientation = simd_mul(
  simd_quatf(
    angle: Deg2Rad(-120 + (90 * eyeBlinkRight!)),
    axis: [1, 0, 0]),
  simd_quatf(
    angle: Deg2Rad((-90 * browRight!) - (-30 * browInnerUp!)),
    axis: [0, 0, 1]))

This updates both the left and right eyelid orientations:

1. To start, you check the robot is currently the active prop, so you can gain access to elements like the left eyelid via robot. You’ll apply two rotations to the orientation of the left eyelid, using quaternion multiplication to combine two quaternions.

2. This is the first rotation around the x-axis. The left eyelid is currently resting at a -120° rotation, so you want to use that as the base rotation. You then multiply the left blink blend shape by 90°, which is the amount of influence the blend shape will have over the eyelid rotation.

3. This is the second rotation around the z-axis. The left brow blend shape has a 90° influence over the eyelid orientation, while the inner brow movement only has a 30° influence.

4. You use the same approach to update the right eyelid orientation. The only difference you’ll notice is that the eyelid tilts in the opposite direction around the z-axis for the brow movement.

Updating the Jaw

The eyelids are done, but you still need to update the jaw orientation with the captured blend shape information:

robot.jaw?.orientation = simd_quatf(
  angle: Deg2Rad(-100 + (60 * jawOpen!)),
  axis: [1, 0, 0])

Similar to how the eyelids work, the jaw sits at a natural -100° with a 60° range of motion linked to the jaw open blend shape.

And that’s it, you’re all done! Time for another build and run test.

You can now blink, frown and control that huge metal jaw. Careful, this robot looks a bit on the angry side! :]

Adding Lasers

The robot is mostly done, but there’s always room for improvement. Wouldn’t it be cool if it could shoot lasers from its eyes when it gets extra angry?

Well, that’s your next task! When the user opens their mouth past a certain point, dangerous lasers will shoot from the robot’s eyes.

Here’s how you add the lasers. Open Experience.rcproject in Reality Composer, then select the Robot scene.

Add a Basic ▸ Cylinder object to the scene.

Select the newly-added cylinder and rename it Laser_L. Under the Transform section, set the Position to (X:4cm, Y:60cm, Z:-4.5cm).

Under the Look section, leave the Material as Glossy Paint, but change the Material Color to Yellow. Set the Diameter to 1 cm, set the Height to 1 m and set the Bevel Radius to 0 cm.

Now, duplicate the cylinder and rename it Laser_R. Then under the Transform section, set the Position to (X:-4 cm, Y:60 cm, Z:-4.5 cm).

Two lasers should now shoot from the robot’s eyes.

Sending & Receiving Notifications

Your next goal is to really bring those lasers to life. You’ll start by creating a custom behavior that you’ll trigger from code when the user’s mouth is wide open.

While the lasers are firing, you have to wait for them to finish before you can fire another laser. To achieve that, you’ll send a notification to your code to indicate that the laser has finished.

The first thing you need to do is to hide the lasers when the scene starts. Open the Behaviors panel, then add a Start Hidden behavior.

Rename the behavior to Start and add the two lasers as the affected objects for the Hide action.

Now, when the robot scene starts, both lasers will be hidden.

With the two lasers selected in the scene, you’ll add a Custom behavior next.

Rename the behavior to Lasers, then add the Trigger as a Notification.

Under the Notification trigger, change the Identifier to ShowLasers.

You can now trigger this behavior from code by using the identifier.

Next, you want the lasers to become visible and make a noise, then disappear again.

To make them appear, add a grouped Show action with a Play Sound action sequence. For the Show action, set the Duration to 3 sec. For the Play Sound action, set the Audio Clip to DJ Scratching and Effect 23.caf.

Now, to make them disappear, add a grouped Hide action with a Play Sound action sequence. For the Hide action, set the Duration to 3 sec. For the Play Sound action, set the Audio Clip to DJ Scratching and Effect 22.caf.

Now, the lasers will appear then disappear over six seconds while making a strange noise.

The last thing you need to do is to create a notification that lets the code know when the action sequence finishes.

With both lasers still selected, under the Lasers behavior, add a Notify action to the Action Sequence.

For the Notify action, set the Identifier to LasersDone.

Now, the Action Sequence can notify the code once the sequence completes.

Save the Reality Composer project, then jump back to Xcode.

Your changes are now visible in the project. You’ll take a look at the coding side of handling notifications next.

Coding the Notifications

Now, you’ll add the code that prevents other things from happening while the lasers are firing.

Start by adding the following property to the top of the ARDelegateHandler class:

var isLasersDone = true

You’ll use this variable to block additional triggers. When this value is false, the lasers are currently active and you have to wait for the action sequence to complete before triggering the lasers again.

Add the following block of code to the bottom of session(_:didUpdate:):

// 1
if (self.isLasersDone == true && jawOpen! > 0.9) {
  // 2
  self.isLasersDone = false
  // 3
  robot.notifications.showLasers.post()
  // 4
  robot.actions.lasersDone.onAction = { _ in
    self.isLasersDone = true
  }
}

Here’s its breakdown:

1. If the user’s jaw is about 90% open and the lasers aren’t currently active, you can trigger the lasers.

2. Once you trigger the lasers, you need to keep isLasersDone set to false. That indicates the action sequence is currently active, preventing the user from triggering multiple sequences at the same time.

3. Trigger the custom behavior by finding it under the available notifications using the identifier name you defined in behavior.

4. Here, you create an action notification handler that triggers once the action sequence completes. You’ll find it under the available actions using the identifier name you defined within the notify action. It sets isLasersDone back to true, letting the user trigger new lasers again.

Congratulations! Build and run now to test the final product.

The robot can blink, look sad and angry, and open and close his jaw. But, best of all, it can shoot lasers from its eyes when the user opens their mouth wide. Awesome!

Key Points

Congratulations, you’ve reached the end of this chapter and section. Before grabbing a delicious cup of coffee, quickly take a look at some key points you’ve learned in this chapter.

To recap:

• Facial blend shapes: You’ve learned about facial blend shapes and how they’re used to track a face’s key points.

• ARSessionDelegate: You learned how to handle scene updates via the ARSessionDelegate. Every time a blend shape updates, it triggers a session update, allowing you to update the entities within the scene.

• Using blend shapes: You’ve learned how to track blend shapes and use the data to update entity orientations.

• Quaternions: You know what quaternions are and how to use helper functions to demystify them, making rotations a breeze to work with.

• Notifications: Triggering actions sequences from code and receiving notify actions from scenes is simple.

Enjoy that cup of coffee. See you in the next section, where you’ll learn more about ARKit and SpriteKit.