15. ARKit & SceneKit

Written by Chris Language

Now that you’ve seen ARKit in action alongside Apple’s general-purpose 2D graphics framework, SpriteKit, it’s time to unlock the third dimension with SceneKit. In this chapter, you’ll continue to learn more about ARKit, but with the focus of using it with SceneKit as its rendering technology.

You’ll start by creating a new SceneKit-based AR project with Xcode. The project will grow into an interactive 3D augmented reality experience: a small AR Airport with basic animations.

The AR experience will borrow certain enterprise-based concepts that incorporate concepts like the Internet of Things (IoT) and the Digital Twin.

Don’t worry, the data component for this project is non-existent. Your main focus is to create an awesome AR experience that will serve as a fun little frontend.

What is SceneKit?

SceneKit is a high-performance rendering engine and 3D graphics framework. It’s built on top of Metal, which delivers the highest performance possible. It leverages the power of Swift to deliver a simple, yet extremely powerful and descriptive, 3D graphics framework. With it, you can easily import, manipulate and render 3D content. With its built-in physics simulation and animation capabilities, creating rich 3D experiences has never been easier.

Best of all, Apple’s platforms all support SceneKit and it integrates extremely well with other frameworks, like GameplayKit and SpriteKit.

Creating a SceneKit AR Project

Start Xcode — it’s time to create a SceneKit-based AR project.

Create a new project in Xcode and, when asked to select your template, choose iOS ▸ Application ▸ Augmented Reality App, then click Next to continue.

Change the Product Name to ARPort and choose SceneKit for the Content Technology. You’ll use a Storyboard UI, so leave the Interface as-is and leave the Language as Swift.

Finally, turn off Include Tests and click Next to continue.

Xcode now generates a bare-bones SceneKit-based Augmented Reality project for you.

Let it finish and, before doing anything else, take the app for a quick spin. Build and run the project to see what the bare-bones app does out of the box.

When the app starts, it uses the phone’s current position in real-world space as the world origin point. It then spawns a big spaceship at that exact position. You’ll need to take a step back to see it fully. My oh my, so shiny! :]

Now, take a moment to explore the contents of the project.

Exploring the Project

In Xcode, with the project open, explore the important components that Xcode generated for you based on the SceneKit Augmented Reality Template project. Although the generated project is similar to a SpriteKit project, there are a few small differences:

AppDelegate.swift

This is the standard starting point of your app.

LaunchScreen.storyboard

The launch screen is a standard part of every app. It’s the first thing the user sees when the app launches. Here, you’ll represent your app with a beautiful splash image.

Main.storyboard

The main storyboard is the view component of your AR app, containing the app’s UI. This is a good place to put buttons and heads-up displays, for example.

Take particular note of the ARSCNView scene view class, which lets you overlay an AR scene over a live background image feed from the camera. It provides seamless integration between ARKit and SceneKit. Also, note that the view is connected to an @IBOutlet defined in ViewController.swift.

ViewController.swift

The view controller contains the code behind the entire AR experience, specifically the main storyboard.

Take note of the frameworks at play:

• UIKit: Contains the required infrastructure for iOS and tvOS apps, including the window and view architecture to implement the UI, event handling and input. It also provides support for animation, document, drawing, printing, device information, text and display, search, accessibility, app extension and resource management.

• SceneKit: Supports 3D graphics.

• ARKit: Provides ARKit support.

The ViewController inherits directly from the standard UIViewController, which provides the infrastructure for managing the views of a basic UIKit-based app.

It also adopts the ARSCNViewDelegate protocol from ARKit, which contains methods you can implement to synchronize your SceneKit content with your AR session.

Take special note of @IBOutlet. It connects to ARSCNView, which is defined in the Main.storyboard.

Look at viewDidLoad(), where the app loads and presents the default SCNScene scene named scene.

viewWillAppear(_:) is where you create an ARWorldTrackingConfiguration instance. This configuration is provided to the view’s ARSession when the user starts the app.

art.scnassets

art.scnassets is a standard folder that was simply renamed by adding the .scnassets extension to it. This type of folder is known as a SceneKit Asset Catalog. Its purpose is to help you manage your game assets separately from the code.

Xcode will copy the contents of this folder to your app bundle at build time. Xcode will also preserve the folder hierarchy, giving you full control over the folder structure.

ship.scn

Within the SceneKit Asset Catalog, you’ll find a b file. This defines the SceneKit Scene containing a model of the ship that you see when the AR experience starts.

The view controller loads and presents this scene.

Assets.xcassets

Here, you find your standard app assets like your app icon, for example.

Info.plist

When your app runs for the first time, it needs to ask for permission to access the camera. ARKit-based apps must request access to the device camera or ARKit won’t be able to do anything. It does this by setting the value of Privacy — Camera Usage Description.

Loading & Exploring the Starter Project

Now that you know how to create a standard SceneKit-based AR project with Xcode… you won’t use the project you just created.

To speed things up a bit, you’ll use a specially prepared project instead, which already has a few basic housekeeping things done for you. This way, you can focus on the important ARKit- and SceneKit-related tasks.

Get started by opening ARPort.xcodeproj in the Starter folder and checking out a few important components you need to know.

ViewController.swift

You’ll find that the view controller has been reduced to its bare bones. Don’t worry, you’ll fill in all the missing code later.

You’ll also find a few important stubs within the code. These act as sections, separating the code into clear, manageable blocks that deal with specific parts of your app. They’ll also make it easier to locate the correct places to enter code as you work through the tutorial.

Here’s a quick breakdown of some of the important sections you’ll find:

• App State Management: Plays a key part in managing your app through various states.
• Properties: You define all class variable members here.
• IB Outlets: Gives you quick access to some key UI elements because they’re already connected to Main.storyboard.
• IB Actions: These actions have also been connected to Main.storyboard. You’ll call these methods when the user interacts with button elements or makes a gesture.
• App Management: Dedicated to methods related to managing the app through its entire lifecycle.
• AR Coaching Overlay: Used to create and manage an AR Coaching Overlay. More on this in just a minute.
• AR Session Management: Focuses on managing the AR session, along with handling possible issues and errors.
• Scene Management: Focuses on SceneKit scene management.
• Focus Node Management: Manages the Focus Node. More on what that is in the next chapter.

Main.storyboard

Next, you’ll look at Main.storyboard.

Of particular interest are these three elements of the storyboard:

• StatusLabel: At the top of the screen, located within a Visual Effect view that blurs the background, you’ll find the status label that you’ll use to keep the user informed at all times. This UI element connects to the statusLabel variable within the view controller.
• ResetButton: Located at the bottom left of the screen, this button will reset the AR experience. This UI element connects to resetButtonPressed() within the view controller.
• TapGesture: A tap gesture recognizer that triggers tapGestureHandler() when the user taps the screen.

App State Management

With all the logistics out of the way, you’ll start with some basic app management. The app requires a state machine to manage its lifecycle through various states. During each individual state, the app will focus on doing one particular thing.

Open ViewController.swift and add the following enum to the App State Management section:

enum AppState: Int16 {
  case DetectSurface
  case PointAtSurface
  case TapToStart
  case Started
}

The enum will manage the app through the following states:

• DetectSurface: During this state, the AR session will actively detect viable horizontal surfaces, known as planes.
• PointAtSurface: The AR session has now successfully detected viable horizontal surfaces. Now, the user has to point towards the detected surface so a focus node can appear.
• TapToStart: The user is pointing towards a viable surface and the focus node is visible. The focus node acts as a visual indicator, showing the user where the 3D content will appear when the AR experience starts.
• Started: The user placed the 3D content and the AR experience has started. The user is now interacting with the 3D content in AR.

With the enum in place, add the following variables under the Properties section of the view controller class:

var trackingStatus: String = ""
var statusMessage: String = ""
var appState: AppState = .DetectSurface

You’ll use appState to keep track of the current active state. Both trackingStatus and statusMessage will help keep the user informed. You’ll use them to show the current AR tracking status along with an instructional message so the user knows what to do at all times.

With the basic app state in place, it’s time to add the following helper functions under the App Management section:

// 1
func startApp() {
  DispatchQueue.main.async {
    self.appState = .DetectSurface
  }
}
//2
func resetApp() {
  DispatchQueue.main.async {
    //self.resetARSession()
    self.appState = .DetectSurface
  }
}

Here’s what these functions do:

1. Sets the appState to DetectSurface. As mentioned above, the AR session will actively detect planes in this state.
2. Resets the AR session, which is currently commented out. You’ll uncomment its code after you create that function. It also sets the appState to DetectSurface.

Basic Scene Management

Now, move on to the SceneKit component of your app. The first thing you need to do is to make the view controller comply with a special protocol.

Add the ARSCNViewDelegate protocol to ViewController so the class definition looks like this:

class ViewController: UIViewController, ARSCNViewDelegate {

ARSCNViewDelegate provides various methods to update your SceneKit content. These methods correspond to ARAnchor objects, which the AR session is tracking. This comes in handy when you want to manage detected surfaces that are represented as planes, for example.

Initializing a New SceneKit Scene

Now, to create the new SceneKit scene.

Add the following function to the Scene Management section:

func initScene() {
  // 1
  let scene = SCNScene()
  sceneView.scene = scene
  // 2
  sceneView.delegate = self
}

Creating a new scene is really easy:

1. You create a new SCNScene instance, which you then set to the scene view’s scene.
2. You then set the view controller as the scene view’s delegate, which now conforms to ARSCNViewDelegate.

Excellent! Now that you’ve created the scene, you’ll provide the user with some helpful information.

Providing Feedback

Feedback helps the user know what the app is doing and what steps they need to take next. To start providing feedback, add the following helper function to the Scene Management section:

func updateStatus() {
  // 1
  switch appState {
  case .DetectSurface:
    statusMessage = "Scan available flat surfaces..."
  case .PointAtSurface:
    statusMessage = "Point at designated surface first!"
  case .TapToStart:
    statusMessage = "Tap to start."
  case .Started:
    statusMessage = "Tap objects for more info."
  }
  // 2
  self.statusLabel.text = trackingStatus != "" ?
      "\(trackingStatus)" : "\(statusMessage)"
}

This helper function keeps the user informed by:

1. Setting a statusMessage based on the current app state.
2. Constructing the final status label’s message for the user by combining trackingStatus and statusMessage.

To put this helper function to good use, add the following function to the Scene Management section:

func renderer(_ renderer: SCNSceneRenderer, 
  updateAtTime time: TimeInterval) {
  DispatchQueue.main.async {
    self.updateStatus()
  }
}

SceneKit calls renderer(_:updateAtTime:) once for every frame update. This ensures that updateStatus() is called regularly, keeping the user constantly informed.

Note: By calling updateStatus() inside DispatchQueue.main.async, you ensure the call executes within the main thread. This is very important when updating any information located on the UI. If you don’t, you might experience some update lag.

Now, to ensure the scene actually initializes when the app starts, add a call to it at the bottom of viewDidLoad():

self.initScene()

Great, now the scene will initialize when the app starts. This is a great time to test everything out. Build and run to deploy the app to your device.

Fantastic, the app started and the scene initialized… but it’s totally dark! That’s because you haven’t started ARKit yet. You’ll look at that component next.

AR Session Management

Now that you’ve created the scene and ensured that the user will be kept informed of what the app’s doing, move on to the AR component.

Main.storyboard contains ARSCNView, which is basically a SceneKit view. It includes ARSession, which is responsible for motion tracking and image processing in ARKit. It’s session-based, which means you have to create an AR session instance, then you have to run that session to start the AR tracking process.

AR Configuration

Before starting an AR session, you have to create an AR session configuration. You use this configuration to establish the connection between the real world, where your device is, and the virtual 3D world, where your virtual content is.

There are six types of configurations:

• AROrientationTrackingConfiguration: Basic three degrees of freedom (3DOF) tracking.
• ARWorldTrackingConfiguration: Six degrees of freedom (6DOF) tracking. Tt also tracks people, known images and objects.
• ARBodyTrackingConfiguration: Tracks human bodies.
• ARImageTrackingConfiguration: Tracks known images.
• ARObjectScanningConfiguration: Tracks known 3D objects.
• ARFaceTrackingConfiguration: Tracks faces and facial expressions using the front-facing camera.

Starting the AR Session

With ViewController.swift still open, add the following extension under the AR Session Management section:

func initARSession() {
  // 1
  guard ARWorldTrackingConfiguration.isSupported else {
    print("*** ARConfig: AR World Tracking Not Supported")
    return
  }
  // 2
  let config = ARWorldTrackingConfiguration()
  // 3
  config.worldAlignment = .gravity
  config.providesAudioData = false
  config.planeDetection = .horizontal
  config.isLightEstimationEnabled = true
  config.environmentTexturing = .automatic
  // 4
  sceneView.session.run(config)
}

Take a look at what’s happening here:

1. isSupported checks if the device supports the required AR configuration. This is a good time to tell the user to upgrade their iPhone, if necessary! :]

2. Creates an ARWorldTrackingConfiguration configuration instance assigned to config. This gives your app six degrees of freedom (6DOF) tracking, as well as tracking people, known images and objects.

3. This sets a few configuration requirements:

   a) worldAlignment: Setting it to gravity sets the coordinate system’s y-axis parallel to gravity, with the origin to the initial position of the device.

   b) providesAudioData: This disables capturing audio during the AR session. You don’t want to sample any audio.

   c) planeDetection: You set it to horizontal, which specifies that the AR session should automatically detect horizontal flat surfaces. More on this in just a second.

   d) isLightEstimationEnabled: By setting this to true, you give the running AR session responsibility for providing scene lighting information.

   e) environmentTexturing: Setting this to automatic lets the AR session automatically determine when and where to generate environment textures.

4. Finally, this calls run(_:options:) on ARSCNView’s ARSession, passing in the freshly created ARWorldTrackingConfiguration. This ultimately starts the AR session.

Resetting the AR Session

At times, you might want to reset the AR session. This comes in handy when you want to restart the AR experience, for example.

To do this, add the following function to the AR Session Management section:

func resetARSession() {
  // 1
  let config = sceneView.session.configuration as!
    ARWorldTrackingConfiguration
  // 2
  config.planeDetection = .horizontal
  // 3
  sceneView.session.run(config,
    options: [.resetTracking, .removeExistingAnchors])
}

Here’s how it breaks down:

1. You can gain access to the existing AR configuration through the AR session configuration. This casts the existing AR configuration back into an ARWordTrackingConfiguration.

2. This ensures that planeDection is still set to horizontal so the AR session will continue to automatically detect horizontal flat surfaces once it resets.

3. Finally, this resets the AR session with the following options:

   a) resetTracking: Simply resets the device’s position from the previous session run.

   b) removeExistingAnchors: Removes all the anchor objects associated with the previous session run.

Handling AR Session State Changes

Now that you can start and reset the AR session, you need to keep the user informed any time the AR session state changes. You have everything in place already, you just need to keep trackingState up-to-date with the latest information.

Add the following function override to the AR Session Management section:

func session(_ session: ARSession, 
  cameraDidChangeTrackingState camera: ARCamera) {
  switch camera.trackingState {
  case .notAvailable: self.trackingStatus = 
    "Tracking:  Not available!"
  case .normal: self.trackingStatus = ""
  case .limited(let reason):
    switch reason {
    case .excessiveMotion: self.trackingStatus = 
      "Tracking: Limited due to excessive motion!"
    case .insufficientFeatures: self.trackingStatus =
      "Tracking: Limited due to insufficient features!"
    case .relocalizing: self.trackingStatus = 
      "Tracking: Relocalizing..."
    case .initializing: self.trackingStatus = 
      "Tracking: Initializing..."
    @unknown default: self.trackingStatus = 
      "Tracking: Unknown..."
    }
  }
}

This interrogates the camera’s current tracking state and populates trackingState with an appropriate message to show the user.

Handling AR Session Issues

Finally, you need to keep the user informed when any issues occur. Again, you’ll use trackingState for this purpose.

Add the following function overrides to the AR Session Management section:

func session(_ session: ARSession, 
  didFailWithError error: Error) {
  self.trackingStatus = "AR Session Failure: \(error)"
}
  
func sessionWasInterrupted(_ session: ARSession) {
  self.trackingStatus = "AR Session Was Interrupted!"
}
  
func sessionInterruptionEnded(_ session: ARSession) {
  self.trackingStatus = "AR Session Interruption Ended"
}

Once any of these session issues occur, trackingState is populated with an appropriate message that will be displayed to the user.

Now, to make sure the AR session actually initializes when the app starts, add a call to it at the bottom of viewDidLoad():

self.initARSession()

Also, connect the Reset button by adding the following line of code to resetButtonPressed(_:):

self.resetARSession()

Finally, uncomment the call to resetARSession(_:) inside resetApp().

Now, do another quick test. Build and run to see what the app looks like this time around.

The black screen has been replaced by the camera feed. That’s because the AR session has been initialized and is now actively scanning for horizontal surfaces. Pressing the Reset button will also work, restarting the tracking when you press it.

Excellent, you’re making great progress!

AR Coaching Overlay

Currently, the app uses the status bar at the top to provide step-by-step instructions to help onboard the user into the AR experience. However, your approach to this onboarding process might differ entirely from another developer’s. This causes massive fragmentation in AR experiences as the user switches from one experience to another.

Apple is curbing this fragmentation with the AR Coaching Overlay View.

What is an AR Coaching Overlay View?

Apple now provides a special overlay view known as the ARCoachingOverlayView. You can easily integrate it into your existing AR experiences to provide the user with a standardized AR onboarding process.

The overlay operates in a few basic states:

• Starting State: When the app starts, the Coaching Overlay facilitates ARKit by taking the user through a standardized onboarding process.

• Goal-based State: Use this to provide the overlay with a specific goal. For example, you can instruct the user to find horizontal or vertical surfaces.

• Relocating State: When the app loses tracking, the overlay takes over again, guiding the user back into a stable tracking state.

Pretty cool stuff! Next, you’ll add it to your app.

Adding AR Coaching Overlay

The first thing to do is to ensure your view controller conforms to the new protocol.

Add the following ARCoachingOverlayViewDelegate protocol to the view controller under the AR Coaching Overlay section:

extension ViewController : ARCoachingOverlayViewDelegate {
}

And that’s all it took to make the view controller conform to the new protocol.

Handling AR Coaching Overlay Events

Next, you need to provide some functions to handle the overlay events.

Add the following functions to the AR Overlay Management section:

// 1
func coachingOverlayViewWillActivate(_ 
  coachingOverlayView: ARCoachingOverlayView) {
}
// 2  
func coachingOverlayViewDidDeactivate(_ 
  coachingOverlayView: ARCoachingOverlayView) {
  self.startApp()
}
// 3
func coachingOverlayViewDidRequestSessionReset(_
  coachingOverlayView: ARCoachingOverlayView) {
  self.resetApp()
}

Here’s what these functions do:

1. coachingOverlayViewWillActivate(_:): This event triggers right before the overlay is activated.
2. coachingOverlayViewDidDeactivate(_:): This event triggers just after the overlay is deactivated, indicating the overlay has found sufficient horizontal surfaces for the app to function. This is a great place to start the app.
3. coachingOverlayViewDidRequestSessionReset(_:): This event triggers when the AR session has lost tracking for some unknown reason. The overlay will kick in again and ensure that there’s sufficient horizontal surface information for the app to function. This is a great spot to reset the app so that the user can place the AR content again.

Initializing the AR Coaching Overlay

Now that you’re handling everything the AR Coaching Overlay will throw at your app, you need to initialize it.

You’ll do this with the following handy helper function. Add it to the AR Coaching Overlay Management section:

func initCoachingOverlayView() {
  // 1
  let coachingOverlay = ARCoachingOverlayView()
  // 2
  coachingOverlay.session = self.sceneView.session
  // 3
  coachingOverlay.delegate = self
  // 4
  coachingOverlay.activatesAutomatically = true
  // 5
  coachingOverlay.goal = .horizontalPlane
  // 6
  self.sceneView.addSubview(coachingOverlay)
}

Here’s what it does:

1. Creates an instance of ARCoachingOverlayView, then stores it in coachingOverlay.
2. Sets the overlay’s session to the same session as the scene view.
3. Sets the view controller as the overlay’s delegate.
4. Configures the overlay to activate automatically. While ARKit is initializing or dealing with tracking issues, the overlay will take over and guide the user automatically. You don’t have to worry about a thing.
5. Tells the overlay that you’re only interested in horizontal surfaces. So the overlay will help guide the user to find horizontal surfaces during the onboarding process.
6. Finally, adds the overlay as a subview of the sceneView so it knows which view it needs to overlay its content.

Adding Constraints to the AR Coaching Overlay

With the overlay initialized, your next step is to provide it with proper constraints.

Do this by adding the following to the bottom of initCoachingOverlayView():

// 1
coachingOverlay.translatesAutoresizingMaskIntoConstraints =
  false
// 3
NSLayoutConstraint.activate([
  NSLayoutConstraint(item: coachingOverlay,
    attribute: .top, relatedBy: .equal,
    toItem: self.view, attribute: .top,
    multiplier: 1, constant: 0),
  NSLayoutConstraint(item:  coachingOverlay, 
    attribute: .bottom, relatedBy: .equal,
    toItem: self.view, attribute: .bottom, 
    multiplier: 1, constant: 0),
  NSLayoutConstraint(item:  coachingOverlay,
    attribute: .leading, relatedBy: .equal,
    toItem: self.view, attribute: .leading,
    multiplier: 1, constant: 0),
  NSLayoutConstraint(item:  coachingOverlay,
    attribute: .trailing, relatedBy: .equal,
    toItem: self.view, attribute: .trailing,
    multiplier: 1, constant: 0)])

Taking a closer look, you can see it:

1. Disables the view’s auto-resizing mask so it’s not translated into auto layout constraints.
2. Provides the overlay with manual constraints that conform to the current view’s constraints.

Great, you’ve now added the overlay. It will be visible when the app starts, to help guide the user through the AR onboarding process.

Finally, to ensure the AR Coaching Overlay initializes when the app starts, add a call to it at the bottom of viewDidLoad():

self.initCoachingOverlayView()

Time for a final build and run. Take a look at the new AR Coaching Overlay View:

If you blink, you might have missed the AR Coaching Overlay View. Don’t worry, you can make it come back. Just block the camera with your finger and the AR Coaching Overlay View should kick back into place. How cool is that? :]

Note: You can find the final version of the project in final/ARPort.

Key Points

Congratulations, you’ve reached the end of this chapter — and your app is shaping up nicely.

Take a look at some key points you’ve picked up so far:

• SceneKit: It’s easy to create a new SceneKit-based AR experience by using Xcode’s available AR app templates.
• Key App Components: You peeked under the hood and learned how key components play their parts in the Xcode project.
• App State Management: You learned how to implement basic app state management for a typical AR experience.
• Scene Creation: You created a blank SceneKit scene.
• AR Session Management: Creating, running and resetting an AR session is really simple.
• AR Coaching Overlay View: Making your AR experience conform to Apple’s standard onboarding process is as simple as implementing an AR Coaching Overlay View into your apps.

Now, you’ve gotten all the groundwork out of the way. In the next chapter, you’ll focus on creating the actual AR experience. See you there!