13. ARKit & SpriteKit

Written by Chris Language

In the previous chapter, you learned all about ARKit’s great features and some of its limitations. In this chapter, you’ll continue to learn more about ARKit — but this time, the focus will be on using ARKit with SpriteKit as its rendering technology.

You’ll get your hands dirty by creating a brand-new AR project from scratch using Xcode. You’ll create a fun AR experience that uses 2D-based emoji graphics. Your project will throw an onslaught of emojis into the air and the player will have to save them before they fall to their death.

Keen on seeing emojis fall to their death? Then what are you waiting for? Jump in and get those hands dirty!

What is SpriteKit?

SpriteKit is Apple’s general-purpose 2D graphics framework. You can use it to draw shapes, particles, text, sprites and video.

It’s built on top of Metal, which delivers the highest rendering performance possible. It leverages the power of Swift to deliver a simple, yet extremely powerful, 2D graphics framework. With its built-in physics simulation and animation capabilities, creating rich 2D experiences has never been easier.

Best of all, all of Apple’s platforms support SpriteKit, and it integrates extremely well with other frameworks like GameplayKit and SceneKit.

So start Xcode, it’s time to create the project.

Creating a SpriteKit AR project

Create a new project in Xcode. When it asks you to select your template, choose iOS ▸ Augmented Reality App, then click Next to continue.

Change the Product Name to EmojiPop and choose SpriteKit for the Content Technology. You’ll use a Storyboard UI, so leave the Interface as-is and leave the Language as Swift.

Turn off Include Tests, then click Next to continue:

Choose a secure location to save your project. The Desktop is a great location for quick projects. You don’t need to create a Git repository, so turn that off for now and click Create to complete the process.

Xcode will now generate a bare-bones SpriteKit-based Augmented Reality project for you. Once it’s done, you’ll have a fully-functional project that looks like this:

Before doing anything else, take the project for a quick spin. Connect your device and do a quick build and run to deploy it.

Take the project for a walk in your garden. Tap the screen to spawn lots of little Space Invaders all over. Nice!

A few things to note:

• Position: When you tap the screen, a critter spawns into existence based on the position of your device in real-world space.

• Orientation: Did you notice how the little critter keeps looking at you, even when you move? Don’t freak out, this is just a cool feature known as billboarding, which makes 2D sprites always face the camera in 3D space. That way, you’ll never see the flat side of the image.

• Anchors: Once they spawn into the world, the little critters maintain their position in real-world space, no matter where you move. This is due to anchoring, which connects the virtual object to the real world, keeping the anchor at a constant position.

• Debug Info: At the bottom-right of the screen, you’ll see some debug information. In this instance, you see how many critters have spawned in the form of nodes. You can also see the current frame rate, running at a smooth 60 frames per second.

OK, enough fresh air, go back to your workspace and take a look at what’s inside the project.

Exploring the project

In Xcode, with the project open, explore the important components that Xcode generated for you based on the SpriteKit Augmented Reality Template project.

AppDelegate.swift

This is the standard starting point of your app.

LaunchScreen.storyboard

The launch screen is another standard part of every app. It’s the first thing the user sees when they launch your app.

This is where you’ll place a beautiful splash image that represents your app.

Main.storyboard

The main storyboard is the view component of your AR app, containing the app’s UI. This is a good place to put buttons and heads-up displays, for example.

Take particular note of the ARSKView scene view class, which lets you overlay an AR scene over a live background image feed from the camera. It provides seamless integration between ARKit and SpriteKit. Also, note that the view is connected to an @IBOutlet defined in ViewController.swift.

ViewController.swift

The view controller contains the code behind the entire AR experience, specifically for the main storyboard.

Take note of the frameworks at play:

• UIKit: This framework provides the required infrastructure for iOS and tvOS apps including the window and view architecture to implement the UI, event handling and input. It also provides support for animation, document, drawing, printing, device information, text and display, search, accessibility, app extension and resource management.

• SpriteKit: The framework for 2D graphic support.

• ARKit: The framework for AR support.

The ViewController inherits directly from the standard UIViewController, which provides the infrastructure for managing the views of a basic UIKit-based app.

It also adopts the ARSKViewDelegate protocol from ARKit, which contains methods you can implement to synchronize your SpriteKit content with your AR session.

Take special note of @IBOutlet. It connects to ARSKView, which is defined in the Main.storyboard.

Look at viewDidLoad() and you’ll see that it enables the showFPS and showNodeCount debug information for the scene view. This is also where the app loads and presents the default SKScene scene named Scene.

viewWillAppear(_:) is where an ARWorldTrackingConfiguration instance is created. This configuration is provided to the view’s ARSession when the user starts it.

Scene.sks

This defines an empty SpriteKit scene.

This is the scene that’s loaded and presented in the view controller.

Scene.swift

This contains the code behind the SpriteKit scene.

It defines a Scene class that inherits from SKScene. It provides overrides like didMove(to:), which is called when the scene is presented, and update(_:), which is called once every frame. As always, this is where you can handle touch input too.

Assets.xcassets

Here, you’ll find your stock-standard app assets like your app icon, for example.

Note: There are a bunch of icons in starter/resources/AppIcon. Feel free to drag and drop them here to give your game a cool-looking icon.

Info.plist

When your app runs for the first time, it has to ask for permission to access the camera. ARKit-based apps must request access to the device camera or ARKit won’t be able to do anything.

Privacy - Camera Usage Description is the message the user will see when your app requests access to the camera upon start. Feel free to change the description to something more human-readable, like: AR experience requires access to camera.

ARSKView & ARSession

The ARSKView (Augmented Reality SpriteKit View) is a special class used to create 2D SpiteKit AR experiences. It allows you to place 2D content into 3D space within the camera view.

The view includes an ARSession object, which is responsible for ARKit’s motion tracking and image processing. It’s session-based, which means you have to create an AR session instance then run it to start the AR tracking process.

Creating a heads-up display (HUD)

For this particular AR experience, you’ll need a basic Heads-Up Display (HUD) to show the player important information.

Note: To save time building the UI, you’ll take a few shortcuts to keep things short and simple, but still functional.

Open Main.storyboard and get ready to add a HUD to it. In this instance, the HUD will just be a Label.

Open the Object Library and search for a UILabel. Drag and drop it onto the ARView in the design space, snapping it nicely at the center of the top of the screen.

Adjust the label size so it fits across the width of the screen and set the height to 40 units.

Add some Constraints to keep the label at the top and stretched across the screen. Constrain the label to the top, left, right and height. Finally, select Add 4 Constraints to apply the constraints so the label is fully responsive.

Under the Attributes Inspector, clear the Text value. Change the Color to White and set the Font to System Bold 21.0. Lastly, set the Alignment to Centered.

Rename the label to HUD, then open a side-by-side view. Select ViewController.swift so it’s open at the side.

Hold down the Control key then, from the storyboard side, click and drag a connection from the HUD label into ViewController.swift to insert an outlet.

Name the outlet hudLabel and select Connect to create the @IBOutlet..

Great, now you can access the label that you’ll use as the HUD.

Updating the HUD

With the HUD in place, you need a way to update the displayed message while the game is running.

Open Scene.swift and add the following helper function to bottom of Scene:

func updateHUD(_ message: String) {
  guard let sceneView = self.view as? ARSKView else {
    return
  }
  let viewController = sceneView.delegate as! ViewController
  viewController.hudLabel.text = message
}

With this function in place, you’ll be able to update the message displayed in the HUD. Now, you can provide valuable instructions to the player.

Adding game state

A good way to control the game is to add some kind of game state management. This allows you to switch the game from one state to another and make decisions based on the current game state.

Open Scene.swift and add the following enum to the top of it, just after the imports section:

public enum GameState {
  case Init
  case TapToStart
  case Playing
  case GameOver
}

Your game will use the following states:

• Init: While in this state, the key components of the game are still being initialized. Once everything is ready to go, the game moves into a TapToStart state.

• TapToStart: While in this state, the HUD will display the message TAP TO START, which is an instruction to the player to tap the screen to start the game. Once the player taps the screen, the app creates an AR anchor and places a little box in view of the player. The box acts as a visual indicator that shows the emojis’ spawn point to the player. The game starts and moves into the Playing state.

• Playing: While in this state, emojis will spawn into existence from the box at the spawn point. The player has to catch each emoji before it falls to its death. During this time, the HUD displays the player’s current score and total lives left. Once all lives are lost, the game moves into a GameOver state.

• GameOver: While in the this state, the HUD displays Game Over along with the player’s final score. The player can tap again to continue, which will move the game back into the TapToStart state.

Declaring game variables

Other than the game state, you’ll use a few other variables to control important aspects of your game.

Declare the following variables at the top of Scene:

var gameState = GameState.Init
var anchor: ARAnchor?
var emojis = "😁😂😛😝😋😜🤪😎🤓🤖🎃💀🤡"
var spawnTime : TimeInterval = 0
var score : Int = 0
var lives : Int = 10

This code declares six new properties. Here’s what each one does:

• gameSate: This maintains the current game state. You’ll use it to control the game.
• anchor: This contains the only AR anchor for the game. When the player starts the game, it creates a single anchor. This acts as the location in the real world where the emojis will spawn.
• emojis: This is a string filled with a bunch of fun emojis. The game will randomly spawn any one of these emojis while the user is playing.
• spawnTime: This timed interval controls the rate of spawn. The app uses this to spawn an emoji every 0.5 seconds.
• score: This stores the player’s current score, incrementing every time the player saves an emoji.
• lives: When the player misses an emoji, the emoji falls to its death and the player loses a life. This variable keeps track of the player’s available lives. Once this number reaches 0, the game ends.

Add the following game state management functions to the bottom of Scene:

public func startGame() {
  gameState = .TapToStart
  updateHUD("- TAP TO START -")
}

public func playGame() {
  gameState = .Playing
  score = 0
  lives = 10
  spawnTime = 0
}

public func stopGame() {
  gameState = .GameOver
  updateHUD("GAME OVER! SCORE: " + String(score))
}

Now you can control the current state of the game by calling these different functions. Take a look at them in detail:

• startGame(): Places the game into the TapToStart state and displays the message: TAP TO START.
• playGame(): Places the game into the Play state and resets the score, lives and spawnTime.
• stopGame(): Places the game into the GameOver state and displays the message: GAME OVER! SCORE: followed by the actual score for the last game played.

Replace the existing contents of touchesBegan(_:with:) with the following switch statement:

switch (gameState)
{
  case .Init:
    break
      
  case .TapToStart:
    playGame()
    break
      
  case .Playing:
    //checkTouches(touches)
    break
      
  case .GameOver:
    startGame()
    break
}

The way you track touches in your game will vary depending on the game’s current state. The switch controls the flow of touch events based on the current game state.

• Init: While in this state, the app ignores all touch input.
• TapToStart: Here, the app is waiting for touch input. When the player touches the screen, the app starts the game.
• Playing: In this state, the app checks if the player touched a spawned emoji. If they did, the app will remove that emoji.
• GameOver: Once in this mode, the game is over. When the player taps the screen, the app restarts the game.

Note: The call to checkTouches() is currently commented out because that function doesn’t exist yet. You’ll add it a little later.

Creating a spawn point

With all that in place, it’s time to start the game. When the app starts, the view controller will load Scene.sks. Once loaded, the app presents the scene to the user and calls didMove(to:). This is a great place to start the game.

With Scene.swift still open, add a call to startGame() in didMove(to:):

startGame()

The game is placed in TapToStart state and the player receives the instruction to tap the screen to start the game.

Now, when the player does tap the screen, the app has to create an anchor along with a spawn point.

Add the following function to the bottom of Scene:

func addAnchor() {
  // 1
  guard let sceneView = self.view as? ARSKView else {
    return
  }
  // 2
  if let currentFrame = sceneView.session.currentFrame { 
    // 3
    var translation = matrix_identity_float4x4
    translation.columns.3.z = -0.5
    let transform = simd_mul(currentFrame.camera.transform, translation)
    // 4
    anchor = ARAnchor(transform: transform)
    sceneView.session.add(anchor: anchor!)
  }
}

Take a closer look at what’s happening here:

1. This casts the view as an SKSView so you can access the current AR session.

2. This gets the current active frame from the AR session, which contains the camera. You’ll use the cameras transform information to create an AR anchor in front of the camera view.

3. This calculates a new transform located 50cm in front of the camera’s view.

4. Finally, this creates an AR anchor with the new transform information and adds it to the AR session.

Now, to call to this function, add the following to the bottom of playGame():

addAnchor()

When the player taps the screen, the game will change state and add an AR anchor 50cm in front of the player.

Now, for the reverse. When the game restarts, you need to remove the previously-added anchor.

To do this, add the following function to the bottom of Scene:

func removeAnchor() {
  guard let sceneView = self.view as? ARSKView else {
    return
  }
  if anchor != nil {
    sceneView.session.remove(anchor: anchor!)
  }
}

This checks whether there’s already an active anchor. If there is, it removes the anchor from the AR session.

Now, add a call to removeAnchor() at the bottom of startGame():

removeAnchor()

Excellent! When the game restarts now, the app removes any existing anchors along with all the SpriteKit nodes associated with it.

ARSKViewDelegate

If you recall, the ViewController adopted the ARSKViewDelegate protocol. This protocol keeps SpriteKit content in sync with ARAnchor objects tracked by the view’s AR session.

It offers the following functions that you can use:

• func view(_:nodeFor:) -> SKNode: Call this when the app adds a new AR anchor. Note that it returns a SKNode, so this is a good place to create and link a SpriteKit node to the newly-added AR anchor.

• func view(_:didAdd:for:): Informs the delegate that a SpriteKit node related to a new AR anchor has been added to the scene.

• func view(_:willUpdate:for:): Informs the delegate that a SpriteKit node will be updated based on changes to the related AR anchor.

• func view(_:didUpdate:for:): Informs the delegate that a SpriteKit node has been updated to match changes on the related AR anchor.

• func view(_:didRemove:for:): Informs the delegate that the SpriteKit node has been removed from the scene on the related AR anchor.

Adding a spawn point

After the app creates the AR anchor, you’ll use the delegate to provide a SKNode for the new anchor. This SpriteKit node acts as the Spawn Point for the game.

Open ViewController.swift, then find view(_:nodeFor:) -> SKNode and replace its contents with the following:

// 1
let spawnNode = SKNode()
spawnNode.name = "SpawnPoint"
// 2
let boxNode = SKLabelNode(text: "🆘")
boxNode.verticalAlignmentMode = .center
boxNode.horizontalAlignmentMode = .center
boxNode.zPosition = 100
boxNode.setScale(1.5)
spawnNode.addChild(boxNode)
// 3    
return spawnNode

Take a look at the code:

1. This creates an empty SpriteKit node and sets its name to SpawnPoint.

2. To give the player a visual indicator of where the spawn point is in the real world, this creates a little SOS box and adds it as a child of the spawn point node.

3. Finally, the spawnNode is provided as the SKNode for the newly-added AR anchor. This also links the spawn node to the AR anchor. Any changes to the AR anchor will be synced to the spawn node.

Do a quick build and run to test how it works.

The game starts and the HUD shows TAP TO START. When the player taps the screen, a small SOS box spawns into view, anchored to that location.

This might not look like much, but you’re making great progress. You’ve gotten all the ground work out of the way.

Handling problems with the AR session

Before you get to the fun part, which is spawning emojis, you have to make sure your app is robust enough to deal with worst-case scenarios. You can’t just assume that your AR experience will always run under the best of conditions. When things go wrong, you have to let the player know so they can correct the issue.

AR issues come in the following forms:

• AR Session Failures: Typically occur when the AR session has stopped due to some kind of failure.

• AR Camera Tracking Issues: These occur when the quality of ARKit’s position tracking has degraded for some reason.

• AR Session Interruptions: This issue happens when the session has temporarily stopped processing frames and device position tracking — typically because the player took a phone call or switched to a different app.

You’ll use an alert message to notify the player of any issues.

With ViewControll.swift open, add the following helper function to the bottom of ViewController:

func showAlert(_ title: String, _ message: String) {
  let alert = UIAlertController(title: title, message: message, 
    preferredStyle: .alert)
  alert.addAction(UIAlertAction(title: "OK",
    style: UIAlertAction.Style.default, handler: nil)) 
  self.present(alert, animated: true, completion: nil)
}

This defines a function named showAlert(_:_:), which creates and presents a basic alert with the provided title and message. You also add an OK action button so the player can dismiss the alert.

Handling AR session failures

The only thing you really can do when an AR session fails is to inform the player of the issue.

Add the following line of code to session(_:didFailWithError:):

showAlert("Session Failure", error.localizedDescription)

When the session fails, the player will see an alert message with the detailed information from the provided error message in error.localizedDescription.

Handling camera tracking issues

When the AR tracking conditions degrade, you can check a few things to try to determine what the problem is. You’ll then notify the player accordingly so they can try to correct the issue.

Add the following function to ViewController:

func session(_ session: ARSession, 
  cameraDidChangeTrackingState camera: ARCamera) {
// 1
switch camera.trackingState {
  case .normal: break
  case .notAvailable:
    showAlert("Tracking Limited", "AR not available")
    break
  // 2
  case .limited(let reason):
    switch reason {
    case .initializing, .relocalizing: break
    case .excessiveMotion:
      showAlert("Tracking Limited", "Excessive motion!")
      break
    case .insufficientFeatures:
      showAlert("Tracking Limited", "Insufficient features!")
      break
    default: break
    }
  }
}

When the camera changes tracking state, the app notifies this delegate function. You then need to interrogate the provided tracking state for more information.

Take a look at what exactly it’s doing:

1. You can access the current tracking state through the provided camera. Th switch statement then handles all possible cases. If there’s a problem, you then notify the player with an alert message.

2. When tracking is limited, you can dig deeper to find out exactly why. Again, you have a few cases to deal with. You’ll then send the player an alert message with the result.

Handling AR session interruptions

If something like a phone call or switching to another app interrupts the AR session, there’s a good chance you’ll have to restart everything. Luckily, there are delegates that help you handle this.

Add the following to sessiongWasInterrupted(_:):

showAlert("AR Session", "Session was interrupted!")

When the player returns to the game, this simply notifies the player that there was an interruption to the game so it stopped.

Add the following to sessionInterruptedEnded(_:):

let scene = sceneView.scene as! Scene
scene.startGame()

This makes sure your game restarts properly, by moving the game into a TAP TO START state. It also removes all the SpriteKit nodes and the spawn point anchor.

Do a final build and run to test all the changes. You can test a few things now:

• Session Interruption: Start the game, switch over to another app, then return to the game. You’ll get an alert message stating Session was Interrupted and the game will return to the TAP TO START state.

• Limited Camera Tracking: Stick your finger over the camera lens so the scene goes dark. This forces a limited tracking issue, and you’ll receive an alert message with the exact reason why.

Key points

Fantastic, you’ve reached the end of this chapter. You can find a copy of the project in its current state under final/EmojiPop.

Here’s a quick recap of what you’ve learned:

• ARKit & SpriteKit: You’ve learned how easy it is to create an ARKit-based project that uses SpriteKit as the key content technology. You also got an in-depth overview of the project content that the AR project template generated for you.

• ARSKView & ARSession: You now know about the AR view that’s responsible for rendering augmented SpriteKit content. You also know about the AR session that’s responsible for ARKit’s motion tracking and image processing.

• HUD: You learned how to create a basic heads-up display using the standard storyboard with a label. This is a simple way to give the player important alerts and updates.

• Game State Management: You implemented basic game state management, which allows you to keep things under control based on the current state of the game.

• ARAnchor & ARSKViewDelegate: You learned how to add an anchor to an AR session and how to keep your SpriteKit content synchronized by using ARSKViewDelegate to track when an anchor is added, updated or removed.

• AR Session Issues: Elegantly handling possible AR session-related issues is vital makes your AR apps robust, delivering a high-quality AR experience for the player.

Go grab yourself a well deserved break, but don’t stay away too long. In the next chapter, you’ll finally get to spawn those emojis — and you’ll get to move them with physics!