Pointer Interaction Tutorial for iOS: Supporting the Mouse and Trackpad

Apple has been steering its iPad product range toward productivity and professional use for several years. This includes support for hardware keyboards. And iOS APIs like UIKeyCommand reinforce keyboard support by allowing you, as a developer, to add keyboard shortcuts to your app.

What about the mouse, though? How do you handle a screen pointer on a device designed for touch input? This is where pointer interactions come in. Ever since iOS 13.4, you can connect a trackpad or mouse to your iPad and use an on-screen pointer with your controls.

But pointer interactions are not limited to the presence of an on-screen pointer. Many standard UIKit controls will react to the presence of the pointer with shapes or animations — with little or no work from you! In this tutorial, you’ll add pointer interactions to a simple game app. By doing this, you’ll learn how to:

• Enable built-in pointer interactions
• Customize UIButton interactions
• Apply pointer interactions to other views
• Configure custom pointer shapes
• Perform coordinated animations for pointer movement
• Respond to hover events

And you’ll get to work with pointer interactions in both UIKit and SwiftUI!

To do this tutorial you’ll need Xcode 11.5 or higher. You don’t need hardware or a physical mouse, as the simulator provides a pointer widget for your use.

Getting Started

Use the Download Materials link at the top or bottom of the tutorial to download the starter project. This project is a simple pattern matching game called RaySays. You’ll be adding pointer interactions in multiple places in the code. The project is ready to run, so you can focus on the task of enhancing it with pointer interactions.

Select the iPad Pro (9.7-inch) iPad simulator in the target selector.

Open the Xcode project in the folder RaySays-Starter, then build and run. The UI has implementations in both SwiftUI and UIKit. You’ll be modifying both UI versions during the tutorial.

Now try out the game. When you tap UIKit or SwiftUI then Play, a sequence will flash at you. You’ll repeat the same sequence back. Be prepared for a challenge — the sequence gets longer at every level!

Enabling UIKit Interactions

In this section, you’ll add pointer interactions to the UIKit version of your user interface. Later, you’ll add similar interactions to the SwiftUI version.

Switching on Built-in Behaviors

In this section, you’ll switch on the built-in pointer behavior for UIButton, UIBarButton and UISegmentedControl.

Run the app again, this time staying on the start screen. It has two buttons to start the game and a segmented control to choose a difficulty level.

Capture the pointer by using the simulator’s Capture Cursor button.

You’ll need to remember to do that with every build and run step in this tutorial, or you won’t see any pointer interactions!

When you’re finished, press the Escape key to release the pointer from the simulator.

Move the pointer over the three controls. The two buttons do nothing, but the segmented control responds with a little wriggle. It must be ticklish!

Tap the UIKit button to enter the game view. The back button in the navigation bar also responds to the pointer. These pointer interactions are built in.

Examine the Project navigator. Locate and open the RaySays folder. Open the UIKit View Controllers folder, and then open Main.storyboard. Find the RaySays Scene, which is the root of the initial navigation controller.

In the View hierarchy, expand the RaySays Scene. Continue to expand the view until you see the buttons UIKit Selector and SwiftUI Selector. Select both the green buttons by clicking on them while holding down the Shift key. Open the Attributes inspector and locate the control to enable pointer interaction. Switch it on by clicking the checkbox.

Build and run and capture the pointer. Now the two buttons respond with movement when you move the pointer over them!

Customizing UIButton Interaction

iOS provides a default visual effect you get when you move the pointer over the buttons. In this section, you’ll find out how to choose your own style.

In the Project navigator, find the UIKit View Controllers folder. Locate and open RootViewController.swift. Add this code at the end of the main class:

func configureButtons() {
  uikitSelector.pointerStyleProvider = { button, effect, shape in
    let preview = UITargetedPreview(view: button)
    return UIPointerStyle(effect: .highlight(preview))
  }

  swiftuiSelector.pointerStyleProvider = { button, effect, shape in
    let preview = UITargetedPreview(view: button)
    return UIPointerStyle(effect: .lift(preview))
  }
}

In this code, you set the pointerStyleProvider on each button to give a custom UIPointerStyle. Pointer styles affect the shape of the pointer and the visual appearance of the button. One way to create a pointer style is to use a UIPointerEffect, which is what you are using here. .highlight and .lift are two different effects. Notice how you create a UITargetedPreview to receive the effect. That allows UIKit to mess around with the appearance of your button during pointer interactions without changing the underlying views.

To apply your styles, add this line at the end of viewDidLoad():

configureButtons()

Build and run and use the pointer to observe the differences between the two effects. .highlight is quite subtle, while .lift is more flamboyant.

Another effect you can experiment with is .hover. Have fun!

Applying Pointer Interactions to Other Views

You’ve learned how to apply a pointer effect to a button, but what about other views? In this section, you’ll apply a UIPointerInteraction to any UIView.

Reviewing the Game Controls

First, before making any changes, look at how the existing code works. In the Project navigator, open the folder UIKit View Controllers. In GameViewController.swift, locate configureGameButtons():

func configureGameButtons() {
  for (index, item) in zip(allColors, allButtons).enumerated() {
    let button = item.1
    let color = item.0
    button.color = color
    button.tag = index

    let tapGesture = UITapGestureRecognizer(
      target: self,
      action: #selector(gameButtonAction(_:))
    )
    button.addGestureRecognizer(tapGesture)
  }
}

This is where you set up the four large color tiles that make up the game controls. You apply a color, a tag and a UITapGestureRecognizer to each of the four controls.

Each of the tiles is an instance of GameButton.

Now return to the Project navigator. In UIKit View Controllers, open GameButton.swift. GameButton is a subclass of UIView. You’ll now add a pointer effect to instances of this view.

Adding Tracking Variables

First, you need to add some tracking variables to GameButton. Add these two properties inside the body of the main class:

var pointerLocation: CGPoint = .zero {
  didSet {
    setNeedsDisplay()
  }
}
  
var pointerInside = false {
  didSet {
    setNeedsDisplay()
  }
}

These properties allow you to keep track of the location of the pointer and whether the pointer is inside the view. Each time they change, you mark the view as needing a redraw. This will come in handy soon!

Adding a Delegate Extension

Next, add this delegate extension to the end of the file:

extension GameButton: UIPointerInteractionDelegate {
  //1
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    regionFor request: UIPointerRegionRequest,
    defaultRegion: UIPointerRegion
  ) -> UIPointerRegion? {
    pointerLocation = request.location
    return defaultRegion
  }

  //2
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    styleFor region: UIPointerRegion
  ) -> UIPointerStyle? {
    return nil
  }

  //3
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    willEnter region: UIPointerRegion,
    animator: UIPointerInteractionAnimating
  ) {
    pointerInside = true
  }

  //4
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    willExit region: UIPointerRegion,
    animator: UIPointerInteractionAnimating
  ) {
    pointerInside = false
  }
}

In this code, you define four delegate methods in an extension to GameButton. The game button object is the view and also the delegate. The app calls these four methods during the various parts of the lifecycle:

1. The pointer has moved, or is about to move, within the view. You, as the delegate, can return a UIPointerRegion. This is a sub-rectangle relative to the view’s coordinate space. You record the current location and return defaultRegion, i.e. bounds.
2. What kind of pointer style do you want the system to apply to this rectangular region you returned? For now, it’s nothing!
3. The pointer is about to enter a region.
4. The pointer is about to leave a region.

These four methods implement UIPointerInteractionDelegate. You use this protocol to define pointer behavior within the pointer interaction lifecycle. This lifecycle has three participants:

• The view receives support for pointer interaction.
• The system calls delegate methods as the pointer interacts with the view.
• The delegate responds to those delegate method calls.

As you can see, UIPointerInteractionDelegate offers very fine-grained control over the behavior of the pointer when it’s within a view.

Adding a Pointer Effect

Finally, locate awakeFromNib(). The system calls this method when you create a view from a XIB or Storyboard. At this point, everything in the storyboard or XIB has been created.

Add these lines to the start of awakeFromNib():

let interaction = UIPointerInteraction(delegate: self)
addInteraction(interaction)

You’ll now use this information to apply some super-fancy and useful drawing to the view. Add this stored property and function to the body of the main class:

let blobSize = CGFloat(60)

override func draw(_ rect: CGRect) {
  if pointerInside {
    let rect = CGRect(center: pointerLocation, size: blobSize)
    let blob = UIBezierPath(ovalIn: rect)
    UIColor.white.set()
    blob.fill()
  }
}

Build and run. Go to the UIKit section of the app and capture the pointer. When the cursor enters any of the four game buttons, you can see that you draw a circle at the cursor point — which is neither useful nor fancy! :]

OK, even though this code isn’t useful and fancy, it does demonstrate the potential to react to the position of the cursor within a view. You’ll have some fun with this circle later in the tutorial.

Supplying a Custom Pointer Shape

The next thing you’ll do is create a custom shape for the cursor when it’s inside the button. Remember UIPointerStyle, which you created to give lift and highlight effects to your buttons? You can also create a style which is all about changing the shape of the cursor. To do this, you use UIPointerShape.

UIPointerShape is an enum. It has three predefined shapes, or it can use any UIBezierPath:

public enum UIPointerShape {
  case path(UIBezierPath)
  case roundedRect(CGRect, radius: CGFloat = UIPointerShape.defaultCornerRadius)
  case verticalBeam(length: CGFloat)
  case horizontalBeam(length: CGFloat)

  public static let defaultCornerRadius: CGFloat
}

You’ll use the path initializer to create a shaped cursor. Still in GameButton.swift, locate pointerInteraction(_:styleFor) inUIPointerInteractionDelegate.

Find the statement:

return nil

Replace it with this code:

let hand = UIBezierPath(svgPath: AppShapeStrings.hand, offset: 24)
return UIPointerStyle(shape: UIPointerShape.path(hand))

Here you create a path using an open-source utility that converts Scalable Vector Graphics (SVG) data to a UIBezierPath. You can export SVG from many vector art applications, such as Sketch. The offset parameter moves the path a bit to center it on the cursor.

Build and run. Go to the UIKit section of the app and capture the pointer. Now, when you enter a game button, the cursor will morph into the new shape. That’s handy! :]

The ability to create a shaped cursor is fun and cool, but it also has practical application. A custom shape can provide extra contextual information about an operation. Your customer will appreciate that!

Coordinating Animations

You’ve seen how we can animate the pointer when it enters and exits a region. In this section, you’ll find out how to perform additional coordinated animations alongside those of the pointer itself.

In GameButton.swift, go to the UIPointerInteractionDelegate delegate extension you added earlier. Take a look at pointerInteraction(_:willExit:animator:) and pointerInteraction(_:willEnter:animator:).

The last parameter, animator, is an opaque object that conforms to UIPointerInteractionAnimating.

This protocol has two methods: addAnimations(_:) and addCompletion(_:)

Now, you’ll supply an animation related to the circle you drew earlier. Add this extension to GameButton.swift:

extension GameButton {
  func animateOut(_ origin: CGPoint)
    -> (animatedView: UIView, animation: () -> Void) {
      //1
      let blob = UIView(frame: CGRect(center: origin, size: blobSize))
      blob.backgroundColor = UIColor.white
      blob.layer.cornerRadius = blobSize / 2
      self.addSubview(blob)

      //2
      return (blob, {
        blob.frame = CGRect(center: self.bounds.center, size: self.blobSize / 10)
        blob.layer.cornerRadius = self.blobSize / 20
        blob.backgroundColor = self.color
      })
  }
}

In this method, you:

1. Create a view that looks the same as the circle you draw in draw(_:), then add that view to the button.
2. Supply a closure that states the end values of frame and color.

Now locate pointerInteraction(_:willExit:animator:). Find the line:

pointerInside = false

Following this line, add the code:

let animation = animateOut(pointerLocation)
animator.addAnimations(animation.animation)
animator.addCompletion { _ in
  animation.animatedView.removeFromSuperview()
}

Here you add a closure to the animator. Then you strip the animated view when the animation has finished.

Build and run. Go to the UIKit section of the app and capture the pointer. Now, each time the pointer leaves a game button, the circle will animate back into the center of the button. animator has done the work of animating the change for you!

Responding to Hover Events

Apple introduced Catalyst with iOS 13.0 and macOS 10.15. Catalyst gave you the ability to build a macOS app with the UIKit API. To provide compatibility with a screen pointer on macOS, Apple added a new gesture, UIHoverGestureRecognizer. It wasn’t until iOS 13.4 and the rest of the pointer interaction changes arrived that this had any effect in iOS.

In the Project navigator, look in the UIKit View Controllers folder. Open Main.storyboard. Pan to the right side of the storyboard, and you’ll see a view controller with a sad cat emoji. This is the view you see when you lose the game.

Once again, look in the UIKit View Controllers folder. Now open LoseViewController.swift.

There’s not much to see in this class. There is already a UITapGestureRecognizer to allow the view to be dismissed. Now you’ll add a hover gesture. This will display a speech bubble when the pointer is inside the central area.

Add this method to LoseViewController:

@objc func hoverOnCentralView(_ gesture: UIHoverGestureRecognizer) {
  let animationSpeed = 0.25
  switch gesture.state {
  //1
  case .began:
    centralLabel.text = happyCat
    UIView.animate(withDuration: animationSpeed ) {
      self.speechBubble.alpha = 1.0
      self.speechBubble.transform = CGAffineTransform(scaleX: 1.1, y: 1.1)
    }
  //2
  case .ended:
    centralLabel.text = sadCat
    UIView.animate(withDuration: animationSpeed ) {
      self.speechBubble.alpha = 0
      self.speechBubble.transform = .identity
    }
  default:
    print("message - unhandled state for hover")
  }
}

In this code, you display and dismiss the speech bubble:

1. When the pointer enters the tracked view, the speech bubble becomes opaque and expands.
2. When the pointer exits the tracked view, the speech bubble becomes transparent and returns to the identity transform.

Next, add these lines to configureGestures():

let hover = UIHoverGestureRecognizer(
  target: self,
  action: #selector(hoverOnCentralView(_:))
)
centralView.addGestureRecognizer(hover)

Here you add the hover gesture to UIStackView, in the middle of the main view.

Finally, in viewDidLoad(), find this line:

speechBubble.alpha = 1

Change it to this:

speechBubble.alpha = UIDevice.current.userInterfaceIdiom == .pad ? 0 : 1

In general, people use a pointer device only with an iPad. On an iPhone, you always want the speech bubble to remain visible.

Build and run and capture the pointer. Go to the UIKit section. Play a game! Now, when you lose, the speech bubble will react to the position of the pointer. Sad cat becomes happy cat!

You’ve now completed the UIKit section of the tutorial. Maybe it’s time for a break while you try to beat your high score!

Enabling SwiftUI Interactions

In this section, you’ll add similar interactions to the SwiftUI version of the user interface. Unfortunately, the hover API in SwiftUI is not yet as rich as that in UIKit. But it still provides you with a great opportunity to enhance your app.

Adding a Simple Hover Effect

In this section you’ll add a lift effect to the Play/Reset button.

Go to the Project navigator. Open the SwiftUI Views folder, and then open GameView.swift. This is the game interface. This interface is very similar to the UIKit version.

In the body of GameView, locate GameKeyboard(gameEngine: gameEngine). Below that, you’ll find Button.

At the end of the button description is a comment, //add button hover here. Delete the comment, and replace it with this line:

.hoverEffect(.lift)

Build and run and capture the pointer. Go to the SwiftUI section of the app. Now the Play button will react whenever the pointer enters the button.

.hoverEffect is a modifier on View. This means it’s not limited to buttons — you can use it for anything on the screen that the pointer passes over!

Adding an OnHover Handler

You can also add more complex pointer interactions to your SwiftUI. To do that, you’ll need to detect the presence of the pointer within a view. In this section, you’ll upgrade the end screen for a lost game, as you did for UIKit.

In the Project navigator, in the SwiftUI Views folder, open LoseView.swift.

First, add these properties in LoseView:

@State var isHoverInside: Bool = false

var shouldDisplayHover: Bool {
  return isHoverInside || UIDevice.current.userInterfaceIdiom == .phone
}

isHoverInside will be updated when the pointer enters the view you’re interested in. shouldDisplayHover decides if you should show the hover state or not. iPhones don’t have pointer interactions, so you will always show the effect on iPhone. Remember, you must not use pointer interactions as the only way to access functionality or information, as they are not available to all users.

Next, locate the comment //add on hover here at the bottom of ZStack. Replace it with this:

//1
.onHover { inside in
  self.isHoverInside = inside
}
//2
if shouldDisplayHover {
  HStack {
    Rectangle().foregroundColor(.clear)
    GeometryReader { _ in
      SpeechBubble(message: "It's OK.\nTap anywhere to try again")
        .offset(self.bubbleOffset)
    }
  }
}

In this block:

1. You add an onHover event to the central VStack. You update the value of isHoverInside during the event. onHover only supplies one piece of information: It tells you whether the pointer is in or out.
2. You use the state of isHoverInside to add SpeechBubble conditionally to the outer ZStack.

Finally, make the cat happy! Find this line in the inner VStack:

Text("\(sadCat)")

Change it to this:

Text("\(isHoverInside ? happyCat : sadCat)")

Build and run and capture the pointer. Go to the SwiftUI section of the app. When you lose, the speech bubble will appear when you place the pointer in the central area.

In the interest of not over-baking the cake, you’ll finish here. You’ve seen that SwiftUI allows you to interact with the pointer, but with less control than UIKit.

Where to Go From Here?

You can download the finished project using the Download Materials button at the top or bottom of this tutorial.

In this Pointer Interaction Tutorial for iOS, you learned how to:

• Add simple automatic pointer effects.
• Provide custom pointer effects.
• Interact with the pointer position and animation sequence.

You’ve seen that pointer interactions can supplement the normal touch interactions in an iOS application. And you’ve learned how you can add these interactions with very little work! You’ve also seen that UIKit has a rich pointer API that SwiftUI doesn’t yet match. Hopefully, you’ll see some of these interesting interactions in future releases of SwiftUI.

For now, most people won’t be using a pointer device with their iPad. So developing complex pointer interactions might not be the best use of your time. But it is a way to add some fun and charm to the UI. Think of the cool Easter eggs you could make!

Remember to check the Human Interface Guidelines for pointer interactions.

I look forward to seeing what you do with this API in your own applications. If you have any suggestions or questions, or if you want to show off what you did to improve this project, join the discussion below.