Detect Hand & Body Poses with Vision in iOS

Learn how to detect hand and body landmarks in live video with help from the Vision framework. Explore the kind of data Vision can provide, and use it to evaluate hand and body poses for simple gestures. By Catie Catterwaul.

Leave a rating/review
Download materials
Save for later
Comments
Share

Who is this for?

This course will be best for experienced iOS developers who are comfortable with Swift. It focuses exclusively on using the Vision framework to detect and evaluate hand and body poses via live video capture. The UI is not a focus of this course, but the project does use SwiftUI and UIKit.

Covered concepts

  • Vision
  • AVFoundation
  • Hand Pose Request & Observation
  • Hand Anatomy
  • VNRecognizedPoint

Part 1: Detect Hand & Body Poses with Vision in iOS

01
Toggle description

Learn a bit about what the Vision framework is, how it relates to other Apple frameworks, and how we’ll use it in this course.

Toggle description

Get a short tour of the sample app, and prepare a live camera feed with AVFoundation.

Toggle description

Start detecting hands from live video with a VNDetectHumanHandPoseRequest, and get back data about all of the visible joints in your hands with VNHumanHandPoseObservation.

Toggle description

Learn about how all your hand bones are connected together in this quick anatomy lesson!

Toggle description

Explore and organize the hand landmark data received from Vision’s VNHumanHandPoseObservations.

Toggle description

Convert Vision’s landmarks for use in a SwiftUI view to display fingertip locations.

Toggle description

Collect specific hand landmarks and evaluate their positions for potential hand gesture matches.

Toggle description

The thumbs are a bit different than the rest of the phalanges! Learn how to handle thumb data.

Toggle description

Detecting whole bodies with Vision is quite similar to the hand detection you’ve already done. Try it out!

Toggle description

Collect specific body landmarks and evaluate their positions for potential body pose matches.