Building a Camera App With SwiftUI and Combine

SwiftUI enables developers to design and create working user interfaces almost as quickly as they could prototype one in Sketch or Photoshop. Think about how powerful this is. Instead of making static mockups, you can create working prototypes for almost the same amount of effort. How cool is that?

Add Combine as a data pipeline to the mix, and you’ve got the Batman and Robin of the programming world. You can decide which one is Batman and which one is Robin. :]

Some people may complain that SwiftUI and Combine aren’t ready for prime time, but do you really want to tell Batman he can’t go out and fight crime? In fact, would you believe you can write a camera app completely in SwiftUI without even touching UIViewRepresentable?

Creating a camera app using SwiftUI and Combine makes processing real-time video easy and delightful. Video processing can already be thought of as a data pipeline. Since Combine manages the flow of data like a pipeline, there are many similarities between these patterns. Integrating them allows you to create powerful effects, and these pipelines are easy to expand when future features demand.

In this tutorial, you’ll learn how to use this dynamic duo to:

• Manage the camera and the video frames it captures.
• Create a data pipeline in Combine to do interesting things with the captured frames.
• Present the resulting image stream using SwiftUI.

You’ll do these with an app called Filter the World. So, get ready to filter the world through your iPhone — even more so than you already do!

Getting Started

Click the Download Materials button at the top or bottom of this tutorial. There’s currently not a lot there, aside from a custom Error, a helpful extension to convert from a CVPixelBuffer to a CGImage, and some basic SwiftUI views that you’ll use to build up the UI.

If you build and run now, you won’t see much.

Currently, there’s a blank screen with the name of the app in the center.

If you want to make a camera-based app, what’s the most important thing you need? Aside from having a cool name, being able to display the camera feed is probably a close second.

So that’s exactly what you’ll do first!

Displaying Captured Frames

If you were going to use a UIViewRepresentable, you’d probably opt for attaching an AVPreviewLayer to your UIView, but that’s not what you’re going to do! In SwiftUI, you’ll display the captured frames as Image objects.

Since the data you get from the camera will be a CVPixelBuffer, you’ll need some way to convert it to an Image. You can initialize an Image from a UIImage or a CGImage, and the second route is the one you’ll take.

Inside the Views group, create an iOS SwiftUI View file and call it FrameView.swift.

Add the following properties to FrameView:

var image: CGImage?
private let label = Text("Camera feed")

When adding FrameView to ContentView in a little bit, you’ll pass in the image it should display. label is there to make your code in the next step a little bit cleaner!

Replace Text in the body with the following:

// 1
if let image = image {
  // 2
  GeometryReader { geometry in
    // 3
    Image(image, scale: 1.0, orientation: .upMirrored, label: label)
      .resizable()
      .scaledToFill()
      .frame(
        width: geometry.size.width,
        height: geometry.size.height,
        alignment: .center)
      .clipped()
  }
} else {
  // 4
  Color.black
}

In this code block, you:

1. Conditionally unwrap the optional image.
2. Set up a GeometryReader to access the size of the view. This is necessary to ensure the image is clipped to the screen bounds. Otherwise, UI elements on the screen could potentially be anchored to the bounds of the image instead of the screen.
3. Create Image from CGImage, scale it to fill the frame and clip it to the bounds. Here, you set the orientation to .upMirrored, because you’ll be using the front camera. If you wanted to use the back camera, this would need to be .up.
4. Return a black view if the image property is nil.

Great work! Now you need to hook it up in the ContentView.

Open ContentView.swift and replace the contents of ZStack with:

FrameView(image: nil)
  .edgesIgnoringSafeArea(.all)

This adds the newly created FrameView and ignores the safe area, so the frames will flow edge to edge. For now, you’re passing in nil, as you don’t have a CGImage, yet.

There’s no need to build and run now. If you did, it would show up black.

It’s still a blank screen. Is that really an improvement?

To display the frames now, you’ll need to add some code to set up the camera and receive the captured output.

Managing the Camera

You’ll start by creating a manager for your camera — a CameraManager, if you will.

First, add a new Swift file named CameraManager.swift to the Camera group.

Now, replace the contents Xcode provides with the following code:

import AVFoundation
// 1
class CameraManager: ObservableObject {
  // 2
  enum Status {
    case unconfigured
    case configured
    case unauthorized
    case failed
  }
  // 3
  static let shared = CameraManager()
  // 4
  private init() {
    configure()
  }
  // 5
  private func configure() {
  }
}

So far, you’ve set up a basic structure for CameraManager. More specifically, you:

1. Created a class that conforms to ObservableObject to make it easier to use with future Combine code.
2. Added an internal enumeration to represent the status of the camera.
3. Included a static shared instance of the camera manager to make it easily accessible.
4. Turned the camera manager into a singleton by making init private.
5. Added a stub for a configure() you’ll fill out soon.

Configuring the camera requires two steps. First, check for permission to use the camera and request it, if necessary. Second, configure AVCaptureSession.

Checking for Permission

Privacy is one of Apple’s most touted pillars. The data captured by a camera has the potential to be extremely sensitive and private. Since Apple (and hopefully you) care about users’ privacy, it only makes sense that the user needs to grant an app permission to use the camera. You’ll take care of that now.

Add the following properties to CameraManager:

// 1
@Published var error: CameraError?
// 2
let session = AVCaptureSession()
// 3
private let sessionQueue = DispatchQueue(label: "com.raywenderlich.SessionQ")
// 4
private let videoOutput = AVCaptureVideoDataOutput()
// 5
private var status = Status.unconfigured

Here, you define:

1. An error to represent any camera-related error. You made it a published property so that other objects can subscribe to this stream and handle any errors as necessary.
2. AVCaptureSession, which will coordinate sending the camera images to the appropriate data outputs.
3. A session queue, which you’ll use to change any of the camera configurations.
4. The video data output that will connect to AVCaptureSession. You’ll want this stored as a property so you can change the delegate after the session is configured.
5. The current status of the camera.

Next, add the following method to CameraManager:

private func set(error: CameraError?) {
  DispatchQueue.main.async {
    self.error = error
  }
}

Here, you set the published error to whatever error is passed in. You do this on the main thread, because any published properties should be set on the main thread.

Next, to check for camera permissions, add the following method to CameraManager:

private func checkPermissions() {
  // 1
  switch AVCaptureDevice.authorizationStatus(for: .video) {
  case .notDetermined:
    // 2
    sessionQueue.suspend()
    AVCaptureDevice.requestAccess(for: .video) { authorized in
      // 3
      if !authorized {
        self.status = .unauthorized
        self.set(error: .deniedAuthorization)
      }
      self.sessionQueue.resume()
    }
  // 4
  case .restricted:
    status = .unauthorized
    set(error: .restrictedAuthorization)
  case .denied:
    status = .unauthorized
    set(error: .deniedAuthorization)
  // 5
  case .authorized:
    break
  // 6
  @unknown default:
    status = .unauthorized
    set(error: .unknownAuthorization)
  }
}

In this method:

1. You switch on the camera’s authorization status, specifically for video.
2. If the returned device status is undetermined, you suspend the session queue and have iOS request permission to use the camera.
3. If the user denies access, then you set the CameraManager‘s status to .unauthorized and set the error. Regardless of the outcome, you resume the session queue.
4. For the .restricted and .denied statuses, you set the CameraManager‘s status to .unauthorized and set an appropriate error.
5. In the case that permission was already given, nothing needs to be done, so you break out of the switch.
6. To make Swift happy, you add an unknown default case — just in case Apple adds more cases to AVAuthorizationStatus in the future.

Now, you’ll move on to the second step needed to use the camera: configuring it. But before that, a quick explanation!

How AVCaptureSession Works

Whenever you want to capture some sort of media — whether it’s audio, video or depth data — AVCaptureSession is what you want.

The main pieces to setting up a capture session are:

• AVCaptureDevice: a representation of the hardware device to use.
• AVCaptureDeviceInput: provides a bridge from the device to the AVCaptureSession.
• AVCaptureSession: manages the flow of data between capture inputs and outputs. It can connect one or more inputs to one or more outputs.
• AVCaptureOutput: an abstract class representing objects that output the captured media. You’ll use AVCaptureVideoDataOutput, which is a concrete implementation of this class.

Apple’s awesome flowchart explains this nicely:

This shows a high-level view of how each of these pieces fit together. In your app, you’ll be using these parts to configure the capture session.

Configuring the Capture Session

For anyone who has never used a capture session, it can be a little daunting to set one up the first time. Each time you do it, though, it gets a little easier and makes more sense. Fortunately, you usually need to configure a capture session just once in your app — and this app is no exception.

Add the following code to CameraManager:

private func configureCaptureSession() {
  guard status == .unconfigured else {
    return
  }
  session.beginConfiguration()
  defer {
    session.commitConfiguration()
  }
}

So far, this is pretty straightforward. But it’s worth noting that any time you want to change something about an AVCaptureSession configuration, you need to enclose that code between a beginConfiguration and a commitConfiguration.

At the end of configureCaptureSession(), add the following code:

let device = AVCaptureDevice.default(
  .builtInWideAngleCamera,
  for: .video,
  position: .front)
guard let camera = device else {
  set(error: .cameraUnavailable)
  status = .failed
  return
}

This code gets your capture device. In this app, you’re getting the front camera. If you want the back camera, you can change position. Since AVCaptureDevice.default(_:_:_:) returns an optional, which will be nil if the requested device doesn’t exist, you need to unwrap it. If for some reason it is nil, set the error and return early.

After the code above, add the following code to add the device input to AVCaptureSession:

do {
  // 1
  let cameraInput = try AVCaptureDeviceInput(device: camera)
  // 2
  if session.canAddInput(cameraInput) {
    session.addInput(cameraInput)
  } else {
    // 3
    set(error: .cannotAddInput)
    status = .failed
    return
  }
} catch {
  // 4
  set(error: .createCaptureInput(error))
  status = .failed
  return
}

Here, you:

1. Try to create an AVCaptureDeviceInput based on the camera. Since this call can throw, you wrap the code in a do-catch block.
2. Add the camera input to AVCaptureSession, if possible. It’s always a good idea to check if it can be added before adding it. :]
3. Otherwise, set the error and the status and return early.
4. If an error was thrown, set the error based on this thrown error to help with debugging and return.

Have you noticed that camera management involves a lot of error management? When there are so many potential points of failure, having good error management will help you debug any problems much more quickly! Plus, it’s a significantly better user experience.

Next up, you need to connect the capture output to the AVCaptureSession!

Add the following code right after the code you just added:

// 1
if session.canAddOutput(videoOutput) {
  session.addOutput(videoOutput)
  // 2
  videoOutput.videoSettings =
    [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
  // 3
  let videoConnection = videoOutput.connection(with: .video)
  videoConnection?.videoOrientation = .portrait
} else {
  // 4
  set(error: .cannotAddOutput)
  status = .failed
  return
}

With this code:

1. You check to see if you can add AVCaptureVideoDataOutput to the session before adding it. This pattern is similar to when you added the input.
2. Then, you set the format type for the video output.
3. And force the orientation to be in portrait.
4. If something fails, you set the error and status and return.

Finally, there’s one last thing you need to add to this method before it’s finished — right before the closing brace, add:

status = .configured

And with that, your capture session can be configured by calling configureCaptureSession()!

You’ll do that now.

Camera Manager Final Touches

There are a couple of small things you need to take care of to hook all the camera logic together.

Remember that configure() you initially added with the class definition? It’s time to fill that in. In CameraManager, add the following code to configure():

checkPermissions()
sessionQueue.async {
  self.configureCaptureSession()
  self.session.startRunning()
}

Now, check for permissions, configure the capture session and start it. All of this happens when CameraManager is initialized. Perfect!

The only question is: How do you get captured frames from this thing?

You’ll create a FrameManager, which will receive delegate calls from AVCaptureVideoDataOutput. However, before you can do that, you need to add one last method to CameraManager:

func set(
  _ delegate: AVCaptureVideoDataOutputSampleBufferDelegate,
  queue: DispatchQueue
) {
  sessionQueue.async {
    self.videoOutput.setSampleBufferDelegate(delegate, queue: queue)
  }
}

Using this method, your upcoming frame manager will be able to set itself as the delegate that receives that camera data.

Pat yourself on the back and take a quick break! You just completed the longest and most complicated class in this project. It’s all smooth sailing from now on!

Next, you’ll write a class that can receive this camera data.

Designing the Frame Manager

FrameManager will be responsible for receiving data from CameraManager and publishing a frame for use elsewhere in the app. This logic could technically be integrated into CameraManager, but splitting these up also divides the responsibility a bit more clearly. This will be especially helpful if you choose to add more functionality — such as preprocessing or synchronization — to FrameManager in the future.

Add a new Swift file named FrameManager.swift in the Camera group. Replace the contents of the file with the following:

import AVFoundation
// 1
class FrameManager: NSObject, ObservableObject {
  // 2
  static let shared = FrameManager()
  // 3
  @Published var current: CVPixelBuffer?
  // 4
  let videoOutputQueue = DispatchQueue(
    label: "com.raywenderlich.VideoOutputQ",
    qos: .userInitiated,
    attributes: [],
    autoreleaseFrequency: .workItem)
  // 5
  private override init() {
    super.init()
    CameraManager.shared.set(self, queue: videoOutputQueue)
  }
}

With this initial implementation, you:

1. Define the class and have it inherit from NSObject and conform to ObservableObject. FrameManager needs to inherit from NSObject because FrameManager will adopt AVCaptureSession‘s video output. This is a requirement, so you’re just getting a head start on it.
2. Make the frame manager a singleton.
3. Add a published property for the current frame received from the camera. This is what other classes will subscribe to to get the camera data.
4. Create a queue on which to receive the capture data.
5. Set FrameManager as the delegate to AVCaptureVideoDataOutput.

Right about now, Xcode is probably complaining that FrameManager doesn’t conform to AVCaptureVideoDataOutputSampleBufferDelegate. That’s kind of the point!

To fix this, add the following extension below the closing brace of FrameManager:

extension FrameManager: AVCaptureVideoDataOutputSampleBufferDelegate {
  func captureOutput(
    _ output: AVCaptureOutput,
    didOutput sampleBuffer: CMSampleBuffer,
    from connection: AVCaptureConnection
  ) {
    if let buffer = sampleBuffer.imageBuffer {
      DispatchQueue.main.async {
        self.current = buffer
      }
    }
  }
}

In this app, you’re checking if the received CMSampleBuffer contains an image buffer and then sets the current frame. Once again, since current is a published property, it needs to be set on the main thread. That’s that. Short and simple.

You’re close to being able to see the fruits of your oh-so-hard labor. You just need to hook this FrameManager up to your FrameView somehow. But to do that, you’ll need to create the most basic form of view model first.

Adding a View Model

You’ll eventually do some fairly intensive business logic around what will be displayed on the screen. While you could put this in the ContentView or even the FrameView, that will get messy quickly. Often, it’s better and cleaner to separate this logic into a view model. The view model will then feed your view all the data it needs to display what you want.

Create a new Swift file named ContentViewModel.swift in the ViewModels group. Then, replace the contents of that file with the following code:

import CoreImage

class ContentViewModel: ObservableObject {
  // 1
  @Published var frame: CGImage?
  // 2
  private let frameManager = FrameManager.shared

  init() {
    setupSubscriptions()
  }
  // 3
  func setupSubscriptions() {
  }
}

In this initial implementation, you set up some properties and methods you need:

1. frame will hold the images that FrameView will display.
2. Data used to generate frame will come from FrameManager.
3. You’ll add all your Combine pipelines to setupSubscriptions() to keep them in one place.

To transform the CVPixelBuffer data FrameManager provides to a CGImage your FrameView requires, you’ll harness the power of Combine! You made this possible when you declared FrameManager.current @Published. Very smart! :]

Add the following code to setupSubscriptions():

// 1
frameManager.$current
  // 2
  .receive(on: RunLoop.main)
  // 3
  .compactMap { buffer in
    return CGImage.create(from: buffer)
  }
  // 4
  .assign(to: &$frame)

In this pipeline, you:

1. Tap into the Publisher that was automatically created for you when you used @Published.
2. Receive the data on the main run loop. It should already be on main, but just in case, it doesn’t hurt to be sure.
3. Convert CVPixelBuffer to CGImage and filter out all nil values through compactMap.
4. Assign the output of the pipeline — which is, itself, a publisher — to your published frame.

Excellent work!

Now, open ContentView.swift to hook this up. Add the following property to ContentView:

@StateObject private var model = ContentViewModel()

You declare the model to be a @StateObject instead of an @ObservedObject, because it’s created within the ContentView as opposed to being passed in. ContentView owns the model and doesn’t merely observe it.

Now replace FrameView(image: nil) with:

FrameView(image: model.frame)

Do you know what time it is? No, it’s not 9:41 AM. It’s time to build and run!

Finally, you can display the frames captured by the camera in your UI. Pretty nifty.

But what happens if there’s an error with the camera or capture session?

Error Handling

Before you can move on to even more fun, take care of any potential errors CameraManager encounters. For this app, you’ll display them to the user in an ErrorView. However, just like the capture frames, you’re going to route the errors through your view model.

Open ContentViewModel.swift. Add the following properties to ContentViewModel:

@Published var error: Error?
private let cameraManager = CameraManager.shared

Next, you’ll add a new Combine pipeline to setupSubscriptions(). Add the following code to the beginning of setupSubscriptions():

// 1
cameraManager.$error
  // 2
  .receive(on: RunLoop.main)
  // 3
  .map { $0 }
  // 4
  .assign(to: &$error)

With this code, you once again:

1. Tap into the Publisher provided automatically for the published CameraManager.error.
2. Receive it on the main thread.
3. Map it to itself, because otherwise Swift will complain in the next line that you can’t assign a CameraError to an Error.
4. Assign it to error.

Now, to hook it up to your UI, open ContentView.swift and add the following line inside your ZStack, below FrameView:

ErrorView(error: model.error)

If you build and run now, you won’t see any difference if you previously gave the app access to the camera. If you want to see this new error view in action, open the Settings app and tap Privacy ▸ Camera. Turn off the camera permissions for FilterTheWorld.

Build and run and see your beautiful error!

The app correctly informs you that camera access has been denied. Success! Or, um, error!

Go ahead and turn camera permissions back on for the app! :]

Now you have a very basic, working camera app, which also displays any encountered errors to the user. Nice. However, the point of this app isn’t to just show the world as it is. After all, the app is called Filter the World…

Creating Filters With Core Image

It’s time to have a little fun. Well, even more fun! You’ll add some Core Image filters to the data pipeline, and you can turn them on and off via some toggle buttons. These will let you add some cool effects to the live camera feed.

First, you’ll add the business logic to your view model. So, open ContentViewModel.swift and add the following properties to ContentViewModel:

var comicFilter = false
var monoFilter = false
var crystalFilter = false
private let context = CIContext()

These will tell your code which filters to apply to the camera feed. These particular filters are easily composable, so they work with each other nicely.

Since CIContexts are expensive to create, you also create a private property to reuse the context instead of recreating it every frame.

Next, open ContentViewModel.swift. Replace the following code inside setupSubscriptions():

// 1
frameManager.$current
  // 2
  .receive(on: RunLoop.main)
  // 3
  .compactMap { buffer in
    return CGImage.create(from: buffer)
  }
  // 4
  .assign(to: &$frame)

With the following:

frameManager.$current
  .receive(on: RunLoop.main)
  .compactMap { $0 }
  .compactMap { buffer in
    // 1
    guard let image = CGImage.create(from: buffer) else {
      return nil
    }
    // 2
    var ciImage = CIImage(cgImage: image)
    // 3
    if self.comicFilter {
      ciImage = ciImage.applyingFilter("CIComicEffect")
    }
    if self.monoFilter {
      ciImage = ciImage.applyingFilter("CIPhotoEffectNoir")
    }
    if self.crystalFilter {
      ciImage = ciImage.applyingFilter("CICrystallize")
    }
    // 4
    return self.context.createCGImage(ciImage, from: ciImage.extent)
  }
  .assign(to: &$frame)

Here, you:

1. Try to convert CVPixelBuffer to a CGImage, and if it fails, you return early.
2. Convert CGImage to a CIImage, since you’ll be working with Core Image filters.
3. Apply the appropriate filters, which have been turned on.
4. Render CIImage back to a CGImage.

Now, to connect this to the UI, open ContentView.swift and add the following code within the ZStack after the ErrorView:

ControlView(
  comicSelected: $model.comicFilter,
  monoSelected: $model.monoFilter,
  crystalSelected: $model.crystalFilter)

Build and run one last time!

Above, you see examples of how the camera feed looks after applying different filters.

Where to Go From Here?

Download the completed project files by clicking the Download Materials button at the top or bottom of the tutorial.

You wrote a lot of code and now have a well organized and extendable start for a camera-based app. Well done!

From here, you could further your knowledge by:

• Diving deeper into Combine by reading Combine: Asynchronous Programming with Swift.
• Researching other Core Image filters to implement.
• Adding some Vision APIs to the frame processing pipeline.

We hope you enjoyed this tutorial. If you have any questions or comments, please join the forum discussion below!