Building a Camera App With SwiftUI and Combine
Learn to natively build your own SwiftUI camera app using Combine and create fun filters using the power of Core Image. By Yono Mittlefehldt.
Sign up/Sign in
With a free Kodeco account you can download source code, track your progress, bookmark, personalise your learner profile and more!
Create accountAlready a member of Kodeco? Sign in
Sign up/Sign in
With a free Kodeco account you can download source code, track your progress, bookmark, personalise your learner profile and more!
Create accountAlready a member of Kodeco? Sign in
Contents
Building a Camera App With SwiftUI and Combine
30 mins
SwiftUI enables developers to design and create working user interfaces almost as quickly as they could prototype one in Sketch or Photoshop. Think about how powerful this is. Instead of making static mockups, you can create working prototypes for almost the same amount of effort. How cool is that?
Add Combine as a data pipeline to the mix, and you’ve got the Batman and Robin of the programming world. You can decide which one is Batman and which one is Robin. :]
Some people may complain that SwiftUI and Combine aren’t ready for prime time, but do you really want to tell Batman he can’t go out and fight crime? In fact, would you believe you can write a camera app completely in SwiftUI without even touching UIViewRepresentable
?
Creating a camera app using SwiftUI and Combine makes processing real-time video easy and delightful. Video processing can already be thought of as a data pipeline. Since Combine manages the flow of data like a pipeline, there are many similarities between these patterns. Integrating them allows you to create powerful effects, and these pipelines are easy to expand when future features demand.
In this tutorial, you’ll learn how to use this dynamic duo to:
- Manage the camera and the video frames it captures.
- Create a data pipeline in Combine to do interesting things with the captured frames.
- Present the resulting image stream using SwiftUI.
You’ll do these with an app called Filter the World. So, get ready to filter the world through your iPhone — even more so than you already do!
Getting Started
Click the Download Materials button at the top or bottom of this tutorial. There’s currently not a lot there, aside from a custom Error
, a helpful extension to convert from a CVPixelBuffer
to a CGImage
, and some basic SwiftUI views that you’ll use to build up the UI.
If you build and run now, you won’t see much.
Currently, there’s a blank screen with the name of the app in the center.
If you want to make a camera-based app, what’s the most important thing you need? Aside from having a cool name, being able to display the camera feed is probably a close second.
So that’s exactly what you’ll do first!
Displaying Captured Frames
If you were going to use a UIViewRepresentable
, you’d probably opt for attaching an AVPreviewLayer
to your UIView
, but that’s not what you’re going to do! In SwiftUI, you’ll display the captured frames as Image
objects.
Since the data you get from the camera will be a CVPixelBuffer
, you’ll need some way to convert it to an Image
. You can initialize an Image
from a UIImage
or a CGImage
, and the second route is the one you’ll take.
Inside the Views group, create an iOS SwiftUI View file and call it FrameView.swift.
Add the following properties to FrameView
:
var image: CGImage?
private let label = Text("Camera feed")
When adding FrameView
to ContentView
in a little bit, you’ll pass in the image it should display. label
is there to make your code in the next step a little bit cleaner!
Replace Text
in the body with the following:
// 1
if let image = image {
// 2
GeometryReader { geometry in
// 3
Image(image, scale: 1.0, orientation: .upMirrored, label: label)
.resizable()
.scaledToFill()
.frame(
width: geometry.size.width,
height: geometry.size.height,
alignment: .center)
.clipped()
}
} else {
// 4
Color.black
}
In this code block, you:
- Conditionally unwrap the optional
image
. - Set up a
GeometryReader
to access the size of the view. This is necessary to ensure the image is clipped to the screen bounds. Otherwise, UI elements on the screen could potentially be anchored to the bounds of the image instead of the screen. - Create
Image
fromCGImage
, scale it to fill the frame and clip it to the bounds. Here, you set the orientation to.upMirrored
, because you’ll be using the front camera. If you wanted to use the back camera, this would need to be.up
. - Return a black view if the image property is
nil
.
Great work! Now you need to hook it up in the ContentView
.
Open ContentView.swift and replace the contents of ZStack
with:
FrameView(image: nil)
.edgesIgnoringSafeArea(.all)
This adds the newly created FrameView
and ignores the safe area, so the frames will flow edge to edge. For now, you’re passing in nil
, as you don’t have a CGImage
, yet.
There’s no need to build and run now. If you did, it would show up black.
It’s still a blank screen. Is that really an improvement?
To display the frames now, you’ll need to add some code to set up the camera and receive the captured output.
Managing the Camera
You’ll start by creating a manager for your camera — a CameraManager
, if you will.
First, add a new Swift file named CameraManager.swift to the Camera group.
Now, replace the contents Xcode provides with the following code:
import AVFoundation
// 1
class CameraManager: ObservableObject {
// 2
enum Status {
case unconfigured
case configured
case unauthorized
case failed
}
// 3
static let shared = CameraManager()
// 4
private init() {
configure()
}
// 5
private func configure() {
}
}
So far, you’ve set up a basic structure for CameraManager
. More specifically, you:
- Created a
class
that conforms toObservableObject
to make it easier to use with future Combine code. - Added an internal enumeration to represent the status of the camera.
- Included a static shared instance of the camera manager to make it easily accessible.
- Turned the camera manager into a singleton by making
init
private. - Added a stub for a
configure()
you’ll fill out soon.
Configuring the camera requires two steps. First, check for permission to use the camera and request it, if necessary. Second, configure AVCaptureSession
.
Checking for Permission
Privacy is one of Apple’s most touted pillars. The data captured by a camera has the potential to be extremely sensitive and private. Since Apple (and hopefully you) care about users’ privacy, it only makes sense that the user needs to grant an app permission to use the camera. You’ll take care of that now.
Add the following properties to CameraManager
:
// 1
@Published var error: CameraError?
// 2
let session = AVCaptureSession()
// 3
private let sessionQueue = DispatchQueue(label: "com.raywenderlich.SessionQ")
// 4
private let videoOutput = AVCaptureVideoDataOutput()
// 5
private var status = Status.unconfigured
Here, you define:
- An error to represent any camera-related error. You made it a published property so that other objects can subscribe to this stream and handle any errors as necessary.
-
AVCaptureSession
, which will coordinate sending the camera images to the appropriate data outputs. - A session queue, which you’ll use to change any of the camera configurations.
- The video data output that will connect to
AVCaptureSession
. You’ll want this stored as a property so you can change the delegate after the session is configured. - The current status of the camera.
Next, add the following method to CameraManager
:
private func set(error: CameraError?) {
DispatchQueue.main.async {
self.error = error
}
}
Here, you set the published error
to whatever error is passed in. You do this on the main thread, because any published properties should be set on the main thread.
Next, to check for camera permissions, add the following method to CameraManager
:
private func checkPermissions() {
// 1
switch AVCaptureDevice.authorizationStatus(for: .video) {
case .notDetermined:
// 2
sessionQueue.suspend()
AVCaptureDevice.requestAccess(for: .video) { authorized in
// 3
if !authorized {
self.status = .unauthorized
self.set(error: .deniedAuthorization)
}
self.sessionQueue.resume()
}
// 4
case .restricted:
status = .unauthorized
set(error: .restrictedAuthorization)
case .denied:
status = .unauthorized
set(error: .deniedAuthorization)
// 5
case .authorized:
break
// 6
@unknown default:
status = .unauthorized
set(error: .unknownAuthorization)
}
}
In this method:
- You switch on the camera’s authorization status, specifically for video.
- If the returned device status is undetermined, you suspend the session queue and have iOS request permission to use the camera.
- If the user denies access, then you set the
CameraManager
‘s status to.unauthorized
and set the error. Regardless of the outcome, you resume the session queue. - For the
.restricted
and.denied
statuses, you set theCameraManager
‘s status to.unauthorized
and set an appropriate error. - In the case that permission was already given, nothing needs to be done, so you break out of the switch.
- To make Swift happy, you add an unknown default case — just in case Apple adds more cases to
AVAuthorizationStatus
in the future.
Now, you’ll move on to the second step needed to use the camera: configuring it. But before that, a quick explanation!
How AVCaptureSession Works
Whenever you want to capture some sort of media — whether it’s audio, video or depth data — AVCaptureSession
is what you want.
The main pieces to setting up a capture session are:
-
AVCaptureDevice
: a representation of the hardware device to use. -
AVCaptureDeviceInput
: provides a bridge from the device to theAVCaptureSession
. -
AVCaptureSession
: manages the flow of data between capture inputs and outputs. It can connect one or more inputs to one or more outputs. -
AVCaptureOutput
: an abstract class representing objects that output the captured media. You’ll useAVCaptureVideoDataOutput
, which is a concrete implementation of this class.
Apple’s awesome flowchart explains this nicely:
This shows a high-level view of how each of these pieces fit together. In your app, you’ll be using these parts to configure the capture session.
Configuring the Capture Session
For anyone who has never used a capture session, it can be a little daunting to set one up the first time. Each time you do it, though, it gets a little easier and makes more sense. Fortunately, you usually need to configure a capture session just once in your app — and this app is no exception.
Add the following code to CameraManager
:
private func configureCaptureSession() {
guard status == .unconfigured else {
return
}
session.beginConfiguration()
defer {
session.commitConfiguration()
}
}
So far, this is pretty straightforward. But it’s worth noting that any time you want to change something about an AVCaptureSession
configuration, you need to enclose that code between a beginConfiguration
and a commitConfiguration
.
At the end of configureCaptureSession()
, add the following code:
let device = AVCaptureDevice.default(
.builtInWideAngleCamera,
for: .video,
position: .front)
guard let camera = device else {
set(error: .cameraUnavailable)
status = .failed
return
}
This code gets your capture device. In this app, you’re getting the front camera. If you want the back camera, you can change position
. Since AVCaptureDevice.default(_:_:_:)
returns an optional, which will be nil
if the requested device doesn’t exist, you need to unwrap it. If for some reason it is nil
, set the error and return early.
After the code above, add the following code to add the device input to AVCaptureSession
:
do {
// 1
let cameraInput = try AVCaptureDeviceInput(device: camera)
// 2
if session.canAddInput(cameraInput) {
session.addInput(cameraInput)
} else {
// 3
set(error: .cannotAddInput)
status = .failed
return
}
} catch {
// 4
set(error: .createCaptureInput(error))
status = .failed
return
}
Here, you:
- Try to create an
AVCaptureDeviceInput
based on the camera. Since this call can throw, you wrap the code in a do-catch block. - Add the camera input to
AVCaptureSession
, if possible. It’s always a good idea to check if it can be added before adding it. :] - Otherwise, set the error and the status and return early.
- If an error was thrown, set the error based on this thrown error to help with debugging and return.
Have you noticed that camera management involves a lot of error management? When there are so many potential points of failure, having good error management will help you debug any problems much more quickly! Plus, it’s a significantly better user experience.
Next up, you need to connect the capture output to the AVCaptureSession
!
Add the following code right after the code you just added:
// 1
if session.canAddOutput(videoOutput) {
session.addOutput(videoOutput)
// 2
videoOutput.videoSettings =
[kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
// 3
let videoConnection = videoOutput.connection(with: .video)
videoConnection?.videoOrientation = .portrait
} else {
// 4
set(error: .cannotAddOutput)
status = .failed
return
}
With this code:
- You check to see if you can add
AVCaptureVideoDataOutput
to the session before adding it. This pattern is similar to when you added the input. - Then, you set the format type for the video output.
- And force the orientation to be in portrait.
- If something fails, you set the error and status and return.
Finally, there’s one last thing you need to add to this method before it’s finished — right before the closing brace, add:
status = .configured
And with that, your capture session can be configured by calling configureCaptureSession()
!
You’ll do that now.
Camera Manager Final Touches
There are a couple of small things you need to take care of to hook all the camera logic together.
Remember that configure()
you initially added with the class definition? It’s time to fill that in. In CameraManager
, add the following code to configure()
:
checkPermissions()
sessionQueue.async {
self.configureCaptureSession()
self.session.startRunning()
}
Now, check for permissions, configure the capture session and start it. All of this happens when CameraManager
is initialized. Perfect!
The only question is: How do you get captured frames from this thing?
You’ll create a FrameManager
, which will receive delegate calls from AVCaptureVideoDataOutput
. However, before you can do that, you need to add one last method to CameraManager
:
func set(
_ delegate: AVCaptureVideoDataOutputSampleBufferDelegate,
queue: DispatchQueue
) {
sessionQueue.async {
self.videoOutput.setSampleBufferDelegate(delegate, queue: queue)
}
}
Using this method, your upcoming frame manager will be able to set itself as the delegate that receives that camera data.
Pat yourself on the back and take a quick break! You just completed the longest and most complicated class in this project. It’s all smooth sailing from now on!
Next, you’ll write a class that can receive this camera data.
Designing the Frame Manager
FrameManager
will be responsible for receiving data from CameraManager
and publishing a frame for use elsewhere in the app. This logic could technically be integrated into CameraManager
, but splitting these up also divides the responsibility a bit more clearly. This will be especially helpful if you choose to add more functionality — such as preprocessing or synchronization — to FrameManager
in the future.
Add a new Swift file named FrameManager.swift in the Camera group. Replace the contents of the file with the following:
import AVFoundation
// 1
class FrameManager: NSObject, ObservableObject {
// 2
static let shared = FrameManager()
// 3
@Published var current: CVPixelBuffer?
// 4
let videoOutputQueue = DispatchQueue(
label: "com.raywenderlich.VideoOutputQ",
qos: .userInitiated,
attributes: [],
autoreleaseFrequency: .workItem)
// 5
private override init() {
super.init()
CameraManager.shared.set(self, queue: videoOutputQueue)
}
}
With this initial implementation, you:
- Define the class and have it inherit from
NSObject
and conform toObservableObject
.FrameManager
needs to inherit fromNSObject
becauseFrameManager
will adoptAVCaptureSession
‘s video output. This is a requirement, so you’re just getting a head start on it. - Make the frame manager a singleton.
- Add a published property for the current frame received from the camera. This is what other classes will subscribe to to get the camera data.
- Create a queue on which to receive the capture data.
- Set
FrameManager
as the delegate toAVCaptureVideoDataOutput
.
Right about now, Xcode is probably complaining that FrameManager
doesn’t conform to AVCaptureVideoDataOutputSampleBufferDelegate
. That’s kind of the point!
To fix this, add the following extension below the closing brace of FrameManager
:
extension FrameManager: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(
_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection
) {
if let buffer = sampleBuffer.imageBuffer {
DispatchQueue.main.async {
self.current = buffer
}
}
}
}
In this app, you’re checking if the received CMSampleBuffer
contains an image buffer and then sets the current frame. Once again, since current
is a published property, it needs to be set on the main thread. That’s that. Short and simple.
You’re close to being able to see the fruits of your oh-so-hard labor. You just need to hook this FrameManager
up to your FrameView
somehow. But to do that, you’ll need to create the most basic form of view model first.
Adding a View Model
You’ll eventually do some fairly intensive business logic around what will be displayed on the screen. While you could put this in the ContentView
or even the FrameView
, that will get messy quickly. Often, it’s better and cleaner to separate this logic into a view model. The view model will then feed your view all the data it needs to display what you want.
Create a new Swift file named ContentViewModel.swift in the ViewModels group. Then, replace the contents of that file with the following code:
import CoreImage
class ContentViewModel: ObservableObject {
// 1
@Published var frame: CGImage?
// 2
private let frameManager = FrameManager.shared
init() {
setupSubscriptions()
}
// 3
func setupSubscriptions() {
}
}
In this initial implementation, you set up some properties and methods you need:
-
frame
will hold the images thatFrameView
will display. - Data used to generate
frame
will come fromFrameManager
. - You’ll add all your Combine pipelines to
setupSubscriptions()
to keep them in one place.
To transform the CVPixelBuffer
data FrameManager
provides to a CGImage
your FrameView
requires, you’ll harness the power of Combine! You made this possible when you declared FrameManager.current
@Published
. Very smart! :]
Add the following code to setupSubscriptions()
:
// 1
frameManager.$current
// 2
.receive(on: RunLoop.main)
// 3
.compactMap { buffer in
return CGImage.create(from: buffer)
}
// 4
.assign(to: &$frame)
In this pipeline, you:
- Tap into the
Publisher
that was automatically created for you when you used@Published
. - Receive the data on the main run loop. It should already be on main, but just in case, it doesn’t hurt to be sure.
- Convert
CVPixelBuffer
toCGImage
and filter out allnil
values throughcompactMap
. - Assign the output of the pipeline — which is, itself, a publisher — to your published
frame
.
Excellent work!
Now, open ContentView.swift to hook this up. Add the following property to ContentView
:
@StateObject private var model = ContentViewModel()
You declare the model to be a @StateObject
instead of an @ObservedObject
, because it’s created within the ContentView
as opposed to being passed in. ContentView
owns the model and doesn’t merely observe it.
Now replace FrameView(image: nil)
with:
FrameView(image: model.frame)
Do you know what time it is? No, it’s not 9:41 AM. It’s time to build and run!
Finally, you can display the frames captured by the camera in your UI. Pretty nifty.
But what happens if there’s an error with the camera or capture session?
Error Handling
Before you can move on to even more fun, take care of any potential errors CameraManager
encounters. For this app, you’ll display them to the user in an ErrorView
. However, just like the capture frames, you’re going to route the errors through your view model.
Open ContentViewModel.swift. Add the following properties to ContentViewModel
:
@Published var error: Error?
private let cameraManager = CameraManager.shared
Next, you’ll add a new Combine pipeline to setupSubscriptions()
. Add the following code to the beginning of setupSubscriptions()
:
// 1
cameraManager.$error
// 2
.receive(on: RunLoop.main)
// 3
.map { $0 }
// 4
.assign(to: &$error)
With this code, you once again:
- Tap into the
Publisher
provided automatically for the publishedCameraManager.error
. - Receive it on the main thread.
- Map it to itself, because otherwise Swift will complain in the next line that you can’t assign a
CameraError
to anError
. - Assign it to
error
.
Now, to hook it up to your UI, open ContentView.swift and add the following line inside your ZStack
, below FrameView
:
ErrorView(error: model.error)
If you build and run now, you won’t see any difference if you previously gave the app access to the camera. If you want to see this new error view in action, open the Settings app and tap Privacy ▸ Camera. Turn off the camera permissions for FilterTheWorld.
Build and run and see your beautiful error!
The app correctly informs you that camera access has been denied. Success! Or, um, error!
Go ahead and turn camera permissions back on for the app! :]
Now you have a very basic, working camera app, which also displays any encountered errors to the user. Nice. However, the point of this app isn’t to just show the world as it is. After all, the app is called Filter the World…
Creating Filters With Core Image
It’s time to have a little fun. Well, even more fun! You’ll add some Core Image filters to the data pipeline, and you can turn them on and off via some toggle buttons. These will let you add some cool effects to the live camera feed.
First, you’ll add the business logic to your view model. So, open ContentViewModel.swift and add the following properties to ContentViewModel
:
var comicFilter = false
var monoFilter = false
var crystalFilter = false
private let context = CIContext()
These will tell your code which filters to apply to the camera feed. These particular filters are easily composable, so they work with each other nicely.
Since CIContext
s are expensive to create, you also create a private property to reuse the context instead of recreating it every frame.
Next, open ContentViewModel.swift. Replace the following code inside setupSubscriptions()
:
// 1
frameManager.$current
// 2
.receive(on: RunLoop.main)
// 3
.compactMap { buffer in
return CGImage.create(from: buffer)
}
// 4
.assign(to: &$frame)
With the following:
frameManager.$current
.receive(on: RunLoop.main)
.compactMap { $0 }
.compactMap { buffer in
// 1
guard let image = CGImage.create(from: buffer) else {
return nil
}
// 2
var ciImage = CIImage(cgImage: image)
// 3
if self.comicFilter {
ciImage = ciImage.applyingFilter("CIComicEffect")
}
if self.monoFilter {
ciImage = ciImage.applyingFilter("CIPhotoEffectNoir")
}
if self.crystalFilter {
ciImage = ciImage.applyingFilter("CICrystallize")
}
// 4
return self.context.createCGImage(ciImage, from: ciImage.extent)
}
.assign(to: &$frame)
Here, you:
- Try to convert
CVPixelBuffer
to aCGImage
, and if it fails, you return early. - Convert
CGImage
to aCIImage
, since you’ll be working with Core Image filters. - Apply the appropriate filters, which have been turned on.
- Render
CIImage
back to aCGImage
.
Now, to connect this to the UI, open ContentView.swift and add the following code within the ZStack
after the ErrorView
:
ControlView(
comicSelected: $model.comicFilter,
monoSelected: $model.monoFilter,
crystalSelected: $model.crystalFilter)
Build and run one last time!
Above, you see examples of how the camera feed looks after applying different filters.
Where to Go From Here?
Download the completed project files by clicking the Download Materials button at the top or bottom of the tutorial.
You wrote a lot of code and now have a well organized and extendable start for a camera-based app. Well done!
From here, you could further your knowledge by:
- Diving deeper into Combine by reading Combine: Asynchronous Programming with Swift.
- Researching other Core Image filters to implement.
- Adding some Vision APIs to the frame processing pipeline.
We hope you enjoyed this tutorial. If you have any questions or comments, please join the forum discussion below!