Vision Framework

Here’s the app you’ve been working with. In this demo, you’ll update the faces tab to recognize faces in an image and draw red rectangles on them. Build and run the app. The classification tab works the same as the last demo, but when you tap the “Detect faces” button, nothing happens.

Open FacesView. You can see that it’s similar to the object view you saw in the last demo. If there’s an image, it gets drawn; otherwise, nothing is drawn.

Over in the ViewModel, there’s a published variable for the array of face-observation rectangles and the current index and an error message string. In this app, one face at a time will have a rectangle on it, and the user can cycle through the faces in an image.

Below, there’s a detectFaces function. The pattern should be pretty familiar by now. First, you’ll make a faceDetectionRequest, and then in the completion handler, you’ll process the observations. Then, you’ll make a vision handler with the image you want to process. Finally, you’ll perform the request on the handler.

Right now, the completion handler of the request just checks for errors. You can add some code to extract all the boundingBoxes from the observations.

let rectangles: [CGRect] = request.results?.compactMap { obs in
  guard let observation = obs as? VNFaceObservation else
    { return nil }
  return observation.boundingBox
} ?? []

Here, you declare an array of CGRect to hold the rectangles. Then you .map the observations and return just the bounding box of each one. Once you have the array of rectangles, you assign it to the faceRectangles property and, if the array is empty, you show an error. You can use code like this:

DispatchQueue.main.async {
  self?.faceRectangles = rectangles
  self?.errorMessage = rectangles.isEmpty
    ? "No faces detected" : nil
}

Remember the completion handler is probably operating on a background queue, so you need to use a dispatch queue before you update any published properties or UI elements.

Build and run. Select an image and click Detect Faces. You can use the Previous and Next buttons to cycle through them. But where did the rectangles come from? You didn’t write that code in your completion handler.

It’s over here in this helper class. You can walk through it now.

The Vision request works with a CGImage version of the UIImage, so this drawing code does the same to ensure the rotation matches with the observation bounding boxes. visionRect is the bounding box CGRect of the face.

First, the code creates a context to draw on and draws the image.

Next, it uses the VNImageRectForNormalizedRect with the bounding box rect and the image height and width to get the bounding box to be in the right location.

Then, it’s standard core graphics drawing code to draw the box and fill it in.

Now, the image gets extracted from the context and the code closes the context so there aren’t any memory leaks. And it gets turned back into a UIImage for display. Notice in the console and here in this logger statement that the orientation needed to be changed. This code uses the method discussed earlier where each original rotation of the UIImage gets the inverse applied. So this image that had a .up rotation to start will have a .downMirrored rotation to end.

Another interesting thing about the simulator is that all the images in the photo roll tend to have an orientation of 0 or 1. That’s because when you drag a photo out of your Photos app on your mac, it gets converted to a jpeg and the rotation gets reset. The same thing happens if you download a jpeg or other image from the internet and drag it in - the orientation will be 0 or 1.

On a real device, the orientation is not usually 0 when you choose from your photo library because the UIImage is created directly. It doesn’t become a jpeg first.