3rd-Party On-Device Models

Processing With Vision Detect

Now, you’ll process what the model finds in the image. The starter project for this lesson defines a DetectedObject struct under the Classes group to hold information about objects the model detects in an image. This struct stores a label and confidence for each detected object. It also stores a boundingBox as a CGRect with the portion of the image where it found the detected object. Add the following state property after cgImage:

@State private var detectedObjects: [DetectedObject] = []

This will provide a location to store information about the detected objects as an array of DetectedObject structs. Replace the // Insert result processing code here line in the runModel() method with:

// 1
if results.isEmpty {
  print("No results found.")
  return
}
// 2
for result in results {
  // 3
  if let firstIdentifier = result.labels.first {
    let confidence = firstIdentifier.confidence
    let label = firstIdentifier.identifier
    // 4
    let boundingBox = result.boundingBox
    // 5
    let object = DetectedObject(
      label: label,
      confidence: confidence,
      boundingBox: boundingBox
    )
    detectedObjects.append(object)
  }
}

This loop gathers information about the detected object and adds it to the detectedObjects array:

1. If everything worked, but the model didn’t find any results, then results will be empty, and you print a debugging message to that effect and return.
2. You’ll loop through the results passed to the closure. Each VNRecognizedObjectObservation contains information on the object detected, the confidence in the detection, and the coordinates on the image for the object.
3. For a detection request like this one, the labels property stores the object classification label. This property can have multiple results for some models, but for the Ultralytics model, you’ll only deal with the first result. You ensure this isn’t nil and store that first label as firstIdentifier. From that variable, you extract the model’s confidence in the detection and the classification label from that firstIdentifier.
4. The boundingBox property of the result stores the bounding box of the detected object. The model returns this using unit coordinates that range from zero to one. The origin where both are zero lies at the bottom-left corner of the image.
5. You create a new DetectedObject and store the values from steps three and four before appending the new object to the array.

To ensure the array is cleared each time the model runs against an image, add the following code inside the handler for the VNCoreMLRequest before the if let error = error { line:

detectedObjects = []

One more change. Whenever the image changes, you want to run this new method. After the existing onChange(of:initial:_:) that deals with the selectedImage property, add another onChange(of:initial:_:) as follows:

.onChange(of: cgImage) {
  runModel()
}

This modifier will call your function to run runModel() each time the cgImage changes, which should occur whenever the user selects a new image from the Photo Library. To show the user the objects detected by the framework, you’ll use the ObjectOverlayView defined in the starter project. This view expects a DetectedObject struct and outlines the bounding box on the view under it while showing the label in the box’s leading-top corner. It also adjusts for the fact that SwiftUI images have their origin at the top-left corner while the values returned by the model are measured from the bottom-left corner. Add the following modifier to the ImageDisplayView(image: image) in ContentView.swift:

.overlay {
  ForEach(detectedObjects, id: \.self) { ident in
    `ObjectOverlayView`(object: ident)
  }
}

The overlay instance method aligns a view on top of another view while letting the original view define its location and size. The view then loops through any detectedObjects and for each, shows the ObjectOverlayView view reflecting that DetectedObject. You’ll see that this nicely outlines each detected object and displays information about the object.

One last step. At the end of the view, add the following code:

ForEach(detectedObjects, id: \.self) { obj in
  Text(obj.label) + Text(" (") + Text(obj.confidence, format: .percent) + Text(")")
}

This code will loop through any currently detected objects and display the label and confidence as a percentage for each one. Run the app and select the sample image you imported into the simulator earlier. The image will appear on the view, and after a short processing time, you should see it outline the location of the two cats and display information about the detected objects, along with some library errors you can ignore. You’ll see the model detects two cats with high confidence, and the boxes around the cats seem accurate.

Select a few other photos to see how it fares. The photo of flowers does detect a lot of flowers, but not all of them and some bounding areas are more accurate than others. In the close-up photo of a waterfall, a single plant among the algae grows on the foreground rock. The photo of the distant waterfall detects one plant while classifying the entire rest of the area as a single plant. As you can see, the model has some success, but the success varies depending on the contents of the image.

While that shows that your code works, as a last step, you’ll update to allow selecting different models.