3rd-Party On-Device Models

Comparing Models in Your App

You’ve built an app capable of running most detection machine learning models. Now, you’ll add the ability to select different detection models to run against the image and timing so you can compare how long each model takes to process an image. Open ContentView.swift and add the following new state properties to the top of the file after detectedObjects:

@State private var startTime: DispatchTime?
@State private var endTime: DispatchTime?

This gives two optional DispatchTime values where you’ll store the start and end times of the processing of each image. This gives a default clock with nanosecond precision that’s more than acceptable for this testing. Now, at the end of runModel(), find the Swift do statement that wraps the try handler.perform([visionRequest]) call and add the following code before the perform() command:

startTime = DispatchTime.now()
endTime = nil

This will set the startTime property to the current DispatchTime when the app starts the Vision request. It also clears the endTime so the app will know a scan is in progress. You now need to set endTime when the model processing completes. Find the VNCoreMLRequest(model: detector) completion handler and add the following code to the start of the block before the detectedObjects = [] code:

self.endTime = DispatchTime.now()

This code sets the endTime when starting the completion handler. Now you can take the difference between the two values when both aren’t nil, as the time of the last Vision request. Find the loop through detectedObjects that you added to the end of the view. Just before it, add the following code:

if let start = startTime, let end = endTime {
  let elapsedNanoseconds = end.uptimeNanoseconds - start.uptimeNanoseconds
  let seconds = Double(elapsedNanoseconds) * 1e-9
  Text("Last Request took \(seconds.roundTwo) seconds")
}

This will get the difference between the two values in nanoseconds and convert it to seconds before displaying it when both values are set. The roundTwo is defined in the Helpers.swift file under the Classes group and rounds the value to two decimal places. Run the app and select an image to see how long it takes with the current model. In the simulator, running the model against the sample image takes 0.69 seconds. Select the same image again, and you’ll see a slightly different value.

Now that you’ve added a timer to the app, you’ll add the ability to select between the different models.

Adding More Models

First, at the top of ContentView.swift, before the view definition, add the following code:

enum SelectedModel {
  case yolo8xfull
  case yolo8int8
  case yolo8m
  case yolo8n
}

This defines an enum for each of the four models in your app. Now add the following new property after endTime:

@State private var currentModel: SelectedModel = .yolo8xfull

This creates a state property to store the currently selected model and defaults to the yolov8x_oiv7 model. Now add the following code at the top of your view before the PhotosPicker view:

Picker("Select Model to Use", selection: $currentModel) {
  Text("yolov8x-oiv7").tag(SelectedModel.yolo8xfull)
  Text("yolov8x-oiv7-int").tag(SelectedModel.yolo8int8)
  Text("yolov8m-oiv7").tag(SelectedModel.yolo8m)
  Text("yolov8n-oiv7").tag(SelectedModel.yolo8n)
}

This Picker view will let the user select between the four models and set the currentModel to the appropriate value. Now, you need to update the runModel() to load the correct model based on the currentModel property. Replace the full guard statement and its three assignments at the start of the method with:

guard let cgImage = cgImage else {
  print("Unable to load photo.")
  return
}

var model: MLModel?
switch currentModel {
case .yolo8xfull:
  model = try? yolov8x_oiv7(configuration: .init()).model
case .yolo8int8:
  model = try? yolov8x_oiv7_int(configuration: .init()).model
case .yolo8m:
  model = try? yolov8m_oiv7(configuration: .init()).model
case .yolo8n:
  model = try? yolov8n_oiv7(configuration: .init()).model
}

guard let model = model,
      let detector = try? VNCoreMLModel(for: model) else {
  print("Unable to load model.")
  return
}

These seem like a lot of lines, but the result differs little from before. You still ensure you have a valid cgImage and then attempt to instantiate the model using the appropriate class for each case. If the model isn’t nil, you then try to create a VNCoreMLModel as before. If that fails, you print a message to that effect and return from the method.

One more change. To ensure the new model is run each time you select one, add the following code after the onChange(of:initial:_:) for cgImage:

.onChange(of: currentModel) {
  runModel()
}

Now, when currentModel changes, the method to run the model against the current image will be used. Run the app and select the sample image. Now select each of the other three models. Your exact values will differ depending on your hardware and other running programs, but you should see that the time for the model to run decreases from the first through the final model in order. That’s also the order of the models in decreasing size.

You should also see the bounding boxes and confidence change between the different models. As a rough guideline, you’ll see little difference between the full yolo8x-oiv model and the one you converted to use Int8 as yolo8x-oiv-int except for the model completing in nearly half the time. When using the yolo8m-oiv model, it should run a bit faster than the Int8 model with similar accuracy. The yolo8n-oiv model runs the fastest, around ten times faster than the original model, but you’ll see the confidence values are much lower.

You’ll see a similar pattern of results in other images. The largest model yolo8x-oiv and its Int8 optimized form yolo8x-oiv-int perform similarly with the other two running faster with smaller model sizes with a tradeoff in lower quality results. For the flower photo, the yolo8x-oiv model finds eight flowers all with over 97% confidence, with the yolo8n-oiv model finding only five with lower confidence, though none find a fraction of the dozens of flowers in the image.

You’ll find the smallest model often misses objects such as the plants in the two waterfall images.

This should show the tradeoffs discussed in the last lesson around reducing model size. While smaller models require less space and consume fewer resources, they lose accuracy. This is on top of the model’s base accuracy, which varies. This model, for instance, doesn’t detect the bed the cats in the sample image lay in, even with the largest model. When developing an app using local models, part of the development process will include determining where the right balance lies for your app and your needs.

As a challenge, take this project and give the user the ability to select one of the detections and only show its bounding box over the image. As a hint to one approach, store the object when a user selects one from the list and only show all of them when the user hasn’t selected one. See the Challenge folder in the sample project for one solution.