Instruction

Now that you have a model, you can integrate this model into your iOS app. Find and open the Starter project for this lesson. You can find the materials for this project in the Walkthrough folder. Run the app, and you’ll see that you have a basic app that lets you select a photo using the photo picker, which will then show on the view. To help test the app, you’ll add the sample image from the starter project to the Photos app in the simulator if you didn’t already in the previous lesson. Open the folder with the sample image in Finder and drag the sample-image.jpg file onto the iOS simulator.

Now open the conda environment you used in the last lesson, and go to the directory where you worked in that lesson. Start the Python interpreter and enter the following one line at a time:

from ultralytics import YOLO
import os

model = YOLO("yolov8x-oiv7")
model.export(format="coreml", nms=True, int8=True)
os.rename("yolov8x-oiv7.mlpackage", "yolov8x-oiv7-int.mlpackage")
model.export(format="coreml", nms=True)

model = YOLO("yolov8n-oiv7")
model.export(format="coreml", nms=True)

model = YOLO("yolov8m-oiv7")
model.export(format="coreml", nms=True)

You’ll find these commands in the materials as download-models.py.

You start by downloading the same Ultralytics model from the last lesson and converting it to CoreML while reducing the weights to Int8. You then rename the resulting .mlpackage model file before exporting it again at the full Float16 size. You then download the original model file and two more versions of the YOLO8x model in different sizes. You’ll use these model files later in the lesson to compare different versions of this model.

Open the starter project for this lesson. Now, in Finder, find the following four model files - yolov8x-oiv7.mlpackage, yolov8x-oiv7-int.mlpackage, yolov8m-oiv7.mlpackage, and yolov8n-oiv7.mlpackage and drag them into the Models group of the Xcode project. Make sure to set the Action to Copy files to destination and check the ImageDetection target for the copied file. Then click Finish.

Using a Model with the Vision Framework

Since you’re dealing with image-related models, you can use the Vision framework to simplify interaction with the model. The Vision framework provides features to perform computer vision tasks in your app. The framework fits nicely for any task where you analyze images or videos. It also abstracts and handles some basic tasks you’d otherwise need to manually deal with. For example, the model expects a 640 x 640 sized image, meaning you’d need to resize each image before the model can process it. The Vision framework will take care of that for you.

To use the framework, open ContentView.swift and add the following import after the others at the top of the file:

import Vision

Now add the following method to the view after the cgImage parameter:

func runModel() {
  guard
    // 1
    let cgImage = cgImage,
    // 2
    let model = try? yolov8x_oiv7(configuration: .init()).model,
    // 3
    let detector = try? VNCoreMLModel(for: model) else {
      // 4
      print("Unable to load photo.")
      return
  }
}

You’ll use this method to perform object detection on the image. Here are the first steps.

1. You first need to set up the model and Vision framework, and since each can fail, you’ll wrap them inside a guard statement. This first step validates that the user has chosen a valid photo that can be represented as a CGImage. The PhotosPicker .onChange(of:initial:_:) modifier for selectedImage handles setting this state property when the user selects an image from the Photo library.
2. You attempt to load one of the models that you added earlier in this section. Note the yolov8x_oiv7 class name shown when you viewed the information about the mlpackage file. You create an instance of the class with the default configuration specified by .init(). For the Vision framework, you need access to the model itself, accessible using the model property on the class. Since this can throw an exception, you use the try? keyword, which returns nil if the call throws an error.
3. With the model loaded, you attempt to create a VNCoreMLModel class using the model loaded in step two. This class encapsulates the information loaded from the model file and acts as the interface between Swift and the model.
4. If any of these steps fail, then the guard/else block falls here, which will print a debugging message and return. In a real app, you’d need to provide more information and assistance to the user.

Now add the following code to the end of your new function:

// 1
let visionRequest = VNCoreMLRequest(model: detector) { request, error in
  if let error = error {
    print(error.localizedDescription)
    return
  }
  // 2
  if let results = request.results as? [VNRecognizedObjectObservation] {
    // Insert result processing code here
  }
}

This code builds the Vision request to run the model against an image but doesn’t execute it. It also defines a closure block that will execute when the detection completes. Here’s what each step does.

1. A VNCoreMLRequest creates the actual request for the Vision framework. You pass the VNCoreMLModel that you instantiated in step three as a parameter. The results of the request will be sent to the closure as a VNRequest named request with any errors sent through the error parameter passed to the closure block. This call can fail if the model has errors or is incompatible with Vision, such as a model that didn’t accept an image as input. If it fails, you print the error and return from the method.
2. If no error occurs, you then attempt to extract the results from the VNRequest as VNRecognizedObjectObservation objects. These objects contain the results of detect actions such as you’re doing in this app with this model. You’ll come back here in the next section to use this information.

Add the following code to finish out the method:

// 1
visionRequest.imageCropAndScaleOption = .scaleFill
// 2
let handler = VNImageRequestHandler(cgImage: cgImage, orientation: .up)
// 3
do {
  try handler.perform([visionRequest])
} catch {
  print(error)
}

This finishes the setup of the Vision framework request and performs the request to run the model against the image:

1. As mentioned earlier, the Vision framework ensures the image fits the size expected by the model, in this case, 640 x 640. The imageCropAndScaleOption property on the visionRequest tells Vision how to resize images with different sizes. In this app, the scaleFill option tells the framework to scale the image so that none of the image is lost when resizing and to pad out any unused area of the image.
2. The VNImageRequestHandler class processes the image-analysis request on a single image. Here, you use the cgImage version of the image selected by the user. You tell the method that the pixel data in the image is laid out so that the original pixel data matches the image’s intended display orientation. In other words, the image isn’t rotated or flipped.
3. At last, you run the model against the image. This can throw errors, so you wrap the call inside a do/catch statement to handle those errors. Here, you just print the error to the debug console. Notice that the call to perform(_:) doesn’t block, and when successful, the end of the function will happen immediately afterward. The results of the processing will be sent to the closure to VNCoreMLRequest, which you’ll work on in the next section.

This method now contains the code needed to identify the image and the model can work with the Vision framework and then send the image through the model. Now that you have the Vision framework in place, it’s time to do something with the results in the next section.