3rd-Party On-Device Models

In the last lesson, you converted a model using CoreML Tools. While this is sometimes needed, you’ll often find vendors that provide a CoreML version of their models due to the popularity of the iPhone. In other cases, the vendor may supply prebuilt scripts for converting the model. You’ll work with such a model named YOLOv8 from Ultralytics. This library performs a number of tasks, including:

• Detection: Identifying the location and class of objects in an image.
• Segmentation: A more exact version of detection and the same as ResNet from the last part of this section.
• Pose Estimation: The identification of certain points in an image, such as identifying the joints of people in an image.

The library is available for free use in academic and learning scenarios but does require licensing for commercial use. While Ultralytics used PyTorch for the native model, they provide a rich set of support and help to convert this into other environments and this will make your job of satisfying the dependencies and getting the model running locally on an iOS device much easier.

Open a terminal and enter the CoreML Tools environment that you created in the last lesson by entering conda activate coremltools. Now enter the following at the prompt:

pip install ultralytics

Running a Model

As you installed the library using the pip package manager, you should expect Ultralytics to support Python. Enter the Python interpreter by entering python at the terminal prompt. The first thing you need is the Ultralytics YOLO library. Enter the following at the prompt:

from ultralytics import YOLO

Depending on the specs of your computer, it can take 30 to 60 seconds for this command to complete. Python divides code into organizational units called modules. This command imports the YOLO submodule of the Ultralytics module. Both were installed when you installed the ultralytics Python package.

To load a model, enter the following command at the Python prompt:

model = YOLO("yolov8x-oiv7.pt")

This is the largest and most advanced detection model from Ultralytics at this time and can identify and classify over 600 different objects in an image, from accordions to zucchinis. When you enter this command, it’ll search for the model file in the current directory. If found, it’ll use it. If not found, as it will not be, since this is the first time you run this command, the YOLO module will download the pre-trained yolov8x-oiv7.pt model from GitHub. The file currently is 131 MB and can take several minutes, depending on your internet speed.

Once the download completes, enter the following at the Python prompt:

model("sample-image.jpg")

This will run the model against the sample image. In the first line of the output, you’ll find it correctly detected two cats in the image. Keep this in mind, as after conversion, you should ensure the model produces the same output for the same input. You’ll learn more about interpreting this output when integrating it into an iOS app.

If python complains that Numpy is not available (even if you have it installed), you may need a newer version of torch and torchvision. If so, exit back to the command line and upgrade those packages:

pip install torch==2.4.0
pip install torchvision==0.19

Then go back into the python interpreter and load the previous commands in again.

Converting a Model

From the previous section, you should remember the .pt extensions identify a PyTorch model. For your iOS app, you want a CoreML model. Fortunately, you can get a .mlpackage CoreML file. Enter the following at the Python prompt:

model.export(format="coreml", nms=True)

You’ll now see some examples of the problems of keeping versions in sync for machine learning projects. You’ll get some warnings and errors related to versions being untested. Eventually, the conversion will start and, depending on your hardware specs, can take several minutes. For about half the time, you’ll see no indication the conversion is still running, but be patient. When testing on a MacBook Pro with an M2 Max and 32 GB of RAM, it takes about four minutes to complete the export.

Exit Python by entering:

exit()

In the folder, you’ll now see a new yolov8x-oiv7.mlpackage file along with the earlier resnet101.mlpackage file. In the next section, you’ll explore these two models.

Adding CoreML Models to a Project

Now that you have a model, you’ll integrate this model into your iOS app. Find and open the Starter project for this lesson. Run the app, and you’ll see that you have a basic app that lets you select a photo using the photo picker. When you select a photo, it’ll show in the view. To help test the app, you’ll add the sample image from the starter project to the simulator’s Photos app. Open the folder with the sample image in Finder. Now drag the sample-image.jpg file on top of the simulator. The simulator will add it to its Photos collection.

In the next lesson, you’ll add the ability to run models against images. For now, you’ll explore a bit about the models.

Now, find the yolov8x-oiv7.mlpackage file you created in the previous section in Finder. Drag that model file into the Models group of the Xcode project. Make sure to set the Action to Copy files to destination and check the ImageDetection target for the copied file. Then click Finish. Do the same with the resnet101.mlpackage file.

With both models added to your Xcode project, click resnet101.mlpackage. Xcode will display several pieces of useful information about the model. At the top, you’ll see the type of model, ML Program in this case, the model’s size, and the compatibility of the model across Apple platforms. The next line displays the Model Class, which gives the name of the auto-generated class Xcode created from the file, resnet101.

Below this information, you’ll see several tabs. The first General will display any metadata. Since you didn’t add that during the conversion, you’ll see it blank. The tab also lists the size of the weights inside the model. The Float16 type consists of a floating point number stored in two bytes. A Float32 stores a floating point number in four bytes, allowing more precision and range at the cost of doubling the space. An Int32 stores information using four bytes. These sizes directly contribute to the size of the model, as larger formats require more space. The Storage parameter is Float16, indicating the weights in the model are stored in that format. To the right, you’ll see information on the operation of the model outside the scope of this lesson.

The Predictions tab will show you the expected input and output of the model. This model expects a MultiArray, an array that stores data in multiple dimensions. This one will have the dimensions of 1 x 3 x 640 x 640. Those numbers should look familiar as they were the size of the im parameter you passed in when creating the model. The first digit tells the model to expect a single image at a time. The second digit defines that each image will consist of three values for the red, green, and blue components of the image. The last two are the size of the expected image.

Now click the yolov8x-oiv7.mlpackage file. You’ll see the size is a bit larger at 137.7 MB, and the class names match the filename with the dash replaced by an underscore. On the General tab, you’ll see the conversion library filled in more of the metadata, and this model also has weights in the Float16 format. On the right, you’ll see all the class labels the model supports. These define the types of objects the model can identify. When you begin working on an app to run this model, you’ll look at the Predictions tab in the next section.

Select the new Preview tab. Here, you can test the model against your own images. It’s good practice to verify that the converted model gives matching results to the original after converting a model. Click the plus sign to the bottom left of the area stating Drag or Add Images and select Add files…. Find and open the sample image from earlier. You’ll see the image load to the right of the Preview tab. After a short processing period, you’ll see the model detects the two cats. Since it detected two cats in both the original PyTorch and CoreML models, you can feel comfortable that the model produces matching results.