Instruction 2

While working with the models in this lesson, you’ve likely noticed that they can be quite large. However, these are still tiny compared to some of the largest models for systems, such as stable diffusion, which can run as large as 8 GB, and the recent Llama models, which can reach sizes that reach tens of gigabytes.

These large sizes can be a poor fit for mobile devices where storage and RAM are at a premium. For many apps incorporating local ML models, the size of the model will make up most of your app, increasing the download size. Putting off the download until later only pushes the problem into the future without solving it.

Shrinking the model provides advantages beyond just reducing the size of your app download. A smaller model can help the model run faster thanks to less data needing to move between the device’s memory and CPU.

The first approach to addressing this problem is to reduce the model size during training. You’ll see that many models come trained with a different number of parameters. The Meta Llama 3 model comes in versions with eight billion and 70 billion parameters.

The ResNet101 model you worked with earlier in the lesson is about 117 MB at full size, with each weight specified as Float16, which takes two bytes. Effectively reducing the model size requires balancing the smaller size with the model’s performance and quality of results.

Reduction Techniques

There are three primary techniques used in Core ML Tools to reduce model size. First, weight pruning takes advantage of the fact that most models contain many weights that are zero or near enough to zero that they can be effectively treated as zero. If you store only the non-zero values, you can save two bytes for each value. For the ResNet101 model, that can save about half the size. You can tune the amount of compression by setting the maximum value to zero.

The second technique is quantization. This technique reduces the precision from a Float16 to a smaller data type, usually Int8. An Int8 stores values between -127 and 128. This will save half the size of the original model.

The third technique refines this further and replaces each weight value with an index to an index table. This is known as palettization, which works by replacing weights with similar values with a single value and storing that value in the index table. You then replace the weight with the index value. The amount of compression depends on the number of values in the index table. For some models, you can deal with as few as four index values, resulting in a compression of 8X. Many model files also support using different index tables for different model parts.

Each method works best for different distributions of model weights. However, all lost information was found in the original model. When doing them, you must balance the amount of compression with the reduction in model accuracy and find the best compression for your use case.

This compression can be done either after the training, as you’ll do in this lesson, or during training. Doing compression during training usually lets you get the same accuracy at a higher compression rate at the cost of adding complexity and time to the training process.

Converting in Practice

CoreML Tools supports applying compression to existing CoreML models. Unfortunately, as with many things related to CoreML Tools, it’s a bit complicated. A separate set of packages works on the older .mlmodel type files compared to the newer .mlpackage files. In this section, you’ll work a bit with the latter.

Open your Python coremltools environment in conda and then start Python. Now enter the following code one line at a time:

import coremltools as ct
import coremltools.optimize as cto

This imports CoreML Tools and the optimization libraries new to Core ML Tools 7. Now enter:

orig_model = ct.models.MLModel("resnet101.mlpackage")

If you recall from the last section, this model stores data in a Float16 format. To save space, you’ll convert it to use Int8. Enter the following:

op_config = cto.coreml.OpLinearQuantizerConfig(
    mode="linear_symmetric", weight_threshold=512
)
config = cto.coreml.OptimizationConfig(global_config=op_config)
compressed_8_bit_model = cto.coreml.linear_quantize_weights(orig_model, config=config)

This code creates a configuration that tells CoreML Tools to quantize the model by converting it using linear symmetric interpolation. You then create an OptimizationConfig object set to the values and run the method to compress the weights. Now enter:

compressed_8_bit_model.save("resnet101-8.mlpackage")

This will save your model to the disk with a different name. If you view the two files, you’ll notice the new file is half the size of the previous one. You can see that converting from a 16-bit value to an eight-bit value should reduce the size by half.

Reducing an Ultralytics Model Size

Again, the Ultralytics package wraps this complexity for you. Enter the following code:

from ultralytics import YOLO
model = YOLO("yolov8x-oiv7.pt")
model.export(format="coreml", nms=True, int8=True)

This differs from your earlier export by adding the int8=True parameter which activates Int8 quantization. This will take a few minutes to run, but when it completes, you’ll have a file that’s roughly half the size of the original file.

How does this compression and optimization affect the speed and accuracy of the models? You’ll explore that in the next lesson as you integrate these models into an iOS app.