10. YOLO & Semantic Segmentation

Written by Matthijs Hollemans

You’ve seen how easy it was to add a bounding box predictor to the model: simply add a new output layer that predicts four numbers. But it was also pretty limited — this model only predicts the location for a single object. It doesn’t work so well when there are multiple objects of interest in the image.

You might think that you could just add more of these output layers, or perhaps predict 8 numbers for two bounding boxes, or 12 for three bounding boxes, etc. Good try, but unfortunately that doesn’t work so well in practice.

Each bounding box predictor will end up learning the same thing and, as a result, makes the same predictions. Instead of finding the locations of multiple objects, such a model will predict the same bounding box multiple times. And chances are, these bounding boxes will not actually enclose any of the objects but all end up somewhere in the middle of the image as a compromise.

To make a proper object detector, you need to encourage the different bounding box predictors to learn different things.

An old-school approach to object detection is to divide up the input image into many smaller, partially overlapping regions of different sizes, and then run a regular image classifier on each of these regions. This definitely works, but it gives a lot of duplicate detections. Even worse: It’s really slow. You need to run the classifier many, many, many times for each image.

A slightly smarter approach is to first try and figure out which parts of the image are potential regions of interest. This is the approach taken by the popular R-CNN family of models. The classifier is still run on multiple image regions, but now only on regions that are at least somewhat likely to have an object in them.

To predict which regions are potentially interesting, the “Faster R-CNN” model uses a Region Proposal Network, which sounds impressive but is really just a bunch of layers on top of the feature extractor — hey, what did you expect? Unfortunately, even though it has “Faster” in its name, this model is still on the slow side and not really suitable for mobile devices.

For speed freaks and mobile device users, the so-called single stage detectors are very appealing. As the name implies, these model types just run the classifier once on the input image and do all of the work in a single pass. Examples of single-stage object detectors are YOLO (You Only Look Once), SSD (Single Shot multi-box Detector) and DetectNet.

Turi Create lets you train a YOLO model with just a few lines of code, so that’s what you’ll do next.

Single stage detectors

The simplest form of a single stage detector, and the one you’ll be training, looks like this:

Again, there’s a feature extractor plus a few layers on top. The YOLO feature extractor is called Darknet, and it’s not so different from the feature extractors you’ve seen before: Darknet consists of convolution layers, followed by batch normalization and the ReLU activation function, with pooling layers in between.

Note: The activation function used by Darknet is actually a variation of ReLU, known as leaky ReLU. Where a regular ReLU completely removes any values that are less than zero, the leaky version makes negative values a lot smaller but still lets them “leak through.”

The extra layers are all convolutional. Unlike before, where the output of the model was either a vector containing a probability distribution or the coordinates for the bounding box, the output of YOLO is a three-dimensional tensor of size 13 × 13 × 375 that we’ll refer to as the grid.

YOLO takes a 416×416 pixel image as input. That’s larger than what you typically use for classification. This way, small details don’t get lost. There are five pooling layers in Darknet that each halve the spatial dimensions of the image, for a total reduction factor of 32. Since 416/32 = 13, the final grid is 13×13 pixels.

Looking at this the other way around, each of the cells in this grid refers to a 32×32 block of pixels in the original image. Each cell is therefore responsible for detecting objects in or around that particular 32×32 region of the input image.

YOLO, therefore, has 13×13 = 169 different bounding box predictors, and each of these is assigned to look only at a specific location in the image. Actually, this isn’t entirely true: Each grid cell has not just one but 15 different predictors, for a total of 169×15 = 2,535 bounding box predictors across the entire image. That’s quite an upgrade over the simple model you made previously!

Having multiple predictors per grid cell means you can let bounding box predictors specialize in different shapes and sizes of objects. Each cell will have a predictor that looks for small objects, a different predictor that looks for large objects, one that looks for wide but flat objects, one that looks for narrow but tall objects, and so on.

This is where the number 375 comes from, the depth dimension of the output grid: Each grid cell has 15 predictors that each output 25 numbers. Why 25? This is made up of the probability distribution over our snack classes, so that’s 20 numbers. It also includes four numbers for the bounding box coordinates. Finally, YOLO also predicts a confidence score for the bounding box: how likely it thinks this bounding box actually contains an object. So there are two confidences being predicted here: one for the class, and one for the bounding box.

Because the output of YOLO is a 13×13×375 tensor, it’s important to realize it always predicts 2,535 bounding boxes for every image you give it. Even if the image doesn’t contain any recognizable objects at all, YOLO still outputs 2,535 bounding boxes — whether you want them or not.

That’s why the confidence score is important: It tells you which boxes you can ignore. In an image with no or just a few objects, the vast majority of predicted boxes will have low confidence scores. So at least YOLO is kind enough to tell you which of these 2,535 predictions are rubbish.

Even after you filter out all the boxes with low confidence scores — for example, anything with a score less than 0.25 — you’ll still end up with too many predictions. This kind of situation is typical:

These are all bounding boxes that the model feels good about since they have high scores, but as a consumer of an object detection model, you really want to have only a single bounding box for each object in the image. This sort of thing happens because nearby cells may all make a prediction for the same object — especially when the object is larger than 32×32 pixels.

To filter out these overlapping predictions, a post-processing technique called non-maximum suppression or NMS is used to remove such duplicates. The NMS algorithm keeps the predictions with the highest confidence scores and removes any other boxes that overlap the ones with higher scores by more than a certain threshold, say an IOU of 45% or more. The model created by Turi Create automatically takes care of this post-processing step for you, so you don’t have to worry about any of this.

Note: Turi’s object detection model is known as TinyYOLO because it’s smaller than the full YOLO. The full version of YOLO has multiple output grids of varying dimensions in order to handle different object sizes better, but this model is also larger and slower. Another popular single-stage detector is SSD. Architecturally, YOLO and SSD are very similar in design and differ only in the details. SSD does not have its own feature extractor and can be used with many different convnets. Particularly suitable for use on mobile is the combination of SSD and MobileNet.

Hello Turi, my old friend

Switch to the turienv Python environment and create a new Jupyter notebook. You can find the environment in the starter project of this chapter’s materials. Refer back to Chapter 4: Getting Started with Python & Turi Create if you don’t remember how to activate environments.

You can also follow along with final/YOLO.ipynb from the chapter’s resources.

First, import the needed packages:

import os, sys, math
import pandas as pd
import turicreate as tc

Training an object detection model with Turi Create is straightforward, but you do need to tell it about the bounding box annotations for the training images.

Turi gets its training data from an SFrame object. You will have to add the ground-truth bounding boxes in a new column named annotations. Unlike with Keras, where each row from the Pandas DataFrame was a separate annotation, in the Turi SFrame there is only one row per image. The annotations column must have all the ground-truth boxes for that image.

You do this by putting the image’s bounding box information into a list of dictionaries, like so:

[ {'coordinates': {'height': 129, 'width': 151, 'x': 75, 'y': 186},
   'label': 'juice'},
  {'coordinates': {'height': 130, 'width': 170, 'x': 228, 'y': 191},
   'label': 'juice'},
  {'coordinates': {'height': 129, 'width': 153, 'x': 76, 'y': 191},
   'label': 'juice'} ],

There is a separate dictionary for each annotation. It has two keys: coordinates, which in turn is another dictionary that holds the bounding box coordinates, and label, which is the class name of the object inside the bounding box. The above annotations are for a single image, ID 06d9c7df75a1a12f, that has three bounding boxes.

The first order of business is to write some code that loads the annotations CSV files and puts them into the format Turi Create expects. Since this is a fairly large function, we’ll describe it here in parts:

def load_images_with_annotations(images_dir, annotations_file):
    # Load the images into a Turi SFrame.
    data = tc.image_analysis.load_images(images_dir, with_path=True)

    # Load the annotations CSV file into a Pandas dataframe.
    csv = pd.read_csv(annotations_file)

First, you create a new SFrame by loading all the images from the specified folder. This is the same as what you did back in Chapter 4, “Getting Started with Python & Turi Create.” The new SFrame contains two columns: image with the image object and path with the image’s folder and filename.

The second line loads the CSV file into a Pandas DataFrame like you did in the previous chapter. Now, you will combine these two sources of data into a single SFrame that Turi can use for training. The function continues:

    all_annotations = []
    for i, item in enumerate(data):
        # Grab image info from the SFrame.
        img_path = item["path"]
        img_width = item["image"].width
        img_height = item["image"].height

        # Find the corresponding row(s) in the CSV's dataframe.
        image_id = os.path.basename(img_path)[:-4]
        rows = csv[csv["image_id"] == image_id]

The for loop looks at all images in the SFrame and then tries to find the corresponding annotations from the CSV’s DataFrame. The match is performed on the image_id field.

This field does not exist in the SFrame but you can use os.path.basename() to get the name of the file from the full path, and use Python’s special [:-4] indexing syntax to strip off the last four characters that say .jpg.

Then csv["image_id"] == image_id finds all the rows in the Pandas DataFrame that match this ID. Now, this doesn’t give you yet what you’re looking for. It returns a new Pandas object with the same number of rows as csv but with every row having the value True or False, depending on whether or not the row matched the predicate.

To get just the rows with the specified image ID, you need to filter csv again based on this True/False mask by writing csv[csv["image_id"] == image_id]. The variable rows is now a brand new dataframe with the actual annotations for just this image.

The loop continues:

        img_annotations = []
        for row in rows.itertuples():
            xmin = int(round(row[2] * img_width))
            xmax = int(round(row[3] * img_width))
            ymin = int(round(row[4] * img_height))
            ymax = int(round(row[5] * img_height))

            # Convert to center coordinate and width/height:
            width = xmax - xmin
            height = ymax - ymin
            x = xmin + math.floor(width / 2)
            y = ymin + math.floor(height / 2)
            class_name = row[6]

            img_annotations.append({"coordinates":
                  {"height": height, "width": width, "x": x, "y": y},
                  "label": class_name})

This looks like a lot of code but it simply reads the bounding box coordinates from rows and converts them into the format that Turi expects. Recall that the CSV file stores the coordinates as normalized numbers between 0 and 1, but Turi wants them in pixel space, so you need to multiply them by the image width and height. Also, Turi describes the bounding boxes using a center coordinate and a width and height. A bit of math is needed to convert the bounding boxes from one format to the other.

Once all the annotations for the current image have been converted and added to the img_annotations list, you append it to the grand list of all annotations:

        if len(img_annotations) > 0:
            all_annotations.append(img_annotations)
        else:
            all_annotations.append(None)

If there were no annotations, you still need to append something to all_annotations, so that this list has exactly the same number of rows as the SFrame. In that case, you append None, which is Python’s version of Swift’s nil.

Finally, once you’ve looped through all images, the all_annotations list contains all their ground-truth bounding boxes in Turi format. You put this into an SArray object and assign it to a new column in the SFrame named "annotations":

    data["annotations"] = tc.SArray(data=all_annotations, dtype=list)
    return data.dropna()

There’s one more thing to do, here. Recall that not all images will have annotations. For such images, the annotations field in the SFrame will be None. You don’t want to include these images during training.

The easiest way to remove those images from the SFrame is to call data.dropna(). This filters out any rows with missing values.

And that’s it for load_images_with_annotations().

Now you can load the training images and their bounding boxes. Make sure the folder you’re working in contains the snacks dataset. If you haven’t downloaded it already, open the snacks-download-link.webloc file from the starter folder, and, once downloaded, move the snacks folder in your working directory.

Load the images with the following code:

data_dir = "snacks"
train_dir = os.path.join(data_dir, "train")

train_data = load_images_with_annotations(train_dir,
                    data_dir + "/annotations-train.csv")

It might take a short while to load all the images. When it’s done, len(train_data) should print 4265 because that’s how many training images you have annotations for.

train_data.head() should show the following:

To view the annotations for a specific training image in more detail you can run a cell that prints train_data[some_index], but even better is Turi’s built-in visualization tool:

util = tc.object_detector.util
train_data["image_with_ground_truth"] = util.draw_bounding_boxes(
  train_data["image"],
  train_data["annotations"])
train_data.explore()

This adds another column to the SFrame named image_with_ground_truth that does exactly what it says: It contains the images with the ground-truth bounding boxes drawn on top.

Now that the data is in order, you’re ready to start training the model.

Training the model

It just takes a single line of code and a whole lot of patience:

model = tc.object_detector.create(train_data, feature="image",
                                  annotations="annotations")

If this is your first time training this kind of model, Turi Create will first download the pre-trained weights for the Darknet feature extractor. And then it starts training:

Setting 'batch_size' to 32
Using GPU to create model (GeForce GTX 1080 Ti)
Setting 'max_iterations' to 13000
+--------------+--------------+--------------+
| Iteration    | Loss         | Elapsed Time |
+--------------+--------------+--------------+
| 1            | 11.276       | 12.7         |
| 36           | 10.892       | 22.8         |
| 71           | 10.506       | 32.8         |
| 107          | 10.517       | 43.1         |
...
| 12999        | 2.106        | 3755.3       |
+--------------+--------------+--------------+

Needless to say, having a GPU for training this model is a must. The author trained on Linux with an NVIDIA GPU, but Turi Create can also use your Mac’s AMD GPU if you have a recent Mac running macOS Mojave. Even on the powerful 1080 Ti, it still took over an hour to train this model. Training on the CPU takes ages — at about eight seconds per iteration, doing 13,000 iterations would need about 29 hours.

If you don’t have 29 hours of patience, don’t worry! We included the trained model in this chapter’s materials that you’ll use in the next section.

Note: In order to use Turi Create with a GPU on Linux, you will need to install MXNet. See the Turi Create user guide at https://github.com/apple/turicreate/blob/master/LinuxGPU.md for instructions.

Once the model is done training, you can save it:

model.save("SnackDetector.model")

Also, export the model to Core ML. It’s possible you get some warnings at this point from coremltools. You can safely ignore these.

model.export_coreml("SnackDetector.mlmodel")

Before you put this model into an app, let’s use Turi Create to evaluate how well it does on the test set.

How good is it?

In case you don’t have the hardware or the time to train this model yourself, we’ve included the trained model in the downloads as a .zip file in the final folder, SnackDetector.model.zip. Unzip this model to your working directory and then load it into the notebook:

model = tc.load_model("SnackDetector.model")

Also, load the test images and their annotations:

test_dir = os.path.join(data_dir, "test")
test_data = load_images_with_annotations(test_dir,
                   data_dir + "/annotations-test.csv")

Then call model.evaluate() on the test_data SFrame:

scores = model.evaluate(test_data)

This predicts the bounding boxes for every image in the test set and then compares these predictions against the ground-truths. It can take a few minutes if you’re running this on a CPU. When it’s done, scores looks something like this:

{'average_precision_50': {
  'apple': 0.52788541232511876,
  'banana': 0.41939129680862453,
  'cake': 0.38973319479991153,
  'candy': 0.36857447872282678,
  ...
  'watermelon': 0.37970409310715819},
 'mean_average_precision_50': 0.38825907147323535}

In the previous chapter, you used the IOU or Intersection-over-Union metric to determine how good the predictions were, but Turi uses a different metric. It computes the average precision for each class, as well as the overall average of these average precisions — yes, you read that right — known as the mean average precision, or mAP.

What’s important for these average precision metrics is that higher is better. If you were to train a different model on this same dataset and it gave a better mAP score, then you can safely draw the conclusion this new model is indeed better than the old one.

IOU only measures by how much the predicted object overlaps the real object. It doesn’t say anything about the classification accuracy — if a predicted box for class “orange” overlaps a ground-truth box for class “apple” with 95%, then the bounding box is very accurate, but the class is totally wrong. For a realistic metric, you want to make sure the class matches, too.

Also, when evaluating an object detector, you want to get some idea of whether the model actually finds all the objects in the image (the recall). The mAP metric combines these different criteria into a single number. It’s not necessarily a very intuitive metric, but it’s useful to rank the quality of different models on a given dataset.

Note: The Pascal VOC dataset (host.robots.ox.ac.uk/pascal/VOC) is one of the standard datasets that people use to benchmark object detectors. At the time of writing, the top-scoring model on Pascal VOC was named DOLO and had an mAP of 81.3. The runner-up was a variant of Faster R-CNN with a score of 81.1. For comparison, YOLO v2 scores “only” 48.8 and SSD scores 64.0. You can view the leaderboards at this link: bit.ly/2ET24Ym. Another popular object detection dataset is COCO: cocodataset.org/#detection-leaderboard.

For more insight into how well your model is doing, take a look at the individual predictions for the test images, using model.predict():

test_data["predictions"] = model.predict(test_data)

This adds a new column to the test_data SFrame. The data in this column looks very similar to the annotations column from train_data, but in addition to the predicted coordinates and class label there is now also the confidence score:

[{'confidence': 0.7225357099539148,
  'coordinates': {'height': 73.92794444010806,
                  'width': 90.45315889211807,
                  'x': 262.2198759929745,
                  'y': 155.496952970812},
  'label': 'dog',
  'type': 'rectangle'},
 ...]

Again, it’s a lot nicer to look at this data using Turi’s visualization tool:

test_data["image_with_predictions"] =
    tc.object_detector.util.draw_bounding_boxes(test_data["image"],
                                                test_data["predictions"])
test_data.explore()

It looks like this:

That’s not half bad! The YOLO model does a pretty good job at finding — and properly classifying — the objects in the test images. It doesn’t always find all objects, and sometimes its predictions are plain wrong, but overall this is a very good result. By the way, if Turi didn’t find any objects in a test image, the predictions column contains an empty list [].

These test set results are all fine and good, so now it’s time to take the model out of the lab and onto the streets, and see how well it does on live video!

The demo app

This is a book about machine learning on iOS, and it’s been a while since we’ve seen the inside of Xcode, so let’s put the trained YOLO model into an app. The book downloads contain a demo app named ObjectDetection.

Note: This example app only works on iOS 12 / Xcode 10 and later.

Open the project in Xcode. This already includes the finished SnackDetector.mlmodel file that was exported from Turi Create.

Select the .mlmodel file in the Project navigator to take a closer look:

The model description indeed says this is an object detector using Darknet and YOLO, with non-maximum suppression.

The type of the model is not a neural network but a pipeline. In machine learning terms, a pipeline is several models that are glued together so that the output of one model is used as the input for the next model in the pipeline. In this case, the object detection model is followed by a non-maximum suppression (NMS) model.

Also, note that the SnackDetector model has three inputs and two outputs. In addition to the regular image input for a 416×416 color image, there are two new inputs named iouThreshold and confidenceThreshold. These two values are used by NMS to decide which bounding boxes it should keep.

The higher you set the confidence threshold, the larger the confidence score on a predicted box has to be in order to keep that box. The IOU threshold determines when overlapping two boxes are too similar. A lower value means that even boxes that only overlap a little bit are considered to be duplicates.

Even though you saw earlier that YOLO produces a single tensor of size 13×13×375, the Core ML model actually has two outputs. That’s because the Core ML pipeline applies NMS to the predictions from YOLO and only outputs the best bounding boxes. That’s also why the first dimension is 0, or unknown because NMS will return a different number of boxes depending on how many objects are in the image. For convenience, Core ML provides the class predictions and coordinates as separate values.

However, you don’t have to worry about these inputs and outputs because you’re going to be using this model through the Vision framework.

Most of the source code in ViewController.swift is exactly the same as in the previous example apps. You make the VNCoreMLModel and VNCoreMLRequest objects the same way as before. You still start the request using the VNImageRequestHandler. The only thing that’s different is the result object returned by Vision.

Previously the result objects were of type VNClassificationObservation, but now they are VNRecognizedObjectObservation objects. This is a new class that was added to Vision with iOS 12, and it exists specifically to handle the results from Turi Create’s YOLO model. All your app needs to do is handle these VNRecognizedObjectObservation instances. In the demo app, we draw a rectangle around any detected objects.

The fun stuff happens in processObservations(for:error:), which is called from the completion handler for the Vision request. This function receives an array of zero or more VNRecognizedObjectObservation instances. If the array is empty, no objects were found. In that case, the app removes any previous rectangles from the screen.

The logic for interpreting the Vision results lives inside the show(predictions:) method. This simply loops through the VNRecognizedObjectObservation instances, converts the predicted coordinates to screen coordinates, and shows rectangles for the detected objects using the BoundingBoxView class.

• The VNRecognizedObjectObservation class has a labels property containing a list of familiar VNClassificationObservation instances, sorted from highest probability to lowest, telling you the most likely classes for the object inside the bounding box. The app simply grabs the first VNClassificationObservation from the list, as this is the best prediction, and puts its identifier and confidence into the rectangle’s label.

• There is also a boundingBox property, a CGRect object that tells you where in the image the object is located. This uses normalized coordinates again, but with the origin of the CGRect in the lower-left corner. That’s a little awkward, but it’s just how Vision does things. In order to draw a rectangle around this object, you need to transform the normalized coordinates to screen coordinates and also flip the y-axis.

Here is a screenshot of the app in action:

Just an FYI: This picture was taken while pointing the iPhone at a picture on a Mac. This is a quick way of testing that the model works — just look up some test pictures on Google Images — but be aware that the interaction of the LEDs in the computer display with the camera’s sensor may cause artefacts to appear in the image that can throw off the model.

If users pointing their phones at computer screens is going to be a major use case for your own apps, then you’ll need to train the model to ignore those artifacts and distortions.

One limitation of Vision was that it could only provide a value for the model’s image input, but as of iOS 13, you can also pass in values for the other inputs to override the defaults from the model. You do this on the VNCoreMLModel object:

if #available(iOS 13.0, *) {
  visionModel.inputImageFeatureName = "image"
  visionModel.featureProvider = try MLDictionaryFeatureProvider(
      dictionary: [
    "iouThreshold": MLFeatureValue(double: 0.45),
    "confidenceThreshold": MLFeatureValue(double: 0.25),
  ])
}

With a bit of effort, it’s also possible to make YOLO work on iOS 11. In that case, Vision does not give you the convenient VNRecognizedObjectObservation instances but a VNCoreMLFeatureValueObservation with the contents of the 13×13×375 grid. You’ll have to decode these contents into actual bounding boxes and perform NMS yourself to find the best boxes. The YOLO models built with Turi Create aren’t compatible with iOS 11, but there are also Keras versions of YOLO available. For an example of how to do run a YOLO model on iOS 11, see github.com/hollance/YOLO-CoreML-MPSNNGraph.

Note: This was only a brief introduction to object detection. As you can imagine, there is a lot going on under the hood that we glossed over here. If you want to learn more about how these single-stage object detectors are trained, see the author’s in-depth blog post at machinethink.net/blog/object-detection/.

Note: The architecture used by Turi Create, “Tiny YOLO v2”, is already a few years old. You can download more modern YOLOv3 and YOLOv3-Tiny models from developer.apple.com/machine-learning/models/. These are trained on the 80 classes from the COCO dataset. To make a version of YOLOv3 that uses your own classes, you’ll need to train it yourself, for example using github.com/ultralytics/yolov3 or one of the many other open source implementations.

Note: The latest version of Turi Create can also do one-shot object detection. The term “one-shot” usually refers to training with only a single example image for each class, or at most a handful of training images. That’s great for many real-life scenarios where you won’t always have hundreds of training images. Turi does this using synthetic data augmentation, where they overlay the training image on a selection of real-world images. Another advantage is that with this method you don’t need to provide any bounding box annotations. See the WWDC 2019 session “Drawing Classification and One-Shot Object Detection in Turi Create” for more details: developer.apple.com/videos/play/wwdc2019/420/.

Semantic segmentation

You’ve seen how to do classification of the image as a whole, as well as classification of the contents of bounding boxes, but it’s also possible to make a separate classification prediction for each individual pixel in the image. This is called semantic segmentation. Here’s what it looks like:

On the right is the segmentation mask for this photo. It shows a different color for each class. Pixels that belong to the class “human” are yellow, pixels that belong to the class “motorbike” are purple. Pixels that don’t belong to any kind of object we care about are classified as belonging to the special background class.

Whereas object detection only gives you a rough idea of where objects are in the image, semantic segmentation tells you exactly what the objects are shaped like.

In this section, you’ll look at a top-of-the-line semantic segmentation model called DeepLab v3+. One of the possible applications of semantic segmentation is replacing a photo’s background with another picture. You’ll see how to do that in the included demo app.

The DeepLab model looks like this:

Not surprisingly, the first part of the neural network is comprised of a feature extractor. Here, you’re using version 2 of MobileNet, which is more powerful and more efficient — but also a little more complicated — than V1. On top of the feature extractor are the layers that perform the segmentation task.

An interesting twist is that for segmentation you want the output of the model to be an image with the same dimensions as the input image. DeepLab expects the input image to be 513×513 pixels and so the predicted segmentation mask should be 513×513 also.

But as you’ve seen, most feature extractors will gradually reduce the spatial dimensions of the images, usually by a factor of 32, using pooling or convolutions with a stride of 2. If that were the case here, too, you’d end up with an output of 16×16 pixels, which is not nearly accurate enough to function as the final segmentation map for the image.

To avoid this, the version of MobileNet used by DeepLab only has an output stride of 8 instead of 32. This means that, instead of five times, it only chops the tensors in half three times, scaling down the input by a factor of 8 instead of 32. Rounded off, that makes the output of the feature extractor 65×65 pixels. The semantic segmentation layers that follow the feature extractor then do their work on this 65×65-pixel tensor, which still has plenty of detail.

Note: DeepLab uses odd image sizes because some of the convolution layers are atrous or dilated, meaning they have holes in them. This is necessary to achieve that output stride of 8. With an odd number of pixels, there’s always a center pixel, and so the math works out better this way. You may immediately forget this.

Finally, there is an upsampling layer at the end of the model that scales the 65×65 tensor back up to 513×513 pixels using bilinear resizing. Obviously, you lose some of the finer details because of this upscaling, which is why, if you look closely at the edges of the objects in the segmentation map, you’ll see that they’re smoothed out.

The output of DeepLab is then an output “image” with the same width and height as the input image, 513×513. However, it does not have three color channels like a regular image. The version of DeepLab that you’re using here is trained on the Pascal VOC dataset of 20 classes, and so the output is a 513×513×21 tensor.

Every pixel has its own 21-element probability distribution from a softmax, because DeepLab does a class prediction for each individual pixel. Why 21 probabilities and not 20? Recall that you need an extra class to signify “background,” for pixels that don’t belong to any of the 20 regular classes.

Note: We’re not going to show you how to train this semantic segmentation model on the snacks dataset. It’s not particularly difficult to train these kinds of models, but unfortunately, we don’t have any ground-truth segmentation masks for the training images. Still, we wanted to show you that semantic segmentation is just another variation of the kinds of models you’ve already seen.

Converting the model

You’re going to be using a pre-trained version of DeepLab that is made freely available as part of the TensorFlow Models repository, at: github.com/tensorflow/models/tree/master/research/deeplab.

This model was trained on the Pascal VOC dataset and can recognize the following 20 classes: person, bird, cat, cow, dog, horse, sheep, aeroplane, bicycle, boat, bus, car, motorbike, train, bottle, chair, dining table, potted plant, sofa and tv/monitor.

The demo app already includes the converted .mlmodel file, but it’s a good idea to try converting this model by yourself.

Note: You can also download a version of DeepLab v3+ from developer.apple.com/machine-learning/models/. It is the same model you’ll be using here.

First, download the file containing the pre-trained TensorFlow model. We used mobilenetv2_coco_voc_trainval from the link download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz. We recommend that you use this exact version, or risk running into errors.

After you unzip the download, look for the file frozen_inference_graph.pb. The extension pb stands for protobuf, which is the format that TensorFlow models — or graphs as they are called — are saved in.

This particular file is a “frozen” graph that has been optimized for inference; all the operations for training have been removed from this graph.

Netron can open such pb files, so take a look inside.

The particular section of the graph shown in the figure on the next page, has the layers that are responsible for performing the semantic segmentation. These are all layer types you’ve seen before: convolution, batch normalization, ReLU and average pooling.

There isn’t some special semantic segmentation layer that performs all the magic — this neural network uses the same building blocks that they all do.

So how come these layers know how to perform the segmentation task — as opposed to classification or object detection or something else? The reason is the training data. During training, the output of the network is compared to the ground-truth segmentation masks from the training data, using a suitable loss function. The training process drives the loss to be lower and lower, and so the longer you train, the more the output of the network starts to resemble the ground-truth masks.

Of course, you can be clever in your choice of layers — for example, in the illustration above you see that DeepLab uses a branching structure called spatial pyramid pooling that makes predictions at different image scales, to capture detail at different levels.

But the point here is that it’s not so much the design of the network that makes it do a task; it’s the data you use to train it. In this case, the training targets were segmentation masks, and so the network has learned to perform semantic segmentation. It’s really that simple.

Usually coremltools is the go-to package for converting models to Core ML. You used it to convert your Keras model in the previous chapter. The bad news is that coremltools does not directly support TensorFlow graphs.

The good news is that Apple and Google have collaborated to bring us tfcoreml, a TensorFlow to Core ML converter. This is an additional Python package that you need to install alongside coremltools.

TensorFlow works at a much lower level of abstraction than Core ML and Keras. With TensorFlow, you build computational graphs that consist of primitive operations such as addition, multiplication, matrix math, array manipulations and so on.

Core ML and Keras, on the other hand, only know about neural network layers. That’s why it’s easy to convert from Keras to Core ML. But it’s not always possible to convert TensorFlow graphs to Core ML. You can only convert graphs that use operations that Core ML supports. The tf-coreml website, github.com/tf-coreml/tf-coreml, lists the supported operations.

Note: Core ML 3 is more capable in this regard than its predecessors. Like TensorFlow, it supports many low-level operations in addition to the neural network layers. Thanks to this, it should now be possible to convert more complex TensorFlow models to Core ML.

We recommend that for this section you use the kerasenv environment, since that already has coremltools and TensorFlow installed. If you don’t already have it installed, you can find kerasenv.yaml in the starter folder. Use conda create -n kerasenv.yaml to create the environment, and then activate it.

But this environment doesn’t have tfcoreml yet, so use pip to install the latest version:

$ pip install -U tfcoreml

You may also need to install the most recent version of coremltools. If using the one from pip gives errors, here’s a handy trick for installing the very latest version straight from GitHub:

$ pip install -U git+https://github.com/apple/coremltools.git

Now, create a new text file, convert_deeplab.py, and write the following:

import tfcoreml as tf_converter

input_path = "deeplabv3_mnv2_pascal_trainval/frozen_inference_graph.pb"
output_path = "DeepLab.mlmodel"
input_tensor = "ImageTensor:0"
input_name = "ImageTensor__0"
output_tensor = "ResizeBilinear_3:0"

tf_converter.convert(tf_model_path=input_path,
         mlmodel_path=output_path,
         output_feature_names=[output_tensor],
         input_name_shape_dict={input_tensor : [1, 513, 513, 3]},
         image_input_names=input_name)

That’s all you need to convert the model. Pretty simple, right? The trick is getting all the arguments to tf_converter.convert() correct.

Most importantly, you need to tell tfcoreml what the model’s input and output tensors are. For DeepLab, those are "ImageTensor:0" and "ResizeBilinear_3:0", respectively. See if you can find these tensors in the graph using Netron.

TensorFlow operators can have multiple outputs and you need to specify which one you want to use. The :0 in the name tells TensorFlow that you want to use the tensor from the operator’s first output.

Think of it as special syntax for indexing an array at element 0. It’s only a detail but if you forget the :0 behind the name, tf-coreml won’t be able to find the tensor in the graph.

The input_name_shape_dict argument tells tf-coreml what the size of the input image will be: 513×513 pixels.

Just like with the Keras conversion you’ve done before, image_input_names is used to inform Core ML that it should treat the input as a proper image instead of an array of numbers. Note that tf-coreml renames :0 to __0 in the names of the model’s inputs and outputs, so "ImageTensor:0" is now "ImageTensor__0".

Run this script from a Terminal:

$ python3 convert_deeplab.py

tf-coreml will now load the TensorFlow model, analyze the graph, and convert all the operations to Core ML layers. When it’s done and everything went well, you will have a brand new DeepLab.mlmodel file.

Double-click to open it in Xcode, and this is what you should see:

DeepLab’s output is a multi-array of size 21×513×513. Multi-array is the term that Core ML uses for tensor. When you need to deal with tensor objects in Core ML, it’s always through the MLMultiArray class. You’ll get a taste of that shortly.

Notice that Core ML puts the number of channels (21) at the front of the tensor, as the outermost dimension. Usually, in this book, we’ve described the size of a tensor as height × width × channels (HWC), but in practice, you’ll also see it done the other way around: channels × height × width (CHW). It’s important to know which format you’re working with at any given time.

Open DeepLab.mlmodel with Netron, too, and put it side-by-side with the original TensorFlow model to see how different/similar the two models are.

The demo app

The downloads for this chapter include an app called Segmentation. The code is very similar to that of the HealthySnacks app from a few chapters ago, except now there are two pairs of camera/photo library buttons, allowing you to select a background image and a foreground image.

After you’ve selected two images, the app will send the “front” image through DeepLab and uses the predicted segmentation mask to composite all the pixels that are not classified as “background” on top of the other image. Who needs a green screen when you’ve got a segmentation model?

Tap the screen to view the actual segmentation mask (right side of the illustration). Note that the results aren’t perfect — there are a few bits of a painting hanging in the background that got mistakenly classified as “person” and are shining through.

The app uses Vision to run the model, so that’s the same code as usual. But there are some interesting code snippets we can look at in more detail. First, the init method of ViewController.swift:

required init?(coder aDecoder: NSCoder) {
  let outputs = deepLab.model.modelDescription.outputDescriptionsByName
  guard let output = outputs["ResizeBilinear_3__0"],
        let constraint = output.multiArrayConstraint else {
    fatalError("Expected 'ResizeBilinear_3__0' output")
  }
  deepLabHeight = constraint.shape[1].intValue
  deepLabWidth = constraint.shape[2].intValue

  super.init(coder: aDecoder)
}

The size of DeepLab’s output image is 513×513 pixels. You could hard-code this into the app, but it’s better to ask the model what these dimensions are. That way the same code can work with models that output other sizes, too.

Here, deepLab is an instance of DeepLab, the class that Xcode automatically generates from the .mlmodel file. It has a model property for an MLModel object. If you don’t want to use Vision, you can also use the MLModel instance to make predictions directly.

More importantly, for our purposes, you can ask this object about the configuration of the model. Here you grab the modelDescription, which contains all the info you see in Xcode, and from that the outputDescriptionsByName. This is a dictionary describing the model’s outputs.

This model has only one output with the name "ResizeBilinear_3__0". Note that this used to be called "ResizeBilinear_3:0" in the TensorFlow graph, but tfcoreml renamed it. That output is of type multi-array, which means you literally get access to the entire 21×513×513 tensor of output values. By grabbing the multiArrayConstraint property, you can look at the size and datatype of this array. Here, you care about the size, given by the shape property.

Because the Core ML API was designed to work with both Swift and Objective-C, using some of these classes can be a little elaborate. For example, shape returns an array of NSNumber objects, and so you need to use .intValue to turn these into integers. The shape array has three values in it: [channels, height, width] and you read the height and width into two properties so you can use them later.

After Vision successfully performs the request, the prediction results arrive as an array of VNCoreMLFeatureValueObservation objects. You get one of these objects for every output that is of type multi-array. The actual predictions are inside an MLMultiArray object. This is how you obtain that MLMultiArray from the Vision results:

func processObservations(for request: VNRequest, error: Error?) {
  if let results = request.results as? [VNCoreMLFeatureValueObservation],
     !results.isEmpty,
     let multiArray = results[0].featureValue.multiArrayValue {

    DispatchQueue.main.async {
      self.show(results: multiArray)
    }
  }
}

The show(results) method then decides whether to show the composited image or just the segmentation mask. It uses two helper methods (createMaskImage or matteImages) to do the actual work. Both of these helper methods follow the same approach:

1. Allocate an array of type UInt8 that will hold the output pixels. The size of this array is deepLabWidth * deepLabHeight * 4 because it will be an RGBA image.
2. Loop through all the 513×513 pixels in the MLMultiArray that holds DeepLab’s predictions.
3. For each pixel, find the index of the winning class. This is done by looping over the 21 probability values for that pixel and finding the largest value (the argmax).
4. To combine the two photos, the color that is used for the output pixel is read from the foreground image if the best class is not 0, the special background class. Otherwise, it is read from the background image. It’s easy to change this logic to only keep certain classes in the image, such as only cats and dogs.
5. In the other mode, where the app is drawing the segmentation mask, you get the color of the output pixel from a lookup table, using the winning class as the index.
6. Finally, convert the pixel array into a UIImage.

You’ll now look at how some of these steps work in detail. The MLMultiArray API is a little tricky to work with. For many model types, such as classification and object detection, Vision will hide away these details from you, but if your model outputs a multi-array then you have no choice but to get your hands dirty.

In the following code, features is the variable with the MLMultiArray object.

let classes = features.shape[0].intValue
let height = features.shape[1].intValue
let width = features.shape[2].intValue
var pixels = [UInt8](repeating: 255, count: width * height * 4)

Just like the model’s output description had a shape, so does the actual MLMultiArray. You use the width and height from this shape to allocate the pixels array.

For this particular model, the output dimensions are fixed at 513×513, but in the model description for YOLO you saw that the first output dimension was 0, or unknown. In that case, the actual shape of the multi-array object isn’t known yet until runtime and can be different from one invocation of the model to the next.

To read a value from the MLMultiArray, you can write the following:

let value = features[[c, y, x] as [NSNumber]].doubleValue

Where c is the channel number (0-20), y is the vertical coordinate (0-512), and x is the horizontal coordinate (also 0-512). Remember that the height dimension comes before the width!

Indexing the multi-array in this manner works fine, but it’s very slow. Note that you’re first creating an array [c, y, x] to hold the three indices. Because this is an Objective-C API, you need to cast that to an array of NSNumbers. MLMultiArray uses this NSNumber array as a subscript, reads the value from the tensor, and returns it as a new NSNumber object that you have to convert back to a Double before you can properly use it.

Now, imagine doing this in a triple nested loop of 21×513×513 iterations. It gets slow really quick.

A better approach is to use a pointer to directly access the MLMultiArray’s memory. After all, it’s just a big array of Double values. Using pointers is not something Swift developers are accustomed to doing, but it’s not a big deal:

let featurePointer = UnsafeMutablePointer<Double>(
                           OpaquePointer(features.dataPointer))
let cStride = features.strides[0].intValue
let yStride = features.strides[1].intValue
let xStride = features.strides[2].intValue

First, you turn features.dataPointer, a raw pointer, into an UnsafeMutablePointer for Double values. To find out where in this memory area the value that you want to read is located, you need to do a little bit of math. That’s what the strides are for. This again is an array of NSNumbers.

The stride for a given dimension tells you how far apart in memory subsequent values from that dimension are. Here, cStride is the stride of the first dimension, which holds the channels. It is 263169 because one channel is made up of 513×513 = 263169 pixel values. The yStride is the distance between two subsequent rows in the image and is 513 because one row contains 513 pixels. And xStride is the distance between two pixels in the same row, which is 1 because they’re right next to each other in memory.

Now, you can forget about these numbers immediately. Important to remember is that the strides are used to index the MLMultiArray’s memory directly when you’re using pointers. Conveniently enough, MLMultiArray has already calculated what the correct stride values are.

To read the value at c, y, x, you can now write:

let value = featurePointer[c*cStride + y*yStride + x*xStride]

That’s all you need to do to directly read the Double value from the MLMultiArray’s memory. It doesn’t get much faster than this!

The main processing loop then looks like the following:

for y in 0..<height {
  for x in 0..<width {

    // Take the argmax for this pixel, the index of the largest class.
    var largestValue: Double = 0
    var largestClass = 0
    for c in 0..<classes {
      let value = featurePointer[c*cStride + y*yStride + x*xStride]
      if value > largestValue {
        largestValue = value
        largestClass = c
      }
    }
    . . .

There are three nested loops: You loop through all the rows (y) and all the columns (x) of the multi-array to look at all the image positions. Each “pixel” is really made up of 21 probability values, and you loop through these (c) to find the largest one. Then you can use the value of largestClass to do something interesting with this pixel.

Note: We told a small lie earlier. The 21 probability values for each pixel aren’t really probabilities yet, but so-called logits. To save time, DeepLab didn’t actually apply a softmax to these numbers. The softmax computation is kind of slow, and it would have to be done for every individual pixel, that is 513×513 times.

All the softmax does is re-scale the logits so that they sum up to 1, but this doesn’t actually change the relative order of these numbers. If you’d sort the logits from before the softmax, and the probabilities from after the softmax, they’d be in the exact same order. Because you only care here which class has the biggest value, and not what probability that value represents, you can skip the softmax step and save some time.

Try it out, run the app and see how well the semantic segmentation model works on your own pictures. The app will run on the simulator or on a device with iOS 12 or newer.

By the way, it can take a few seconds for the model to run. Semantic segmentation is a harder job than classification. Also, the code for compositing the two images is not necessarily the most optimal way to do this. We just wanted to keep the example code readable. In practice, you’d use optimized routines from Core Image, the vImage framework, or even Metal GPU shaders to draw these images.

There are also different versions of DeepLab v3+. There is one that uses the Xception network as the feature extractor instead of MobileNet. This gives higher quality results but it also comes at a cost: The MobileNet version of DeepLab is only 8.6 MB, while the Xception version is easily ten times bigger.

Note: With semantic segmentation, pixels only know which class they belong to. There is also a different kind called instance segmentation, where the pixels not only know their class but also which distinct object they belong to. For example, in a photo of two people who are sitting side by side and are touching, semantic segmentation will only see a single blob in which all the pixels are of class “person.” Instance segmentation will be able to distinguish between person 1 and person 2.

A popular model for instance segmentation is Mask R-CNN, which adds segmentation capabilities to the Faster R-CNN object detector. It makes sense to think of instance segmentation as being a combination of object detection and segmentation. As should be obvious by now, all these techniques are closely related.

Challenges

Challenge 1: Create a dataset for object detection

If you collected your own classification dataset for one of the previous challenges, then use a tool such as RectLabel to add bounding box annotations for these images. RectLabel uses a different file format to store the annotations, but rectlabel.com has code examples that show how to use these files with Turi Create.

Challenge 2: Train MobileNet+SSD on the snacks dataset

The size of the YOLO model you trained with Turi Create is 64.6 MB. That’s pretty hefty! This is reaching the upper limit of what is acceptable on mobile devices. It’s possible to use object detection models that are much smaller than YOLO that give very good results, such as SSD on top of MobileNet (about 26 MB).

Try training MobileNet+SSD on the snacks dataset. The easiest way to do this is with the TensorFlow object detection API at github.com/tensorflow/models/tree/master/research/object_detection. This is a good project to get some experience working directly with TensorFlow. The main difficulty will be converting the training data into a format that this API understands.

To convert the final TensorFlow model to Core ML, you can use tf-coreml and the following repo: github.com/vonholst/SSDMobileNet_CoreML.

Compare the mAP of this model with the mAP reported for YOLO earlier this chapter to see if MobileNet+SSD compares favorably or not. Good luck!

Challenge 3: Change the semantic segmentation demo app

Change the semantic segmentation demo app to only keep pixels that belong to cats and dogs — or whatever your favorites are from the 20 Pascal VOC classes.

Key points

• To create a model that can detect multiple objects, it’s not enough to just add extra bounding box predictors. Single-stage detectors like YOLO and SSD put the predictors into a grid so that they only look at specific portions of the input image. They also train different predictors to specialize in various object shapes and sizes.

• Non-maximum suppression (NMS) is a post-processing step used to only keep the best bounding box predictions. The YOLO model that is trained by Turi Create will automatically apply NMS.

• Semantic segmentation lets you make a unique class prediction for every pixel in the image. Instead of a single probability distribution, this predicts as many probability distributions as there are pixels.

Where to go from here?

Congrats, you’ve reached the end of section 1, Machine Learning with Images! Of course, we hope this is really only the beginning of your journey into the wonderful world of computer vision and deep learning.

Some fun new areas to explore are:

• Models that can create new images. Style transfer is a technique of taking a photo and making it look like a famous painting. Colorization adds color to old black-and-white photos. Generative models can produce totally new works of art, such as an infinite number of unique anime characters, youtube.com/watch?v=PUkQbGaL4Fg.

• Instead of working with still images you can also add a time dimension and make predictions on video. One example is tracking moving objects, which is like object detection but over time. Another futuristic application — literally! — is predicting what will happen in the next few frames of the video.

• Human pose detection. You can use a neural network to find keypoints on the human body, such as where a person’s hands and feet are. It’s even possible to reconstruct realistic 3D models of the human body from photos, see densepose.org.

And much more… The possibilities are endless!

One cool project that was published recently is Everybody Dance Now, which combines techniques of human pose detection and generative models to copy a professional dancer’s moves onto the body of a regular person, turning anyone into a dancing pro. See the amazing video at youtube.com/watch?v=PCBTZh41Ris.

Believe it or not, you already understand 90% of the techniques used in that project and many others. Everything builds on what you’ve learned in the first part of this book.

If you want to know exactly how Everybody Dance Now works, check out the paper. You can find it here: arxiv.org/pdf/1808.07371.pdf. A lot of the collective knowledge in machine learning isn’t written down in books, articles and blog posts, but in academic research papers. If you’re serious about machine learning, get into the habit of reading those papers.

To be honest, it can be hard to get into reading research papers. It’s quite likely half the stuff in the Everybody Dance Now paper won’t make sense to you at first reading. Don’t fret! Simply read a few other papers on the same topic. Skip the math and any parts that don’t make sense to you yet. Gradually, you’ll get comfortable with the way these papers are written. And once you speak the language of a certain subfield such as human pose detection or generative models, new papers become easier to read.

A good place to find papers is on arxiv.org. Even better is the Arxiv Sanity Preserver at arxiv-sanity.com, which also has a “top hype” section where you can find what the latest buzz is about. To stay up-to-date on what’s happening in machine learning, the papers are where it’s at.

Note: We heartily recommend watching the fast.ai videos once you’ve finished this book. This is one of the best online courses about computer vision, natural language processing, and other applications of deep learning — and it’s free! Not only will you gain a deeper understanding of machine learning, but this course is also packed with handy tips and tricks, and advice on how to get state-of-the-art results. 5 out of 5 stars!