5. Digging Deeper into Turi Create

Written by Audrey Tam & Matthijs Hollemans

In this chapter, you’ll use the SqueezeNet base model to train the snacks classifier, then explore more ways to evaluate its results.

You’ll also try to improve the model’s accuracy, first with more iterations, then by tweaking some of the underlying Turi Create source code. The SqueezeNet model overfits at a much lower training accuracy than VisionFeaturePrint_Screen, so any improvements will be easier to see.

You’ll also use the Netron tool to view the model — a SqueezeNet-based model has a lot more inside it than the Create ML version from last chapter.

Getting started

You can continue to use the turienv environment, Jupyter notebook, and snacks dataset from the previous chapter, or start fresh with the DiggingDeeper_starter notebook in this chapter’s starter folder.

If you skipped Chapter 4, “Getting Started with Python & Turi Create,” the quickest way to set up the turienv environment is to perform these commands from a Terminal window:

$ cd /path/to/chapter/resources
$ conda env create --file=starter/turienv.yaml
$ conda activate turienv
$ jupyter notebook

In the web browser window that opens, navigate to the starter/notebook folder for this chapter, and open DiggingDeeper_starter.ipynb.

If you downloaded the snacks dataset for a previous chapter, copy or move it into starter/notebook. Otherwise, double-click starter/notebook/snacks-download-link.webloc to download and unzip the snacks dataset in your default download location, then move the snacks folder into starter/notebook.

Note: In this book we’re using Turi Create version 5.6. Other versions may give different results or even errors. This is why we suggest using the turienv that comes with the book.

Transfer learning with SqueezeNet

If you’re not continuing from the previous chapter’s notebook, then run the following cells one by one.

Note: If you are continuing from last chapter’s notebook, but have shut down Jupyter in the mean time, then you’ll also need to re-run these cells. Jupyter does not automatically restore the Python state. You can skip the data exploration cells.

1. Import the required Python modules:

import turicreate as tc
import matplotlib.pyplot as plt

2. Load training and testing data and display lengths to make sure there’s data:

train_data = tc.image_analysis.load_images("snacks/train",
                                           with_path=True)
len(train_data)

test_data = tc.image_analysis.load_images("snacks/test", with_path=True)
len(test_data)

3. Extract labels from the image paths and display label count values:

import os
train_data["label"] = train_data["path"].apply(
              lambda path: os.path.basename(os.path.split(path)[0]))

test_data["label"] = test_data["path"].apply(
              lambda path: os.path.basename(os.path.split(path)[0]))

train_data["label"].value_counts().print_rows(num_rows=20)
test_data["label"].value_counts().print_rows(num_rows=20)

Now that the dataset has been loaded, you can create the image classifier. If you don’t want to wait for your Mac to train this model, load the pre-trained model instead, from the starter/notebook folder:

model = tc.load_model("MultiSnacks.model")

But if you’ve got some cycles to spare, feel free to train the model. This is done the same as before, except now you’ll use the arguments model="squeezenet_v1.1" to use the SqueezeNet feature extractor:

model = tc.image_classifier.create(train_data, target="label",
                                   model="squeezenet_v1.1",
                                   verbose=True, max_iterations=100)

When you run this cell, you’ll be pleasantly surprised at how fast the feature extraction is, compared to last chapter. This is because SqueezeNet extracts only 1000 features from 227×227-pixel images, compared with VisionFeaturePrint_Screen’s 2,048 features from 299×299 images.

However, you’ll probably be disappointed by the training and validation accuracies:

Note: It’s likely you’ll get slightly different training results than what are shown in this book. Recall that untrained models, in this case the logistic regression part of the model, are initialized with random numbers. This can cause variations between different training runs. Just try it again if you get a training accuracy that is much less than 65%. Advanced users of machine learning actually take advantage of these differences between training runs to combine multiple models into one big ensemble that gives more robust predictions.

Like Create ML, Turi Create randomly chooses 5% of the training data as validation data, so validation accuracies can vary quite a bit between training runs. The model might do better on a larger fixed validation dataset that you choose yourself (which you’ll do later in this chapter).

Evaluate the model and display some metrics:

metrics = model.evaluate(test_data)
print("Accuracy: ", metrics["accuracy"])
print("Precision: ", metrics["precision"])
print("Recall: ", metrics["recall"])

No surprises here — accuracy is pretty close to the validation accuracy:

Accuracy:  0.6470588235294118
Precision:  0.6441343963604582
Recall:  0.6445289115646259

Getting individual predictions

So far, you’ve just repeated the steps from the previous chapter. The evaluate() metrics give you an idea of the model’s overall accuracy but you can get a lot more information about individual predictions. Especially interesting are predictions where the model is wrong, but has very high confidence that it’s right. Knowing where the model is wrong can help you improve your training dataset.

To get some more insight into what’s going on, run the following cell:

metrics.explore()

This opens a new window (Mac only) that lets you examine the accuracy and other metrics visually. It also shows examples of images that were correctly — and more interestingly, incorrectly — classified. Very handy!

Predicting and classifying

Turi Create models have other functions, in addition to evaluate(). Enter and run these commands in the next cell, and wait a while:

model.predict(test_data)

It displays the actual prediction for each individual image from the test set:

['apple', 'grape', 'orange', 'orange', 'orange', 'apple', 'orange', 'apple', 'candy', 'apple', 'grape', 'apple', ’strawberry', 'apple', 'apple', 'carrot', 'candy', 'ice cream', 'apple', 'apple', 'apple', ...

The first prediction corresponds to the image from test_data[0], the second to the image from test_data[1], and so on. The first 50 test images are all apples, but the model classified the second image as “grape,” so take a look at the image. Enter and run this command in the next cell:

plt.imshow(test_data[1]["image"].pixel_data)

This displays the second image — does it look like grapes?

Maybe the model isn’t really sure, either. Enter and run these commands, and wait a while:

output = model.classify(test_data)
output

The classify() function gets you the probability for each prediction, but only the highest-probability value, which is the model’s confidence in the class it predicts:

So the model is 69.96% confident that the second image is “grape”! And 93% confident the fourth image is “orange”! But it’s less than 50% confident about the other images it labelled “orange.”

It’s helpful to see the images that correspond to each prediction. Enter and run these commands:

imgs_with_pred = test_data.add_columns(output)
imgs_with_pred.explore()

The first command adds the output columns to the original test_data columns. Then you display the merged SFrame with explore().

The label column is the correct class, and class is the model’s highest-confidence prediction, which you can see on the next page:

The most interesting images are the rows where the two labels disagree, but the probability is very high — over 90%, for example. Enter the following commands:

imgs_filtered = imgs_with_pred[(imgs_with_pred["probability"] > 0.9) &
                 (imgs_with_pred["label"] != imgs_with_pred["class"] )]
imgs_filtered.explore()

This command filters the SFrame to include only those rows with high-probability wrong predictions. The first term selects the rows whose probability column has a value greater than 90%, the second term selects the rows where the label and class columns are not the same.

The subset of matching rows is saved into a new SFrame, then displayed:

The true label of the highlighted image is “strawberry,” but the model is 97% confident it’s “juice,” probably because the glass of milk(?) is much larger than the strawberries.

You can learn a lot about how your model sees the world by looking at these confident-but-wrong predictions: Sometimes the model gets it completely wrong, but sometimes the predictions are actually fairly reasonable — even if it is strictly speaking “wrong,” since what was predicted wasn’t the official label.

But if the image contains more than one object, such as the example with the drink and the strawberries, you could argue that the training label is actually wrong — or at least, misleading.

Sorting the prediction probabilities

Turi Create’s predict() method can also give you the probability distribution for each image. Enter and run these lines, then wait a while:

predictions = model.predict(test_data, output_type="probability_vector")

You add the optional argument output_type to get the probability vector for each image — the predicted probability for each of the 20 classes. Then let’s look at the second image again, but now display all of the probabilities, not just the top one:

print("Probabilities for 2nd image", predictions[1])

This outputs something like the following:

array('d', [0.20337662077520557, 0.010500386379535839, 2.8464920324200633e-07, 0.0034932724790819624, 0.0013391166287066811, 0.0005122369124003818, 5.118841868115829e-06, 0.699598450277612, 2.0208374302686123e-07, 7.164497444549948e-07, 2.584012081941193e-06, 5.5645094234565224e-08, 0.08066298157942492, 0.00021689939485918623, 2.30074608705137e-06, 3.6511378835730773e-10, 5.345215832976188e-05, 9.897270575019545e-06, 2.1477438456101293e-08, 0.00022540187389448156])

The probabilities are sorted alphanumerically by name of the class in the training set, so the first value is for “apple,” the second is “banana,” the third is “cake” … Ack! — you need to add class labels to make this useful! Enter and run the following:

labels = test_data["label"].unique().sort()
preds = tc.SArray(predictions[1])
tc.SFrame({"preds": preds, "labels": labels}).sort([("preds", False)])

First, you get the set of labels from the test_data SFrame, sort them so they match the order in the probability vector, and store the result in labels, which is an SArray — a Turi Create array. Then you create another SArray from the probability vector of the second image. In the last line, you merge the two SArrays into an SFrame, then sort it on the preds column, in descending order (ascending = False).

Here are the top five rows from this output:

So the model does at least give 20% confidence to “apple.” Top-three or top-five accuracy is a fairer metric for a dataset whose images can contain multiple objects.

Using a fixed validation set

Turi Create extracts a random validation dataset from the training dataset — 5% of the images. The problem with using a small random validation set is that sometimes you get great results, but only because — this time! — the validation dataset just happens to be in your favor.

For example, if the model is really good at predicting the class “waffle” and the validation set happens to be mostly images of waffles, the validation accuracy will be higher than the true performance of the model. That’s bad, because it may lead you to overestimate how good the model really is.

If you repeat training the model a few times, you’ll see the validation accuracy vary a lot. Sometimes it’s better than the 67% you saw before, sometimes it’s way worse — on one of the author’s training runs, it went as low as 59%. It’s hard to understand how well the model is doing when there’s so much variation between different runs.

To get more reliable estimates of the accuracy, use your own validation set instead of letting Turi Create randomly select one. By using a collection of validation images that is always the same, you can control your experiments better and get reproducible results. You can now train the model with a few different configuration settings, also known as the hyperparameters, and compare the results to determine which settings work best. If you were to use a different validation set each time, then the variation in the chosen images could obscure the effect of the changed hyperparameter.

The snacks dataset already comes with a val folder containing images for this purpose. Load these images into their own SFrame, using the same code as before:

val_data = tc.image_analysis.load_images("snacks/val", with_path=True)
val_data["label"] = val_data["path"].apply(lambda path:
      os.path.basename(os.path.split(path)[0]))
len(val_data)

The last statement should output 955, which is almost the same number of images as in test_data, and a lot more than 5% of the 4838 train_data images.

To train the model on your own validation set, write the following:

model = tc.image_classifier.create(train_data, target="label",
                                   model="squeezenet_v1.1",
                                   verbose=True, max_iterations=100,
                                   validation_set=val_data)

You should now always get the same validation accuracy — about 63% — no matter how often you repeat the training. The large fluctuations are gone.

Because the model is initialized with random numbers at the start of training, there are still small differences between each training run. To get exactly the same results each time, pass the seed argument to tc.image_classifier.create() to fix the seed for the random number generator, for example seed=1234.

Note: With this fixed validation set, training has become a little slower. That’s because computing the validation accuracy takes up a significant amount of time. Previously, Turi only used 5% of the training set for this, or about 240 images. Now it uses 955 images, so it takes about 4 times as long to compute the validation score. But getting more trustworthy estimates is worth the extra wait.

Increasing max iterations

So, is a validation accuracy of 63% good? Meh, not really. Turi Create knows it, too — at the end of the training output it says:

This model may not be optimal. To improve it, consider increasing `max_iterations`.

Turi Create has recognized that this model still has some issues. (It’s possible you won’t get this message, this seems to vary with Turi Create versions.)

Let’s train again, this time with more iterations — 200 instead of 100:

model = tc.image_classifier.create(train_data, target="label",
                                   model="squeezenet_v1.1",
                                   verbose=True, max_iterations=200,
                                   validation_set=val_data)

Note: Like Create ML, Turi Create has to extract the features again. It does not keep those feature vectors around — if it had, training the model again would be a lot quicker. If 100 iterations already took a very long time on your Mac, feel free to load the pre-trained model from starter/notebook:

model = tc.load_model("MultiSnacks_200.model")

The number of iterations is an example of a hyperparameter. This is simply a fancy name for the configuration settings for your model. Why “hyper”? The things that the model learns from the training data are called the “parameters” or learned parameters. The things you configure by hand, which don’t get changed by training, are therefore the “hyperparameters.” The hyperparameters tell the model how to learn, while the training data tells the model what to learn, and the parameters describe that which has actually been learned.

The max_iterations setting determines how long the model will be trained for. Like all hyperparameters, it’s important to set it to a good value or else the resulting model may not be as good as you’d hoped. If the training time is too short, the model won’t have had the opportunity to learn all it could; if the training time is too long, the model will overfit.

After 200 iterations of training, the final score is:

The training accuracy is now 90%! This means on the training set of 4582 examples it only gets 10% wrong, as opposed to 20% before (when the training accuracy was about 80%).

That seems pretty good, but remember that you shouldn’t put too much faith in the training accuracy by itself. More important is the validation accuracy. As you can see, this briefly went up and then down again.

The sweet spot for this model seems to be somewhere around 150 iterations where it gets a validation accuracy of 63.7%. If you train for longer, the validation accuracy starts to drop and the model becomes worse, even though the training accuracy will slowly keep improving. A classic sign of overfitting.

Overfitting has a bad rap, and it’s certainly an issue you’ll run into when you start training your own models. But overfitting isn’t necessarily a bad thing to experience, as it means that your model still has capacity to learn more. It’s just learning the wrong things, and techniques such as regularization will help your model to stay on the right path. (More about regularization later in this chapter.)

Unfortunately, Turi Create does not let you save the iteration of the model with the best validation accuracy, only the very last iteration, and so you’ll have to train again with max_iterations=150, to get the best possible result.

Run the usual code to evaluate the model on the test set and display the metrics:

metrics = model.evaluate(test_data)
print("Accuracy: ", metrics["accuracy"])
print("Precision: ", metrics["precision"])
print("Recall: ", metrics["recall"])

Evaluating this model on the test dataset produces metrics around 65%, which is slightly higher than before:

Accuracy:  0.6554621848739496
Precision:  0.6535792163681828
Recall:  0.6510697278911566

Increasing the number of iterations did help a little, so apparently the initial guess of 100 iterations was too low. Tip: For the best results, first train the model with too many iterations, and see at which iteration the validation accuracy starts to become worse. That’s your sweet spot. Now train again for exactly that many iterations and save the model.

Confusing apples with oranges?

A picture says more than a thousand numbers, and a really useful visualization of how well the model does is the confusion matrix. This matrix plots the predicted classes versus the images’ real class labels, so you can see where the model tends to make its mistakes.

In the previous chapter, you ran this command:

print("Confusion Matrix:\n", metrics["confusion_matrix"])

This displayed a table:

+--------------+-----------------+-------+
| target_label | predicted_label | count |
+--------------+-----------------+-------+
|    cookie    |      juice      |   1   |
|    carrot    |    watermelon   |   1   |
|   pretzel    |     pretzel     |   14  |
|     cake     |    ice cream    |   2   |
|  pineapple   |      carrot     |   1   |
|   doughnut   |      muffin     |   1   |
|    muffin    |     doughnut    |   7   |

The target_label column shows the real class, while predicted_label has the class that was predicted, and count is how many of this particular mistake were made.

The table shows the model predicted “muffin” 7 times when the image was really “doughnut,” predicted “cake” twice when the image was really “ice cream,” and so on.

However, presented this way, the confusion matrix doesn’t look much like a matrix, and we promised to show you how to get a better visualization.

Start by entering and running the following code:

import numpy as np
import seaborn as sns

def compute_confusion_matrix(metrics, labels):
    num_labels = len(labels)
    label_to_index = {l:i for i,l in enumerate(labels)}

    conf = np.zeros((num_labels, num_labels), dtype=np.int)
    for row in metrics["confusion_matrix"]:
        true_label = label_to_index[row["target_label"]]
        pred_label = label_to_index[row["predicted_label"]]
        conf[true_label, pred_label] = row["count"]

    return conf

def plot_confusion_matrix(conf, labels, figsize=(8, 8)):
    fig = plt.figure(figsize=figsize)
    heatmap = sns.heatmap(conf, annot=True, fmt="d")
    heatmap.xaxis.set_ticklabels(labels, rotation=45,
                                 ha="right", fontsize=12)
    heatmap.yaxis.set_ticklabels(labels, rotation=0,
                                 ha="right", fontsize=12)
    plt.xlabel("Predicted label", fontsize=12)
    plt.ylabel("True label", fontsize=12)
    plt.show()

You define two new functions: one to compute the confusion matrix and one to draw it.

compute_confusion_matrix() looks at all the rows in the metrics["confusion_matrix"] table, and fills up a 2D-array with the counts of each pair of labels. It uses the NumPy package for this.

Then, plot_confusion_matrix() takes this NumPy array, and plots it as a heatmap using Seaborn, a plotting package that adds useful plot types to Matplotlib. You installed Seaborn when you created the turienv environment in the previous chapter.

Now, enter and run the following commands to call these functions:

conf = compute_confusion_matrix(metrics, labels)
plot_confusion_matrix(conf, labels, figsize=(16, 16))

And enjoy the display!

A heatmap shows small values as “cool” colors — black and dark purple — and large values as “hot” colors — red to pink to white. The larger the value, the brighter it gets. In the confusion matrix, you expect to see a lot of high values on the diagonal, since these are the correct matches.

For example, the row for the “pretzel” class shows 14 correct matches and 11 wrong ones. The wrong predictions are one “apple,” two “cookie,” one “doughnut,” and seven “hotdog.” Notice that apples often get mistaken for oranges, and cookie, doughnut, and muffin also get mixed up often.

The confusion matrix is very useful because it shows potential problem areas for the model. From this particular confusion matrix, it’s clear the model has learned a great deal already, since the diagonal really stands out, but it’s still far from perfect. Ideally, you want everything to be zero except the diagonal. It may be a little misleading from the picture since at first glance it appears that there aren’t that many mistakes. But all the small numbers in the dark squares add up to 340 misclassified images out of 952 total, or 36% wrong.

Keep in mind that some categories have more images than others. For example, pretzel has only 25 images in the test set, while most of the other classes have 50, so it will never have as many correct matches. Still, it only scores 14 out of 25 correct (56%), so overall the model actually does poorly on pretzels.

Computing recall for each class

Turi Create’s evaluate() function gives you the overall test dataset accuracy but, as mentioned in the AI Ethics section of the first chapter, accuracy might be much lower or higher for specific subsets of the dataset. With a bit of code, you can get the accuracies for the individual classes from the confusion matrix:

for i, label in enumerate(labels):
    correct = conf[i, i]
    images_per_class = conf[i].sum()
    print("%10s %.1f%%" % (label, 100. * correct/images_per_class))

For each row of the confidence matrix, the number on the diagonal is how many images in this class that the model predicted correctly. You’re dividing this number by the sum over that row, which is the total number of test images in that class.

This gives you the percentage of each class that the model classified correctly — for example, how many “apple” images did the model find among the total number of “apple” images?

This the recall metric for each class:

     apple 64.0%
    banana 68.0%
      cake 54.0%
     candy 58.0%
    carrot 66.0%
    cookie 56.0%
  doughnut 62.0%
     grape 84.0%
   hot dog 76.0%
 ice cream 44.0%
     juice 74.0%
    muffin 50.0%
    orange 74.0%
 pineapple 67.5%
   popcorn 62.5%
   pretzel 56.0%
     salad 72.0%
strawberry 67.3%
    waffle 62.0%
watermelon 64.0%

The best classes are grape (84% correct) and hot dog (76%). At 74%, juice and orange are also good. The worst performing classes are ice cream (44%), muffin (50%), cake (54%), and pretzel (56%). These would be the classes to pay attention to, in order to improve the model — for example, by gathering more or better training images for these classes.

Note: As always, the numbers you’ll get for your own version of this model might be slightly different. This is due to the choice of hyperparameters, such as the number of iterations. But also because untrained models are initialized with random numbers, and therefore two trained models are never exactly the same (unless you set the random seed to a fixed number).

Training the classifier with regularization

A typical hyperparameter that machine learning practitioners like to play with is the amount of regularization that’s being used by the model. Regularization helps to prevent overfitting. Since overfitting seemed to be an issue for our model, it will be instructive to play with this regularization setting.

Enter and run this statement:

model = tc.image_classifier.create(train_data, target="label",
                                   model="squeezenet_v1.1",
                                   verbose=True, max_iterations=200,
                                   validation_set=val_data,
                                   l2_penalty=10.0, l1_penalty=0.0,
                                   convergence_threshold=1e-8)

You’ve added three additional arguments: l2_penalty, l1_penalty and convergence_threshold. Setting the convergence_threshold to a very small value means that the training won’t stop until it has done all 200 iterations.

l2_penalty and l1_penalty are hyperparameters that add regularization to reduce overfitting.

What’s regularization? Recall that a model learns parameters — also called weights or coefficients — for combining feature values, to maximize how many training data items it classifies correctly. Overfitting can happen when the model gives too much weight to some features, by giving them very large coefficients. Setting l2_penalty greater than 0 penalizes large coefficients, encouraging the model to learn smaller coefficients. Higher values of l2_penalty reduce the size of coefficients, but can also reduce the training accuracy.

Setting l1_penalty greater than 0 also penalizes large coefficients. In addition, it discards features that have very small coefficients, by setting these to 0. Typically, you’d use either l2_penalty or l1_penalty, but not both in the same training session.

In the author’s training session, the model has stopped overfitting:

The training accuracy doesn’t race off to 100% anymore but tops out at about 79%. More importantly, the validation accuracy doesn’t become worse with more iterations. Note that it is typical for the training accuracy to be higher than the validation accuracy. This is OK — it’s only bad if the validation accuracy starts going down.

Question: Is l2_penalty=10.0 the best possible setting? To find out, you can train the classifier several times, trying out different values for l2_penalty and l1_penalty. This is called hyperparameter tuning.

Selecting the correct hyperparameters for your training procedure can make a big difference in the quality of the model you end up with. The validation accuracy gives you an indication of the effect of these hyperparameters. This is why you’re using a fixed validation set, so that you can sure any change in the results is caused by the change in the hyperparameters, not by chance.

Hyperparameter tuning is more trial and error than science, so play with these settings to get a feeling for how they affect your model. Try setting l2_penalty to 100: you’ll note that the training accuracy won’t go over 65% or so, as now you’re punishing the model too hard.

Unfortunately, every time you train the model, Turi Create has to extract the features from all the training and validation images again, over and over and over. That makes hyperparameter tuning a very slow affair. Let’s fix that!

Wrangling Turi Create code

One of the appealing benefits of Turi Create is that, once you have your data in an SFrame, it takes only a single line of code to train the model. The downside is that the Turi Create API gives you only limited control over the training process. Fortunately, Turi Create is open source, so you can look inside to see what it does, and even hack around some of its limitations.

The code for tc.image_classifier.create() is in the file turicreate/src/python/turicreate/toolkits/image_classifier/image_classifier.py in the GitHub repo at github.com/apple/turicreate. You’re simply going to copy-paste some of that code into the notebook, and play with the hyperparameters.

Saving the extracted features

Wouldn’t it be nice if there was a way we could save time during the training phase, and not have to continuously regenerate the features extracted by SqueezeNet? Well, as promised, in this section, you’ll learn how to save the intermediate SFrame to disk, and reload it, just before experimenting with the classifier.

Note: If you don’t want to wait for the feature extraction, just load the features from the starter/notebook folder:

extracted_train_features = tc.SFrame("extracted_train_features.sframe")

extracted_val_features = tc.SFrame("extracted_val_features.sframe")

First, load the pre-trained SqueezeNet model and grab its feature extractor:

from turicreate.toolkits import _pre_trained_models
from turicreate.toolkits import _image_feature_extractor

ptModel = _pre_trained_models.MODELS["squeezenet_v1.1"]()
feature_extractor = _image_feature_extractor.MXFeatureExtractor(ptModel)

MXFeatureExtractor is an object from the MXNet machine learning framework that Turi Create is built on. In Python, names starting with an underscore are considered to be private, but you can still import them. Next, enter and run this code statement:

train_features = feature_extractor.extract_features(train_data,
                                          "image", verbose=True)

You’re using the MXFeatureExtractor object to extract the SqueezeNet features from the training dataset. This is the operation that took the most time when you ran tc.image_classifier.create(). By running this separately now, you won’t have wait for feature extraction every time you want to train the classifier. Next, enter and run this code statement:

extracted_train_features = tc.SFrame({
    "label": train_data["label"],
    "__image_features__": train_features,
    })

Here, you’re just combining the features of each image with its respective label into a new SFrame. This is worth saving for later use! Enter and run this code statement:

extracted_train_features.save("extracted_train_features.sframe")

You’re saving extracted_train_features to a file. The next time you want to do more training with these same features, you can simply load the SFrame again, which takes a fraction of the time it took to extract the features:

# Run this tomorrow or next week
extracted_train_features = tc.SFrame("extracted_train_features.sframe")

Inspecting the extracted features

Let’s see what these features actually look like — enter and run this command:

extracted_train_features.head()

Each row has the extracted features for one training image.

The __image_features__ column contains a list with numbers, while the label column has the corresponding class name for this row. Enter and run this command:

extracted_train_features[0]["__image_features__"]

This shows you what a feature vector looks like — it prints something like:

array('d', [6.1337385177612305, 10.12844181060791, 13.025101661682129, 7.931194305419922, 12.03809928894043, 15.103202819824219, 12.722893714904785, 10.930903434753418, 12.778315544128418, 14.208030700683594, 16.8399658203125, 11.781684875488281, ...

This is a list of 1,000 numbers — use the len() function to verify this. They all appear to be numbers between 0 and about 30. What do they represent? I have no idea, but they are features that SqueezeNet has determined to be important — how long, round, square, orange, etc. the objects are. All that matters is that you can train a logistic classifier to learn from these features.

In the same way, extract the features for the images from the validation dataset, and save this SFrame to a file too:

val_features = feature_extractor.extract_features(val_data,
                                      "image", verbose=True)

extracted_val_features = tc.SFrame({
    "label": val_data["label"],
    '__image_features__': val_features,
    })

extracted_val_features.save("extracted_val_features.sframe")   

Training the classifier

Now you’re ready to train the classifier! Enter and run this statement:

lr_model = tc.logistic_classifier.create(extracted_train_features,
                             features=["__image_features__"],
                             target="label",
                             validation_set=extracted_val_features,
                             max_iterations=200,
                             seed=None,
                             verbose=True,
                             l2_penalty=10.0,
                             l1_penalty=0.0,
                             convergence_threshold=1e-8)

This is the Turi Create code that creates and trains the logistic regression model using the extracted_train_features SFrame as the input data, and extracted_val_features for validation.

Want to try some other values for these hyperparameters? Simply change them in the above cell and run it again. It’s a lot quicker now because the feature extraction step is skipped.

There are a few other hyperparameters you can set here as well: feature_rescaling, solver, step_size and lbfgs_memory_level. To learn what these do, type the following in a new cell or check out the comments in the Turi Create source code.

tc.logistic_classifier.create?

It turns out that, with regularization, training for 400 or so iterations keeps slowly improving the validation score of this model to 65%. It’s only improved by a small amount, but every little bit helps. The only way we found this was by experimenting with the hyperparameters. Perhaps you can find hyperparameters for this model that do even better?

To make sure you’re not chasing phantoms and that this validation score really is representative of the model’s true performance, you should also check the accuracy on the test set.

First, turn your model into a valid ImageClassifier object:

from turicreate.toolkits.image_classifier import ImageClassifier

state = {
    'classifier': lr_model,
    'model': ptModel.name,
    'max_iterations': lr_model.max_iterations,
    'feature_extractor': feature_extractor,
    'input_image_shape': ptModel.input_image_shape,
    'target': lr_model.target,
    'feature': "image",
    'num_features': 1,
    'num_classes': lr_model.num_classes,
    'classes': lr_model.classes,
    'num_examples': lr_model.num_examples,
    'training_time': lr_model.training_time,
    'training_loss': lr_model.training_loss,
}
model = ImageClassifier(state)

This combines the base model with the classifier you trained into the state structure, and creates an ImageClassifier object from this.

Calculate the test set metrics as before:

metrics = model.evaluate(test_data)
print("Accuracy: ", metrics["accuracy"])
print("Precision: ", metrics["precision"])
print("Recall: ", metrics["recall"])

This prints out:

Accuracy:  0.6712184873949579
Precision:  0.6755916486674352
Recall:  0.6698818027210884

Regularization and hyperparameter tuning won’t always work miracles, but they do improve the model, even if only a little bit.

Note: model.evaluate() performs feature extraction on the test set each time you run it. Exercise for the adventurous reader: Try digging into the Turi Create source code to see if you can make a version of evaluate() that accepts an SFrame with already extracted features. Hint: type model.evaluate?? into a new cell (yes, two question marks) to view the source code of this method.

Saving the model

You can save the model as a Turi Create model:

model.save("MultiSnacks_regularized.model")

Or export a Core ML model:

model.export_coreml("MultiSnacks_regularized.mlmodel")

To learn more about the model, run the following:

model

This shows you some high-level information about the model and its training:

Class                                    : ImageClassifier

Schema
------
Number of classes                        : 20
Number of feature columns                : 1
Input image shape                        : (3, 227, 227)

Training summary
----------------
Number of examples                       : 4838
Training loss                            : 3952.4993
Training time (sec)                      : 59.2703

Training loss — the overall error over the training dataset — changes when you change the hyperparameters. Enter and run this to see a bit more information:

model.classifier

This shows you information about the classifier portion of the model:

Class                          : LogisticClassifier

Schema
------
Number of coefficients         : 19019
Number of examples             : 4838
Number of classes              : 20
Number of feature columns      : 1
Number of unpacked features    : 1000

Hyperparameters
---------------
L1 penalty                     : 0.0
L2 penalty                     : 10.0

Training Summary
----------------
Solver                         : lbfgs
Solver iterations              : 200
Solver status                  : Completed (Iteration limit reached).
Training time (sec)            : 59.2703

Settings
--------
Log-likelihood                 : 3952.4993

Highest Positive Coefficients
-----------------------------
(intercept)                    : 1.8933
(intercept)                    : 1.4506
(intercept)                    : 0.6717
(intercept)                    : 0.5232
(intercept)                    : 0.4072

Lowest Negative Coefficients
----------------------------
(intercept)                    : -1.6521
(intercept)                    : -1.5588
(intercept)                    : -1.4143
(intercept)                    : -0.8959
(intercept)                    : -0.5863

This information is mostly useful for troubleshooting or when you’re just curious about how the logistic regression classifier works.

Notable is Number of coefficients — 19,019 — the number of parameters this model learned in order to classify images of snacks into the 20 possible categories. Here’s where that number comes from: each input feature vector has 1,000 numbers, and there are 20 possible outputs, so that is 1,000 × 20 = 20,000 numbers, plus 20 “bias” values for each output, making 20,020 coefficients.

However, if there are 20 possible classes, then you actually only need to learn about 19 of those classes, giving 19,019 coefficients. If the prediction is none of these 19 classes, then it must be the 20th class. Interestingly, in the Core ML .mlmodel file, the logistic regression layer does have 20,020 parameters. You can see this for yourself with Netron in the next section.

Under the Settings heading, Log-likelihood is the more mathematical term for Training loss. Below this are the highest and lowest coefficients — remember, the purpose of the regularization hyperparameter is to reduce the size of the coefficients.

To compare with the coefficients of the original no-regularization model, enter and run these lines:

no_reg_model = tc.load_model("MultiSnacks.model")
no_reg_model.classifier

This reloads the pre-trained model (see the starter/notebook folder), and you inspect its classifier. This model had higher training accuracy, so Log-likelihood aka Training loss is lower: 2,400. As you’d expect, its highest and lowest coefficients are larger — in absolute value — than the model with regularization:

Settings
--------
Log-likelihood                 : 2400.3284

Highest Positive Coefficients
-----------------------------
(intercept)                    : 0.3808
(intercept)                    : 0.3799
(intercept)                    : 0.1918
__image_features__[839]        : 0.1864
(intercept)                    : 0.15

Lowest Negative Coefficients
----------------------------
(intercept)                    : -0.3996
(intercept)                    : -0.3856
(intercept)                    : -0.3353
(intercept)                    : -0.2783
__image_features__[820]        : -0.1423

In the next chapter we’ll talk more about what all of this means, as you’ll be writing code to train your own logistic regression from scratch, as well as a complete neural network that will outperform Turi Create’s SqueezeNet-based model.

A peek behind the curtain

SqueezeNet and VisionFeaturePrint_Screen are convolutional neural networks. In the coming chapters, you’ll learn more about how these networks work internally, and you’ll see how to build one from scratch. In the meantime, it might be fun to take a peek inside your Core ML model.

There is a cool free tool called Netron (github.com/lutzroeder/Netron) that creates a nice visualization of the model architecture. On the GitHub page, scroll down to the Install instructions, and click the macOS Download link. On the next page, click the Netron-x.x.x.dmg link, then run this file to install Netron.

Open your .mlmodel file in Netron. This shows all the transformation stages that go into you model’s pipeline.

The input image is at the top, followed by convolutions, activations, pooling, and so on. These are the names of the different types of transformations — or layers — used by this kind of neural network.

Notice how this pipeline sometimes branches and then comes back together again — that’s the “squeeze” feature that gives SqueezeNet its name.

Click on one of these building blocks to learn more about its configuration, its inputs and its output.

At the very end of the pipeline is an innerProduct layer followed by something called a softmax — these two blocks make up the logistic classifier. Everything up until the flatten block is the SqueezeNet feature extractor.

In Chapters 6 and 7, you’ll learn all about what these different kinds of layers do, but for now we suggest that you spend a few minutes playing with Netron to get a rough idea of what these models look like on the inside.

Netron works with any Core ML model, as well as models from many other machine learning frameworks. If you downloaded a model from Apple’s website in the last chapter, also take a look at that.

It should look quite similar to this one, as all neural networks are very alike at their core. Often what is different is the number of layers and the branching structure.

Note: Apple’s own models such as VisionFeaturePrint_Screen are included in iOS 12 and do not get bundled into the .mlmodel file. The .mlmodel file itself doesn’t contain any of the VisionFeaturePrint_Screen layers. For customized models based on these built-in feature extractors, Netron can’t show you anything more than what you see in Xcode’s description: inputs, outputs, metadata. The internal architecture of these models remains a mystery and a secret.

Challenges

Challenge 1: Binary classifier

Remember the healthy/unhealthy snacks model? Try to train that binary classifier using Turi Create. The approach is actually very similar to what you did in this chapter. The only difference is that you need to assign the label “healthy” or “unhealthy” to each row in the training data SFrame.

healthy = [
    'apple', 'banana', 'carrot', 'grape', 'juice', 'orange',
    'pineapple', 'salad', 'strawberry', 'watermelon'
]

unhealthy = [
    'cake', 'candy', 'cookie', 'doughnut', 'hot dog',
    'ice cream', 'muffin', 'popcorn', 'pretzel', 'waffle'
]

train_data["label"] =
  train_data["path"].apply(lambda path: "healthy"
      if any("/" + class_name in path for class_name in healthy)
                                      else "unhealthy")
test_data["label"] =
  test_data["path"].apply(lambda path: "healthy"
      if any("/" + class_name in path for class_name in healthy)
                                      else "unhealthy")

First, you assign each class into a healthy or unhealthy array — there are 10 classes in each array. Then, you set each image’s label column to "healthy" or "unhealthy", depending on which array the image’s path name is in. The result is, you’ve divided 20 classes of images into two classes, based on the name of the subdirectory they’re in.

Note: The process to do this same exercise in Create ML is much more manual. You’d have to create a new train folder with subfolders healthy and unhealthy, then copy or move all the images from each of the 20 food-labelled folders into the correct healthy or unhealthy folder. You’d do this either in Finder or Terminal.

Verify that the resulting model gets about 80% accuracy on the test dataset.

You may wonder why you can’t use the multi-class snacks model for this, and simply look if the predicted category is in the list of healthy or unhealthy classes. This is possible but, by training from scratch on just these two categories, the model has a chance to learn what healthy/unhealthy means, and it might use a more intricate rule than just “this class label is in the list of healthy categories.”

If you want to be sure which approach works better, use the 20-class model to classify() the healthy/unhealthy test dataset, and merge its output with test_data as before. The label column contains “healthy” or “unhealthy,” while the class column contains “apple,” “banana,” etc.

Then use filter_by(healthy, "class") to find images the model predicts to be in a class listed in the healthy array. Filter these images with filter_by(["unhealthy"], "label") to find images that are really in unhealthy classes. Manually calculate the accuracy of the 20-class model in predicting healthy/unhealthy. I got 47%.

Challenge 2: ResNet50-based model

Train the 20-class classifier using the ResNet-50 model and see if that gets a better validation and test set score. Use model_type="resnet-50" when creating the classifier object. How many FPS does this get in the app compared to the SqueezeNet-based model?

Challenge 3: Use another dataset

Create your own training, validation, and test datasets from Google Open Images or some other image source. I suggest keeping the number of categories limited.

Key points

• In this chapter, you’ve gotten a taste of training your own Core ML model with Turi Create. In fact, this is exactly how the models were trained that you used in chapter 2, “Getting Started with Image Classification”.

• Turi Create is pretty easy to use, especially from a Jupyter notebook. It only requires a little bit of Python code. However, we weren’t able to create a super accurate model. This is partly due to the limited dataset.

• More images is better. We use 4,800 images, but 48,000 would have been better, and 4.8 million would have been even better. However, there is a real cost associated with finding and annotating training images, and for most projects, a few hundred images or at most a few thousand images per class may be all you can afford. Use what you’ve got — you can always retrain the model at a later date once you’ve collected more training data. Data is king in machine learning, and who has the most of it usually ends up with a better model.

• Another reason why Turi Create’s model wasn’t super is that SqueezeNet is a small feature extractor, which makes it fast and memory-friendly, but this also comes with a cost: It’s not as accurate as bigger models. But it’s not just SqueezeNet’s fault — instead of training a basic logistic regression on top of SqueezeNet’s extracted features, it’s possible to create more powerful classifiers too.

• Turi Create lets you tweak a few hyperparameters. With regularization, we can get a grip on the overfitting. However, Turi Create does not allow us to fine-tune the feature extractor or use data augmentation. Those are more advanced features, and they result in slower training times, but also in better models.

In the next chapter, we’ll look at fixing all of these issues when we train our image classifier again, but this time using Keras. You’ll also learn more about what all the building blocks are in these neural networks, and why we use them in the first place.