8. Advanced Convolutional Neural Networks

Written by Matthijs Hollemans

SqueezeNet

What you did in the previous chapter is very similar to what Create ML and Turi Create do when they train models, except the convnet they use is a little more advanced. Turi Create actually gives you a choice between different convnets:

• SqueezeNet v1.1
• ResNet50
• VisionFeaturePrint_Scene

In this section, you’ll take a quick look at the architecture of SqueezeNet and how it is different from the simple convnet you made. ResNet50 is a model that is used a lot in deep learning, but, at over 25 million parameters, it’s on the big side for use on mobile devices and so we’ll pay it no further attention.

We’d love to show you the architecture for VisionFeaturePrint_Scene, but, alas, this model is built into iOS itself and so we don’t know what it actually looks like.

This is SqueezeNet, zoomed out:

SqueezeNet uses the now-familiar Conv2D and MaxPooling2D layers, as well as the ReLU activation. However, it also has a branching structure that looks like this:

This combination of several different layers is called a fire module, because no one reads your research papers unless you come up with a cool name for your inventions. SqueezeNet is simply a whole bunch of these fire modules stacked together.

In SqueezeNet, most of the convolution layers do not use 3×3 windows but windows consisting of a single pixel, also called 1×1 convolution. Such convolution filters only look at a single pixel at a time and not at any of that pixel’s neighbors. The math is just a regular dot product across the channels for that pixel.

Convolutions with a 1×1 kernel size are very common in modern convnets. They’re often used to increase or to decrease the number of channels in a tensor. That’s exactly why SqueezeNet uses them, too.

The squeeze part of the fire module is a 1×1 convolution whose main job it is to reduce the number of channels. For example, the very first layer in SqueezeNet is a regular 3×3 convolution with 64 filters. The squeeze layer that follows it, reduces this back to 16 filters. What such a layer learns isn’t necessarily to detect patterns in the data, but how to keep only the most important patterns. This forces the model to focus on learning only things that truly matter.

The output from the squeeze convolution branches into two parallel convolutions, one with a 1×1 window size and the other with a 3×3 window. Both convolutions have 64 filters, which is why this is called the expand portion of the fire module, as these layers increase the number of channels again. Afterwards, the output tensors from these two parallel convolution layers are concatenated into one big tensor that has 128 channels.

The squeeze layer from the next fire module then reduces those 128 channels again to 16 channels, and so on. As is usual for convnets, the number of channels gradually increases the further you go into the network, but this pattern of reduce-and-expand repeats several times over.

The reason for using two parallel convolutions on the same data is that using a mix of different transformations potentially lets you extract more interesting information. You see similar ideas in the Inception modules from Google’s famous Inception-v3 model, which combines 1×1, 3×3, and 5×5 convolutions, and even pooling, into the same kind of parallel structure.

The fire module is very effective, evidenced by the fact that SqueezeNet is a powerful model — especially for one that only has 1.2 million learnable parameters. It scores about 67% correct on the snacks dataset, compared to 40% from the basic convnet of the previous section, which has about the same number of parameters.

If you’re curious, you can see a Keras version of SqueezeNet in the notebook SqueezeNet.ipynb in this chapter’s resources. This notebook reproduces the results from Turi Create with Keras. We’re not going to explain that code in detail here since you’ll shortly be using an architecture that gives better results than SqueezeNet. However, feel free to play with this notebook — it’s fast enough to run on your Mac, no GPU needed for this one.

The Keras functional API

One thing we should mention at this point is the Keras functional API. You’ve seen how to make a model using Sequential, but that is limited to linear pipelines that consist of layers in a row. To code SqueezeNet’s branching structures with Keras, you need to specify your model in a slightly different way.

In the file keras_squeezenet/squeezenet.py, there is a function def SqueezeNet(...) that defines the Keras model. It more-or-less does the following:

img_input = Input(shape=input_shape)

x = Conv2D(64, 3, padding='valid')(img_input)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(x)
x = fire_module(x, squeeze=16, expand=64)
x = fire_module(x, squeeze=16, expand=64)
x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(x)
...

model = Model(img_input, x)
...
return model

Instead of creating a Sequential object and then doing model.add(layer), here a layer is created by writing:

x = LayerName(parameters)

Then this layer object is immediately applied to the output from the previous layer:

x = LayerName(parameters)(x)

Here, x is not a layer object but a tensor object. This syntax may look a little weird, but in Python, you’re allowed to call an object instance (the layer) as if it were a function. This is actually a very handy way to define models of arbitrary complexity.

To create the actual model object, you need to specify the input tensor as well as the output tensor, which is now in x:

model = Model(img_input, x)

You can see how the branching structure is made in the fire_module function, shown here in an abbreviated version:

def fire_module(x, squeeze=16, expand=64):
    sq = Conv2D(squeeze, 1, padding='valid')(x)
    sq = Activation('relu')(sq)

    left = Conv2D(expand, 1, padding='valid')(sq)
    left = Activation('relu')(left)

    right = Conv2D(expand, 3, padding='same')(sq)
    right = Activation('relu')(right)

    return concatenate([left, right])

This has four tensors: x that has the input data, sq with the output of the squeeze layer, left for the left branch and right for the right branch. At the end, left and right are concatenated into a single tensor again. This is where the branches come back together.

A lot of Keras code will use both Sequential models and models defined using this functional API, so it’s good to be familiar with it.

Note: The SqueezeNet implementation we used here was taken from the GitHub repo github.com/rcmalli/keras-squeezenet.

MobileNet and data augmentation

The final classification model you’ll be training is based on MobileNet. Just like SqueezeNet, this is an architecture that is optimized for use on mobile devices — hence the name.

MobileNet has more learned parameters than SqueezeNet, so it’s slightly bigger but it’s also more capable. With MobileNet as the feature extractor, you should be able to get a model that performs better than what Turi Create gave you in Chapter 5, “Digging Deeper into Turi Create.” Plus you’ll also be using some additional training techniques to make this model learn as much as possible from the dataset.

Follow along with MobileNet.ipynb from the chapter’s resources, or create a new notebook and import the required packages:

import os
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import *
from keras import optimizers, callbacks
import keras.backend as K

%matplotlib inline
import matplotlib.pyplot as plt

Keras already includes a version of MobileNet, so creating this model is easy:

image_width = 224
image_height = 224

from keras.applications.mobilenet import MobileNet

base_model = MobileNet(
  input_shape=(image_height, image_width, 3),
  include_top=False,
  weights="imagenet",
  pooling=None)

Keras’s MobileNet has been trained on the famous ImageNet dataset. But you want to use MobileNet only as a feature extractor, not as a classifier for the 1000 ImageNet categories. That’s why you need to specify include_top=False and pooling=None when creating the model. That way Keras leaves off the classifier layers.

You can use base_model.summary() to see a list of all the layers in this model, or run the following code to save a diagram of the model to a PNG file (this requires the pydot package to be installed):

from keras.utils import plot_model
plot_model(base_model, to_file="mobilenet.png")

If you look at the architecture diagram of MobileNet in that PNG file, you’ll see that it is made up of the following repeating structure:

First, there is a so-called DepthwiseConv2D layer with kernel size 3×3, followed by a BatchNormalization layer, and a ReLU activation. Then there is a Conv2D layer with kernel size 1×1, which is also followed by its own BatchNormalization and ReLU. MobileNet consists of 13 of these building blocks stacked together.

There are a few new things going on, here:

• A depthwise convolution is a variation of convolution wherein each filter only looks at a single input channel. With a regular convolution, the filters always compute their dot products over all the input channels. But a depthwise convolution treats the input channels as separate from one another. Because it doesn’t combine the input channels, depthwise convolution is simpler and faster than Conv2D and uses much fewer parameters.

• The combination of a 3×3 DepthwiseConv2D followed by a 1×1 Conv2D is called a depthwise separable convolution. You can think of this as a 3×3 Conv2D layer that has been split up into two simpler layers: the depthwise convolution filters the data, while the 1×1 convolution — also known as a pointwise convolution — combines the filtered data into a new tensor. This gives an approximation of a “real” 3×3 Conv2D but at much lower cost: there are fewer parameters in total and it also performs fewer computations. This is why MobileNet is so suitable for mobile devices.

• The batch normalization layer, BatchNormalization, is what makes it possible to have these very deep networks. This layer helps to keep the data “fresh” as it moves between the layers. Without batch normalization, the data in the tensors would eventually disappear in deep networks because the numbers become too small — known as the problem of the vanishing gradients — and then the model won’t be able to learn anything anymore. You’ll see BatchNormalization in pretty much any modern convnet.

Note: Depending on your version of Keras, you may also see ZeroPadding2D layers before the DepthwiseConv2D layer, which adds padding around the input tensor so that the convolution works correctly for pixels at the edges. Another small detail: The activation function used is actually ReLU6, a variation of the ReLU activation you’ve seen before. It works in the same way as ReLU but also prevents the output of the convolution from becoming too large — it limits to output to 6.0, hence the name — which allows for the use of faster limited-precision computations on mobile and embedded devices.

Looking at the model.summary(), you may have noticed that MobileNet does not use any pooling layers, yet the spatial dimensions of the image tensor do become smaller over time — from 224×224 at the beginning to only 7×7 at the end.

MobileNet achieves this pooling effect by setting the stride of some of the Conv2D and DepthwiseConv2D layers to 2 instead of 1.

The stride is the size of the steps the convolution kernel takes as it slides through the image. Usually, this step size is 1 and the convolution looks at all the pixels.

With a stride of 2, the window will skip every other pixel, thereby only computing dot products for half the pixels in both the width and height directions. This way you don’t need a special pooling layer to make the image smaller.

You’ll see both techniques, pooling and larger strides, used in practice.

The final layer in this model outputs a tensor of size (7, 7, 1024). This tensor contains the features that MobileNet has extracted from the input image. You’re simply going to add a logistic regression on top of these extracted features, exactly like you’ve done before.

Note: MobileNet has more learned parameters than SqueezeNet, so it takes up more space in your app bundle and also more RAM at runtime. However, thanks to these additional parameters, MobileNet produces higher quality results than SqueezeNet. Even better, it’s also faster than SqueezeNet due to the depthwise separable convolutions.

Choosing a feature extractor is always a trade-off between quality, storage size and runtime speed. If MobileNet is too large for your app — it adds between 8 and 16 MB to your app bundle — then SqueezeNet might be a better choice. But the predictions of a SqueezeNet-based model may be worse and it runs slower.

The VisionFeaturePrint_Scene model that is built into iOS 12 is even more powerful than MobileNet, and it doesn’t even take up any space in your app bundle, but again is slower. And you can’t use it on iOS 11 or other platforms.

Which model is “best” comes down to what you care most about: speed, download size or results. As they say, there is no free lunch in machine learning.

Adding the classifier

You’ve placed the MobileNet feature extractor in a variable named base_model. You’ll now create a second model for the classifier, to go on top of that base model:

num_classes = 20

top_model = Sequential()
top_model.add(base_model)
top_model.add(GlobalAveragePooling2D())
top_model.add(Dense(num_classes))
top_model.add(Activation("softmax"))

This should look familiar by now: it’s a logistic regression.

Just like before it has a Dense layer followed by a softmax activation at the end.

The GlobalAveragePooling2D layer shrinks the 7×7×1024 output tensor from MobileNet to a vector of 1024 elements, by taking the average of each individual 7×7 feature map.

Note: If you had used Flatten instead of global pooling, the Dense layer would have had 49 times more parameters. That simple change would add another one million parameters to the model. Not only does global pooling give you a smaller model than using Flatten, it also works better because too many parameters is the main cause for overfitting.

Note: You used a Dense layer for the logistic regression, but modern convnets often have a 1×1 Conv2D layer at the end instead. If you do the math, you’ll see that a 1×1 convolution that follows a global pooling layer is equivalent to a Dense or fully-connected layer. These are two different ways to express the same operation. However, this is only true after a global pooling layer, when the image is reduced to just a single pixel. Anywhere else, a 1×1 convolution is not the same as a Dense layer.

Next up, you need to freeze all MobileNet layers:

for layer in base_model.layers:
    layer.trainable = False

You’re not going to be training the MobileNet feature extractor. This has already been trained on the large ImageNet dataset, just like SqueezeNet was. All you have to train is the logistic regression that you’ve placed on top. This is why it’s important to set the layers from the feature extractor to be not trainable. Yup, you guessed it, this again is transfer learning in action.

When you do top_model.summary() it should now show this:

_______________________________________________________________
Layer (type)                 Output Shape              Param #
===============================================================
mobilenet_1.00_224 (Model)   (None, 7, 7, 1024)        3228864
_______________________________________________________________
global_average_pooling2d_2 ( (None, 1024)              0       
_______________________________________________________________
dense_1 (Dense)              (None, 20)                20500   
_______________________________________________________________
activation_1 (Activation)    (None, 20)                0       
===============================================================
Total params: 3,249,364
Trainable params: 20,500
Non-trainable params: 3,228,864
_______________________________________________________________

The number of trainable params is only 20,500 since that’s how big the Dense layer is. The other 3.25 million parameters are from MobileNet and will not be trained.

Note that the first “layer” in this new model is MobileNet, so if you ask top_model to make a prediction on an image, it will first send the image through base_model and then applies the final logistic regression layers.

Finally, compile the model just like before:

top_model.compile(loss="categorical_crossentropy",
                  optimizer=optimizers.Adam(lr=1e-3),
                  metrics=["accuracy"])

Before you start training this model, let’s first talk about a handy trick that can make your training set ten times larger with almost no effort on your part.

Data augmentation

We only have about 4800 images for our 20 categories, which comes to 240 images per category on average. That’s not bad, but these deep learning models work better with more data. More, more, more! Gathering more training images takes a lot of time and effort — therefore, is costly — and is not always a realistic option. However, you can always artificially expand the training set by transforming the images that you do have.

Here’s a typical training image:

Notice how it’s pointing to the left? One easy way to instantly double the number of training images is to horizontally flip them so that the model also learns to detect bananas that point to the right. There are many more of these transformations, such as rotating the image, shearing by a random amount, zooming in or out, changing the colors slightly, etc. It’s smart to include any transformations that you want your model to be invariant to.

This is what we call data augmentation: You augment the training data through small random transformations. This happens on-the-fly during training. Every time Keras loads an image from the training set, it automatically applies this data augmentation to the image. For that you have to make an ImageDataGenerator object.

from keras.applications.mobilenet import preprocess_input

train_datagen = ImageDataGenerator(
                    rotation_range=40,
                    width_shift_range=0.2,
                    height_shift_range=0.2,
                    shear_range=0.2,
                    zoom_range=0.2,
                    channel_shift_range=0.2,
                    horizontal_flip=True,
                    fill_mode="nearest",
                    preprocessing_function=preprocess_input)

val_datagen = ImageDataGenerator(
                    preprocessing_function=preprocess_input)

test_datagen = ImageDataGenerator(
                    preprocessing_function=preprocess_input)

You’ve already used ImageDataGenerator in previous notebooks, where it was only responsible for loading the images and normalizing them.

Here, you tell the ImageDataGenerator that it should also rotate the images, flip them horizontally, shift the images up/down/sideways, zoom in/out, shear, and change the color channels by random amounts. That’s a lot of different transformations, and you don’t want to go overboard and make the images unrecognizable, but doing this really helps to grow the amount of available training data.

For normalizing the image data, you previously used your own function, but here you use the preprocess_input function from the Keras MobileNet module because that knows exactly how MobileNet expects the input data.

Note: MobileNet’s preprocess_input() actually does the exact same thing you’ve done in the previous chapters: divide the pixel values by 127.5 and subtract 1, so that the new values are in the range [-1, 1]. However, not all models use this particular method of preprocessing. Another common way to normalize images is to use the mean and standard deviation of all the pixel values in the training set. If you’re using a pretrained model, make sure to use the correct preprocessing for that model, or risk getting incorrect predictions.

For the validation and test sets, you create a plain ImageDataGenerator object that does not apply any of the data augmentations. You always want to evaluate the performance of the model on the exact same set of images.

Given these datagen objects, you can now make the generators that will read the images from their respective folders. This works just like before:

images_dir = "snacks/"
train_data_dir = images_dir + "train/"
val_data_dir = images_dir + "val/"
test_data_dir = images_dir + "test/"
batch_size = 64

train_generator = train_datagen.flow_from_directory(
                    train_data_dir,
                    target_size=(image_width, image_height),
                    batch_size=batch_size,
                    class_mode="categorical",
                    shuffle=True)

val_generator = val_datagen.flow_from_directory(
                    val_data_dir,
                    target_size=(image_width, image_height),
                    batch_size=batch_size,
                    class_mode="categorical",
                    shuffle=False)

test_generator = test_datagen.flow_from_directory(
                    test_data_dir,
                    target_size=(image_width, image_height),
                    batch_size=batch_size,
                    class_mode="categorical",
                    shuffle=False)

And now you’re ready to train!

Training the classifier layer

Training this model is no different than what you’ve done before: you can run model.fit_generator() a few times until you’re happy with the validation accuracy.

But before you rush off to train this fancy new model, allow us to introduce a very handy Keras feature: callbacks. A callback is a Python function that is called at various points in the training process, for example when a new epoch begins or an epoch has just finished.

You’ve seen that if you train for too long, the model will eventually start to overfit and the validation accuracy becomes worse.

It’s hard to say beforehand exactly when this will happen, but it does mean that the last epoch isn’t necessarily the best one. Ideally, you’d stop training just before overfitting starts to happen.

For this, you can add an EarlyStopping callback that will halt the training once the "val_acc" metric, the validation accuracy, stops improving. The patience argument is the number of epochs with no improvement after which the training will be stopped.

It’s also smart to save a model checkpoint every so often. This is a copy of the model’s weights it has learned up to that point.

For this you’d use the ModelCheckpoint callback. It saves a copy of the trained model whenever the metric you’re interested in has improved. Here you’re monitoring "val_acc", so every time the validation accuracy goes up, a new model checkpoint is saved.

checkpoint_dir = "checkpoints/"
checkpoint_name = (checkpoint_dir
    + "multisnacks-{val_loss:.4f}-{val_acc:.4f}.hdf5")

if not os.path.exists(checkpoint_dir):
    os.makedirs(checkpoint_dir)

def create_callbacks():
    return [
        callbacks.EarlyStopping(
            monitor="val_acc",
            patience=10,
            verbose=1),
        callbacks.ModelCheckpoint(
            checkpoint_name,
            monitor="val_acc",
            verbose=1,
            save_best_only=True),
    ]

my_callbacks = create_callbacks()

Note: You need to make sure the checkpoints directory already exists, or Keras will give an error message when it tries to save the checkpoint. That’s why you do os.makedirs() first. By the way, if you ever wanted to save the current state of the model by hand, you can always write model.save("convnet.h5"). HDF5, with the extension .hdf5 or .h5, is the file format used by Keras to save its models. You can view these files with Netron.

Now you can train the model. You need to pass the array with the callback objects to fit_generator()’s callbacks argument.

histories = []
histories.append(top_model.fit_generator(
  train_generator,
  steps_per_epoch=len(train_generator),
  epochs=10,
  callbacks=my_callbacks,
  validation_data=val_generator,
  validation_steps=len(val_generator),
  workers=8))

Training this model should be pretty speedy on a computer with a GPU since you’re only training the one Dense layer for the logistic regression. On the author’s iMac, however, it takes about six minutes per epoch. That’s too slow to be practical, which is why he’s glad to also have a Ubuntu machine with a fast GPU.

Remember that Create ML and Turi Create trained their models using a two-step process:

1. First, they extract the features from all the training images. This can take a while.
2. But once they have those feature vectors, training the logistic regression is fast.

By doing the feature extraction just once, Turi and Create ML could save a lot of time in the training stage. It is also possible to do this with Keras, see the SqueezeNet notebook for details. But because you’re doing a lot of data augmentation, it’s not really worth the trouble.

Having feature extraction as a separate step only makes sense if you plan to reuse the same images in every epoch. But with random data augmentation — where images are rotated, flipped and distorted in many other ways — no two images are ever the same. And so all the feature vectors will be different for every epoch.

That’s why this MobileNet-based model is trained end-to-end and not in two separate stages. In every epoch, Keras needs to compute all the feature vectors again because all the training images are now slightly different from last time. It’s a bit slower, but that’s a small price to pay for having a much larger training set with very little effort.

After training for 10 epochs, the validation accuracy stops going up. Here is the code for plotting the accuracy again (same as in the last chapter):

def combine_histories():
    history = {
        "loss": [],
        "val_loss": [],
        "acc": [],
        "val_acc": []
    }

    for h in histories:
        for k in history.keys():
            history[k] += h.history[k]
    return history

history = combine_histories()

def plot_accuracy(history):
    fig = plt.figure(figsize=(10, 6))
    plt.plot(history["acc"])
    plt.plot(history["val_acc"])
    plt.xlabel("Epoch")
    plt.ylabel("Accuracy")
    plt.legend(["Train", "Validation"])
    plt.show()

plot_accuracy(history)

The plot looks like this. You can see the validation accuracy flattens out:

In fact, Keras prints a message that says as much:

Epoch 00010: val_acc did not improve from 0.70262

Because of the EarlyStopping callback, if there are more than 10 of such epochs in a row, Keras will stop training. But, at this point, you’ve only trained for 10 epochs in total, so that callback didn’t kick in here yet. The other callback, ModelCheckpoint, did do its job and saved a new version of the model whenever the validation accuracy improved:

Epoch 00009: val_acc improved from 0.69215 to 0.70262, saving model to
checkpoints/multisnacks-1.0450-0.7026.hdf5

The two numbers in the filename, 1.0450 and 0.7026 respectively, are the validation loss and accuracy. After only nine epochs, this model already got up to 70% accuracy. Sweet! That’s a lot better than your previous models and also improves on Turi’s results already. But you’re not done yet…

Fine-tuning the feature extractor

At this point, it’s a good idea to start fine-tuning the feature extractor. So far, you’ve been using the pre-trained MobileNet as the feature extractor. This was trained on the ImageNet dataset, which contains a large variety of photos from 1,000 different kinds of objects.

The pretrained MobileNet knows a lot about photos in general, including photos of food items. This is why you’ve trained a classifier on top of MobileNet’s layers so that it can translate this general knowledge about photos to your own 20 categories of snacks.

But the pretrained feature extractor contains a lot of irrelevant knowledge, too, about animals, vehicles and all kinds of other things that are not snacks. We don’t need this knowledge for our task of classifying snacks.

With fine-tuning, you can adjust the knowledge inside the feature extractor to make it more relevant to your own data. Now the feature extractor itself already understands more about this specific task.

To fine-tune the MobileNet layers, first set them to trainable and then compile the model again:

for layer in base_model.layers:
    layer.trainable = True

top_model.compile(loss="categorical_crossentropy",
                  optimizer=optimizers.Adam(lr=1e-4),
                  metrics=["accuracy"])    

It’s important to use a lower learning rate now, lr=1e-4. That’s because you don’t want to completely throw away everything the MobileNet layers have learned already — you only want to tweak these values a little.

It’s better to set the learning rate too low than too high at this point, or you might end up destroying useful knowledge. The author found 1e-4 by experimenting a bit.

Run top_model.summary() and you’ll see that there are now over 3 million trainable parameters instead of just 20,500. There are still also non-trainable parameters; these are used by the BatchNormalization layers to keep track of internal state.

Simply run the cell with top_model.fit_generator() again to start fine-tuning. The loss may bounce around a bit in the beginning because suddenly the optimizer has a lot more work to do.

It also lost track of where it was because you compiled the model again. But you should see the training and validation accuracy start to improve quite quickly again. If not, lower the learning rate.

Note: Training is suddenly a lot slower now because this time Keras needs to train all the layers, not just the Dense layer. On the author’s iMac, the estimated time for a single epoch went up from six to 20 minutes. On the Linux machine with the GPU, the time went from 10 seconds per epoch to 30 seconds — not nearly as bad. It’s also possible that you will get an out-of-memory error at this point. There are more parameters to update and so the GPU needs more RAM. If that happens, make the batch size smaller and run the cells that create the generators again.

After about 10 epochs, the validation loss and accuracy no longer appear to improve. When that happens, it’s useful to reduce the learning rate. Here, you make it three times smaller:

K.set_value(top_model.optimizer.lr,
            K.get_value(top_model.optimizer.lr) / 3)

Now, train again for five or so epochs. For the author, the validation accuracy immediately shot up from 0.79 to 0.81, even though it had stopped improving earlier.

When the learning rate is too large, the optimizer may not be able to hone in on a good solution. This is why you start with a large-ish learning rate, to quickly get in the neighborhood of a good solution, and then make the learning rate smaller over time, in order to get as close to this solution as you can.

You can repeat this process of lowering the learning rate and training for a few epochs several more times until the loss and accuracy are no longer noticeably improving.

Tip: Keras also has a LearningRateScheduler callback that can automatically reduce the learning rate, which is especially useful for training sessions with hundreds of epochs that you don’t want to babysit. The ReduceLROnPlateau callback will automatically lower the learning rate when the validation accuracy or loss has stopped improving. Very handy!

The final loss and accuracy plots will look like this:

This was over a combined 30 epochs of training. Notice how there’s a bump in the lines at the points where you reduced the learning rate. Eventually, the curves flatten out, meaning that the model has learned all it can from the data.

top_model.evaluate_generator(test_generator,
    steps=len(test_generator))

The final accuracy on the test set is 82%. That’s a lot better than the SqueezeNet model from Turi Create. There are two reasons for this: 1) MobileNet is more powerful than SqueezeNet; and 2) Turi Create does not use data augmentation. Granted, 82% is still not as good as the model from Create ML, which had 91% accuracy, but that in turn uses a proprietary feature extractor that is more powerful than MobileNet. As we said before, it’s all about finding a compromise between results, speed and size.

Note: Notice that in the first few epochs, the validation loss and accuracy are actually a bit better than the training loss and accuracy. This is not unusual, especially with a relatively small validation set. It can also happen when you have a Dropout layer, which is only active for the training set but not for testing on the validation set. You’ll learn about dropout in the next section.

Regularization and dropout

So you’ve got a model with a pretty decent score already, but notice in the above plots that there is a big gap between the training loss and validation loss. Also, the training accuracy keeps increasing — reaching almost 100% — while the validation accuracy flattens out and stops improving.

This doesn’t necessarily mean that the model is overfitting. The training accuracy is always a little higher than the validation accuracy because it’s always easier for the model to make good predictions on the training images than on images it has never seen before.

However, this is only a bad thing when the validation loss or accuracy becomes worse over time. That doesn’t appear to be happening here… while the validation score isn’t as good as the training score, it doesn’t actually become worse — it just flattens out.

Still, it would be better if the validation curves were closer to the training curves. You can do this by adding regularization to the model. This makes it harder for the model to get too attached to the training images. Regularization is very useful, but keep in mind that it isn’t some magic trick that makes your validation score suddenly a lot better — it actually does the opposite and makes the training score a bit worse.

There are different methods for regularization, but what they all have in common is that they make learning more difficult. This discourages the model from learning unnecessary details, which may cause overfitting, and forces it to focus only on what is truly important.

You’ll use the following forms of regularization:

• Batch normalization
• Dropout
• L2 penalty

The MobileNet portion of the model already has a BatchNormalization layer after every convolution layer. These batch norm layers act as a type of regularizer. The main purpose of batch normalization is to make sure that the data that flows between the layers stay healthy.

The calculations involved introduce a small amount of noise, or random variations in the data, into the network. This noise prevents the model from memorizing specific image details. Regularization is not the main purpose of batch normalization, but it’s a nice side benefit.

You will add the other two types of regularization to the logistic regression portion of the model. Create this new classifier model:

from keras import regularizers
top_model = Sequential()
top_model.add(base_model)
top_model.add(GlobalAveragePooling2D())
top_model.add(Dropout(0.5))              # this line is new
top_model.add(Dense(num_classes,
              kernel_regularizer=regularizers.l2(0.001))) # new
top_model.add(Activation("softmax"))

There are only two new things here: a Dropout layer after the global pooling layer and the Dense layer now has a kernel regularizer.

Dropout is a special kind of layer that randomly removes elements from the tensor by setting them to zero. It works on the 1,024-element feature vector that is the output from the global pooling layer. Since you used 0.5 as the dropout percentage, Dropout will randomly set half of the feature vector’s elements to zero. This makes it harder for the model to remember things, because, at any given time, half of its input data is randomly removed — and it’s a different half for each training image.

Randomly removing elements from the feature vector seems like an odd thing to do, but it keeps the neural network from becoming lazy. The connections from the Dense layer cannot depend too much on any given feature since that feature might drop out of the network at random. Using dropout is a great technique to stop the neural network from relying too much on remembering specific training examples.

Aurélien Géron, in Hands-on Machine Learning with Scikit-Learn & TensorFlow at oreil.ly/2nzmN8L, compares this to a workplace where, on any given day, some percentage of the people might not come to work. In such a workplace, everyone must be able to do critical tasks and must cooperate with more co-workers. This makes the company more resilient and less dependent on any single worker.

The dropout rate is a hyperparameter, so you get to decide how high or low it should be. 0.5 is a good default choice. To disable dropout, simply set the rate to zero.

Note: Dropout is always disabled at inference time. This layer is only active during training. We wouldn’t want half of our predictions to randomly disappear!

The other form of regularization you’re using is an L2 penalty on the Dense layer. You’ve already briefly seen this in the chapter, “Digging Deeper Into Turi Create.” When you use a kernel regularizer, as Keras calls it, the weights for that layer are added to the loss term. L2 means that it actually adds the square of the weights to the loss term, so that large weights count as extra heavy.

Since it’s the optimizer’s job to make the loss as small as possible, it is now encouraged to keep the weights small, too, because large weights result in a large loss value. This prevents situations where some features get really large weights, making them seem more important than features with very small weights. Thanks to the L2 penalty, the weights are more balanced, reducing the chance of overfitting.

The value 0.001 is a hyperparameter called weight decay. This lets you tweak how important the L2 penalty is in the loss function. If this value is too large, then the L2 penalty overshadows the rest of the loss terms and the model will have a hard time learning anything. If it’s too small, then the L2 penalty doesn’t really have any effect.

Now, you can compile this new model again and train it. Make sure to first train a few epochs with the MobileNet layers frozen, and then set trainable = True to fine-tune. And don’t forget to periodically lower the learning rate! When you plot the loss curves, you’ll notice that the validation loss now stays much closer to the training loss.

Note: With an L2 penalty, the initial loss can be much higher than the expected np.log(num_classes). This is not so strange, because it adds the L2-norm of the weights to the loss as well. Starting out with a high loss value is usually no problem, as long as it goes down during training. If the loss doesn’t go down, the first thing to try is using a lower learning rate. Note that the validation loss does not include this extra L2 term.

Tune those hyperparameters

You’ve seen three different hyperparameters now:

• the learning rate
• the dropout probability
• the weight decay factor for L2 regularization

Choosing appropriate values for these settings — known as hyperparameter tuning — is essential for getting the training process to work optimally.

The way most people do hyperparameter tuning, is just by trying stuff and then seeing how the validation loss or accuracy changes. If you have a lot of hyperparameters, this can be a time-consuming job. There are ways to automate this, by using a grid search or a random search, which will try all possible combinations of the hyperparameters.

It’s very important that you use the validation set for tuning the hyperparameters, not the training set or the test set. The test set should only be used to verify how well your final model works, not for experiments with the hyperparameters.

There is a very good reason for this: When you tweak the hyperparameters based on the validation results, train the model with the new settings, tweak the hyperparameters again, and so on… then you’re indirectly training the model on the validation set, too.

You’re now manually influencing the training process by making changes based on the validation results. In a way, the images from the validation set are “leaking” into the training process. That’s OK since that’s what the validation set is for. But you don’t want this to happen to your test set, otherwise it can no longer paint a realistic picture of how well your model generalizes on images it has never seen before — because indirectly it will have already seen these images.

You can keep tweaking these hyperparameters to squeeze a bit more performance out of the model, but, at some point, you have to call it good enough. The author got the best results with a dropout rate of 0.7 and a weight decay of 0.01. This model scored 85% on the test set, which is again a few percentage points better than before.

How good is the model really?

The very last training epoch is not necessarily the best — it’s possible the validation accuracy didn’t improve or even got much worse — so in order to evaluate the final model on the test set, let’s load the best model back in first:

from keras.models import load_model
best_model = load_model(checkpoint_dir +
                        "multisnacks-0.7162-0.8419.hdf5")

This loads the model from a checkpoint file that was saved by the ModelCheckpoint callback. This HDF5 file contains the learned parameters for the model but also the architecture definition. (Replace the filename with your own best checkpoint.)

Note: The multisnacks-0.7162-0.8419.hdf5 file is included in this chapter’s resources under final/MobileNet/checkpoints. If you were unable to train the model on your own computer, feel free to load this version.

Note: The above instructions are for Keras version 2.2.4 and Keras-Applications 1.0.8. It may or may not work with newer or older versions.

Now you can evaluate this best model against the test set:

best_model.evaluate_generator(test_generator,
                              steps=len(test_generator))

For the author’s best model, this printed [0.6338246429667753, 0.8634453781512605]. The first number is the loss, which isn’t really that interesting, here.

The second number is the accuracy, over 86%. The Turi Create SqueezeNet model scored 67%, so we’re doing quite a bit better, here. And it gets in the neighborhood of Create ML’s score of 91%.

Take a closer look at what the model predicts:

test_generator.reset()
probabilities = best_model.predict_generator(test_generator,
                                    steps=len(test_generator))
predicted_labels = np.argmax(probabilities, axis=-1)

The predict_generator() function runs the model on all the images from the test set and puts the predicted probabilities in the probabilities array. Then you take the argmax over every result to find the index of the class with the highest probability.

Note: Before using predict_generator() you must first call reset() on the generator object. Otherwise, predict_generator() may not start at the right image and the predictions won’t make any sense.

The variable predicted_labels is a NumPy array with 952 numbers, one for each test set image. These are the predicted class indices. The correct, or ground-truth, class indices can be obtained from the test set generator:

target_labels = test_generator.classes

Now, you can compare these two arrays to find out where the classifier was correct and where it made mistakes, the so-called confusion matrix:

from sklearn import metrics
conf = metrics.confusion_matrix(target_labels, predicted_labels)

The conf variable is another NumPy array, of shape 20×20. It’s easiest to interpret when plotted as a heatmap:

import seaborn as sns

def plot_confusion_matrix(conf, labels, figsize=(8, 8)):
    fig = plt.figure(figsize=figsize)
    heatmap = sns.heatmap(conf, annot=True, fmt="d")
    heatmap.xaxis.set_ticklabels(labels, rotation=45,
                                 ha="right", fontsize=12)
    heatmap.yaxis.set_ticklabels(labels, rotation=0,
                                 ha="right", fontsize=12)
    plt.xlabel("Predicted label", fontsize=12)
    plt.ylabel("True label", fontsize=12)
    plt.show()

# Find the class names that correspond to the indices
labels = [""] * num_classes
for k, v in test_generator.class_indices.items():
    labels[v] = k

plot_confusion_matrix(conf, labels, figsize=(14, 14))

This plots the following confusion matrix:

On the diagonal — the bright squares — are the images that were correctly matched. Everything else is an incorrect match. Ideally, there are only numbers on the diagonal and zeros everywhere else. From this confusion matrix, you can immediately see that apples are often wrongly predicted to be oranges (four times), and cookies and muffins got mixed up three times.

Note: Earlier, we mentioned that the generator for the test set should not use data augmentation. Otherwise, running evaluate_generator() more than once would give different scores each time. You can actually use such differences to your advantage, known as TTA, or Test Time Augmentation.

For example, instead of making only one prediction for each test image, you could do it once for the normal image and once for the image flipped. Then the final score is the average of these two predictions. The more different variations of the test image you use, the better the average score will be.

This trick is often used in competitions to squeeze a few extra points out of the model’s performance. Of course, making multiple predictions per image is also slower and therefore not really suitable for mobile apps.

Precision, recall, F1-score

It’s also useful to make a precision-recall report:

print(metrics.classification_report(target_labels,
                     predicted_labels, target_names=labels))

This prints the following:

              precision    recall  f1-score   support

       apple       0.95      0.80      0.87        50
      banana       0.91      0.96      0.93        50
        cake       0.70      0.76      0.73        50
       candy       0.90      0.88      0.89        50
      carrot       0.92      0.88      0.90        50
      cookie       0.81      0.78      0.80        50
    doughnut       0.88      0.90      0.89        50
       grape       0.94      0.96      0.95        50
     hot dog       0.90      0.88      0.89        50
   ice cream       0.88      0.74      0.80        50
       juice       0.94      0.96      0.95        50
      muffin       0.85      0.83      0.84        48
      orange       0.85      0.82      0.84        50
   pineapple       0.71      0.88      0.79        40
     popcorn       0.85      0.85      0.85        40
     pretzel       0.79      0.88      0.83        25
       salad       0.81      0.94      0.87        50
  strawberry       0.93      0.80      0.86        49
      waffle       0.94      0.90      0.92        50
  watermelon       0.81      0.88      0.85        50

    accuracy                           0.86       952
   macro avg       0.86      0.86      0.86       952
weighted avg       0.87      0.86      0.86       952

Precision means: how many of the images that were classified as being X really are X? For example, the precision on hot dog is pretty good, 0.90. Most of the time when the model thinks something is a hot dog, it really is a hot dog.

Precision is rather low on pineapple, 0.71, which means the model found a lot of objects that it thinks are pineapple that really aren’t. You can see this in the confusion matrix in the column for pineapple. When you sum up the numbers in this column, you get 49 total pineapple predictions, of which only 35 are correct, so the precision is 35/49 or 0.71. Almost one out of four images that the model thinks is a pineapple, actually isn’t a pineapple. Ouch, there’s room for improvement there!

By the way, instead of counting up these numbers by hand, it’s much simpler to write some Python:

# Get the class index for pineapple
idx = test_generator.class_indices["pineapple"]

# Find how many images were predicted to be pineapple
total_predicted = np.sum(predicted_labels == idx)

# Find how many images really are pineapple (true positives)
correct = conf[idx, idx]

# The precision is then the true positives divided by
# the true + false positives
precision = correct / total_predicted
print(precision)

This should print 0.71, just as in the report. As you can tell from the math, the more false positives there are, i.e. images the model thinks belong to class X but that aren’t, the lower the precision.

Recall means: how many of the images of class X did the model find? This is in some ways the opposite of precision.

Recall for banana is high, 0.96, so the images that contained bananas were often correctly found by the model. The recall for ice cream is quite low at 74%, so over one-fourth of the ice cream images were classified as something else. To verify this in Python:

# Get the class index for ice cream
idx = test_generator.class_indices["ice cream"]

# Find how many images are supposed to be ice cream
total_expected = np.sum(target_labels == idx)

# How many ice cream images did we find?
correct = conf[idx, idx]

# The recall is then the true positives divided by
# the true positives + false negatives
recall = correct / total_expected
print(recall)

This should print 0.74. The more false negatives there are, i.e., things that are wrongly predicted to not be class X, the lower the recall for X.

The classification report also includes the F1-score. This is a combination of precision and recall and is useful if you want to get an average of the two.

The classes with the highest F1-score are grape and juice, both at 0.95. You can safely say that this classifier works very well for images with grapes or juices. The class with the lowest F1-score, 0.73, is cake. If you wanted to improve this classifier, the first thing you might want to do is find more and better training images for the cake category.

Note: It’s quite useful to be able to write a bit of Python code. Often you’ll need to write short code snippets like the above to take a closer look at the predictions. Get comfortable with Python if you’re interested in building your own models!

What are the worst predictions?

The confusion matrix and precision-recall report can already give hints about things you can do to improve the model. There are other useful things you can do. You’ve already seen that the cake category is the worst overall. It can also be enlightening to look at images that were predicted wrongly but that have very high confidence scores. These are the “most wrong” predictions. Why is the model so confident, yet so wrong about these images?

For example, you can use the following code to find the images that the model was the most wrong about. It uses some advanced NumPy sorcery:

# Find for which images the predicted class is wrong
wrong_images = np.where(predicted_labels != target_labels)[0]

# For every prediction, find the largest probability value;
# this is the probability of the winning class for this image
probs_max = np.max(probabilities, axis=-1)

# Sort the probabilities from the wrong images from low to high
idx = np.argsort(probs_max[wrong_images])

# Reverse the order (high to low), and keep the 5 highest ones
idx = idx[::-1][:5]

# Get the indices of the images with the worst predictions
worst_predictions = wrong_images[idx]

index2class = {v:k for k,v in test_generator.class_indices.items()}

for i in worst_predictions:
    print("%s was predicted as '%s' %.4f" % (
        test_generator.filenames[i],
        index2class[predicted_labels[i]],
        probs_max[i]
    ))

This will output:

strawberry/09d140146c09b309.jpg was predicted as 'salad' 0.9999
apple/671292276d92cee4.jpg was predicted as 'pineapple' 0.9907
muffin/3b25998aac3f7ab4.jpg was predicted as 'cake' 0.9899
pineapple/0eebf86343d79a23.jpg was predicted as 'banana' 0.9897
cake/bc41ce28fc883cd5.jpg was predicted as 'waffle' 0.9885

It can also be instructive to actually look at those images:

from keras.preprocessing import image
img = image.load_img(test_data_dir +
        test_generator.filenames[worst_predictions[0]])
plt.imshow(img)

Yep, it’s not hard to see why the model got confused, here. You could make a good case that this image is labeled wrong in the test set — or at least is very misleading:

A note on imbalanced classes

There is much more to say about image classifiers than we have room for in this book. One topic that comes up a lot is how to deal with imbalanced data.

In a binary classifier that needs to distinguish between disease present (positive) and not present (negative) in X-ray images, most X-rays will not show any disease at all. That’s a good thing for the patients involved, but it also makes a harder job for the classifier. If the disease happens to only 1% of the patients, the classifier could simply always predict “disease not present” and it would be correct 99% of the time. But such a classifier is also pretty useless… 99% correct sounds impressive, but it’s not always good enough.

Or let’s say you want to train a classifier that can distinguish between the following cases: cat, dog, neither cat or dog. In order to train such a classifier, you’ll obviously need pictures of cats and dogs, but also pictures of things that are not cats and dogs. This last category must be much larger because it needs to cover a wide variety of objects, and the classifier will need to lump all of these into the “not cat or dog” category. The risk here is that the classifier will only learn about that one big category and not about the cat and dog categories, which have many fewer images.

There are various techniques you can use to deal with class imbalance, such as oversampling where you use the images from the smaller categories more often, undersampling where you use fewer images from the larger categories, or setting weights on the classes so that the bigger category has a smaller effect on the loss.

Turi Create and Create ML currently have no options for this, so if you need to build a classifier for an imbalanced dataset, Keras is a better choice.

Here ends our discussion of how to train image classifiers. Next up, you’ll learn how to convert the trained Keras model to a Core ML model that you can use in your iOS and macOS apps.

Converting to Core ML

When you write model.save("name.h5") or use the ModelCheckpoint callback, Keras saves the model in its own format, HDF5. In order to use this model from Core ML, you have to convert it to a .mlmodel file first. For this, you’ll need to use the coremltools Python package.

The kerasenv environment already has coremltools installed. Just in case you need to install it by hand, type this into a command line prompt:

pip install -U coremltools

You can enter the following commands into the Jupyter notebook or just follow along with MobileNet.ipynb. This chapter’s resources also include a separate Python script, convert-to-coreml.py that first loads the model from the best checkpoint and then does the conversion. Using a separate script makes it easy to add the model conversion step to a build script or CI (Continuous Integration) server.

First, import the package:

import coremltools

You may get some warning messages at this point about incompatible versions of Keras and TensorFlow. These tools change quicker than coremltools can keep up with, but usually, these warnings are not a problem. (If you get an error during conversion, you may need to downgrade your Keras install to the last supported version.)

Since this is a classifier model, coremltools needs to know what the label names are. It’s important that these are in the same order as in train_generator.class_indices:

labels = ["apple", "banana", "cake", "candy", "carrot",
          "cookie", "doughnut", "grape", "hot dog",
          "ice cream", "juice", "muffin", "orange",
          "pineapple", "popcorn", "pretzel", "salad",
          "strawberry", "waffle", "watermelon"]

Now, you can use the Keras converter to create a Core ML model:

coreml_model = coremltools.converters.keras.convert(
    best_model,
    input_names="image",
    image_input_names="image",
    output_names="labelProbability",
    predicted_feature_name="label",
    red_bias=-1,
    green_bias=-1,
    blue_bias=-1,
    image_scale=2/255.0,
    class_labels=labels)

This has quite a few arguments, so let’s look at them in turn:

• The first argument is the Keras model object. Here you’re using the best_model object that you loaded in the previous section.

• input_names tells the converter what the inputs should be named in the .mlmodel file. Since this is an image classifier, it makes sense to use the name "image". This is also the name that’s used by Xcode when it automatically generates the Swift code for your Core ML model.

• image_input_names tells the converter that the input called "image" should be treated as an image. This is what lets you pass a CVPixelBuffer object to the Core ML model. If you leave out this option, the input is expected to be an MLMultiArray object, which is not as easy to work with.

• output_names and predicted_feature_name are the names of the two outputs. The first one is "labelProbability" and contains a dictionary that maps the predicted probabilities to the names of the classes. The second one is "label" and is a string that contains the class label of the best prediction. These are also the names that Turi Create used.

• red_bias, green_bias, blue_bias, and image_scale are used to normalize the image. MobileNet, like the other models you’ve trained, expects the pixels to be in the range [-1, 1] instead of the usual [0, 255]. The chosen values are equivalent to the normalization function you’ve used before: image / 127.5 - 1. If these settings are incorrect, Core ML will make bogus predictions.

• class_labels contains the list of label names you defined earlier.

When you run this code, coremltools goes through the Keras model layer-by-layer and prints its progress.

You can also supply metadata, which can be helpful for the users of your model, especially the descriptions of the inputs and outputs:

coreml_model.author = "Your Name Here"
coreml_model.license = "Public Domain"
coreml_model.short_description = "Image classifier for 20 different types of snacks"

coreml_model.input_description["image"] = "Input image"
coreml_model.output_description["labelProbability"]= "Prediction probabilities"
coreml_model.output_description["label"]= "Class label of top prediction"

At this point, it’s useful to write print(coreml_model) to make sure that everything is correct. The input should be of type imageType, not multiArrayType, and there should be two outputs: one a dictionaryType and the other a stringType.

Finally, save the model to an .mlmodel file:

coreml_model.save("MultiSnacks.mlmodel")

If you weren’t on your Mac already, then download this .mlmodel file to your Mac.

Double-click the file to open it in Xcode:

Put it in the app and try it out!

Challenges

Challenge 1: Train using MobileNet

Train the binary classifier using MobileNet and see how the score compares to the Turi Create model. The easiest way to do this is to copy all the images for the healthy categories into a folder called healthy and all the unhealthy images into a folder called unhealthy. (Or maybe you could train a “foods I don’t like” vs. “foods I like” classifier.)

Note: For a binary classifier, you can keep using softmax and the loss function "categorical_crossentropy", which gives you two output values, one for each category. Alternatively, you can choose to have just a single output value, in which case the final activation should not be softmax but Activation("sigmoid"), the logistic sigmoid. The corresponding loss function is "binary_crossentropy". If you feel up to a challenge, try using this sigmoid + binary cross-entropy for the classifier. The class_mode for the ImageDataGenerator should then be "binary" instead of "categorical".

Challenge 2: Add more layers

Try adding more layers to the top model. You could add a Conv2D layer, like so:

top_model.add(Conv2D(num_filters, 3, padding="same"))
top_model.add(BatchNormalization())
top_model.add(Activation("relu"))

**Tip**: To add a `Conv2D` layer after the `GlobalAveragePooling2D` layer, you have to add a `Reshape` layer in between, because global pooling turns the tensor into a vector, while `Conv2D` layers want a tensor with three dimensions.

top_model.add(GlobalAveragePooling2D())
top_model.add(Reshape((1, 1, 1024)))
top_model.add(Conv2D(...))

Feel free to experiment with the arrangement of layers in this top model. In general, adding more layers will make the classifier more powerful, but too many layers will make the model big and slow. Keep an eye on the number of trainable parameters!

Challenge 3: Experiment with optimizers

In this chapter and the last you’ve used the Adam optimizer, but Keras offers a selection of different optimizers. Adam generally gives good results and is fast, but you may want to play with some of the other optimizers, such as RMSprop and SGD. You’ll need to experiment with what learning rates work well for these optimizers.

Challenge 4: Train using MobileNetV2

There is a version 2 of MobileNet, also available in Keras. MobileNet V2 is smaller and more powerful than V1. Just like ResNet50, it uses so-called residual connections, an advanced way to connect different layers together. Try training the classifier using MobileNetV2 from the keras.applications.mobilenetv2 module.

Challenge 5: Train MobileNet from scratch

Try training MobileNet from scratch on the snacks dataset. You’ve seen that transfer learning and fine-tuning works very well, but only because MobileNet has been pre-trained on a large dataset of millions of photos. To create an “empty” MobileNet, use weights=None instead of weights="imagenet". You’ll find that it’s actually quite difficult to train a large neural network from scratch on such a small dataset. See whether you can get this model to learn anything, and, if so, what sort of accuracy it achieves on the test set.

Challenge 6: Fully train the model

Once you’ve established a set of hyperparameters that works well for your machine learning task, it’s smart to combine the training set and validation set into one big dataset and train the model on the full thing. You don’t really need the validation set anymore at this point — you already know that this combination of hyperparameters will work well — and so you might as well train on these images too. After all, every extra bit of training data helps! Try it out and see how well the model scores on the test set now. (Of course, you still shouldn’t train on the test data.)

Key points

• MobileNet uses depthwise convolutions because they’re less expensive than regular convolution. Ideal for running models on mobile devices. Instead of pooling layers, MobileNet uses convolutions with a stride of 2.

• Training a large neural network on a small dataset is almost impossible. It’s smarter to do transfer learning with a pre-trained model, but even then you want to use data augmentation to artificially enlarge your training set. It’s also a good idea to adapt the feature extractor to your own data by fine-tuning it.

• Regularization helps to build stable, reliable models. Besides increasing the amount of training data, you can use batch normalization, dropout and an L2 penalty to stop the model from memorizing specific training examples. The larger the number of learnable parameters in the model, the more important regularization becomes.

• You can use Keras callbacks to do automated learning rate annealing, save model checkpoints, and many other handy tasks.

• Try your model on the test set to see how good it really is. Use a confusion matrix and a precision-recall report to see where the model makes mistakes. Look at the images that it gets most wrong to see if they are really mistakes, or if your dataset needs improvement.

• Use coremltools to convert your Keras model to Core ML.