13. Sequence Classification

Written by Chris LaPollo

If you’ve followed along with the last couple chapters, you’ve learned some things about how working with sequences differs from other types of data, and you got some practice collecting and cleaning datasets. You also trained a neural network to recognize user gestures from iPhone sensor data. Now you’ll use your trained model in a game where players have just a few seconds to perform an activity announced by the app. When you’ve finished, you’ll have learned how to feed data from your device into your model to classify user activity.

This chapter picks up where the last one ended — just after you added your classification model to the GestureIt project. If you didn’t go through the previous chapter and train your own model, don’t fret! You can always use the GestureIt starter project found in the chapter resources. Either way, once you have the project open in Xcode, you’re ready to go!

Classifying human activity in your app

You trained a model and added it to the GestureIt project in the last chapter, and you learned a bit about how that model works. Now take a quick look through the project to see what else is there. The project’s Info.plist file already includes the keys necessary to use Core Motion, explained earlier when you built the GestureDataRecorder project.

GestureIt’s interface (not shown here) is even simpler than GestureDataRecorder’s — it’s just two buttons: Play and Instructions. Choosing Instructions shows videos of each gesture, and Play starts a game.

While playing, the game speaks out gestures for the player to make, awarding one point for each correctly recognized gesture. The game ends when the app recognizes an incorrect gesture or if the player takes too long.

The project already includes the necessary gameplay logic, but if you play it now you’ll always run out of time before scoring any points. If you want it to recognize what the player is doing, you’ll need to wire up its brain.

All the code you write for the rest of this chapter goes in GameViewController.swift, so open that file in Xcode to get started.

This file already imports the Core Motion framework and includes all the necessary code to use it. Its implementations of enableMotionUpdates and disableMotionUpdates are almost identical to what you wrote in the GestureDataRecorder project. The differences are minor and you should have no problem understanding them. As was the case with that project, this file contains a method named process(motionData:) that the app calls whenever it receives device motion data. At the moment it’s empty, but you’ll implement it later. For now, import the Core ML framework by adding the following line with the other imports near the top of the file:

import CoreML

In order to keep your code tidy and more easily maintainable, you’ll store numeric configuration values as constants in the Config struct at the top of the class, just like you did in the GestureDataRecorder project. To start, add the following three constants to that struct:

static let samplesPerSecond = 25.0
static let numberOfFeatures = 6
static let windowSize = 20

These values must match those of the model you trained. You’ll use samplesPerSecond to ensure the app processes motion data at the same rate your model saw it during training. The dataset provided in this chapter’s resources was collected at 25 samples per second, so that’s the value used here. However, change this value if you train your own model using data fed to it at a different rate.

Note: In case it’s not clear why the app’s samplesPerSecond must match that of the dataset used to train your model, consider this example: Imagine you trained your model using a prediction window of 200 samples, on data collected at 100 samples per second. That means the model would learn to recognize actions seen in highly detailed, two-second chunks. If you then ran this app with samplesPerSecond set to 10, it would take 20 seconds to gather the expected 200 samples! Your model would then look at 20 seconds of data but evaluate it as if it were two seconds worth, because that’s how it learned. This would almost certainly make the patterns in these sequences appear different from what the model saw during training. Remember, machine learning models only work well with data that is similar to what they saw during training, so getting the sampling rate wrong here could make a perfectly good model seem completely broken.

Likewise, the model discussed in this chapter expects data in blocks of 20 samples at a time, with six features for each sample. The windowSize and numFeatures constants capture those expectations.

Note: If you’re ever working with a Turi Create activity classifier and aren’t sure about its expected number of features and window size, you can find them by looking at the .mlmodel file in Xcode’s Project Navigator. However, this does not include information about the rate at which motion data needs to be processed, so that you’ll just need to know.

Now that you’ve added those constants, you can complete the starter code’s implementation of enableMotionUpdates by setting the CMMotionManager’s update interval. To do so, add the following line inside enableMotionUpdates, just before the call to startDeviceMotionUpdates:

motionManager.deviceMotionUpdateInterval = 1.0 / Config.samplesPerSecond

Just like you did in GestureDataRecorder, this tells motionManager to deliver motion updates to your app 25 times per second — once every 0.04 seconds.

Core ML models, such as GestureClassifier, expect their input in the form of MLMultiArray objects. Unfortunately, working with these objects involves quite a bit of type casting. Swift’s type safety is great, and explicit type casting forces developers to be more thoughtful about their code — but I think we can all agree code gets pretty ugly when there’s too much casting going on. To keep that ugliness — and the extra typing it requires — to a minimum, you’ll be isolating any MLMultiArray-specific code within convenience methods. Add the first of these methods below the MARK: - Core ML methods comment in GameViewController:

static private func makeMLMultiArray(numberOfSamples: Int) -> MLMultiArray? {
  try? MLMultiArray(
    shape: [1, numberOfSamples, Config.numberOfFeatures] as [NSNumber],
    dataType: .double)
}

This function takes as input the number of samples the array should contain. It then attempts to make an MLMultiArray with a shape and data type that will work with our model: [1, numSamples, Config.numFeatures] and double, respectively. Notice how the shape needs to be cast as an array of NSNumbers — you’ll see a lot of those types of casts when dealing with MLMultiArrays.

Attempting to create an MLMultiArray can fail by throwing an exception. If that occurs here, the try? causes this function to return nil. This might occur in situations such as when there is insufficient memory to create the requested array. Hopefully it doesn’t ever happen, but you’ll add some code to deal with that possibility a bit later.

Now that you have that handy function, you’ll use it to create space to store motion data to use as input to your model. Add the following property, this time to the area under the // MARK: - Core ML properties comment:

let modelInput: MLMultiArray! =
  GameViewController.makeMLMultiArray(numberOfSamples: Config.windowSize)

This creates the modelInput array, appropriately sized for the model you trained. Later you’ll populate this array with motion data prior to passing it to your model for classification.

Note: You may have noticed that modelInput is declared as an implicitly unwrapped optional, but makeMLMultiArray can return nil. Doesn’t that mean you run the risk of crashing your app elsewhere if you try to unwrap modelInput when it’s nil? Normally, that would be a problem, but later you’ll add some code that ensures this can never happen.

Overlapping prediction windows

Now, you could work with just a single MLMultiArray like modelInput, repeatedly filling it up over time and passing it to the model.

The diagram below shows what it would look like making two predictions with a window size of 20:

As the diagram above shows, the array would fill up between times T1 and T20, then you’d pass it to your model to make your first prediction. After that you’d reuse the array between times T21 and T40, before passing it to your model again to make your second prediction.

This technique is the simplest to code and is fine for many apps. However, there are times when doing this would cause some problems. Consider the situation shown in the following diagram, where an activity you want to recognize spans across prediction boundaries:

In this case, a few things might happen. If the amount of data in the first prediction window is sufficient for the model to recognize the activity, then no problem — it returns the correct classification. But if the model needs to see more activity data than is available in the first window, it won’t be able to classify it correctly until its second prediction.

In that case it takes longer than necessary to report the result, which makes your app feel sluggish. Or worse yet, all the non-activity data in the second window might make the second prediction fail to recognize the activity, too.

Delayed responses or inaccurate predictions — take your pick, but neither is a great option.

Now consider another problematic scenario, shown in the following diagram:

Here there’s one activity that spans across two predictions, just like before. But now a second activity occurs only within the second prediction window. In this case, assume the first prediction did not recognize anything, so now it’s up to the second window to handle everything. How will it classify the two activities?

It can only make one prediction, so it will either correctly predict one of the activities, or it will become so confused that it fails to predict either of them. This isn’t necessarily incorrect — it really depends on the app — but it’s something you need to consider carefully.

In many cases it would be better if you could make predictions more often. You might try smaller prediction windows, but that isn’t always an option because your model might need to see larger chunks of data to successfully recognize activities — that depends entirely on your specific data, model, and use case. But it turns out you can make predictions more often without changing the window size if you overlap your prediction windows, as shown in the following diagram:

In this case, the first prediction sees data from times T1 to T20, and the third prediction sees the data from times T21 to T40. But now a second prediction window overlaps each of those, spanning the data from times T11 through T30.

Because this is like sliding the prediction window along the data (using offsets of 10 in this case), many people call these “sliding” windows.

An app using this design responds more quickly because it makes more predictions, and it’s more accurate because it considers individual samples as part of multiple possible sequences. The first prediction window still might not recognize anything, but the second prediction would see the first activity — and predict it at T30 instead of waiting until T40. And then the third prediction would recognize the second activity only 10 samples later. The app ends up feeling more responsive and it doesn’t miss either activity.

Overlapping predictions mostly solves all of the problems mentioned earlier. But depending on how much data your model needs to see in order to make a prediction, and how much you overlap your windows, you still might run into missed or erroneous classifications. It’s a matter of finding the best amount of overlap for your app.

You’ll be implementing overlapping predictions in Gesture It, because you’ll want fast response times to quickly evaluate the player’s gestures.

But if you were making an app that tracks the amount of time you spend jogging, for example, you would probably be fine with non-overlapping predictions made over longer periods of time (maybe even once every several seconds).

Note: How much you overlap your predictions directly affects more than just accuracy and response time. More overlap means running inference with your model more often, and that extra processing could increase battery drain. And depending on how long it takes your model to make predictions, it might not even keep up with the pace of requests, causing your app to exhibit other performance problems. So test various options and settle on making predictions only as often as is necessary to achieve your goals.

To help define your prediction windows, add the following constants to the Config struct at the top of the file:

static let windowOffset = 5
static let numberOfWindows = windowSize / windowOffset

Here you define windowOffset as five. This is not how much the window overlaps, but rather how far to offset the start of the window from the start of the previous window.

With the windowSize of 20 you defined earlier, this makes numberOfWindows equal four. That’s how many prediction windows you’ll have before you essentially wrap back around to the first one again.

This should be clearer if you refer to the the following diagram, which shows how your predictions would overlap for the first 40 samples:

With the settings you’ve made so far, Gesture It will take 0.8 seconds to respond with its first prediction, but then each successive prediction will occur every 0.2 seconds after that. That’s because samplesPerSecond is 25, so each sample takes 0.04 seconds to arrive. A windowSize of 20 looks at 20 x 0.04s = 0.8 seconds of data, and a windowOffset of 5 means each prediction occurs 5 x 0.04s = 0.2 seconds after the last one.

Notice how different prediction windows overlap with various different combinations of other predictions. For example, Prediction Two sees the last 15 samples in Prediction One, and the first five samples in Prediction Five, along with 15 and 10 samples seen by Predictions Three and Four, respectively. And starting from Prediction Five, each window will process varying numbers of samples from six other prediction windows! All this overlap should help your model classify gestures quickly and accurately.

Note: The integer division used to calculate numWindows means you’ll never have a partial window. For example, if windowOffset were 20 with a windowSize of 50, you’d have two windows, one from T1 to T50 and another from T21 to T70. The code you write in this app will handle that situation fine, but keep in mind that the predictions will not occur at a steady rate unless windowSize is evenly divisible by windowOffset. In this example, an offset of 20 would result in 20 samples between predictions one and two but 30 samples between predictions two and three.

The previous diagrams show what samples each prediction window should use, but how do you implement it? At the moment you’ve got a single MLMultiArray the size of one window, but now you need four.

While you could create four different arrays to store this data, that would waste memory. Instead, you’ll make one slightly larger array that will act as a buffer area for the most recent motion data, and each prediction window will look at the appropriate subset of that larger buffer when necessary.

Add the following constant to the Config struct, which defines the size of the buffer you’ll create:

static let bufferSize =
  windowSize + windowOffset * (numberOfWindows - 1)

You define a buffer size large enough to hold one full window plus the space taken up by the offsets for the other windows. So for the settings you’ve used so far, Gesture It’s buffer will hold 35 samples. Don’t worry if it’s not yet clear why this is the right size — you’ll see soon.

Now add the following properties to manage the buffer. Put them with the other ML-related properties in GameViewController:

let dataBuffer: MLMultiArray! =
  GameViewController.makeMLMultiArray(numberOfSamples: Config.bufferSize)
var bufferIndex = 0
var isDataAvailable = false

You create dataBuffer using the convenience method you wrote earlier. As new motion data arrives from the device, you’ll use bufferIndex to determine where to store that data within the buffer. You’ll set the isDataAvailable flag to true once the buffer contains enough data to perform its first prediction.

For the remainder of this discussion, please refer to the following diagram, which shows the buffer’s contents at each prediction over the first 40 time steps:

Think of the buffer as having two halves, with a full prediction window on the left and auxiliary storage on the right. The second “half” isn’t a true half in this case, because it’s smaller than the first, but that won’t be a problem.

You’ll increment bufferIndex as new data arrives, moving it across the first half of the buffer, and you’ll reset it to the beginning whenever it reaches the buffer’s midpoint. That is, bufferIndex will always point to the next location to fill within the first prediction window. But whenever you store an item in the left half of the buffer, you’ll also store it in the equivalent location in the right half. (You’ll skip updates on the right side that would be out of bounds due to the size mismatch. You could make both sides the same size and then always store values in both places, but the approach used here saves some memory — usually a good thing for mobile apps.)

The top row of the diagram shows what the buffer looks like after 20 timesteps. The left side contains data from times T1 to T20, and the right side contains copies of times T1 to T15. It’s at this point that you’ll reset bufferIndex to zero, set isDataAvailable to true and perform the first prediction using times T1 to T20.

As data continues to arrive, you’ll keep filling the left and right sides of the buffer simultaneously. After five more timesteps, you’ll be ready to make the second prediction. As you can see in the second row of the diagram, the first five items of the buffer contain data from times T21 to T25, but the next 15 items still contain data from times T6 to T20. And because you’ve been updating both sides of the buffer, the first five items on the right contain data from times T21 to T25, too.

So you can now make your second prediction using times T6 to T25 by looking at a window that crosses into the second half of the buffer.

This process continues indefinitely, but the diagram shows the contents of the buffer when making each of the first five predictions. The key point to realize is that after the first time bufferIndex reaches the midpoint of the buffer and resets to the start, it is always the case that the the next 20 items starting at bufferIndex contain data from the previous 20 time steps.

Phew. That was a lot of discussion about such a small bit of code, so hopefully you’re still here. Now back to the app!

Buffering motion data

Now you’re going to add code to handle MLMultiArrays that end up as nil. Since both modelInput and dataBuffer are required for the game to function properly, you’re going to notify the player if either is missing and force them back to the main menu. However, you may want to make your own apps more robust. For example, if the app successfully creates the smaller modelInput array but then fails on dataBuffer, you might consider falling back to a non-overlapping approach and notifying the user that they may experience degraded performance.

Add the following code inside viewDidLoad, immediately above the call to enableMotionUpdates:

guard modelInput != nil, dataBuffer != nil else {
  displayFatalError("Failed to create required memory storage")
  return
}

Here you check to ensure that the app was able to create each of its required MLMultiArray properties. If not, you call displayFatalError, a method in the starter code that alerts the player with the given error message and then dismisses the GameViewController.

Note: The starter code enables motion updates when it loads the game view and stops them when the game is over. However, your production apps should be more robust than that. Be sure your apps are good iOS citizens and have them properly handle situations such as getting paused for incoming phone calls, etc.

The app will receive motion updates Config.samplesPerSecond times each second. For each update, you’ll need to store the appropriate features in dataBuffer, the MLMultiArray you created earlier. You’ll wrap this logic in helper methods to keep things easier to read. Add the first helper method to the class:

@inline(__always) func addToBuffer(
  _ sample: Int, _ feature: Int, _ value: Double) {
  dataBuffer[[0, sample, feature] as [NSNumber]] =
    value as NSNumber
}

The addToBuffer function isolates the NSNumber casts to one line, which keeps the code you’ll add later easier to read. Declaring it with @inline(__always) tells the Swift compiler to replace any calls to this function with the contents of the function itself, ensuring your code executes as quickly as possible.

Swift is good about inlining these one-line functions on its own, but including this tag makes your intention clear.

This method sets a single value inside dataBuffer. That MLMultiArray is arranged as a 3-dimensional tensor, indexed as [batch, sample, feature]. The model’s batch size is always one, so the first index value here is always 0. The sample and feature indices are passed as arguments to this method.

Next, add the following helper method:

// 1
func buffer(motionData: CMDeviceMotion) {
  // 2
  for offset in [0, Config.windowSize] {
    let index = bufferIndex + offset
    if index >= Config.bufferSize {
      continue
    }
    // 3
    addToBuffer(index, 0, motionData.rotationRate.x)
    addToBuffer(index, 1, motionData.rotationRate.y)
    addToBuffer(index, 2, motionData.rotationRate.z)
    addToBuffer(index, 3, motionData.userAcceleration.x)
    addToBuffer(index, 4, motionData.userAcceleration.y)
    addToBuffer(index, 5, motionData.userAcceleration.z)
  }
}

While this methods are essentially just updating an array, there are some important things to note:

1. You’ll call buffer from within process(motionData:), which you’ll write next. It copies motion data into the correct locations in the large buffer backing the overlapping prediction windows described earlier.

2. This for loop ensures each value is stored at the position indexed by bufferIndex, as well as a position that is one window-span later in the buffer. The continue statement ensures that second write attempt is not outside the buffer’s bounds, which would crash the app. For more details about how the overlapping windows work, refer to the discussion earlier in this chapter.

3. Here you call addToBuffer repeatedly to save the relevant data from the CMDeviceMotion object passed to this method. It’s extremely important to store only the features your model expects, and in exactly the order it expects them. This was all determined when you trained the model, but you can verify the information by inspecting the .mlmodel file in Xcode’s Project Navigator.

   Be sure to double check this step, because mistakes here will make your model function incorrectly — sometimes failing with a crash, sometimes by underperforming, and even sometimes by appearing to work! That last one might sound ok, but it just means you’ve got some lucky input and it’s unlikely to work well for long.

Your code so far only adds data to dataBuffer, but you’ll eventually need to pass modelInput to your ML model. That’s because your model expects to see an MLMultiArray with modelInput’s specific shape, not the larger buffer you created to implement overlapping windows. So, you’ll need to copy data between these structures.

To make those copies as fast as possible, you’ll be using low level pointers to copy chunks of memory directly. To do that, you need to know the exact number of bytes you want to access, so add the following constants to the Config struct:

static let windowSizeAsBytes = doubleSize * numberOfFeatures * windowSize
static let windowOffsetAsBytes = doubleSize * numberOfFeatures * windowOffset

Here you calculate the number of bytes it takes to represent a prediction window within an MLMultiArray, as well as the number of bytes necessary to represent the offset between prediction windows. The constant doubleSize referenced in these calculations already exists in the starter code — it stores how many bytes are used by one double. You’ll use these constants soon.

You’re now all set to fill in the placeholder process(motionData:) method. Insert the following code into that method:

// 1
guard expectedGesture != nil else {
  return
}
// 2
buffer(motionData: motionData)
// 3
bufferIndex = (bufferIndex + 1) % Config.windowSize
// 4
if bufferIndex == 0 {
  isDataAvailable = true
}
// 5
if isDataAvailable &&
   bufferIndex % Config.windowOffset == 0 &&
   bufferIndex + Config.windowOffset <= Config.windowSize {
  // 6
  let window = bufferIndex / Config.windowOffset
  // 7
  memcpy(modelInput.dataPointer,
         dataBuffer.dataPointer.advanced(
           by: window * Config.windowOffsetAsBytes),
         Config.windowSizeAsBytes)
  // 8
  // TODO: predict the gesture
}

This is the meat of your data pipeline, so look carefully at what’s going on here:

1. The starter project uses expectedGesture to keep track of what gesture the player should be making. This value will be nil whenever the game is not expecting a gesture, and this guard statement ensures this method doesn’t process motion data in those cases.

2. Here’s where you call the method you recently added, buffer. You pass it the CMDeviceMotion object given to this method, and it stores the motion data in the appropriate locations within dataBuffer.

3. Next, you update bufferIndex to keep track of the next available space in the buffer. You’re incrementing it by one, and looping it back around to zero when it reaches the end of the first window.

4. Here you check to see if bufferIndex is zero. Because bufferIndex is updated before this line, it can only ever be zero after it has exceeded Config.windowSize and wrapped back around at least once. At that point, you update isDataAvailable to indicate you have at least one full window’s worth of data.

5. This if-statement ensures you make predictions at the correct times. It first checks isDataAvailable to make sure at least one window is full. Then, it checks to see if bufferIndex is at the boundary of a window. Because bufferIndex resets when it reaches the end of the first window, you can only reliably check when it’s at the start of most windows, not the end.

   This line determines that by checking to see if bufferIndex is some multiple of the window offset. It also verifies that there is a full windowOffset worth of space after this position in the window. That final check is just a precaution in case you ever use a window size that is not evenly divisible by the offset size. Without that check, the code at 7 would crash your app when it tried to access invalid memory. If all these checks pass, then the function knows it’s OK to make a prediction.

6. Here you determine which prediction window you’re working with so you’ll know which data to access from the buffer.

7. Now you need to copy the samples for window from dataBuffer into modelInput. Conveniently, MLMultiArray objects expose a pointer for low level access to their backing memory via their dataPointer property, so here you take advantage of that fact and use memcpy to copy a window-sized chunk of memory directly from dataBuffer into modelInput.

   To locate the start of the window, you use the pointer’s advanced(by:) method and some math to move it the appropriate number of bytes from the start of the buffer. Be extremely careful with memcpy: Getting anything wrong here will at best give you the wrong results, and at worst will crash your app.

8. Here is where you will eventually attempt to make your prediction. But you’ll need to write just a bit more code before you do.

Making predictions with your model

At long last, your project is ready to start recognizing gestures. Almost. So far the app contains a lot of data processing and business logic — it still needs the machine learning bit!

Add your gesture recognition model into the app by initializing the following property with the other ML-related properties in GameViewController:

let gestureClassifier = GestureClassifier()

Xcode autogenerated the GestureClassifier class when you first dragged the .mlmodel file into the project, so all you have to do is instantiate it like this and then later call its prediction method with the appropriate inputs. It’s almost too easy, right?

Well, it would be if that’s all it took. Recall from the previous chapter’s discussion about the model’s inputs and outputs, the LSTM portion of the network requires you to provide it with the internal memory and output from its previous prediction. That means you’ll need to store that information each time you make a prediction and then pass it back to the model when making the next one. To help with that, Xcode generated the GestureClassifierOutput class at the same time it made GestureClassifier. This class conveniently encapsulates all four of the model’s outputs so you can save them for later use.

However, you’ve implemented your predictions using four overlapping windows, which means consecutive predictions aren’t actually continuations of each other. That is, the first sensor reading in a prediction window is not the reading immediately after the last one in the previous window. Instead, it’s a value within the previous window, offset from its start by Config.windowOffset samples. Because of that fact, it wouldn’t make sense for the LSTM’s internal state to carry over from the previous prediction — it needs to use the state from four predictions ago instead. To keep track of all these outputs, you’ll maintain an array of GestureClassifierOutputs, so add the following property for that:

var modelOutputs = [GestureClassifierOutput?](
  repeating: nil,
  count: Config.numberOfWindows)

This array will hold one GestureClassifierOutput for each prediction window. The values are optional and will be nil for any window before you’ve used it. You can see the code for GestureClassifierOutput by selecting GestureClassifier.mlmodel in the Project Navigator, and then clicking the small arrow icon next to GestureClassifier in the Model Class section. It basically just provides properties to access the model’s various outputs.

One last thing before you actually use your model. Earlier in the book, you read about how Core ML predictions come with probabilities which are essentially the model’s confidence in the prediction. And, you saw how the model will always produce some prediction, but not necessarily with much confidence.

To avoid reacting to low probability predictions, you’ll define a threshold that the probability must exceed to be considered sure enough to act upon. Add the following constant to Config at the top of the file:

static let predictionThreshold = 0.9

This basically means the model needs to be over 90% sure of a prediction before the app responds. This threshold was chosen after some playtesting, but it’s mostly personal preference beyond a certain threshold. Values too low will make the app hallucinate gestures where there are none, so definitely avoid that, but other than that it’s a matter of how touchy or picky you want the app to feel. Later, when you’re done writing the app, try out different values here to see how they affect the gameplay.

With those small additions in place, it’s now time to write the method that uses your trained model to recognize gestures. Add the following code to the end of GameViewController:

func predictGesture(window: Int) {
  // 1
  let previousOutput = modelOutputs[window]
  let modelOutput = try?
    gestureClassifier.prediction(
      features: modelInput,
      hiddenIn: previousOutput?.hiddenOut,
      cellIn: previousOutput?.cellOut)
  // 2
  modelOutputs[window] = modelOutput

  guard
    // 3
    let prediction = modelOutput?.activity,
    let probability = modelOutput?.activityProbability[prediction],
    // 4
    prediction != Config.restItValue,
    // 5
    probability > Config.predictionThreshold
  else {
      return
  }

  // 6
  if prediction == expectedGesture {
    updateScore()
  } else {
    gameOver(incorrectPrediction: prediction)
  }
  // 7
  expectedGesture = nil
}

You’ve written quite a bit of code already, but this method is really the only part of the app that actually uses machine learning. Here’s what it does:

1. First it calls prediction on gestureClassifier to try to classify the motion data, and stores the result as modelOutput. Notice that you provide both modelInput, which you populated in processMotionData, as well as the LSTM’s ouput and internal cell state from the previous prediction for this window. These values will be nil for each window’s first prediction, and that’s fine — this tells the classifier there is no history and it should initialize itself accordingly.

2. Then, it stores the model’s response in modelOutputs so you can access it next time you make a prediction for this window.

3. Next, it grabs the predicted activity, along with the probability assigned to that prediction, from the model’s output.

4. It checks for predictions of non-gestures — i.e. resting — and just ignores them.

5. For non-rest predictions, it checks to see if the probability exceeds the threshold you previously defined. If so, it considers it a real prediction; otherwise it does nothing and the app will continue processing motion events.

6. The next bit of code is game logic, but any app you write with a classification model will have something similar — a spot where you actually use the predicted value. If the model thinks the player made the correct gesture (i.e. the predicted gesture matches expectedGesture), then it calls updateScore to add a point; otherwise, the app thinks the player messed up and it calls gameOver.

7. Regardless of the prediction, the method resets expectedGesture to nil so that the app stops processing motion data for a while. The starter project’s existing game logic will set this to a new gesture when appropriate.

Note: The model class Xcode generates includes three different prediction methods, as well as a predictions (with an “s”) method that batches multiple predictions in one call. This code uses the version that takes MLMultiArrays directly, but you might find situations where you’d prefer to use one of the other versions in your own apps, so be sure to check the generated code for options.

Now go back to that comment you added earlier — // TODO predict the gesture — and replace it with a call to the method you just wrote:

predictGesture(window: window)

You already calculated the correct prediction window inside process, and here you pass that to predictGesture to perform inference.

Now build the app and run it on your iPhone. (Sorry, no motion data in the simulator!) You might succeed with the first gesture, but it won’t take long before the app calls out a gesture and then immediately complains that you got it wrong. What gives?

Remember those fancy overlapping prediction windows? Well, that backing storage buffer you made still contains data from the previous sequences you were processing. So when the app asks for a new gesture, there’s already a prediction window’s worth of data just sitting there ready to be recognized — collected while you were making the previous gesture. And don’t forget, recurrent models use state from previous predictions to help them make new predictions, because they assume the data is related. But when this app asks for a new gesture, it no longer wants the model to consider the prior data. Each new gesture needs a clean slate.

To correct this, you need to reset the buffer and the model’s previous output states. Add the following method to GameViewController:

func resetPredictionWindows() {
  // 1
  bufferIndex = 0
  // 2
  isDataAvailable = false
  // 3
  for i in 0..<modelOutputs.count {
    modelOutputs[i] = nil
  }
}

It’s not much code, but it’s vital in order for your app to function properly. Here’s what it does:

1. Reset bufferIndex to zero to start filling the buffer from the beginning again. This ensures new predictions are based on relevant sequence data, rather than data in the buffer left over from prior sequences.
2. Reset isDataAvailable to false to keep the app from trying to perform another prediction before it has at least one full window.
3. Set everything in modelOutputs to nil to clear out any internal model state built up from previous predictions. This ensures the underlying LSTM cells in your GestureClassifier model don’t remember anything from sequences related to earlier gestures and then try to use that information when making new predictions.

Now that you’ve defined that method, call it at the top of startTimer(forGesture:):

resetPredictionWindows()

The existing game logic already calls startTimerForGesture whenever it notifies the player to perform a new gesture. With this addition, you ensure the predictions made for new gestures are not using any data that arrived while the app was processing earlier gestures.

That’s it! Build and run again, and have fun Gesturing It! If the game times out too quickly for you to respond, increase the value of Config.gestureTimeout. Or, if you want to increase the challenge, see how low you can decrease it. How many correctly recognized gestures can you get in a row?

Challenges

Challenge 1: Expanding Gesture

It would be a good way to get some practice with activity recognition. Adding new gesture types to the GestureDataRecorder project is a straightforward process, so start there, and then collect some data. Next, add your new data to the provided dataset and train a new model. Replace the model in the GestureIt project with your newly trained model, and make the few modifications necessary to add your new gesture to the game.

Challenge 2: Recognizing activites

After that, you could try recognizing activities other than gestures. For example, you could make an app that automatically tracks the time a user spends doing different types of exercises. Building a dataset for something like that will be more difficult, because you have less control over the position of the device and more variation in what each activity looks like. In those cases, you’ll need to collect a more varied dataset from many different people to train a model that will generalize well.

Challenge 3: Using other devices

Keep in mind, these models work on other devices, too. The Apple Watch is a particularly fitting choice — a device containing multiple useful sensors, that remains in a known position on the user and is worn for all or most of the day. If you have access to one, give it a try!

Key points

• Use overlapping prediction windows to provide faster, more accurate responses.
• Call your model’s prediction method to classify data.
• Pass multi-feature inputs to your models via MLMultiArray objects.
• Arrange input feature values in the same order you used during training. The model will produce invalid results if you arrange them in any other order.
• When processing sequences over multiple calls to prediction, pass the hidden and cell state outputs from one timestep as additional inputs to the next timestep.
• Ignore predictions made with probabilities lower than some reasonable threshold. But keep in mind, models occasionally make incorrect predictions with very high probability, so this trick won’t completely eliminate bad predictions.