11. Data Collection for Sequence Classification

Written by Chris LaPollo

You worked exclusively with images throughout the first section of this book, and for good reason — knowing how to apply machine learning to images lets you add many exciting and useful features to your apps. Techniques like classification and object detection can help you answer questions like “Is this snack healthy?” or “Which of these objects is a cookie?”

But you’ve focused on individual images — even when processing videos, you processed each frame individually with complete disregard for the frames that came before or after it. Given the following series of images, can the techniques you’ve learned so far tell me where my cookies went?

Each of the above images tells only part of the story. Rather than considering them individually, you need to reason over them as a sequence, applying what you see in earlier frames to help interpret later ones.

There are many such tasks that involve working with sequential data, such as:

• Extracting meaning from videos. Maybe you want to make an app that translates sign language, or search for clips based on the events they depict.

• Working with audio, for example converting speech to text, or songs to sheet music.

• Understanding text, such as these sentences you’ve been reading, which are sequences of words, themselves sequences of letters (assuming you’re reading this in a language that uses letters, that is).

• And countless others. From weather data to stock prices to social media feeds, there are endless streams of sequential data.

With so many types of data and almost as many techniques for working with it, this chapter can’t possibly cover everything. You’ll learn ways to deal with text in later chapters, and some of the techniques shown here are applicable to multiple domains. But to keep things practical, this chapter focuses on a specific type of sequence classification — human activity detection. That is, using sensor data from a device worn or held by a person to identify what that person is physically doing. You’ve probably already experienced activity detection on your devices, maybe checking your daily step count on your iPhone or closing rings on your Apple Watch. Those just scratch the surface of what’s possible.

In this chapter, you’ll learn how to collect sensor data from Apple devices and prepare it for use training a machine learning model. Then you’ll use that data in the next chapter, along with Turi Create’s task-focused API for activity detection, to build a neural network that recognizes user activity from device motion data. Finally, you’ll use your trained neural net to recognize player actions in a game.

Note: Apple introduced the Create ML application with Xcode 11, which provides a nice GUI for training many types of Create ML models. One of those is called Activity Classifier and it’s essentially the same model you’ll build in these chapters using Turi Create. So why not use the Create ML app here?

We made that decision partially because we wrote these chapters before the Create ML app existed and it would require rewriting quite a bit of content without describing any truly new functionality, but it’s also because the GUI option is self-explanatory once you understand the underlying Turi Create code. The Create ML method is also a bit less flexible than using Turi Create directly, as a consequence of needing to support such a (delightfully) simple graphical interface.

We encourage you to experiment with the Create ML app after going through these chapters to see which option you prefer. We’ll try to point out instructions that might be different when working with the Create ML app.

The game you’ll make is similar to the popular Bop It toy, but instead of calling out various physical bits to bop and twist, it will call out gestures for the player to make with their iPhone. Perform the correct action before time runs out! The gestures detected include a chopping motion, a shaking motion and a driving motion (imagine turning a steering wheel).

We chose this project because collecting data and testing it should be comfortably within the ability of most readers. However, you can use what you learn here for more than just gesture recognition — these techniques let you track or react to any activity identifiable from sensor data available on an Apple device.

Modern hardware comes packed with sensors — depending on the model, you might have access to an accelerometer, gyroscope, pedometer, magnetometer, altimeter or GPS. You may even have access to the user’s heart rate!

With so much data available, there are countless possibilities for behaviors you can detect, including sporadic actions like standing up from a chair or falling off a ladder, as well as activities that occur over longer durations like jogging or sleeping. And machine learning is the perfect tool to make sense of it all. But before you can fire up those neural nets, you’ll need a dataset to train them.

Building a dataset

So you’ve got an app you want to power using machine learning. You do the sensible thing and scour the internet for a suitable, freely available dataset that meets your needs.

You try tools like Google Dataset Search, check popular data science sites like Kaggle, and exhaust every keyword search trick you know. If you find something — great, move on to the next section! But if your search for a dataset turns up nothing, all is not lost — you can build your own.

Collecting and labeling data is the kind of thing professors make their graduate students do — time consuming, tedious work that may make you want to cry. When labeling human activity data, it’s not uncommon to record video of the activity session, go through it manually to decide when specific activities occur, and then label the data using timecodes synced between the data recordings and the video. That may sound like fun to some people, but those people are wrong and should never be trusted.

This chapter takes a different approach — the data collection app automatically adds labels. They may not be as exact — manual labeling lets you pinpoint precise moments when test subjects begin or end an activity — but in many cases, they’re good enough.

To get started, download the resources for this chapter if you haven’t already done so, and open the GestureDataRecorder starter project in Xcode.

Note: The chapter resources include data files you can use unchanged, so you aren’t required to collect more here. However, the experience will help later when working on your own projects. Plus, adding more data to the provided dataset should improve the model you make later in the chapter.

Take a look through the project to see what’s there. ViewController.swift contains most of the app’s code, and it’s the only file you’ll be asked to change. Notice the ActivityType enum which identifies the different gestures the app will recognize:

enum ActivityType: Int {
	case none, driveIt, shakeIt, chopIt
}

If you run the app now, it will seem like it’s working but it won’t actually collect or save any data. The following image shows the app’s interface:

GestureDataRecorder probably won’t win any design awards, but that’s OK — it’s just a utility app that records sensor data. Users enter their ID, choose what activity and how many short sessions of that activity to record, and then hit Start Session to begin collecting data. The app speaks instructions to guide users through the recording process. And the Instructions button lets users see videos demonstrating the activities.

Note: For some datasets, it may be better to randomize activities during a session, rather than having users choose one for the entire thing. My test subjects didn’t seem to enjoy having to pay that much attention, though.

Why require a user ID? You’ll learn more about this later, but it’s important to be able to separate samples in your dataset by their sources. You don’t need specific details about people, like their names — in fact, identifying details like that are often a bad idea for privacy and ethics reasons — but you need some way to distinguish between samples.

GestureDataRecorder takes a simple but imperfect approach to this problem: It expects users to provide a unique identifier and then saves data for each user in separate files. To support this, the app makes users enter an ID number and then includes that in the names of the files it saves. If any files using that ID already exist on this device, the app requests confirmation and then appends new data to those files. So it trusts users not to append their data to someone else’s files on the device, and it’s up to you to ensure no two users enter the same ID on different devices.

The starter code supports the interface and other business logic for the app — you’ll add the motion-related bits now so you get to know how that all works.

Accessing device sensors with Core Motion

You’ll use Core Motion to access readings from the phone’s motion sensors, so import it by adding the following line along with the other imports in ViewController.swift:

import CoreMotion

This lets you access Core Motion within your code, but it’s not enough to allow your app to do so. Apple rightly wants users to decide which apps can access their data, so it requires developers to include an explanation for why they want it. The starter project’s Info.plist file already includes this explanation as a value for the key Privacy - Motion Usage Description. And because motion data is required for this app to function, rather than just a nice additional feature, both accelerometer and gyroscope have been added to Info.plist‘s Required device capabilities list, too. Don’t forget to provide the appropriate properties in your own apps.

Next, to interact with Core Motion, add the following properties inside ViewController. Keep things organized by putting them under the existing comment that reads // MARK: - Core Motion properties:

let motionManager = CMMotionManager()
let queue = OperationQueue()

Here you create a CMMotionManager to access the device’s motion data. Each app should contain only one such object, regardless of how many sensors it plans to use. You’ll use queue to keep sensor update callbacks off the main thread, which helps the device remain responsive while processing these high frequency events. Using a separate OperationQueue like this also ensures your app doesn’t miss updates if it is temporarily too busy to process events.

Before you go any further, find the following two lines inside startRecordingSession and delete them:

/* TODO: REMOVE THIS LINE
...
TODO: REMOVE THIS LINE */

These lines were commenting out a guard statement that ensures the app has access to device motion, and alerts the user otherwise. They were commented out because they require motionManager, which you just added.

You need to tell motionManager how often to produce sensor data. ViewController stores its configuration-related constants inside its Config enum, so add the following constant there:

static let samplesPerSecond = 25.0

Here you set samplesPerSecond to 25, which you’ll use later to specify you want the device to send you 25 sensor updates every second. This number is important because it determines how much data your model looks at based on how often you perform predictions. That is, if you classify the user’s activity once per second, this gives you 25 samples per classification; if you do it once every four seconds, this rate gives you 100 samples.

But why 25? The sensors in Apple devices are capable of producing updates many times per second — at least 100, according to the docs — so shouldn’t you just use the max? After all, aren’t people always saying that when it comes to machine learning, more data is always better?

There are a few reasons why you shouldn’t necessarily increase the update frequency too high:

• More updates means more data processing, which means less CPU available for whatever else your app needs to do.
• Faster updates usually means feeding more data into your model per prediction. That requires more complex ML models, which run more slowly — maybe too slowly to keep up with those faster updates.
• Higher frequency updates increase battery usage. You don’t want users deleting your app because it sucks the life out of their devices.

It’s true that higher frequency updates let you perceive finer details within the data, so there are times when you may need them. But not always — some activities involve slower changes over a longer time, where sensor readings might be necessary only a few times per second, or even less. The value of 25 used here was chosen arbitrarily — it works fine, but experiments to find the lowest usable update rate were not performed.

Note: There’s another option you aren’t using here, but you may want to consider for your own projects. Perform data collection at a high rate, and then downsample it to train multiple models and find the lowest rate that works well. For example, collect data at 80Hz and then train multiple models — 80Hz using all the data, 40Hz using every other sample, 20Hz using every fourth sample, etc. This let’s you collect data once and then have different options for how to use it, which is better than having to recollect it multiple times to experiment with update rates. Once you find the lowest rate that still works well, use that in your production app.

In this app, you’ll store all the collected sensor data in memory and then write it out to disk at the end of the recording session. Add the following array with the other properties under the comment that reads // MARK: - Core Motion properties in ViewController:

var activityData: [String] = []

You’ll create a single string containing all the data you want to record for a sample, and append it to activityData. The entire recording session will live inside this array as one long sequence, and GestureDataRecorder calls saveActivityData at the end of the session to save all these strings to file. However, you need to add the following few lines to actually save the array. Put it at the end of confirmSavingActivityData in ViewController:

do {
  try self.activityData.appendLinesToURL(fileURL: dataURL)
  print("Data appended to \(dataURL)")
} catch {
  print("Error appending data: \(error)")
}

This writes all the strings in the array out to the appropriate file using a helper function from inside StringArrayExtensions.swift. This function creates the file if it doesn’t already exist, or appends to the file otherwise.

One important aspect of GestureDataRecorder is that it keeps recording sessions very short. As such, there’s no fear of running out of memory while storing data in activityData. That also means it’s not a big deal if something goes wrong while recording and you need to throw out some data — it’s never much more than a minute’s worth. Shorter sessions are also easier on your test subjects — it’s probably a bit much to ask someone to shake their phone for an hour straight, but doing lots of tiny sessions isn’t so bad.

However, when working with longer lasting activities, where data collection takes several minutes or more, you don’t want to risk having to throw away too much data. In that case, you should write your data out to disk periodically rather than at the end of the session. You should also consider making your app more robust, by saving data when the app gets interrupted from things like incoming phone calls, for example.

You haven’t enabled motion updates just yet, but eventually the app will receive them in the form of CMDeviceMotion objects. Add the following method to ViewController to process them:

func process(data motionData: CMDeviceMotion) {
  // 1
  let activity = isRecording ? currendActivity : .none
  // 2
  let sample = """
  \(sessionId!)-\(numberOfActionsRecorded),\
  \(activity.rawValue),\
  \(motionData.attitude.roll),\
  \(motionData.attitude.pitch),\
  \(motionData.attitude.yaw),\
  \(motionData.rotationRate.x),\
  \(motionData.rotationRate.y),\
  \(motionData.rotationRate.z),\
  \(motionData.gravity.x),\
  \(motionData.gravity.y),\
  \(motionData.gravity.z),\
  \(motionData.userAcceleration.x),\
  \(motionData.userAcceleration.y),\
  \(motionData.userAcceleration.z)
  """
  // 3
  activityData.append(sample)
}

This method creates samples for your dataset from CMDeviceMotion objects. Here’s how it works:

1. You label each sample with the activity it represents. This line checks to see if there is an activity being recorded or if this data is arriving in-between activities. In the latter case, you label it as ActivityType.none. The current activity is set from within the starter code after the app announces the activity to the user.
2. Here you create one big string representing a single data sample. It includes a session ID, the current activity and the sensor readings extracted from motionData, all separated by commas.
3. This line appends the string to activityData. The entire array gets saved to disk later, when the recording session ends.

Along with the session ID and the activity type, you’re saving 12 different values at each moment in time. These were chosen because they seem like they could be relevant to the task at hand. However, you might not use all of them when you train your model.

But it’s a good idea to record as much data as you can, because it gives you more options later when building your model. You can always remove data you don’t need, but there’s no way to go back to these moments and record additional data — adding features requires a new data collection effort.

Notice the session ID gets created by combining sessionId, which is a timecode created when recording starts, and the number of which recording the user is currently doing. That means that each time a user runs the app, they’ll choose between creating one, two or three sessions, even though to the user it will seem like just one session.

Why is that important? You’ll be using Turi Create’s activity classification API, and it currently requires a few things when training. (Comments from its developers on GitHub seem to indicate they would like to make it more flexible in the future.) First, it doesn’t like super short sessions. Without going into detail here, you’ll want your sessions to be at least as long as 20 predictions worth of data.

Note: This is one area where Create ML differs from Turi Create. Create ML expects each file to contain an uninterrupted sequence of data demonstrating a single activity. To use it you’ll need to break up your recordings into multiple files, each containing a single sample sequence.

So if you plan on predicting once per second, for example, sessions should be at least 20 seconds long. It doesn’t need to be exact, but sessions much shorter than that may not work well.

Secondly, Turi Create seems to prefer a lot of sessions. So instead of fewer, longer sessions, this app opts for creating more, shorter ones. Note however that sessions do not need to contain just a single activity. In fact, the sessions for this app will each contain two activities — the gesture itself, as well as a period of none data recorded before the gesture. In your own apps you can record any number of activities within a single session, but labeling them like this was an easy way to get more sessions with fewer actual user recordings.

You’ve got a method to process CMDeviceMotion objects, but you still need Core Motion to send them. Add the following to ViewController to enable device motion updates:

func enableMotionUpdates() {
  // 1
  motionManager.deviceMotionUpdateInterval =
    1 / Config.samplesPerSecond
  // 2
  activityData = []
  // 3
  motionManager.startDeviceMotionUpdates(
    using: .xArbitraryZVertical,
    to: queue,
    withHandler: { [weak self] motionData, error in
      // 4
      guard let self = self, let motionData = motionData else {
        let errorText = error?.localizedDescription ?? "Unknown"
        print("Device motion update error: \(errorText)")
        return
      }
      // 5
      self.process(data: motionData)
  })
}

Here’s what’s going on in the method above:

1. Use samplesPerSecond that you defined earlier to set how often motionManager sends updates to your app. In this case, you’re setting it to update every 0.04 seconds, or 25 times per second.
2. Set activityData to an empty array. The project starter code calls this function each time the user starts a new recording session — this line ensures each session starts with a fresh array.
3. This line instructs motionManager to start sending device motion updates, passing a block to execute on queue for each update. The using parameter tells Core Motion to use .xArbitraryZVertical as the device position relative to which the device’s attitude values should be reported. Check out CMAttitudeReferenceFrame’s documentation (https://apple.co/2RNdTT5) for the available options.
4. This guard statement ensures the callback received motion data. If not, you log an error message if one is available. If you find yourself getting many errors, then you may need a more robust solution here. For example, receiving too many errors in a row could trigger the session to stop and discard the data.
5. Call process, which you added earlier, to extract features from the sensor data and append them to activityData.

In this app you use Core Motion’s device motion API. CMMotionManager also allows you to access accelerometer, gyroscope and magnetometer data directly, but the device motion API is often a better choice. Data directly from the sensors is often quite noisy and requires some preprocessing to smooth it out. But the good folks at Apple have already worked out some nice preprocessing steps and do them for you if you access the device motion data instead. Another nice touch — it separates acceleration due to the user from acceleration due to gravity, which makes the motion represented by the data easier to decipher.

However, if you ever want raw data from those sensors, CMMotionManager provides APIs that match that of device motion. So deviceMotionUpdateInterval, startDeviceMotionUpdates, etc., become accelerometerUpdateInterval, startAccelerometerUpdates, and so on. Similar methods exist for each sensor.

Note: There are also versions of startDeviceMotionUpdates, startAccelerometerUpdates, etc. that take no parameters. These methods quietly update properties on the CMMotionManager, such as deviceMotion and accelerometerData. For some apps, it makes sense to use these methods instead of the ones that take parameters, and then poll the properties directly when you want sensor data.

Now that you’ve defined enableMotionUpdates, find the comment that reads // TODO: enable Core Motion inside the Utterances.sessionStart case in speechSynthesizer, and add a call to your new method there:

case Utterances.sessionStart:
  // TODO: enable Core Motion
  enableMotionUpdates()
  queueNextActivity()

Most of the timing in GestureDataRecorder actually comes from logic in speechSynthesizer. The app’s AVSpeechSynthesizer calls this function whenever it finishes uttering a phrase, and the app uses the finished utterance to determine what to do next. In the case of the sessionStart message, it enables motion updates and calls queueNextActivity to get the recording started.

You’ve started motion updates, so you’ll need to stop them at some point. Add the following method to ViewController to do that:

func disableMotionUpdates() {
  motionManager.stopDeviceMotionUpdates()
}

This function tells motionManager to stop sending motion updates. Add a call to it inside the following case statement in speechSynthesizer:

case Utterances.sessionComplete:
  disableMotionUpdates()
  ...

This statement executes after the recording session completes. You disable the motion updates and then the rest of the case statement saves the data to a file.

Collecting some data

Now go collect some data, ideally from multiple people. Invite your friends over, serve some nice canapés and make it a phone shaking party. If your friends are anything like my kids, they’ll be willing to record data at least once before losing interest.

Note: If you don’t hear any sound coming from your device, make sure to turn off the mute switch on the device.

Keep in mind, performing activities incorrectly while recording data will reduce your model’s performance. That’s because you aren’t manually labeling things, so you’ll end up with mislabeled sequences in your dataset.

In the next section you’ll see how to get rid of mislabeled data, but it’s much better to avoid recording it in the first place. That’s why GestureDataRecorder presents a confirmation window at the end of each recording session — it gives you the chance to discard data without saving it if you know something went wrong during the session.

Any files GestureDataRecorder saves will be accessible from the Files app on your iPhone, and inside the File Sharing area in iTunes. This works because the starter project’s Info.plist includes the keys Application supports iTunes file sharing and Supports opening documents in place, both with values of YES.

Get any data you’ve collected from the device(s) and onto your computer, and store the files in one of the following three folders, all within the notebooks folder of the resources you downloaded: data/train, data/valid or data/test.

These folders hold the files from which you’ll create the three datasets you’ll use when building your model: train, validation and test. You’ll read more about why later, but try not to store data collected from one person in more than one of these folders. You should put data from most people in data/train, while putting data from about 10% of your users in each of the other two folders. If you end up recording data from only one person — be honest, it was just you, right? — it’s probably best to put it in data/train.

Note: The device’s orientation affects the data you collect. For example, imagine holding an iPhone out in front of you and then moving it up and down, side to side, and toward and away from you. Sensor data collected while doing so would be different if the phone was held in portrait or landscape (including variations based on home button position), with the screen facing toward or away from you, to the left, right, up, down or some angle in between. The gravity fields you stored are enough to determine orientation — that’s actually how iOS knows when to rotate your app’s UI — so your model can learn to identify activities in any of these situations. However, you’ll need to provide plenty of training data to cover all the possibilities well enough for it to recognize them.

For your own projects, you can handle this in one of three ways: Instruct users to position their devices a specific way and accept the model may not work well if they fail to do so, collect a much larger dataset that includes data from devices in all probable orientations, or apply a preprocessing step that transforms values into a known orientation. The projects in this chapter settle for the first option.

Analyzing and preparing your data

So you’ve got some data. You’ve collected it yourself or acquired it from elsewhere, but either way, your next step is to look at it. Don’t try reading every number — that way lies madness — but do some analysis to see exactly what you’re working with.

You want to ensure there aren’t any problems that might ruin the models you try to build.

So what are you looking for? Here are a few things to consider:

• If you didn’t create the dataset yourself, it’s important to see what’s there.
• Mislabeled data. Data is often labeled manually and mistakes are common.
• Poorly collected data. Sometimes mistakes are made while recording, such as misplaced sensors, incorrectly followed instructions, etc.
• Source errors. Sometimes the data source introduces errors, such as a damaged or malfunctioning device reporting bad data. And datasets made by people often contain data entry mistakes.
• Incorrect data types. For example, strings where there should be numbers.
• Missing values. It’s common for some rows to have values missing. You’ll need to decide how to handle those — remove such rows or insert reasonable values. The choice depends on your project, and there are many options for how to fill the values if you go that route. For example, you might use that feature’s mean, median or mode value, or perhaps calculate a new value based on values from nearby rows.
• Outliers. Some variation is required to make a good dataset, but there are cases when a few samples may be too rare to be worth including in your dataset. Training with them can confuse the model, reducing its overall performance, and it’s sometimes better to accept that there are some things your model just won’t handle.

Note: You don’t have to remove such samples — you may very well want your model to support them. But it’s something to consider.

You’ll work with Python for the rest of this and the next chapter, so no more Xcode for a while. You’ll also need Juptyer and Turi Create, so if you don’t already have an environment that includes these from earlier in the book, then create one now using the file at projects/notebooks/turienv.yaml. If you’re unsure how to do so, take a look at Chapter 4, “Getting Started with Python & Turi Create.”

From here on out, we’ll assume your environment with Turi Create is named turienv, so keep that in mind when you see it mentioned.

Open up Terminal and, before you get started, activate the turienv environment:

conda activate turienv

Next, launch Jupyter from within your turienv environment.

jupyter notebook

Create a new notebook in the notebooks folder of the chapter resources. Or if you’d prefer to follow along in a completed notebook, you can open notebooks/Data_Exploration_Complete.ipynb instead.

Get started by entering the following code in a cell and running it with Shift+Return:

%matplotlib inline
import turicreate as tc
import activity_detector_utils as utils

This gives you access to the turicreate package as well as some helper functions provided in activity_detector_utils.py, which you can find in the notebooks folder. The first line is what’s known as a “magic” and it tells Jupyter to display any Matplotlib plots inside the notebook instead of in separate windows.

Now run the following code to load your datasets:

train_sf = utils.sframe_from_folder("data/train")
valid_sf = utils.sframe_from_folder("data/valid")
test_sf = utils.sframe_from_folder("data/test")

Here you use the sframe_from_folder function from activity_detector_utils.py to load your datasets. It takes the path to a folder — given here relative to the notebooks folder in which your notebook resides — and attempts to parse all the CSV files it finds there.

We’ve provided enough data to make the project work, but hopefully you’ve used GestureDataRecorder to collect some more. If so, whatever files you’ve added to these folders get loaded here as well.

Note: If you reuse utils.sframe_from_folder in your own projects, you’ll need to modify it slightly — it currently contains some project-specific details.

After running that cell, the variables train_sf, valid_sf and test_sf will be Turi Create SFrame objects, which are data structures designed to work efficiently with structured data, such as huge tables of numbers collected from an iPhone’s motion sensors.

These three SFrames contain the data you’ll use for your training, validation and test sets, respectively. Take a peek at some samples by running the following code:

train_sf.head()

This displays the first 10 rows of the dataset, along with their column names. These names were assigned in sframe_from_folder but could also have come from the CSV files directly.

The following image shows an example of some output from head, edited slightly to fit here:

Note: If you’ve included your own data in any of these datasets, your results may vary from those shown here. This is true for all the screenshots in this section.

Notice how there is a column named userId. This was added inside sframe_from_folder — the values are derived from the names of your data files. This only works if your files each contain data from just one user, and their names are prefixed with the user’s ID followed by a hyphen (-).

For example, all data read from a file named “bob-data.csv” would be assigned a userId value of “bob.” You could have stored the user ID in each row when you were collecting the data, but saving each user’s data into separate files keeps them smaller and makes them easier to organize. Either way, it’s important to know the source of your data — you’ll see why later.

Here’s another thing about head‘s output — the values in the activity column are all 0. That’s nothing to worry about — it’s just because you’re only looking at the first few rows, which represent less than one second of activity. But what does 0 even mean?

Inside GestureDataRecorder, you stored activity types as numeric values. Turi Create can deal with that just fine, but we humans sometimes interpret words more easily than numbers.

To convert those integers into something more readable, enter the following code in a cell and run it:

# 1
activity_values_to_names = {
  0 : 'rest_it',
  1 : 'drive_it',
  2 : 'shake_it',
  3 : 'chop_it'
}
# 2
def replace_activity_names(sframe):
  sframe['activity'] = sframe['activity'].apply(
    lambda val: activity_values_to_names[val])
# 3
replace_activity_names(train_sf)
replace_activity_names(valid_sf)
replace_activity_names(test_sf)

This replaces the numeric activity values in your datasets with the names of the gestures they represent. Here’s how it works:

1. You create a dictionary that maps numeric activity values to strings. These strings were chosen arbitrarily, but they should describe the values clearly — that’s the whole point of replacing them, right?

   Also note, the app you write later uses these values, too, so you’ll need to modify code there if you change these strings.

2. You use the activity column’s apply function to run a lambda function on the value in each row. Lamdba functions are similar to closures in Swift. This one replaces the column’s integers with their corresponding strings from the dictionary. SFrame columns are represented by SArray objects, so check out that class in the Turi Create class if you’d like to see what’s available. You define this line as a function just to make the next lines cleaner.

3. You call replace_activity_names for each of your dataset SFrames.

After running this cell, you’ve modified your datasets to make them easier to interpret, which you can see by calling train_sf.head() again:

Note: You certainly could have stored these strings directly when you created the files in GestureDataRecorder, saving yourself the trouble of changing them now.

However, using integers conserves a bit of disk space. And more importantly, it gave you the chance to see an example of modifying some data in an SFrame, which you might want to do while preparing future datasets.

It’s helpful to plot your data to examine it, so run the following code to look at your test set:

utils.plot_gesture_activity(test_sf)

Here you call plot_gesture_activity from inside activity_detector_utils.py. It uses Matplotlib to display an SFrame’s contents as a line chart. The following image shows the plot generated when you run that code:

Note: The plots shown in this chapter may be difficult to read, especially in the black-and-white printed version. They are all from notebooks/Data_Exploration_Complete.ipynb — you are encouraged to open it in Jupyter to get a better look at these plots as well as several others not included here.

There’s too much data in this plot to see much detail. But even at this zoomed-out scale, it’s already clear there are distinct patterns present here.

With the plot_gesture_activity helper function, you can plot data for a single activity by specifying its name. The following example would show data just for the drive_it gesture:

utils.plot_gesture_activity(test_sf, activity="drive_it")

And you can zoom in on chunks of data by specifying a slice of the dataset, like so:

utils.plot_gesture_activity(
  test_sf[11950:12050], activity="drive_it")

The following three plots were created using code similar to the line above, showing slices of 100 samples for each of the three gestures in the test set:

The actual values aren’t important in these plots. The important thing to notice is how each gesture appears as a clearly discernable pattern. It certainly seems like we should be able to recognize when a user performs these gestures, but imagine trying to write your own algorithm to do it using if/else statements — it might be pretty difficult! But don’t worry — machine learning makes it much easier.

Removing bad data

Now you’ll see one way to find and remove errors from your dataset. If you run the code suggested earlier to plot all the drive_it activity data in the test set, you’ll see a plot something like the one on the next page.

While much of this data looks similar, some of it stands out as different. Particularly, the last two blocks of activity seem odd. The following code looks at a small section in the second one of those areas:

utils.plot_gesture_activity(
  test_sf[22200:22300], activity="drive_it")

Remember, these specific slice numbers might not be the same in your dataset, but hopefully you can come up with values to find a slice within this section of the data.

This produces the following output:

If you compare this to the examples you plotted earlier, you’ll see it looks more like a shake_it than a drive_it action. It seems someone performed the wrong gesture while recording, essentially mislabeling your data.

The second area of concern is a bit more difficult to see because it mostly looks the same as the good data. But if you look closely you may notice an area of green at the top of the data — green that you don’t see in any of the other drive_it data. The following code zooms in on this area and plots only a few features:

utils.plot_gesture_activity(
  test_sf[21200:21500], activity="drive_it",
  features=["gravX", "gravY", "gravZ"])

This call uses another one of plot_gesture_activity‘s optional parameters to specify a list of features to plot. So rather than showing all the data in this slice, it shows just the data for the device’s gravity readings. The following image was made using code similar to the line above (with some slight adjustments to help with formatting).

The plot on the left shows a 100 sample sequence from the suspicious looking area, and the plot on the right shows a 100 sample sequence similar to the majority of the drive_it data:

These plots show similar readings for gravity along the X and Y axes. The scale is slightly different for gravity along the Y axis, but the two plots are still basically the same. However, the gravity readings along the Z axis seem to be quite different. The patterns are the same, but the values are negative in the left example and positive in the right one. This indicates the user was not holding the phone in the correct orientation while performing the motion — the screen was facing up instead of down.

Both of these sessions contain data that will only serve to confuse your model, reducing its performance, so it’s best to remove them from your dataset before continuing. To do so, run the following code, replacing the index values with ones that work for your dataset:

# 1
bad_session_1 = test_sf[21350]["sessionId"]
bad_session_2 = test_sf[22250]["sessionId"]
# 2
test_sf = test_sf.filter_by(
  [bad_session_1, bad_session_2],
  column_name='sessionId', exclude=True)

Here’s what that does:

1. Grabs the session ID from a row in the middle of each area of bad data. Each session contains data for only one activity, so once you know the session ID for one, you know it for all the rows you want to delete.
2. Calls SFrame’s filter_by method to return a new SFrame that excludes any rows where the sessionId column contains the value of either of the bad sessions.

Plotting the test set’s drive_it data again shows the suspect sessions are now gone. The plot isn’t included here to save space, but the Data_Exploration_Complete.ipynb notebook includes this plot if you’d like to compare it to your results.

This section included a few examples demonstrating some things to look for, but you should spend time thoroughly exploring all three of your datasets, both to clean up problems and to better understand your data. And don’t neglect any particular dataset — testing with bad data can be just as problematic as training with it.

Note: The erroneous data you removed from the test set all comes from one file: notebooks/data/test/bad-drive-it-data.csv. You can safely remove that file if you don’t want to go through this exercise again.

Optional: Removing non-activity data

What about motions that have nothing to do with gestures? You know, all those sensor readings that arrive between the gestures? Take a look at that data by plotting the rest_it activity. Here’s how you do so for the test set:

utils.plot_gesture_activity(test_sf, activity="rest_it")

This plots all samples in the test set labeled as rest_it, which means data that is not a gesture. Here are the results:

Unlike with the gestures you plotted earlier, the resting data shows no clear pattern. That makes sense — users can do whatever they want between gestures, so there are basically an infinite number of possible sequences that could appear with this label.

Depending on how similar the resting and activity data are, a model might have trouble learning to classify them both well. In those cases, it often helps to increase the size of your dataset set. However, in many cases — such as this one — the model will learn to recognize both resting and activities. This is probably because the sequences related to the other gestures are so much more distinct. That is, it will likely learn to classify the other gestures well, and then learn that anything else is resting. It will get some samples wrong — users sometimes perform the gestures while GestureDataRecorder is recording rest data, essentially adding mislabeled data to your dataset — but the juxtaposition of the messy resting data and the patterned gestures should make the model even more confident about its gesture predictions.

For this app, train with all your data, including the resting samples. However, you’re encouraged to try making another model that excludes the resting data to see which you prefer. The results might vary depending on exactly what your datasets look like.

If you ever want to try removing that data, you can do so with the following code:

train_sf = train_sf.filter_by(
  ["rest_it"], 'activity', exclude=True)
test_sf = test_sf.filter_by(
  ["rest_it"], 'activity', exclude=True)
valid_sf = valid_sf.filter_by(
  ["rest_it"], 'activity', exclude=True)

Much like how you removed the bad sessions, this would create new SFrames that do not contain any samples whose activity value was rest_it.

Balancing your classes

After you are satisfied you’ve cleaned your data, there’s one final thing you should check: How many examples of each class do you have? Run the following code to count the examples in each dataset:

utils.count_activities(train_sf)
utils.count_activities(valid_sf)
utils.count_activities(test_sf)

Here you call count_activities, another helper function defined in activity_detector_utils.py. It displays a table showing how many sessions are present for each activity, both per user and total.

The following shows the counts for the datasets we provided:

Here you can see that each dataset contains the same three gestures, and no gesture is represented more than any other within a specific dataset. Users within a dataset are represented equally as well. For example, each of the training set’s two users supplied 50% of the training data. Things are looking great! You won’t always have such perfectly balanced datasets, but you want them to be as well balanced as possible. If any gesture or user is overrepresented in the training set, your model may bias itself toward those samples. But unbalanced validation or test sets can be a problem, too, because they’ll skew your evaluation results, making it more difficult to judge your model.

Note: The rest_it activity takes up half of each dataset — you might want to remove some of those samples to bring it in line with the other gestures, but it wasn’t a problem when training the model included with the book.

The dataset included in the resources contains 216 actions for training, 24 for validation and 27 for testing. It’s not a lot of data, but it’s as much as the author’s family was willing to put up with collecting. :[ Still, it’s a reasonable balance, with about 80% of your data for training, and around 10% each for validation and testing.

Once you’re convinced your datasets are good to go, run the following code to save the cleaned up SFrames for later use:

train_sf.save('data/cleaned_train_sframe')
test_sf.save('data/cleaned_test_sframe')
valid_sf.save('data/cleaned_valid_sframe')

The save method lets you save SFrames in several different formats, such as CSV and JSON. Here you’re using a format that creates the given folder and stores various binary files in it. It’s convenient because it’s smaller and loads faster than the others, but feel free to use any format you like. And remember, you still have your original files, so you can always start over if you decide you don’t like something about your cleaned data.

Note: Turi Create has many options for data exploration and manipulation, as do Pandas and NumPy. And it provides methods to convert to and from the data structures used by these other libraries, so if there’s something you prefer to do in one package over another, you can freely move back and forth. It’s a good idea to spend some time looking through the documentation for these various frameworks to see what’s available, but don’t try to learn everything all at once — as you do more with machine learning, you’ll continue to discover new things about it and all these supporting frameworks, too.

Key points

• Core Motion provides access to motion sensors on iOS and WatchOS devices.
• When building a dataset, prefer collecting less data from more sources over more data from fewer sources.
• Inspect and clean your data before training any models to avoid wasting time on potentially invalid experiments. Be sure to check all your data — training, validation and testing.
• Try isolating data from a single source into one of the train, validation or test sets.
• Prefer a balanced class representation. In cases where that’s not possible, evaluate your model with techniques other than accuracy, such as precision and recall.

Where to go from here?

You have a bunch of motion data sequences organized into training, validation and test sets. Now it’s time to make a model that can recognize specific gestures in them. In the next chapter, you’ll use Turi Create to do just that.