Custom Models with Create ML

MoodTracker: Creating and Evaluating Emotion Classification Models

In this demo, you’ll work on building emotion classification models for the MoodTracker app. The objective is to create three image classifiers: one with default settings using a dataset with two labels, one with all augmentations enabled for the same two-label dataset, and a third using a three-label dataset with all augmentations. This approach will help clarify the effects of augmentations and additional labels on model accuracy.

In the starter folder, you’ll find two folders: one containing training and testing datasets for two emotions (happy and sad), and the other for three emotions (happy, sad, and neutral). You can use these datasets for your project or choose to use your own images.

Building the Image Classifiers

Open Xcode and then select “Xcode” > “Open Developer Tool” > “Create ML” to access the Create ML app. Click “New Document” or navigate to “File” > “New” > “Project”. Select the “Image Classification” template and proceed by clicking “Next”. Name it EmotionsImageClassifier and choose the save location. The Create ML app will now display the three main parts as seen previously.

In the “Main Screen”, select the “Settings” tab, then drag the Training (2 Labels) folder containing the training dataset from the Emotions Data (2 Labels) folder into the Training Data area. Observe that the “Testing Data” area becomes active to accept input. Also, note that the training model will appear under the “Data Sources” area in the left menu.

Keep the “Validation Data” as is, which means it will automatically split the training data to create validation data. Drag the testing dataset folder - Testing (2 Labels) - from the Emotions Data (2 Labels) folder into the “Testing Data” area. Press the “Train” button at the top left of the main area and monitor the progress of the training and testing process. The time required will vary depending on the number and size of images used as well as the configurations of the classifier.

Once training is complete, the app navigates to the evaluation tab, showing the accuracy and results for training, validation, and testing data at the top of the Activity side bar. For this first classifier (default settings with 2 labels), you might see a training accuracy at 100 percent, validation accuracy at 83 percent, and testing accuracy at 77 percent. Before delving deeper into the evaluation, train another classifier with a different configuration to observe accuracy changes.

Press the “+” button next to “Model Sources” in the left menu to create a new classifier. Drag the same training and testing data as used in the first classifier. This time, select “Image Feature Print V2” and enable all augmentation options to diversify the dataset. Click the “Train” button to start training the second model. Monitor the progress and compare the accuracy graphs with those of the first classifier. You might notice that this second model has improved both validation accuracy and test accuracy, achieving 100 percent and 83 percent, respectively. This demonstrates one way to enhance the training process.

Next, repeat the process with the Emotions Data (3 Labels) folder. Create a new classifier using the same steps, but this time drag the training and testing datasets from the Emotions Data (3 Labels) folder into the respective areas. Select “Image Feature Print V2” and enable all augmentation options again to train this third model.

You might observe that this third classifier, which includes the additional neutral label, shows a decrease in accuracy, with training accuracy at 100 percent, validation accuracy at 78 percent, and testing accuracy at 67 percent. This reduction in accuracy likely results from the model’s increased complexity with the additional label. To improve accuracy, consider adding more diverse data to the training set. Achieving near 100 percent accuracy in real-world scenarios is a long and challenging process that often requires significant time and effort.

Note: It’s important to mention that each time you train the model, you might observe variations in accuracy. These differences occur due to the random nature of the training process, especially when dealing with non-deterministic models or when data augmentation is applied. Only in cases where the model is clear-cut and achieves 100 percent accuracy might the results remain consistent across different training attempts. Keep this in mind as you experiment with different configurations and datasets.

Evaluating Model Performance

After training the models, evaluate their performance by checking metrics and testing with real data in the preview section. Open the “Evaluation” tab of the second classifier and press “Testing” to review the results for the testing data. You can see the test accuracy and other statistics, including the lowest precision type.

In the “Evaluation” tab, there are several key metrics used to assess the performance of your model. False Positives occur when the model incorrectly labels a negative instance as positive, while False Negatives happen when the model misses a positive instance, labeling it as negative instead. To evaluate the model’s accuracy, calculate Precision by dividing the number of true positives by the sum of true positives and false positives. Recall is determined by dividing the number of true positives by the sum of true positives and false negatives. Finally, the F1 Score provides a balanced measure of Precision and Recall by calculating their harmonic mean, offering a single metric to assess the model’s accuracy.

Click the “Incorrect” number to filter the images. Create ML displays the incorrect images along with the classifier’s predictions and the correct answers. Review these images to identify areas for improvement in your data. For instance, if many incorrect samples are white and black images, and your training data does not include these, adding such images might enhance accuracy. If you find a certain type of error consistently, it might be worth rechecking to ensure that the images are correctly categorized.

Next, open the “Preview” tab. Drag and drop a few images for both emotions to see a live evaluation of the model with confidence levels. This is a good place to test specific images and identify weak points in your data.

With your model trained and evaluated, you’re now ready to export and integrate it into the MoodTracker app. You’ll cover the export and integration process in the next lesson.