15. Natural Language Transformation, Part 1

Written by Alexis Gallagher

The previous chapter showed you how to use Apple’s Natural Language framework to perform some useful NLP tasks. But Apple only covers the basics — there are many other things you might like to do with natural language. For example, you might answer questions, summarize documents or translate between languages.

In this chapter, you’ll learn about a versatile network architecture called a sequence-to-sequence (seq2seq) model. You’ll add one to the SMDB app you already built, using it to translate movie reviews from Spanish to English, but the same network design has been used for many types of problems, from question answering to generating image captions. Don’t worry if you didn’t already make SMDB — we provide a starter project if you need it. But you can forget about Xcode for a while — seq2seq models require a lower-level framework, so you’ll work with Python and Keras for most of this chapter.

Getting started

Some of the Keras code in this project was initially based on the example found in the file examples/lstm_seq2seq.py inside the Keras GitHub repository github.com/keras-team/keras. This chapter makes stylistic modifications, explains and expands on the code, and shows how to convert the models you build to Core ML and use them in an app.

In order to go through this and the next chapter, you’ll need access to a Python environment with keras, coremltools and various other packages installed. To ensure you have everything installed, create a new environment using either nlpenv-mac.yml or nlpenv-linux.yml, which you’ll find in projects/notebooks. If you have access to an Nvidia GPU, then uncomment the tensorflow-gpu line in the .yml file to greatly increase training speed.

Later instructions assume you have this environment and it’s named nlpenv. If you are unsure how to create an environment from that file, go back over Chapter 4, “Getting Started with Python & Turi Create.”

Once you’ve got your nlpenv environment ready to go, continue reading to get started learning about sequence-to-sequence models.

The sequence-to-sequence model

Inside the chapter resources, you’ll find a text file named spa.txt in projects/notebooks/data/. This file comes originally from manythings.org at http://www.manythings.org/anki/, which provides sentence pairs for many different languages. These pairs were culled from an even larger dataset provided by the Tatoeba Project at www.tatoeba.org.

The file contains lines that look like this:

Each line has an English sentence — they aren’t all one-word long like in the image — followed by one possible Spanish translation of that sentence. You’re going to use this data to train a neural network to ingest Spanish text, like “¡Corre!”, and translate it into English text, like “Run!”

Note: As you can see in the image of samples from the data file, the same phrase may appear multiple times with different translations. If you were training a model to translate from English to Spanish, then this would likely confuse it, possibly forcing it to learn only one of the translations. However, your model will translate from Spanish to English, and there are far fewer duplicate Spanish phrases in the file, so it shouldn’t be an issue.

Encoder-decoder models

There are multiple ways to accomplish this task. The network architecture you’ll use here is called a sequence-to-sequence, or seq2seq, model. At it’s most basic level, it works like this:

The seq2seq model consists of two networks operating together; these are called the encoder and the decoder.

The encoder processes some input — in this case, Spanish text — and produces some vector of output values that captures the essence of the input. The decoder then processes the encoder’s output and produces its own output — here, English text — which represents its interpretation of the information captured by the encoder.

In other words, a lot of the magic happens in that intermediate vector of values that passes from the encoder to decoder. What’s the right way to think about it? One intuition you can use is to think of that intermediate vector of values as the meaning of the text, independent of language.

While this intuition may be helpful, it is also wise not to take it too literally. A word like “meaning” suggests your model is doing more than it is. Does it understand meaning in the way that, say, a person does? And when you “train” a model and it “learns” to predict, has it done anything like what we mean when we say a person has learned? Of course not. The trap is that these words are evocative because they suggest analogies to what people do. But they are merely analogies.

Such evocative analogies are like friends: it is a good idea to have many of them, so that you always have a few different, competing perspectives on an issue.

So, more prosaically, you might also think of this encoder-decoder model as acting like a file compression algorithm. You start with an original file — the encoder’s input — and compress it into some new format, which is the encoder’s output. Later, you can uncompress that formatted data to recover the contents of the original file; this is essentially what the decoder does. But, this is a “lossy” compression scheme, like JPEG. Just as a low quality JPEG image compression will lose details of the image and keep the gross features, here we are losing the the details of the input text, such as the exact wording, and keeping the meaning. We are also decoding it into a different language along the way.

Finally, from a deep-learning perspective, you can think of this as a bit like the transfer learning you used in the computer vision chapters. There, you started with a pre-trained network and passed inputs through it to extract some set of features as output. Those features are really just the output activations from a specific layer in the network. Then you trained a new model to accept those output values as inputs and produce some new output, like a healthy/unhealthy label prediction.

The seq2seq model works in a similar way, with the encoder acting as the feature extractor and the decoder acting as the classification model. The difference here is that you’re going to build and train the encoder and decoder together from scratch.

Seq2seq in depth

Digging a little deeper, the seq2seq model works with sequences both for inputs and outputs. That’s where it gets its name — it transforms a sequence to another sequence. To accomplish this, the encoder and decoder usually both rely on recurrent layers — specifically, this chapter uses the LSTM layer introduced in the sequence classification chapter.

These LSTMs aren’t shown in these diagrams, but keep in mind that the boxes labeled Encoder and Decoder each represent neural networks that include such layers.

The following image gives some more details about how your model will work, showing how it will predict its first output character “H” when translating “Hola.” to “Hello.”:

As you can see, the encoder will process its input as a sequence of individual characters. Each character is labeled in the image with its timestep so it’s clear what order the model sees the input. After ingesting the entire sequence, only then will it pass its output on to the decoder. In other words, the encoder and the decoder each work one sequence at a time, rather than one token a time.

But how is the information actually passed from the encoder to the decoder? You learned in the sequence classification project how LSTMs maintain state that lets them keep track of information between timesteps within a sequence. The decoder will take advantage of that fact and use the final state from the encoder’s LSTM layer as the initial state for its own LSTM layer. In other words, not only does the decoder never see the input sequence fed into the encoder; it also never sees how the encoder responded to early tokens in that input sequence. It only sees the final state of the encoder. Along with that state, the decoder will also take as input a special START token.

The decoder will then produce a single character as output. That is its first output character.

Note: This chapter’s model uses individual characters as inputs and outputs, and as such may use the terms “character” and “token” interchangeably. However, seq2seq models do not need to work at the character level. You’ll see how to use full-word tokens in the next chapter, and find out about some other options, too. So keep in mind: The images of characters in this chapter apply to any size tokens.

The encoder is no longer involved after predicting the first character. As you can see in the following image, the decoder continues predicting the rest of its output sequence by passing its own output character back into itself as its next input character, in what is effectively a loop:

The arrows pointing down in the above image, going directly from Decoder block to Decoder block, represent the output state from the decoder’s LSTM layer. At each timestep after the first, the decoder uses the output state from the previous timestep as its new initial state. Not shown here is the initial state used to process the START token, which comes from the encoder. Likewise, each timestep after the first takes as input the previous timestep’s output token. This process continues until the decoder produces a special STOP token, or until you stop it yourself if you want to limit the length of its output sequences.

Teacher forcing

That is how inference works. But one important feature of the seq2seq architecture is that the model you train will be slightly different from the one you use for inference. During training, your model will actually process each sample like this:

Individual timesteps aren’t shown in order to simplify the diagram, but these sequences are still processed one token at a time. The important thing to notice is how, in training, the decoder pictured above receives as input the encoder’s output and the full English translation of the encoder’s input surrounded by START and STOP tokens.

At each timestep, the decoder takes a single input token and produces a single output token. As you saw earlier, the trained decoder uses its own output from one timestep as its input for the next one, but during training you’ll always input to each timestep what the output should have been at the previous timestep.

The decoder learns to produce a specific character as output when it sees the START token in conjunction with a specific encoder state. From there, it learns to associate its own internal LSTM states with each new character to produce the next one, eventually learning to produce the entire sequence. Ensuring each timestep starts with the correct values — rather than what the decoder actually outputs while training — greatly increases the speed at which a recurrent network trains. This technique is known as teacher forcing.

There’s one last subtlety visible in the diagram above: there’s no START token in the target output. Why not? The target outputs you use to calculate the loss during training don’t include the START token, because the decoder should only ever learn to produce what follows it in the sequence.

Phew! That was a lot of preliminary information, but hopefully getting the idea down first will make the rest of the chapter easier to follow. There are still quite a few details left to discuss, so let’s get started.

Prepare your dataset

First, you need to load your dataset. Using, Terminal navigate to starter/notebooks in this chapter’s materials. Activate your nlpenv environment and launch a new Jupyter notebook. Then run a cell with the following code to load the Spanish-English sequence pairs:

# 1
start_token = "\t"
stop_token = "\n"
# 2
with open("data/spa.txt", "r", encoding="utf-8") as f:
  samples = f.read().split("\n")
samples = [sample.strip().split("\t")
           for sample in samples if len(sample.strip()) > 0]
# 3
samples = [(es, start_token + en + stop_token)
           for en, es in samples if len(es) < 45]

Now, samples contains almost 100,000 sequence pairs. Here’s how you set it up:

1. You define constants here indicating the tab and newline characters will act as the decoder’s START and STOP tokens, respectively. You can assign anything as the START and STOP tokens, but they must not appear elsewhere in any output sequences.
2. You read each line of the data file, then split them around the tab character to create a list of English-Spanish sentence pairs. Notice how it only processes rows that include more than whitespace — this avoids accidentally adding bad entries for empty rows in spa.txt, but it’s certainly not robust file processing. For example, this would still add bad entries for rows that don’t include exactly one tab.
3. This line loops over the list you just created and swaps the order of the elements, creating a list of tuples with the Spanish phrases first. It also adds a START token at the beginning of each English phrase and a STOP token at the end.

Notice you only kept pairs where the Spanish sentence is less than 45 characters long. You’ll read more about how sequence length affects things in the section on training your model, but for now just know we chose 45 based on the length of the Spanish sentences in the SMDB app.

You can display some of the list to be sure things look as expected:

Note: The two lines you wrote to create samples use a Python construct called list comprehension. If you aren’t familiar with this syntax, it’s an optimized way to create a list by iterating over a collection. The anatomy of a comprehension is [a for b in c]. It creates a new list filled with items made by calling the expression a with each item b from the collection c. Optionally, you can add a condition to limit which items from c you process, like this: [a for b in c if d] where d is some conditional expression involving b. You should get comfortable with this syntax if you plan on using Python since it’s extremely common — plus the Python interpreter runs these more efficiently than for loops that build lists with append.

In and out of vocabulary

If you’ve followed along with the book thus far, then you already know it’s best to have separate training, validation, and test sets when building machine learning models. Keras can randomly select samples from your training data to use for validation when you train your model, but you won’t rely on that here.

That’s because you will need to do some special processing on the validation set in order to setup your vocabulary. As you might guess, a sequence model’s vocabulary is simply the model’s set of allowed tokens, which in your case means the set of possible characters.

If you picked the validation set at random, then it might contain characters that never appeared in your training set. To avoid this, and to handle this situation in general, you are going to find the vocabulary and use it to preprocess sequences fed to our model, replacing any unsupported characters with out-of-vocabulary (OOV) tokens.

So for this model, you must separate the data into training and validation sets yourself. Run a cell with the following code to do so:

from sklearn.model_selection import train_test_split

train_samples, valid_samples = train_test_split(
  samples, train_size=.8, random_state=42)

This uses scikit-learn’s train_test_split to randomly assign samples from your dataset into training and validation sets, with 80 percent going towards training. Providing a random_state is optional, but specifying 42 here generates the same datasets we used when training our models.

Before you can deal with OOV tokens, you need to know what’s in the vocabulary. You’ll train your model to recognize a specific set of input tokens (the input vocabulary) and to produce values from a specific set of output tokens (the output vocabulary). But don’t worry: Even with a limited set of tokens, it can ingest and produce an infinite number of different sequences.

Run the following code in your notebook to gather the unique tokens in your dataset to act as your model’s vocabulary:

# 1
in_vocab = set()
out_vocab = set()

for in_seq, out_seq in train_samples:
  in_vocab.update(in_seq)
  out_vocab.update(out_seq)
# 2
in_vocab_size = len(in_vocab)
out_vocab_size = len(out_vocab)

Here’s how you built the input and output vocabularies for your model:

1. You loop over every sample in the training data and store each Spanish or English character in its corresponding vocabulary set. You call update instead of add on the sets so that it adds the individual characters rather than the entire strings. update identifies a string argument as iterable, and iterates over the characters, treating them as individual elements to be added to the set.
2. When you define your Keras model, you’ll need to specify how many tokens each vocabulary contains — you’ll learn why later when you read the discussion about one-hot encoding — so you grab those sizes, here. If you check you’ll see in_vocab_size is 101 and out_vocab_size is 87.

Now, in_vocab contains every character present in the Spanish sequences from your training set, while out_vocab includes every character from the English sequences. If you’re curious to see what you’re working with, you can display the vocabulary by running a line like the following:

print(sorted(in_vocab))

Using sorted isn’t necessary but it makes the list more readable. You’ll see the input vocabulary consists of the following characters:

[' ', '!', '"', '$', '%', "'", '(', ')', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '¡', '«', '°', 'º', '»', '¿', 'Á', 'É', 'Ó', 'Ú', 'á', 'è', 'é', 'í', 'ñ', 'ó', 'ö', 'ú', 'ü', 'ś', 'с', '—', '€']

And running a similar line for out_vocab displays these characters:

['\t', '\n', ' ', '!', '"', '$', '%', "'", ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '°', 'á', 'ã', 'è', 'é', 'ö', '‘', ''', '₂', '€']

It looks like the training data includes a good selection of characters, but notice how there are differences between the input and output vocabularies. This isn’t surprising; after all, these are vocabularies for two different languages.

However, it’s important to understand: This vocabulary defines every character your model will ever handle. That means it will never know what to do with an input character not found in in_vocab, and will never produce a sentence using a character not in out_vocab.

For example, notice the Spanish set contains parenthesis but the English one does not. You know parenthesis should be valid characters in English, but your model will never produce an output that includes them, no matter how many might appear in a given input sequence.

Now that you know which tokens your model can handle, any other tokens are considered OOV. There are different ways to handle OOV tokens, but none work perfectly and it’s an open area of research. For this model, you’ll take the most basic approach and remove OOV tokens from any inputs before processing them. For the validation set — and test set, if you had one — you’ll need to remove them from both the inputs and target outputs.

Note: This chapter’s model processes sequences at the character level, so removing OOV tokens should not remove much information from a sequence. In the next chapter, where you’ll learn about working with full word tokens, dealing with OOV tokens becomes much more difficult.

When you use your model in iOS, you’ll preprocess each sequence before passing it to the model. But you’ll test against your entire validation set every epoch while training, so it’s more efficient to preprocess all of them at once beforehand. Run a cell with the following code to remove OOV tokens from the validation set:

tmp_samples = []
for in_seq, out_seq in valid_samples:
  tmp_in_seq = [c for c in in_seq if c in in_vocab]
  tmp_out_seq = [c for c in out_seq if c in out_vocab]
  tmp_samples.append(
    ("".join(tmp_in_seq), "".join(tmp_out_seq)))
valid_samples = tmp_samples

Here, you iterated over all the validation samples and created new sequences that only include characters that are found in the appropriate vocabulary set — in_vocab for the Spanish inputs and out_vocab for the English outputs.

Note: If you compare your validation samples before and after removing the OOV tokens, you might find little or no difference. That’s just happenstance based on what random split you got earlier from train_test_split, but the premise here still holds true; your model cannot handle tokens it doesn’t see while training, so you need to do some preprocessing when working with sequences containing OOV tokens. This is true while testing and at runtime in your iOS app.

You aren’t quite ready to train a model yet, but you’ve prepared enough to at least define the architecture. The next section walks you through that process.

Build your model

In this section, you’ll use Keras to define a seq2seq model that translates Spanish text into English, one character at a time. Get started by importing the Keras functions you’ll need with the following code:

import keras
from keras.layers import Dense, Input, LSTM, Masking
from keras.models import Model

You’ll read more about these as you use them in the upcoming cells. But to summarize, you are importing Keras, the high-level machine learning library, and then some basic buildings blocks which Keras provides for defining a model.

Note: When you execute this cell and the following one, you may see a small parade of warnings in your notebook, such as “FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’” or “The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead”. Never fear. These are warnings, not errors, and they are safe to ignore. They are coming not from your code but from inside the TensorFlow machine learning library, as Tensorflow (which is developed by Google) complains that Keras (which is also developed by Google) is not using TensorFlow in the newest possible way. Google develops so fast, it cannot even keep up with itself. Remarkable.

Building the encoder

Now, let’s start building the model. Run the following code in your notebook to define the encoder portion of the seq2seq model:

# 1
latent_dim = 256
# 2
encoder_in = Input(
  shape=(None, in_vocab_size), name="encoder_in")
# 3
encoder_mask = Masking(name="encoder_mask")(encoder_in)
# 4
encoder_lstm = LSTM(
  latent_dim, return_state=True, recurrent_dropout=0.3,
  name="encoder_lstm")
# 5
_, encoder_h, encoder_c = encoder_lstm(encoder_mask)

It takes only these five lines to define your encoder, but there’s a lot to say about them:

1. The latent_dim variable defines how many nodes your LSTM uses to represent each recurrent step internally. This, in turn, defines the size of the feature vectors used to store the encoding produced by the encoder. In machine learning, we refer to the process of converting an input into a set of features as mapping it into latent space. So latent_dim defines the number of dimensions in the encoder’s latent space as 256. This value directly affects the size and speed of your model: Larger values produce larger, slower models. However, models with more dimensions can technically learn more complicated relationships so they might produce better results. That is, if you can train them — the larger your model, the more data you need to train it. There are a lot of things involved with finding the right value here, but we chose this value arbitrarily. We encourage you to experiment with other values after you’ve finished the chapter.

2. Here, you make an Input layer, to which you’ll pass batches of sequences during training. You specify two dimensions — None and in_vocab_size. The None tells Keras that you want it to support variable length inputs. That is, you’ll be able to pass sequences of any length, which is good because sentences can come in any length. The in_vocab_size tells it how large the vectors are that hold each character — the same size as how many different possible tokens exist in the vocabulary. Essentially, these two dimensions tell Keras that each input sequence will consist of any number of characters, where each character is represented by a vector of length in_vocab_size. This will be more clear after you read about one-hot encoding later.

3. You pass the input layer through a Masking layer. This layer tells the network to ignore any timesteps in the sequence that are filled with zeros. It’s common to train on batches of samples rather than one sample at a time, mainly to take advantage of the parallelism of the GPU. Batches are stored as tensors, and tensors have specific dimensions. But your model can handle variable length sequences, right? So how do you store variable length sequences in a fixed dimension tensor? You’ll see the details later, but you’ll end up padding the end of shorter sequences with zeros — essentially meaningless timesteps added to make all the sequences in a batch the same size. The Masking layer tells Keras not to train on those padding timesteps. This layer isn’t absolutely necessary, but see the upcoming Note explaining a bit more about the choice to use a Masking layer here.

4. You create an LSTM layer to process the input sequence, passing latent_dim to define the size of its output. Setting return_state to True makes the LSTM layer output its hidden and cell states along with its regular output; you’ll use these as the initial state for the decoder. Refer back to the discussion of the seq2seq model if you don’t recall the role of the encoder’s output states. You also use the recurrent_dropout parameter to add some dropout between timesteps in the LSTM. This can help your model generalize better to unseen data later. There are many parameters available to the LSTM initializer, so you should explore the Keras documentation as you dig deeper into the subject.

5. Here, you actually connect the masking layer as the input to the LSTM. Keras’s functional API allows you to create the function encoder_lstm as an object in the previous line and then call it here with encoder_mask as its input. It will return three values: The first is the actual output from the LSTM layer, but you don’t actually need that for a seq2seq model so you ignore it by assigning it to an underscore. The others are the LSTM’s hidden and cell states, which you store in encoder_h and encoder_c, respectively. These two outputs will each hold vectors of length latent_dim containing whatever information the encoder extracts from its input. (Go back over the model description in the sequence classification chapter if you need a reminder about LSTMs and their hidden and cell states.)

If you look around online, you’ll likely come across examples of networks that do not use Masking layers but still pad sequences to store them in batches. These still work, but there’s a subtle issue with them: These networks consider the padding tokens to be just as important as the real tokens. This causes three problems.

First, they appear to have lower loss values when training, but that’s only because such a large percentage of the tokens they test against are the same padding token and it learns to output a lot of them. The second, more important issue, is that it dilutes the information stored by the encoder because the weights are modified by essentially meaningless tokens. This reduces the power of your encoder. If that isn’t clear now, don’t worry, it will be by the end of the chapter. Finally, for these networks to produce their expected results, they require the same padding at inference time. For example, if you trained a model to translate “Hola.” as part of a batch with 10-character-long sequences, you’d have five padding tokens. Your model would likely then only produce the correct result during inference if you tried to translate it with those same five padding tokens, because the same letters without the padding tokens would get encoded differently and likely translate to something else.

Note: One drawback of using Masking layers is that, although Core ML 3.0 does support them (meaning they are available for iOS 13), the latest version of Apple’s Core ML conversion tool (release 3.3) does not support them when converting from all Keras models. It won’t be a problem for this project — you’ll see why later — but it means you won’t be able to perform batched translations in iOS because it’s unlikely that you can create a batch of sequences of equal length without padding.

Building the decoder

Your decoder definition will look quite similar to that of your encoder, with a few minor but important differences. Run a cell with the following code now:

# 1
decoder_in = Input(
  shape=(None, out_vocab_size), name="decoder_in")
decoder_mask = Masking(name="decoder_mask")(decoder_in)
# 2
decoder_lstm = LSTM(
  latent_dim, return_sequences=True, return_state=True,
  dropout=0.2, recurrent_dropout=0.3, name="decoder_lstm")
# 3
decoder_lstm_out, _, _ = decoder_lstm(
  decoder_mask, initial_state=[encoder_h, encoder_c])
# 4
decoder_dense = Dense(
  out_vocab_size, activation="softmax", name="decoder_out")
decoder_out = decoder_dense(decoder_lstm_out)

Here are the differences from what you wrote for the encoder:

1. The decoder starts with an Input layer passed through a Masking layer, just like the encoder did. The only difference is that it’s sized for the output vocabulary.

2. It uses an LSTM layer just like the encoder, but it adds an additional dropout parameter. Unlike recurrent dropout, which works only on values passed between timesteps, this dropout value affects the original input passed in from the encoder. It’s just an additional bit of regularization that might improve your model’s ability to generalize to unseen data.

3. You connect the masking layer as the input to the LSTM, and set the LSTM’s initial state to the output states from the encoder’s LSTM. The decoder won’t need to access its hidden or cell states while training — it already has access to them internally – so you ingore them by assigning them to underscore variables. You grab the rest of the outputs in decoder_lstm_out.

4. You pass the output from the LSTM into the decoder’s final, fully connected layer. It has a node for each token in the output vocabulary and uses a softmax activation to produce a probability distribution over them. You’ll use this to predict the next character in the output sequence.

Connecting the encoder and decoder

With the encoder and decoder defined, run the following code to combine them into a seq2seq model:

# 1
seq2seq_model = Model([encoder_in, decoder_in], decoder_out)
# 2
seq2seq_model.compile(
  optimizer="rmsprop", loss="categorical_crossentropy")

Here’s how you combine the encoder and decoder to prepare them for training:

1. You construct a Keras Model with the input layers for both the encoder and decoder, as well as the decoder’s output layer. Since you have more than one input layer, you combine them in a list.

2. The RMSProp optimizer uses an adaptive, per-parameter learning rate. We won’t go into details about it here; it’s enough to know that it’s a good choice for training recurrent neural networks. And as you’ve seen with the various other classification models you’ve created throughout this book, categorical cross entropy is a good loss function when your outputs represent probability distributions across classes; in this case, the classes are the tokens in the output vocabulary.

Run the following line to display a summary of how the layers in your model connect:

seq2seq_model.summary()

Here’s what you’ll see:

This is one reason it’s a good idea to name your layers: It makes it easier to read summaries like this one. You can see which layers are connected and how data flows through the network —encoder_in connects to encoder_mask which connects to encoder_lstm, and decoder_in connects to decoder_mask, which connects to decoder_in, along with the second two outputs from encoder_lstm. Then decoder_lstm‘s output connects to decoder_out, which produces the model’s output.

The summary’s a bit misleading because it doesn’t display the details of all the outputs for the LSTMs. Notice the Output Shape column for encoder_lstm shows the tuple (None, 256) with an opening bracket “[” before it and a comma “,” after it, followed by the start of a second tuple showing (None,. You can see a similarly incomplete tuple for decoder_lstm. These were meant to show lists of outputs, but there’s a glitch with the output displayed for the summary function.

Train your model

So far, you’ve defined your model’s architecture in Keras and loaded a dataset. But before you can train with that data, you need to do a bit more preparation.

Numericalization

OK, full disclosure: Neural networks can’t process text. It might seem like a bad time to bring this up, well into a chapter about natural language processing with neural networks, but there it is. Remember from what you learned elsewhere in this book: Neural networks are really just a bunch of math, and that means they only work with numbers. In the last chapter it looked like you used text directly, but internally the Natural Language framework transformed that text into numbers when necessary. This process is sometimes called numericalization. Now you’ll learn one way to perform such conversions yourself.

Run the following code in your notebook in order to create dictionaries that map text to or from integers:

# 1
in_token2int = {token : i
                for i, token in enumerate(sorted(in_vocab))}
# 2
out_token2int = {token : i
                 for i, token in enumerate(sorted(out_vocab))}
out_int2token = {i : token
                 for token, i in out_token2int.items()}

Here’s how you created your conversion maps:

1. This dictionary comprehension — like list comprehensions, but they create dict objects — maps each character in in_vocab to a unique integer. Sorting the vocabulary before assigning the integer values makes the values easier for you to reason about — “A” comes before “B”, etc. — but it isn’t actually necessary for the mapping to work.
2. Here, you create two dictionaries, one that maps each character in out_vocab to a unique integer, and one that maps back from integers to characters. You only needed one mapping for the Spanish characters because the model only translates from Spanish, not to it. But remember, you defined the model to use teacher forcing, which requires you to feed the target English phrases into the decoder along with the output from the encoder. That means you need to convert English characters to and from integers.

Now you’ve got Python dictionaries you can use to easily convert Spanish characters into integers, as well as convert English characters both to and from integers. For example, calling out_token2int['A'] returns the value 25, and calling out_int2token[25] gets you back 'A'.

You’ve got a way to turn text into numbers — so far, so good. But here’s some more full disclosure: You won’t want to use those numbers for machine learning, either.

One-hot encoding

While neural networks require numeric input, they don’t want just any numbers. In this case, the numbers are stand-ins for text. But if you use these values as is, it will confuse the network because it appears as though some ordinal relationship exists that doesn’t. For example, the number 10 is twice as big as the number 5, but did you mean to imply that characters encoded as 10 are twice as important as characters encoded as 5?

No, you didn’t, but there’s no way for a machine learning algorithm to know that. Larger values for a given feature affect the model’s calculations more than smaller ones. The essential problem is that you want the numerical value to indicate a particular token, not to measure the magnitude of a token. At worst this will ruin your model, but at best it will slow down the training process as the model attempts to undo those implied relationships.

There are a couple ways to resolve this encoding problem. You’ll see a different option in the next chapter, but here you’ll convert each token’s integer value into what’s called a one-hot encoding.

One-hot encoding a feature involves replacing each value with a vector the same length as the number of all possible values. These vectors are filled with zeros in all but one position, which contains a one. Imagine you wanted to use just the days Monday through Friday as possible inputs to a model.

The following image shows what it looks like to one-hot encode those values:

As you can see, each vector is has a length of five — the same length as the number of possible values. And each of these vectors is filled with zeros except for a single one in a unique location. This essentially turns one feature — the day of the week — into five mutually exclusive features representing boolean flags indicating their absence or presence.

For your seq2seq model, there are in_vocab_size possible input values, and out_vocab_size possible output values. Rather than pass a sequence of integers to the model, you’ll pass a matrix wherein each row represents a single character, one-hot encoded as a vector the same size as the corresponding vocabulary.

Batching and padding

To keep things in more manageable chunks, you’ll split the logic to one-hot encode training batches into two functions. The first will create appropriately sized NumPy arrays filled with zeros, and the second will place ones into those arrays at the correct locations to encode the sequences.

Run the following code to import NumPy and define the first of those two functions:

import numpy as np

def make_batch_storage(batch_size, in_seq_len, out_seq_len):
  enc_in_seqs = np.zeros(
    (batch_size, in_seq_len, in_vocab_size),
    dtype=np.float32)
  dec_in_seqs = np.zeros(
    (batch_size, out_seq_len, out_vocab_size),
    dtype=np.float32)
  dec_out_seqs = np.zeros(
    (batch_size, out_seq_len, out_vocab_size),
    dtype=np.float32)

  return enc_in_seqs, dec_in_seqs, dec_out_seqs

You declare make_batch_storage to take three parameters, which — along with the vocabulary sizes — define the dimensions of the storage tensors it creates. It returns three NumPy arrays, sized to hold batch_size sequences that are each in/out_seq_len characters long, with each character being in/out_vocab_size wide. Specifying dtype=np.float32 keeps NumPy from defaulting to 64-bit floats.

The in_seq_len and out_seq_len parameters to make_batch_storage deserve more explanation. Consider this: If you create a batch with random samples from your training set, are all their sentences guaranteed to have the same length?

The answer is no, but consider trying to store the following input sequences together in a batch:

Each sequence has a different length, but tensors need fixed dimensions. That means each item in a batch needs to fill the same amount of space in the tensor, regardless of how many tokens are in the actual sequences. To accomplish this, you’ll use special padding tokens at the end of each input sequence shorter than in_seq_len, and at the end of each output sequence shorter than out_seq_len. So those same examples would look more like this in a batch:

This image shows padding as crossed-out boxes, but you could use anything that isn’t already in the vocabulary. These padding tokens are the ones that the masking layer will instruct the network not to weight too excessively while learning. For this project, you’ll pad sequences with zero-filled vectors. Since make_batch_storage returns a batch filled with zeros, that means it’s basically pre-padded and there’s no need to add new padding-specific tokens to the vocabularies.

However, while the figure shows a batch as a two-dimensional array (a matrix) to clarifying padding, recall that because of one-hot encoding every batch will in fact be a three-dimensional array. One dimension’s length is batch_size since its index specifies the batch. Another dimension’s length is the maximum sequence length, since its index specifies position in a sequence. And the third dimension’s length is the size of the vocabulary (the set of possible characters), since this is the one-hot encoding vector and its index specifies which character is represented. So every batch is a three-dimensional array, a cube of ones and zeroes.

Now, run the following code to define the second of the two functions for one-hot encoding — the one that actually encodes the batch:

def encode_batch(samples):
  # 1
  batch_size = len(samples)
  max_in_length = max([len(seq) for seq, _ in samples])
  max_out_length = max([len(seq) for _, seq in samples])

  enc_in_seqs, dec_in_seqs, dec_out_seqs = \
    make_batch_storage(
      batch_size, max_in_length, max_out_length)
  # 2
  for i, (in_seq, out_seq) in enumerate(samples):
    for time_step, token in enumerate(in_seq):
      enc_in_seqs[i, time_step, in_token2int[token]] = 1

    for time_step, token in enumerate(out_seq):
      dec_in_seqs[i, time_step, out_token2int[token]] = 1
    # 3
    for time_step, token in enumerate(out_seq[1:]):
      dec_out_seqs[i, time_step, out_token2int[token]] = 1

  return enc_in_seqs, dec_in_seqs, dec_out_seqs

Here’s how this function one-hot encodes a list of samples as a training batch:

1. You find the batch size and the lengths of the longest input and output sequences in the batch, then use those values to create empty tensors to store the batch data.
2. You loop over the samples and one-hot encode each of their sequences into enc_in_seqs, dec_in_seqs and dec_out_seqs as appropriate. That is, for each token in a sequence, you place a 1 in its corresponding location within a vector of zeros.
3. Notice how when you populate dec_out_seqs, you use out_seq[1:] to skip the START token in the sequence. That’s because this tensor contains the outputs your model learns to predict. It will never predict the START token because it’s always given that as its first input, and it learns to predict the next token.

The only thing left to do is one-hot encode your datasets, then you’re ready to train. You could encode them all at once and then let Keras randomly sample batches —  that works and it’s how many people do it. However, if you ensure batches contain only sequences of similar lengths, you can minimize the amount of padding necessary.

Why does that matter? The Masking layers you added to seq2seq_model ensure the padding doesn’t affect your training results, but it still takes time to process those steps in the sequences. With this model and dataset, minimizing padding makes training go nearly twice as fast.

The chapter resources include a file called seq2seq_util.py in the notebooks folder. It defines a class called Seq2SeqBatchGenerator that you’ll use to generate properly randomized batches while minimizing necessary padding. We won’t go over the details of its code here; read through the code and comments in the file if you’re interested. Run the following code in a new cell to import the Seq2SeqBatchGenerator class and create instances of it for your training and validation datasets:

from seq2seq_util import Seq2SeqBatchGenerator

batch_size = 64
train_generator = Seq2SeqBatchGenerator(
  train_samples, batch_size, encode_batch)
valid_generator = Seq2SeqBatchGenerator(
  valid_samples, batch_size, encode_batch)

You’ll use these objects while training your model to create batches with the given batch size. Seq2SeqBatchGenerator needs a way to one-hot encode each batch of samples, so you pass it the encode_batch function.

Training with early stopping

Warning: Running the following cell will take considerable time. Expect it to run for multiple hours even with a GPU. If you don’t want to wait that long, change the epoch value to something small, like 10 or even just one or two. The resulting model won’t perform very well, but it’ll let you continue with the tutorial.

Now, run a cell with the following code to train your model:

# 1
from keras.callbacks import EarlyStopping
early_stopping = EarlyStopping(
  monitor="val_loss", patience=5, restore_best_weights=True)
# 2
seq2seq_model.fit_generator(
  train_generator, validation_data=valid_generator,
  epochs=500, callbacks=[early_stopping])

There are many parameters available to control training models. Here’s how you trained this one:

1. Keras lets you add callbacks to monitor and modify the training process between epochs. You create an EarlyStopping callback that will stop training if the validation loss stops improving. The patience parameter tells it to keep training for that number of epochs even if the loss isn’t improving, and restore_best_weights tells it to use the model weights from the epoch with the best value rather than the last epoch.

2. You call fit_generator on the seq2seq_model object you created earlier, passing it the two Seq2SeqBatchGenerator objects you just made. This function trains the model, using those objects to create batches for training and validation. You told it to train for 500 epochs, but the early_stopping callback should stop it long before it reaches that number.

The model provided with the chapter had its best validation loss of 0.5905 at epoch 179. If you try training your own model, you’ll get different results, but probably not too different. The specific values aren’t really important, here, just that you understand the architecture so you can apply it to your own problems in the future.

Now that you have a trained model, continue on to the next section to learn how to perform inference with seq2seq models, which requires some changes from what you did for training.

Inference with sequence-to-sequence models

The model you’ve trained so far isn’t actually useful for inference — at least, not in its current form. Why is that? Because the decoder portion of the model requires the correctly translated text as one of its inputs! What good is a translation model that needs you to do the translations?

But don’t worry: You won’t have to throw out all your hard work. The model has learned something useful, you just have to access it a different way.

Assembling an inference model

First, separate the encoder and decoder into two models. Keras makes this easy. You declare a new Model and pass it the input and output layers you want to use, like this:

inf_encoder = Model(encoder_in, [encoder_h, encoder_c])

Running a cell with the above code creates a new encoder model called inf_encoder, short for “inference encoder”. It uses the same input layer and encoder state output layers that you created earlier: encoder_in, encoder_h and encoder_c. Keras maintains a graph of layer connections, so it automatically adds to this Model any existing layers necessary to connect these inputs and outputs. This means you’re actually using the same LSTM and Masking layers that you trained as part of seq2seq_model, too. In essense, inf_encoder accesses only the encoder portion of your original trained model.

You can run the summary function to see what Keras actually built:

inf_encoder.summary()

Which produces the following output:

It shows encoder_in feeds into encoder_mask, which feeds into encoder_lstm, and the final layer outputs two length 256 vectors. These are all layers you created earlier to train seq2seq_model, just repurposed in a new Model object.

You’ve isolated the encoder, so now do the same for the decoder. Run the following code:

# 1
inf_dec_h_in = Input(shape=(latent_dim,), name="decoder_h_in")
inf_dec_c_in = Input(shape=(latent_dim,), name="decoder_c_in")
# 2
inf_dec_lstm_out, inf_dec_h_out, inf_dec_c_out = decoder_lstm(
  decoder_in, initial_state=[inf_dec_h_in, inf_dec_c_in])
# 3
inf_dec_out = decoder_dense(inf_dec_lstm_out)
# 4
inf_decoder = Model(
  [decoder_in, inf_dec_h_in, inf_dec_c_in],
  [inf_dec_out, inf_dec_h_out, inf_dec_c_out])

This looks a lot more complicated than what you did for the encoder, but it’s really only a little more complicated. Let’s go through it step by step:

1. You create two new Input layers with the same shapes as the encoder’s two outputs — inf_dec_h_in and inf_dec_c_in. (The names are getting longer so we’re abbreviating more heavily now.) Remember how in the full seq2seq model, the decoder’s initial state came directly from the encoder. For this new model, you’ll provide the initial state programmatically, and you’ll use these inputs to do it.

2. Here, you connect the new inputs as the initial state for your trained LSTM layer, along with the original decoder_in that lets you give the decoder a one-hot encoded sequence. You passed entire sequences to decoder_in while training seq2seq_model, but during inference you’ll only give it the most recently predicted character for each new prediction. You also grab the LSTM’s state outputs as inf_dec_h_out and inf_dec_c_out, rather than discard them like you did in seq2seq_model. You’ll pass these outputs from one prediction as inputs for the next one.

3. This line connects the new output from the LSTM layer to the dense layer you trained earlier, giving you a new output layer for the decoder model.

4. Finally, you build the new decoder Model. Notice it includes outputs from different layers in the network — the dense layer’s character probabilities and the LSTM’s output states. There’s nothing about neural networks that says the outputs can only come from the last layer!

Once again, the summary function shows you how Keras connected the model’s layers:

inf_decoder.summary()

Which outputs this:

As you can see, decoder_in, decoder_h_in and decoder_c_in all flow into decoder_lstm, which in turn leads to decoder_out. Once again, the summary doesn’t display all the values in the Output Shape column, but you get the idea.

You’re going to write a function that translates sequences using the separate encoder and decoder models you just created. But before you do, run the following code to define a few useful constants:

max_out_seq_len = max(len(seq) for _, seq in samples)
start_token_idx = out_token2int[start_token]
stop_token_idx = out_token2int[stop_token]

The values for these three constants are all specific to this project, but you’ll need to think about them in your own projects as well. Here’s what they’re for:

• max_out_seq_len: This value defines the maximum length of a translation. Ideally, your decoder will predict a STOP token at some point, but this specifies how long you’re willing to wait for one before giving up. You could choose any value here — the model has no limit to the length of sequence it can process — but this line uses the length of the longest English sentence in the training set. Why? Because you know the model never got any practice creating sequences longer than this, so it seems like as good a place as any to call it quits.

• start_token_idx: This is the integer encoding of the START token. You’ll need to know this to signal the decoder to start translating a new sequence.

• stop_token_idx: This is the integer encoding of the STOP token. You’ll need to know this because it’s how the decoder signals to you that it’s done translating a sequence.

Running inference

With those constants defined, you’re ready to actually use your models to translate text. Define the following function in your notebook. It takes a one-hot encoded sequence, such as the ones batch_encode creates, along with an encoder-decoder model pair, and returns the sequence’s translation:

def translate_sequence(one_hot_seq, encoder, decoder):
  # 1
  encoding = encoder.predict(one_hot_seq)
  # 2
  decoder_in = np.zeros(
    (1, 1, out_vocab_size), dtype=np.float32)
  # 3
  translated_text = ""
  done_decoding = False
  decoded_idx = start_token_idx
  while not done_decoding:
    # 4
    decoder_in[0, 0, decoded_idx] = 1
    # 5
    decoding, h, c = decoder.predict([decoder_in] + encoding)
    # 6
    encoding = [h, c]
    # 7
    decoder_in[0, 0, decoded_idx] = 0
    # 8
    decoded_idx = np.argmax(decoding[0, -1, :])
    # 9
    if decoded_idx == stop_token_idx:
      done_decoding = True
    else:
      translated_text += out_int2token[decoded_idx]
    # 10
    if len(translated_text) >= max_out_seq_len:
      done_decoding = True

  return translated_text

This logic drives the translation process, so let’s go over it carefully:

1. The function receives a NumPy array containing a one-hot encoded sequence — one_hot_seq — and passes it to encoder‘s predict function to process it. The encoder model passed to this function should output its LSTM’s h and c states as a list, which you save as encoding.
2. Next, you create a NumPy array to store a one-hot encoded character you’ll give the decoder. Remember from the diagrams earlier in the chapter, you’ll call the decoder repeatedly with the most recently predicted character as input.
3. These variables keep track of the translation so far and control the decoding loop. You’ll set decoded_idx to the one-hot encoding index of the decoder’s most recently predicted character. However, you initialize it to the START token’s index, because you trained the decoder to start decoding sequences using the encoder’s output and START as the initial token.
4. The loop starts by one-hot encoding the current character, indicated by decoded_idx. Important: This must initially equal the index of the START token.
5. It calls predict on the decoder, passing in the most recently predicted character and the most recent cell state. Remember, the first time through this loop, decoder_in contains the START token and encoding contains the outputs from the encoder.
6. Next, you save the decoder’s output h and c states as encoding. You’ll pass these back to the decoder as inputs to predict when predicting the next character.
7. Finally, you clear out the one-hot encoded index, so now decoder_in contains all zeros once again.
8. The decoder doesn’t return a character. Instead, it returns a probability distribution over all possible characters. So which one do you choose? Here you take a greedy approach and always choose the character predicted with the highest probability. This isn’t necessarily the best approach, which you’ll read about later.
9. Given the index of the predicted character, you check to see if it’s the STOP token. If so, the translation is complete and you stop the loop. Otherwise, you convert the index to text and add it to the translation.
10. The final check ensures the loop doesn’t go on forever by stopping it if the translation length reaches its limit.

Now you’re ready to see what your model can do. The notebook folder’s seq2seq_util.py includes a function that loops over a list of sample tuples and displays the predictions along with the correct translations for comparison. It takes as arguments the encoder and decoder models, along with a function to one-hot encode sequences and a decoding function like the one you just wrote.

To see how your model performs, use code like the following, which displays your model’s output for the first 100 samples in the validation set:

from seq2seq_util import test_predictions

test_predictions(valid_samples[:100],
                 inf_encoder, inf_decoder,
                 encode_batch, translate_sequence)

Let’s look at some of the results we got on the validation set when training the model included with the chapter resources. Yours will likely be different but should be in the same quality range, provided you trained for roughly the same number of epochs.

First, there are several like the following which produced the expected results perfectly. Who needs Google Translate, amirite?

Then there are several like the following, which seem like reasonable translations even if they aren’t exactly what the human translators wrote:

Sadly, there are quite a few where our model basically spit out nonsense, like these:

And, finally, there are some translations like these, that start off looking great and then take horrible turns:

Considering eating children? Laughing at poor Mary’s eyes day in and day out? We’ve created an AI monster!

In all seriousness, it’s pretty amazing that with so little effort you’ve created a piece of software that learned to look at Spanish text — one character at a time — and generate English — again, one character at a time — that consists of properly spelled words, mostly arranged in grammatically correct sentences complete with proper punctuation!

And the fact that it generates text that also sometimes translates between languages correctly? That’s a bit mind blowing.

Let’s save the discussion about model quality until the end of the chapter. For now, go on to the next section to learn how to convert your seq2seq model for use in iOS.

Converting your model to Core ML

So far, you’ve used teacher forcing to train a Keras seq2seq model to translate Spanish text to English, then you used those trained layers to create separate encoder and decoder models that work without you needing to provide them with the correct translation. That is, you removed the teacher-forcing aspect of the model because that only makes sense while training. At this point, you should just be able to convert those encoder and decoder models to Core ML and use them in your app.

But is it ever that easy?

Currently, there are issues with Core ML and/or coremltools (the Python package that converts models into Core ML format), preventing you from exporting the models you’ve made. Don’t worry — this section shows you how to work around each of them and convert your models to Core ML.

Start with the encoder: Run the following code to do a bunch of stuff you shouldn’t have to do, which is all explained after the code block:

# 1
coreml_enc_in = Input(
  shape=(None, in_vocab_size), name="encoder_in")
coreml_enc_lstm = LSTM(
  latent_dim, return_state=True, name="encoder_lstm")
coreml_enc_out, _, _ = coreml_enc_lstm(coreml_enc_in)
coreml_encoder_model = Model(coreml_enc_in, coreml_enc_out)
# 2
coreml_encoder_model.output_layers = \
  coreml_encoder_model._output_layers
# 3
inf_encoder.save_weights("Es2EnCharEncoderWeights.h5")
coreml_encoder_model.load_weights("Es2EnCharEncoderWeights.h5")

Everything you just wrote is to work around a conversion problem you’d otherwise have if you didn’t do these things. Here’s what’s going on:

1. This bit creates a new, untrained encoder model, completely separate from the one you trained. This is necessary to work around two issues you would encounter without it. First, Core ML does not currently support Masking layers, so you need to create a new connection from the Input layer directly to the LSTM to remove the Masking layer from the network. The second issue is a bug that causes the converter to crash when exporting models that contain shared layers. That is, layers used by more than one model. Currently, you’re sharing several layers between your seq2seq model and the encoder and decoder models you made for inference. By creating new Input and LSTM layers, this encoder now contains no shared layers.
2. This line is a bit ridiculous, but when using the versions of Keras and coremltools that we used for this book, the Core ML converter looks for the layers using the name output_layers instead of its actual name, _output_layers. This super-hacky line just adds a new property on the model, using the name the converter expects. Hopefully, this bug will get fixed soon and this will no longer be necessary.
3. Finally, you extract the weights from your original, trained encoder and apply them to the new, untrained one. The load_weights function attempts to match weights by layer names, like “encoder_lstm”, but if the layers don’t have identical names then it will try its best to match them based on the architecture. In the end, your new coreml_encoder_model is separate from the models you trained earlier, but contains the same trained weights so it will produce the same results for a given input.

The coremltools Python package provides converters and other utilities to help get models from various machine learning frameworks into Core ML’s format. Run the following code to use the Keras converter to export your encoder model:

import coremltools

coreml_encoder = coremltools.converters.keras.convert(
  coreml_encoder_model,
  input_names="encodedSeq", output_names="ignored")
coreml_encoder.save("Es2EnCharEncoder.mlmodel")

After importing the coremltools package, you use the Keras converter to create a Core ML definition of your model and save it to disk. Notice the input_names and output_names parameters: These are used by Xcode to name the inputs and outputs in the classes it generates, so it’s a good idea to put something descriptive here. You named them “encodedSeq” and “ignored”, respectively, to indicate the input is a one-hot encoded sequence and the output is unused by the app.

Note: You do not mention the LSTM’s h and c states that you intend to pass from your encoder to your decoder — the converter adds those automatically and currently doesn’t let you change their names. You’ll see the final set of names later in Xcode.

With your encoder exported, it’s time to turn to the decoder. Run the following code to perform the same workarounds to prepare your decoder for export to Core ML:

coreml_dec_in = Input(shape=(None, out_vocab_size))
coreml_dec_lstm = LSTM(
  latent_dim, return_sequences=True, return_state=True,
  name="decoder_lstm")
coreml_dec_lstm_out, _, _ = coreml_dec_lstm(coreml_dec_in)
coreml_dec_dense = Dense(out_vocab_size, activation="softmax")
coreml_dec_out = coreml_dec_dense(coreml_dec_lstm_out)
coreml_decoder_model = Model(coreml_dec_in, coreml_dec_out)

coreml_decoder_model.output_layers = \
  coreml_decoder_model._output_layers

inf_decoder.save_weights("Es2EnCharDecoderWeights.h5")
coreml_decoder_model.load_weights("Es2EnCharDecoderWeights.h5")

This code does for the decoder all the same things you did for the encoder. It makes a new model that mirrors the one you trained but without any Masking or shared layers, performs the output_layers hack, and copies the weights from the trained decoder onto the new one.

Then export the decoder like you did for the encoder:

coreml_decoder = coremltools.converters.keras.convert(
  coreml_decoder_model,
  input_names="encodedChar", output_names="nextCharProbs")
coreml_decoder.save("Es2EnCharDecoder.mlmodel")

Here, you convert the decoder model to Core ML and save it to disk, the same way you did for the encoder. The descriptive names for the input and output will make your iOS code more readable later; they remind you that the model takes as input a single one-hot encoded character, and outputs a probability distribution for a single character.

Quantization

The models you’ve saved are fine for use in an iOS app, but there’s one more simple step you should always consider. With apps, download size matters. Your model stores its weights and biases as 32-bit floats. But you could use 16-bit floats instead. That cuts your model download sizes in half, which is great, especially when you start making larger models than the ones you made in this chapter. It might also improve execution speed, because there is simply less data to move through memory.

This process of reducing the floating point accuracy of parts of a model in order to improve its size and performance is called quantizing a model, and the result is called a quantized model. You might expect that simply throwing away half your numerical precision would ruin the accuracy of the model. But this generally turns out not to be the case. Some have even experimented with quantizing models to 8- or 4-bit floats. The limits of such quantization are still an active research topic. For now, let us stick to 16 bits.

In order to quantize your model, run a cell with the following code to define a function you can use to convert existing Core ML models from 32 to 16-bit floating point weights:

def convert_to_fp16(mlmodel_filename):
  basename = mlmodel_filename[:-len(".mlmodel")]
  spec = coremltools.utils.load_spec(mlmodel_filename)

  spec_16bit = coremltools.utils.\
    convert_neural_network_spec_weights_to_fp16(spec)

  coremltools.utils.save_spec(
    spec_16bit, f"{basename}16Bit.mlmodel")

This takes advantage of functions from coremltools to load an existing Core ML model, convert its weights into 16-bit floats, and then save a new version back to disk. It derives a new filename so it won’t overwrite the original model.

Now, call that function for each of your models to create 16-bit versions:

convert_to_fp16("Es2EnCharEncoder.mlmodel")
convert_to_fp16("Es2EnCharDecoder.mlmodel")

The conversion tools will tell you that it is quantizing the layers. If everything worked, you should now have four model files saved in your notebooks folder: Es2EnCharEncoder.mlmodel, Es2EnCharDecoder.mlmodel, Es2EnCharEncoder16Bit.mlmodel and Es2EnCharDecoder16Bit.mlmodel. It’s nice to keep both versions in case you want to compare their performance, but the rest of this chapter will use the models with 16-bit weights.

Numericalization dictionaries

One last thing: When you use your models in your iOS app, you’ll need to do the same one-hot encoding you did here to convert input sequences from Spanish characters into the integers your encoder expects, and then convert your decoder’s numerical output into English characters.

To ensure you use the correct values, run the following code to save out the numericalization dictionaries you’ve been using:

import json

with open("esCharToInt.json", "w") as f:
  json.dump(in_token2int, f)
with open("intToEnChar.json", "w") as f:
  json.dump(out_int2token, f)

Using Python’s json package, you save in_token2int and out_int2token as JSON files. You’ll use these files, along with the Core ML versions of your encoder and decoder models, in an iOS app in the next section.

Using your model in iOS

Most of this chapter has been about understanding and building sequence-to-sequence models for translating natural language. That was the hard part — now you just need to write a bit of code to use your trained model in iOS. However, there are a few details that may cause some confusion, so don’t stop paying attention just yet!

You’ll continue working on your finished project from the previous chapter, so open it now in Xcode. If you skipped that chapter, you can use the SMDB starter project in this chapter’s resources.

Drag your trained Core ML model files — Es2EnCharEncoder16Bit.mlmodel and Es2EnCharDecoder16Bit.mlmodel — into Xcode to add them to the SMDB project. Or, if you’d prefer to use the larger versions, use the models with the same name minus the “16Bit”. Keep in mind that if you choose not to use the 16-bit versions, you’ll need to remove 16Bit from any code instructions that include it.

Note: If you didn’t train your own models, you can find all the necessary files in the pre-trained folder of this chapter’s materials.

Select Es2EnCharEncoder16Bit.mlmodel in the Project Navigator to view details about the encoder. You’ll see the following, which should remind you a bit of the model you saw in the sequence classification chapters.

As you can see, the summary includes the input value encoderSeq, and the output value ignored, which you specified when you exported the encoder as Core ML. But notice it also includes h and c state vectors for the LSTM; you didn’t specify these but Core ML always adds them automatically for recurrent networks.

The most important thing to point out here is the misleading size shown for encodedSeq. According to this report, it expects an MLMultiArray of 101 Doubles. That’s not entirely untrue; it can accept such an input. However, recall from the one-hot encoding section that you’re storing each character in a sequence as a length 101 vector, so this makes it appear as though your model can only take a single character as input. This is not the case.

The encoder can take a one-dimensional array or a three-dimensional array. You’ll use the second option to provide the entire sequence at once rather than feeding it one character at a time. You’ll go over more details about this when you get to the code.

To be thorough, select Es2EnCharDecoder16Bit.mlmodel in the Project navigator to view the decoder model:

This summary shouldn’t hold any surprises for you. Notice again the encodedChar input claims to be an MLMultiArray with a single dimension. In this case, it’s telling you the truth: You actually will provide inputs to the decoder one character at a time.

With your models in Xcode, you can finally write some code to use them. The first problem you need to solve: How to one-hot encode your inputs?

When you one-hot encode characters in iOS, you’ll need to ensure each character maps to the same integer you used when you trained your model. Fortunately, you saved your conversion mappings as JSON files — esCharToInt.json and intToEnChar.json. Add those files to your Xcode project now.

Note: In the following step, you’ll add some globals to NLPHelper.swift, along with all the global functions you’ve been writing in this and the previous chapter. Rest assured, we aren’t proposing you forget everything you’ve probably learned about avoiding globals. You should continue to organize and encapsulate your own code well, but we chose to structure SMDB this way so that you could see working results quickly without dealing with details of app architecture.

Add the following code to load the mapping files as Swift dictionaries to the top of NLPHelper.swift:

let esCharToInt = loadCharToIntJsonMap(from: "esCharToInt")
let intToEnChar = loadIntToCharJsonMap(from: "intToEnChar")

The two functions you call here are provided in Util.swift. They load the JSON files and convert their contents to the expected data types.

Next, add the following import to the file:

import CoreML

With the conversion maps loaded, add the following function that builds inputs for your encoder model:

func getEncoderInput(_ text: String) -> MLMultiArray? {
  // 1
  let cleanedText = text
    .filter { esCharToInt.keys.contains($0) }

  if cleanedText.isEmpty {
    return nil
  }

  // 2
  let vocabSize = esCharToInt.count
  let encoderIn = initMultiArray(
    shape: [NSNumber(value: cleanedText.count),
            1,
            NSNumber(value: vocabSize)])

  // 3
  for (i, c) in cleanedText.enumerated() {
    encoderIn[i * vocabSize + esCharToInt[c]!] = 1
  }

  return encoderIn
}

Here’s how the function one-hot encodes text for use with your encoder:

1. First, you remove any OOV tokens from the text, and return nil if you end up removing everything.
2. Then you use initMultiArray, a helper function provided in Util.swift, to create an MLMultiArray filled with zeros. Notice the number of dimensions of the array. You might expect two but it’s three because of a quirk of Core ML. The first dimension’s length is the length of the sequence because this dimension represents the sequence itself. The second dimension always has a length of one. This dimension exists only as a side effect of Core ML’s computer-vision-focused design; it doesn’t affect how much space you allocate, but your app will crash without it. The third dimension’s length is the input vocabulary size, since each character needs to be one-hot encoded into a vector of that length.
3. Finally, you loop over the characters in the cleaned text and set the appropriate item in the array to one. You index the multi-dimensional MLMultiArray as if it’s a standard flat array, because that’s how its memory is arranged. And remember it’s filled with zeros, so you only have to worry about where to put the ones and you’ll end up with properly one-hot encoded vectors.

That does it for the encoder’s input. Now add another function, this time to process encoded inputs and produce the initial input for your decoder model:

func getDecoderInput(encoderInput: MLMultiArray) ->
  Es2EnCharDecoder16BitInput {
  // 1
  let encoder = Es2EnCharEncoder16Bit()
  let encoderOut = try! encoder.prediction(
    encodedSeq: encoderInput,
    encoder_lstm_h_in: nil,
    encoder_lstm_c_in: nil)
  // 2
  let decoderIn = initMultiArray(
    shape: [NSNumber(value: intToEnChar.count)])
  // 3
  return Es2EnCharDecoder16BitInput(
    encodedChar: decoderIn,
    decoder_lstm_h_in: encoderOut.encoder_lstm_h_out,
    decoder_lstm_c_in: encoderOut.encoder_lstm_c_out)
}

Here’s how this function produces the input you’ll pass to your decoder model:

1. First, you create your encoder model and pass encoderInput to its prediction function. This call returns an Es2EnCharEncoder16BitOutput object that contains the encoder’s latent state after processing the input sequence.

2. You create a zero-filled MLMultiArray just large enough for a single, one-hot encoded character. Notice you use intToEnChar.count because the decoder has a different vocabulary from the encoder. You leave it as all zeros for now.

3. Finally, you create and return an Es2EnCharDecoder16BitInput object, using the empty MLMultiArray you just built as storage for the encodedChar field. You’ll reuse this object for each character you pass to the decoder, but for now you set its initial state inputs to the states from encoderOut. Remember this is what you did earlier in the Jupyter notebook, passing the input text to the encoder and then using the encoder’s output as the initial h and c states for the decoder.

Before writing the guts of the translation logic, add the following constants in NLPHelper.swift:

let maxOutSequenceLength = 87
let startTokenIndex = 0
let stopTokenIndex = 1

These are the values you found for the similarly named constants in the Jupyter notebook. A quick refresher:

• maxOutSequenceLength: Defines the maximum length of a translation. If the decoder produces this many tokens without predicting a STOP token, you’ll stop translating to avoid an infinite loop.
• startTokenIndex: Int value of START token in intToEnChar. Used for one-hot encoding.
• stopTokenIndex: Int value of STOP token in intToEnChar. Used for one-hot encoding.

The starter project already includes an empty function called spanishToEnglish in NLPHelper.swift, and that’s where you’ll put together everything you’ve added so far to translate reviews. This logic essentially duplicates the translate_sequence function you wrote in Python earlier, but we’ll go over the details again.

Start by adding the following code at the beginning of the function:

// 1
guard let encoderIn = getEncoderInput(text) else {
  return nil
}
// 2
let decoderIn = getDecoderInput(encoderInput: encoderIn)
// 3
let decoder = Es2EnCharDecoder16Bit()
var translatedText: [Character] = []
var doneDecoding = false
var decodedIndex = startTokenIndex

Here’s what you’ve done so far:

1. First, you call getEncoderInput to one-hot encode the input text, and exit the function if that fails.

2. Using getDecoderInput and encoderIn, you create the initial input for the decoder.

3. Then you create the decoder model, along with some variables you’ll use to keep track of the translation’s progress. Notice you initialize decodedIndex to the index of the START token.

Now add the following code, still inside spanishToEnglish but after what you just added and before the return statement:

while !doneDecoding {
  // 1
  decoderIn.encodedChar[decodedIndex] = 1
  // 2
  let decoderOut = try! decoder.prediction(input: decoderIn)
  // 3
  decoderIn.decoder_lstm_h_in = decoderOut.decoder_lstm_h_out
  decoderIn.decoder_lstm_c_in = decoderOut.decoder_lstm_c_out
  // 4
  decoderIn.encodedChar[decodedIndex] = 0
}

You aren’t done writing this while loop yet, so don’t worry about the fact that it currently has no way to stop. Here’s what it does do, so far:

1. The loop starts by one-hot encoding the most recently predicted character, indicated by decodedIndex, and storing it in decoderIn.
2. It calls prediction on the decoder model. Remember, the first time through this loop, decoderIn contains the output state from the encoder that was set in getDecoderInput and its encodedChar is the START token.
3. Next, decoderIn’s h and c states are set to the h and c output states from the call to prediction. These will serve as the initial state when the loop repeats to predict the next character in the translation.
4. Finally, you clear out the one-hot encoded index to ensure decoderIn’s encodedChar contains only zeros.

Now, add the following code, at the end — but still inside — of the while loop you were just writing:

// 1
decodedIndex = argmax(array: decoderOut.nextCharProbs)
// 2
if decodedIndex == stopTokenIndex {
  doneDecoding = true
} else {
  translatedText.append(intToEnChar[decodedIndex]!)
}
// 3
if translatedText.count >= maxOutSequenceLength {
  doneDecoding = true
}

This code extracts the predicted character, as well as stops the while loop when appropriate. Here are some details:

1. Here you use argmax, provided in Util.swift, to find the index of the highest value in the probability distribution returned by the decoder. Remember we mentioned earlier that this greedy approach does not necessarily produce the best results.

2. Check to see if the decoder predicted the STOP token. If so, stop the loop; otherwise, add the token to the translation.

3. This check stops the loop if the translation length reaches its limit. Without this, you run the risk of an infinite loop.

Finally, replace the return nil statement included with the starter code with the following line. It converts the list of Character predictions into a String:

return String(translatedText)

The final version of spanishToEnglish should look like this:

func spanishToEnglish(text: String) -> String? {
  guard let encoderIn = getEncoderInput(text) else {
    return nil
  }

  let decoderIn = getDecoderInput(encoderInput: encoderIn)

  let decoder = Es2EnCharDecoder16Bit()
  var translatedText: [Character] = []
  var doneDecoding = false
  var decodedIndex = startTokenIndex

  while !doneDecoding {
    decoderIn.encodedChar[decodedIndex] = 1

    let decoderOut = try! decoder.prediction(input: decoderIn)
    decoderIn.decoder_lstm_h_in = decoderOut.decoder_lstm_h_out
    decoderIn.decoder_lstm_c_in = decoderOut.decoder_lstm_c_out
    decoderIn.encodedChar[decodedIndex] = 0

    decodedIndex = argmax(array: decoderOut.nextCharProbs)
    if decodedIndex == stopTokenIndex {
      doneDecoding = true
    } else {
      translatedText.append(intToEnChar[decodedIndex]!)
    }

    if translatedText.count >= maxOutSequenceLength {
      doneDecoding = true
    }
  }

  return String(translatedText)
}

With that function done, there’s just one final bit of code you need to write — a helper function to break reviews into sentences. Replace getSentences in NLPHelper.swift with the following:

func getSentences(text: String) -> [String] {
  let tokenizer = NLTokenizer(unit: .sentence)
  tokenizer.string = text
  let sentenceRanges = tokenizer.tokens(
    for: text.startIndex..<text.endIndex)
  return sentenceRanges.map { String(text[$0]) }
}

This uses NLTokenizer from the Natural Language framework, which you explored in the previous chapter. It attempts to divide the given text into sentences and returns them as a list. You’re going to translate reviews one sentence at a time for two reasons: First, you only trained your model on single sentence examples, so it’s unlikely it will ever produce anything other than a STOP token after a sentence-ending punctuation mark. And secondly, your model’s performance degrades as its input sequences get longer. By giving it only one sentence at a time, your model has a better chance to produce reasonable results. More discussion on this later.

Note: While it’s fine for this project, and might be good enough in many other situations, translating sentences individually without maintaining any context between them is unlikely to produce state of the art results. There are many cases where information in one sentence might influence the translation of another one. Still, reduced performance in exchange for simpler models that are easier to train is a reasonable concession, here.

The project’s starter code uses the getSentences and spanishToEnglish functions you just wrote to translate reviews when possible. For each review, code inside ReviewsManager.swift calls translateReview, which does the following:

1. Ensures it’s a Spanish-language review, because that’s all it currently knows how to handle. You could modify it to handle other languages if you train a model that supports them.

2. Calls getSentences to tokenize the review into sentences. This gives the model shorter text chunks to translate, which should improve the results because you only trained the model on short sentences.

3. Trims whitespace from the ends of the tokenized sentences. This is important because your model didn’t train with any extra whitespace and you will get different translations for a sentence if it includes even a single extra space at the end.

4. Translates each sentence with spanishToEnglish, then stores the translation as part of the Review. The logic in ReviewsTableViewController.swift ensures table cells display the translation, along with the original text, for reviews that have one.

So how well does your model actually perform? Build and run the app, go to the By Language tab and choose Spanish from the list.

You should see something like the following results, though they likely won’t be exactly the same unless you are running with the pre-trained model we provided.

Yikes! Even if you don’t know Spanish, it’s pretty clear these translations aren’t very good. Does that mean there’s no hope for this model?

Let’s talk translation quality

Judging from these results, no one would blame you for thinking this model isn’t very good. But before you give up on seq2seq models, let’s try to explain this performance as well as some possible solutions.

First, this chapter described the most basic version of a seq2seq model — the encoder and decoder each consist of just a single LSTM layer. LSTMs don’t stack as well as convolutional layers do, but you can still see improvement using more than one. In the next chapter, you’ll try a slightly different encoder that should improve things a bit.

Secondly, we performed absolutely no hyperparameter tuning. How would it perform if the LSTMs had more units, like 512 instead of 256? What about a different optimizer or learning rate? Tuning the model’s hyperparameters would almost certainly lead to at least slightly better results, even training with this same dataset.

And that brings us to the dataset. There are several reasons why this particular dataset likely won’t lead to great results:

• It’s way too small. To create a reasonable Spanish-to-English translator, you’d need a dataset with many millions — or better yet, billions — of words, which would make training far too resource intensive for this book. Why? Because the possibilities of language are infinite and each individual sample is extremely sparse. That is, each sample sentence you train with covers such a small portion of all the sentences that could exist, so your model learns very little from each sample. Or it learns too much and overfits your training data, which is just as bad. However, remember that if you have a more restricted use case — translating words you might find on common street signs, for example — then you can get away with a small dataset.

• It’s biased toward certain types of phrases, so it likely won’t translate just any sentence. For example, Tom and Mary are the only two names used — they each appear thousands of times, with Tom appearing five times more than Mary. Your model learns what letters are likely to follow other letters, so seeing just these names so many times biases it, especially when dealing with proper nouns. Notice in the image, the model’s translation for the review of the Sound of MusicKit ends up mentioning Tom, and the one for Night of the Living Deadlocks mentions Mary! Those two are everywhere!

Note: To be fair, please keep in mind the dataset you used here was not created for machine learning. It’s actually made up of flashcards meant to help people learning English as a foreign language. However, it’s well formatted and easy to work with so it serves as a good starting point for exploring machine translation.

So what if you had a huge dataset, and you performed rigorous hyperparameter tuning to create the best version of this model possible? Would it be good enough to translate any Spanish text to English? Not exactly, due to some other important issues.

First, there’s one major drawback to encoder-decoder models implemented the way we do here: The encoder processes the entire input sequence, and then outputs a fixed size vector meant to encode it. Why’s that a problem? If you consider that longer sequences contain more information, it means as sequences get longer it gets more difficult to include all their information into that fixed-size vector. You can increase the size of this vector to improve performance to a point, but there will always be some limit to how much information you can store in any fixed-sized space. The best current solution to this problem involves something called attention. We won’t implement it in this book, but we’ll explain the concept in the next chapter.

The second issue with your current implementation involves the greedy algorithm for choosing each character. If you always take the next token with the highest probability, you force the results down well-trodden paths. For example, imagine an English sentence that starts with the letters “Th.” If you just go by probability, the next letter is likely “e” because “The” is such a common word, but maybe this sentence starts with “Though.” Hopefully, the encoder signals to the decoder something about this sentence to make it predict “o” instead of “e”, but there will always be times when the correct result doesn’t end up being the top prediction. The next chapter will discuss a method called beam search that can help improve this situation.

Finally, there’s one more detail about this model that warrants discussion: It tokenizes text at the character level. Is that a good idea? Don’t we lose all sorts of information about words when we look at them as individual letters? Yes we do, but it’s a good way to introduce the topic because it simplifies some details like how we deal with OOV tokens. There are other benefits of character-level models, too, but we’ll save that discussion for the next chapter, where we’ll also point out the changes necessary to work on word-level tokens.

Key points

• Encoder-decoder pairs are powerful and versatile; they’ve been used for many tasks, including translation, image captioning and question answering.

• Encoders map their inputs into a latent space. Decoders map an encoder’s latent space to a desired output. You train them together in an end-to-end process, passing inputs to the encoder and minimizing a loss function against the decoder’s outputs.

• The sequence-to-sequence model architecture is an instance of the encoder-decoder architecture. It takes a sequence as input and produces one as output.

• Use padding tokens to make all sequences in one batch the same length.

• The Keras Masking layer ensures models ignore padding tokens. Without this layer, models will assume the padding tokens are just as significant as all the others.

• Speed up training by using batches of similarly-sized sequences to reduce padding.

• When translating languages, you’ll always need some way to deal with OOV tokens. In this chapter, you dropped them, but the next chapter considers other options.

• You can one-hot encode nominal categorical features, like characters in text, to represent them as numbers without implying any ordinal relationships.

• Greedy algorithms for choosing the decoder’s predicted token do not alway produce the best results. The next chapter briefly describes an alternative.

• Core ML and the coremltools Python package are actively evolving and using them sometimes requires workarounds. For example, coremltools currently cannot convert Keras models based on TensorFlow 1 that include shared layers or masking layers, requiring you to build new encoder and decoder models and set their weights from your trained seq2seq model.

• Reduce your app’s download size by converting your model weights from 32-bit floats down to 16-bit instead.

• In model summaries, Xcode displays sequential inputs as an MLMultiArray big enough for one element in a sequence, but models can actually accept sequences of any length. To pass a sequence to a model, use a three-dimensional MLMultiArray with shape SEQUENCE_LENGTH x 1 x ITEM_SIZE. When one-hot encoding, ITEM_SIZE is the number of possible values, such as the vocabulary sizes used in this chapter.

Where to go from here?

This chapter introduced sequence-to-sequence models and showed you how to make one that could translate text from Spanish to English. Sometimes. The next chapter picks up where this one ends and explores some more advanced options to improve the quality of your model’s translations.