16. Natural Language Transformation, Part 2

Written by Alexis Gallagher

The previous chapter introduced sequence-to-sequence models, and you built one that (sort of) translated Spanish text to English. This chapter introduces other techniques that can improve performance for such tasks. It picks up where you left off, so continue using the same nlpenv environment and SMDB project you already made. It’s inadvisable to read this chapter without first completing that one, but if you’d like a clean starter project, use the final version of SMDB found in Chapter 15’s resources.

Bidirectional RNNs

Your original model predicts the next character using only the characters that appear before it in the sentence. But is that really how people read? Consider the following two English sentences and their Spanish translations (according to Google Translate):

The first five words are the same in the English versions of both sentences, but only the first two words end up the same in the Spanish translations. That’s because the meaning of the word “bank” is different in each sentence, but you cannot know that until you’ve read past that word in the sentence. That is, its meaning comes from its context, including the words both before and after it.

In order to consider the full context surrounding each token, you can use what’s called a bidirectional recurrent neural network (BRNN), which processes sequences in both directions, like this:

The forward and reverse layers themselves can be any recurrent type, such as the LSTMs you’ve worked with elsewhere in this book. However, in this chapter, you’ll use a new type called a gated recurrent unit, or GRU.

GRUs were invented after LSTMs and were meant to serve the same purpose of learning longer-term relationships while training more easily than standard recurrent layers. Internally, they are implemented differently from LSTMs, but, from a user’s standpoint, the main difference is that they do not have separate hidden and cell states. Instead, they only have hidden states, which makes them a bit less complicated to work with when you have to manage state directly — like you do with the decoder in a seq2seq model.

So now you’ll try a new version of the model you trained in the previous chapter — one that includes a bidirectional encoder. The Python code for this section is nearly identical to what you wrote for your first seq2seq model. As such, the chapter’s resources include a pre-filled Jupyter notebook for you to run at notebooks/Bidir-Char-Seq2Seq-Starter.ipynb. Or, you can just review the contents of notebooks/Bidir-Char-Seq2Seq-Complete.ipynb, which shows the output from the run used to build the pre-trained bidirectional model included in the notebooks/pre-trained/BidirCharModel/ folder.

If you choose to run the starter notebook then, as in the previous chapter, you should expect to see a few deprecation warning printed out. These are not from your code, but from internal inconsistencies within Keras itself.

The rest of this section goes over the important differences between this and the previous model you built.

The first difference isn’t out of necessity, but this model uses a larger latent_dim value:

latent_dims = 512

The previous model used 256 dimensions, which meant you passed 512 features from your encoder to your decoder — the LSTM produced two 256-length vectors, one for the hidden state and one for the cell state. GRU layers don’t have a cell state, so they return only a single vector of length latent_dim. Rather than send only half the amount of information to the decoder, the author chose to double the size of the GRUs.

The biggest differences for this model are in the encoder, so let’s go over its definition:

# 1
encoder_in = Input(
  shape=(None, in_vocab_size), name="encoder_in")
encoder_mask = Masking(name="encoder_mask")(encoder_in)
# 2
fwd_enc_gru = GRU(
  latent_dim, recurrent_dropout=0.3, name="fwd_enc_gru")
rev_enc_gru = GRU(
  latent_dim, go_backwards=True, recurrent_dropout=0.3,
  name="rev_enc_gru")
fwd_enc_out = fwd_enc_gru(encoder_mask)
rev_enc_out = rev_enc_gru(encoder_mask)
# 3
encoder_out = Concatenate(name="encoder_out")(
  [fwd_enc_out, rev_enc_out])

This encoder uses a bidirectional RNN with GRU layers. Here’s how you set it up:

1. The Input and Masking layers are identical to the previous chapter’s encoder.
2. Rather than creating one recurrent layer, you create two — one that processes the sequence normally and one that processes it in reverse because you set go_backwards=True. You feed the same masking layer into both of these layers.
3. Finally, you concatenate the outputs from the two GRU layers so the encoder can output them together in a single vector. Notice that, unlike in the previous chapter, here you don’t use the h states and instead use the layer outputs. This wasn’t mentioned before, but that works because the hidden states are the outputs. The reason you used the states for the LSTM was to get at the cell states, which are not returned as outputs like the hidden states are.

As far as the decoder goes, one important difference is in the size of the inputs it expects. We define a new variable called decoder_latent_dim, like this:

decoder_latent_dim = latent_dim * 2

The decoder’s recurrent layer needs twice as many units as the encoder’s did because it accepts a vector that contains the concatenated outputs from two of them — forward and reverse.

The only other differences with the decoder are in the following lines:

decoder_gru = GRU(
  decoder_latent_dim, return_sequences=True,
  return_state=True, dropout=0.2, recurrent_dropout=0.3,
  name="decoder_gru")
decoder_gru_out, _ = decoder_gru(
  decoder_mask, initial_state=encoder_out)

Once again, you use a GRU layer instead of an LSTM, but use decoder_latent_dim instead of latent_dim to account for the forward and reverse states coming from the encoder. Notice the GRU only returns hidden states, which you ignore for now by assigning them to an underscore variable. This differs from the LSTM you used in the previous chapter, which returned both hidden and cell states.

Note: One important detail is that the decoder does not implement a bidirectional network like the encoder does. That’s because the decoder doesn’t actually process whole sequences — it just takes a single character along with state information.

If you run this notebook, or look through the completed one provided, you’ll see a few things. First, this model is much larger than the last one you built — about 5.4 million parameters versus 741 thousand. Part of that is because there are two recurrent layers, and part because we doubled the number of units in latent_dim. Still, each epoch only takes a bit longer to train.

The other thing that stands out is the performance. This model trained to a validation loss of 0.3533 by epoch 128 (before automatically stopping training at epoch 133). Compare that to the previous model, which only achieved a 0.5905 validation loss, and it took 179 epochs to do it. So this model achieved lower loss in fewer epochs, thanks mostly to the additional information gleaned from the bidirectional encoder.

For inference, the only difference is with the encoder’s output. Instead of outputting the encoder’s latent state, you use the concatenated layer encoder_out, like this:

inf_encoder = Model(encoder_in, encoder_out)

The notebook includes code to export your encoder and decoder models to Core ML. There are slight differences to match the new model architecture, but nothing should look unfamiliar to you. It includes the same workarounds you used in the last chapter.

Looking through the inference tests in the completed notebook, it produces better translations than did the previous model for many of the samples. For example:

It does about as well on most — but not all — of the other tests, too. Some of the most interesting are those it gets wrong, but less wrong than the last model did. Such as:

Notice that, in each of these examples, the bidirectional model does better then the previous chapter’s model when translating words that appear near the end of the sentences. That makes sense, since it looks at the sequence in both directions, letting it encode more context for the decoder.

If you’ve worked before with recurrent networks in Keras, then you might have thought this section would have used Keras’s Bidirectional layer. Before trying out your new model in Xcode, take a look at this brief discussion of why we didn’t use that class, here.

Why not use Keras’s Bidirectional layer?

Keras includes a Bidirectional layer that simplifies the creation of bidirectional RNNs. You initialize it with a single recurrent layer, like an LSTM or GRU layer, and it handles duplicating that as a reversed layer for you. To use it, you’d write something like this for a bidirectional LSTM:

encoder_lstm = Bidirectional(
  LSTM(latent_dim, return_state=True, recurrent_dropout=0.3),
  name="encoder_lstm")
encoder_out, fwd_enc_h, fwd_enc_c, rev_enc_h, rev_enc_c = \
  encoder_lstm(encoder_mask)

Or like this for a bidirectional GRU:

encoder_gru = Bidirectional(
  GRU(latent_dim, return_state=True, recurrent_dropout=0.3),
  name="encoder_gru")
encoder_out, fwd_enc_h, rev_enc_h = encoder_gru(encoder_mask)

Each of these examples sets return_state=True so the layers return their hidden and cell states (for LSTMs, which have cell states). So when you connect the Bidirectional layer to a network, it returns the output along with the forward and reverse states. Then you would concatenate the states like this:

encoder_h = Concatenate(name="encoder_out")(
  [fwd_enc_h, rev_enc_h])
encoder_c = Concatenate(name="encoder_out")(
  [fwd_enc_c, rev_enc_c])

Finally, you’d pass those concatenated states to the decoder as its initial state. And in Keras all of that works great.

Unfortunately, just like with masking layers, here you encounter a major roadblock to using the Bidirectional class: coremltools. If you try to setup a bidirectional GRU layer, coremltools will emit a helpful error message that “Keras bi-directional wrapper conversion supports only LSTM layer at this time”. This is true for coremltools version 3.3, which is the latest releases at the time of this writing.

Until this issue is resolved, it’s a better idea to create bidirectional models like you did here, using multiple recurrent layers explicitly.

Using your bidirectional model in Xcode

Open the SMDB project you’ve been working with for the past couple chapters in Xcode, or use the starter project found in this chapter’s resources. Then, add the Es2EnBidirGruCharEncoder16Bit.mlmodel and Es2EnBidirGruCharDecoder16Bit.mlmodel models to SMDB like you’ve done before. If you didn’t train your own, you can find the ones we trained in the notebooks/pre-trained/BidirCharModel folder.

Here’s the encoder’s model summary, minus the unimportant bits to conserve space:

One big difference is just how big the encoder is — 3.8MB versus the 734KB encoder from the previous chapter. The other thing to notice are the input and output names, which are different from what you used in your first encoder.

The decoder’s summary (not shown) is similar to the decoder in the previous chapter, but it’s 7MB instead of 750KB. That’s nearly 10 times larger!

Note: If you’ve followed along building this and the previous model, or are using the provided resources, then there is no need to add the vocabulary JSON files esCharToInt.json and intToEnChar.json to the project. That’s because the files from the previous chapter are exactly the same. But if you split your training data differently, then there’s a chance your JSON files are different, too. In that case, add your new JSON files to the project with different names and then modify the declarations of esCharToInt and intToEnChar in NLPHelper.swift to load them.

Now to actually use the new models. Rather than remove the code you wrote in the previous chapter, you’ll create a separate function that creates the decoder input suitable for your bidirectional model. Add the following function inside NLPHelper.swift:

func getBidirDecoderInput(encoderInput: MLMultiArray) ->
  Es2EnBidirGruCharDecoder16BitInput {
  let encoder = Es2EnBidirGruCharEncoder16Bit()
  let encoderOut = try! encoder.prediction(
    oneHotEncodedSeq: encoderInput,
    fwd_enc_gru_h_in: nil,
    rev_enc_gru_h_in: nil)

  let decoderIn = initMultiArray(
    shape: [NSNumber(value: intToEnChar.count)])

  return Es2EnBidirGruCharDecoder16BitInput(
    encodedChar: decoderIn,
    decoder_gru_h_in: encoderOut.decodersIntialState)
}

This is almost identical to what you wrote in getDecoderInput in the previous chapter. This version only changes class and parameter names to match the ones Xcode created for your new models. Notice you set decoder_gru_h_in to the encoder output’s decodersInitialState value — that’s the name of the output that concatenates the outputs of the forward and reverse GRUs.

Now, you need to call this new function and pass its output to the decoder model that pairs with your bidirectional encoder. To do that, find the following two lines inside spanishToEnglish:

let decoderIn = getDecoderInput(encoderInput: encoderIn)
let decoder = Es2EnCharDecoder16Bit()

And replace them with these two:

let decoderIn = getBidirDecoderInput(encoderInput: encoderIn)
let decoder = Es2EnBidirGruCharDecoder16Bit()

Nothing new, here: You’re just calling a different function to get the decoder’s input and using your new bidirectional decoder model.

Finally, you need to modify the parameters on the decoder’s input. Find these two lines:

decoderIn.decoder_lstm_h_in = decoderOut.decoder_lstm_h_out
decoderIn.decoder_lstm_c_in = decoderOut.decoder_lstm_c_out

And replace them with this one:

decoderIn.decoder_gru_h_in = decoderOut.decoder_gru_h_out

That’s it. Build and run the app, then choose Spanish in the By Languages tab to see your model in action!

Better than the model from last chapter? Um, kind of, in places, but not much. It got close on some sentences, such as, “I don’t usually like music,” instead of the expected translation, “I don’t usually like musicals,” and “How boring!” which is close to the correct “Very boring!” It also included additional correct words within translations, such as “worst” and “movie” in its translation of, “Es mi peor pesadilla hecha película,” and “bad” when translating, “Muy mala película.”

While this doesn’t seem like much of an improvement, keep in mind that you haven’t addressed most of the problems mentioned at the end of the last chapter. For example, the dataset is still too small, and you’re still predicting characters with a greedy algorithm. But bidirectional models generally produce better results for tasks like translation, where useful context can appear before and after an item in a sequence.

The rest of this chapter shows some other techniques that might lead to better performance. In the next section, you’ll learn about a popular alternative to greedy decoding called beam search. In the process, you’ll find out this model wasn’t as far off as it seems with many of its translations.

Beam search

This and the previous chapter have both implied there’s a better option than greedily choosing the token predicted with the highest probability at each timestep. The solution most commonly used is called beam search, and you should strongly consider implementing it if you want to improve the quality of a model’s generated sequences.

Beam search is a heuristic-based searching method that considers many possible sequences, and it ranks them based on the total probability of the sequence, regardless of the probability of individual choices at any particular timestep.

What’s that mean? Consider the following contrived example, wherein a model attempting to predict the first word of a sequence tries both choices for the first character instead of just the most probable one:

The numbers in the above image have all been fabricated, and they make the unrealistic assumption that all the unlisted characters have zero probability, but it demonstrates the issue. As you can see, choosing the highest probability character for the first choice — “H” — does not lead to the highest probability sequence: “Why.”

Each prediction the model makes affects all its future predictions in that sequence, so any single bad choice can ruin the rest of the translation. That’s why using the greedy approach doesn’t usually lead to the best results. There’s no perfect neural network, so do you really want to count on a prediction with probability 0.0134 being definitely the right choice over one with probability 0.0133?

Beam search attempts to work around this issue by not committing to any one sequence. Instead, it maintains multiple possible search paths, and it expands the most promising sequences, eventually returning the one it finds with the best overall probability score. So in the above example, beam search would predict “Who” instead of “How,” even though it seemed like the correct first character should have been “H” when the model first started translating the sequence.

The basic algorithm goes like this:

1. Define the number of sequences you’ll maintain, called the beam width, B.

2. Make a prediction with the model, then take the values with the top B probabilities and store them as the start of B different sequences.

3. For each of the current B sequences, make another prediction with the model and extend the sequence with the top B prediction results, giving you BxB sequences.

4. Store the top B sequences from this set.

5. Repeat steps 3–4 until you have B sequences that have all predicted STOP tokens or reached their maximum length.

6. Return the top sequence from the final B sequences.

There are variations you can make to this basic algorithm, such as sometimes sampling randomly to include lower probability sequences (in case they surprise you and improve later), or storing previously discarded sequences in case you want to return to them if they look better after exploring other search paths.

Note: It’s important to realize that beam search is not guaranteed to find the best result. It does not try every possible sequence — that would be too computationally expensive — so it returns the best result if can find using its heuristics and available resources.

Implementing beam search isn’t really related to machine learning; it’s just a useful algorithm for dealing with joint probabilities. As such, we won’t provide code for it, here. However, we do want to point out a few potential gotchas that you should be aware of for when you try to implement it yourself.

The probability of a sequence is the joint probability of all the predictions used to create the sequence. You calculate it by multiplying these probabilities together. For example, there’s a 0.5 probability of getting heads when flipping a coin once, so the probability of getting three heads in a row is 0.5 x 0.5 x 0.5 = 0.125.

The trouble is, probabilities are small numbers between 0 and 1, and multiplying them produces even smaller numbers. It doesn’t take long before the limited precision available on computers for floating point arithmetic introduces errors, and eventually underflows and ruins everything by multiplying by zero.

You can avoid this by adding the logarithms of the probabilities instead of multiplying the probabilities directly. So in the earlier example, taking the log probabilities and adding them together gives you these values:

The logs are all negative, but notice they still work out so that higher values correspond to higher probabilities. Maximizing the sum of log probabilities — or minimizing the sum of the negatives of the log probabilities, if you prefer to implement it that way — produces the same sequences as multipyling the probabilities, but the math is stable.

The next issue relates to sequence length. Each additional token lowers the total probability for the sequence, so comparing these totals directly would penalize longer sequences. You’ll need to normalize them before making any comparisons.

One way to do that is to divide each of the sums by the length of its sequence. However, according to Andrew Ng, it’s common to divide by the length raised to some power between zero and one; using zero turns off normalization completely, and using one normalizes by the length directly, but any value between those normalizes by the length while still preferring shorter sequences to some degree.

Ng claims there is no good mathematically rigorous way to choose this value; it’s just something you experiment with until you find a value you like, but he says 0.7 seems to work well.

Finally, you’ll need to decide how many resources you can dedicate to running the beam search. The more sequences you try, the longer it will take. And, depending on your design, wider beams could lead to a great deal more memory use, too. You’ll have to experiment and make performance tradeoffs, especially to run it on mobile.

So if you implemented beam search and used it with the bidirectional model you trained earlier in the chapter, without training with more data or making any other changes, would it help?

Consider when it translates this sentence: “Entonces esta película te sorprenderá.” By choosing the character with the highest prediction probability at each step, it currently outputs, “So what this movie will surprise you.” This sentence has a normalized sum of log probabilities of -0.341630.

However, the correct translation — according to Google Translate — is, “This movie will surprise you.” And this sentence has a normalized sum of log probabilities of -0.300456.

Producing this better translation requires making three choices that are not the highest probability character for those steps — the fourth highest for the first character, and the second highest for two others. Depending on how you implement it and the beam width you use, beam search could find this correct translation. That means the current model can produce better translations just with the help of some smarter decoding processing.

Note: This chapter’s bidirectional model can translate quite a few of the review and test sentences correctly if we occasionally choose lower-probability characters. For example, choosing “u” instead of “o” when translating, “Hay un gato debajo de la mesa,” correctly outputs, “There is a cat under the table,” instead of, “There is a cat on the table.” Just adding beam search doesn’t mean you’ll get these translations. That’s because they still end up with lower total probabilities than what the model finds using greedy search.

However, that’s because of the training data more than the model or the search algorithm; training with larger datasets means gathering better language usage statistics, so once you have a model trained with a lot of data, coupling it with beam search gives you the best chance of producing high quality results.

Attention

The previous chapter mentioned an important problem with the encoder portion of your seq2seq model: It needs to encode the entire sequence into a single, fixed-length vector. That limits the length of the input sequences it can successfully handle, because each new token essentially dilutes the stored information.

To combat this issue, you can use a technique called attention. The most basic implementation of attention works like this: Instead of using one vector to represent the entire sequence, the encoder uses a vector per input token. Then the decoder learns to apply different weights to each vector at each output timestep, essentially paying attention to specific combinations of words. The weights are often visualized as attention maps, like in the following image:

There’s a column for each token in the input sequence — in this case, the tokens are words, not characters — and a row for each output token. The colors in the rows indicate how much each column’s input token was considered for that timestep, from black (zero) to white (one).

Notice how it doesn’t necessarily focus on the word aligned with the same timestep. For example, it needs to look at terms out of order when translating “the European Economic Area” to “la zone économique européenne.” Using attention, models learn to relate specific parts of their output with specific parts of their input, which produces much better results than the basic seq2seq models you’ve built here, especially on longer sequences. It has proven so powerful that it’s now used in most state-of-the-art models for NLP, as well as for other tasks like some computer vision problems.

There’s a variation of attention called self-attention that gives even better results. Whereas regular attention works between the encoder and decoder, self-attention gains additional information by allowing the encoder to apply attention between the input tokens, and it lets the decoder apply attention between its output tokens. That is, regular attention only relates the inputs to output tokens, but self-attention relates tokens within each sequence to other tokens in the same sequence, as well.

Self-attention performs even better than regular attention because it lets the encoder fine tune its encodings based on relationships it finds between input tokens, such as noun-verb agreement or to whom a pronoun refers in a sentence. In fact, self-attention is so good that current state-of-the-art models, often based on a network architecture called a Transformer, do away with the recurrent portions of the encoder and decoder and rely entirely on self-attention. Not only do they perform better, but they train faster, too!

These state-of-the-art models are quite large, so they usually run in the cloud. However, you can take advantage of these techniques in smaller models, too, depending on the task and the details of your implementation. If you decide to explore adding attention to your own networks, you may want to look at some of the important papers related to it:

• “Neural Machine Translation by Jointly Learning to Align and Translate” arxiv.org/abs/1409.0473

• “A Structured Self-attentive Sentence Embedding”, arxiv.org/abs/1703.03130

• “Attention Is All You Need”, arxiv.org/abs/1706.03762

We glossed over how the previous image showed whole word tokens, not characters like what you’ve used in your models. The next section discusses the choice to work with characters instead of words in these chapters.

Why use characters at all?

The seq2seq models you’ve made in this book work with sequences at the character level, but why? How much information does a model get from each token when it views sequences this way? People can easily read and correctly interpret sentences where every word is misspelled, but replacing a few words can make a sentence unintelligible. It seems like most individual characters don’t add much information to a sentence, whereas most words do, so shouldn’t translation models consider words instead?

The answer to that is yes — and also maybe not.

Researchers are always exploring different options, e.g., combining words into phrases or going the other way and breaking them into subwords. There are hybrid approaches that look at tokens in multiple ways, e.g., as words and as characters, which can help when dealing with OOV tokens. They’ve even made models that work with text as sequences of bytes! Working with whole words is probably the most common approach, but there are some difficulties involved that you should be aware of before attempting to build such models.

First, there’s vocabulary size. It may not be obvious if you haven’t dealt much with these models, but vocabulary size matters quite a bit, for these three reasons:

1. The model’s output layer performs a softmax calculation across the entire vocabulary to produce the probabilities for each token — the more tokens, the longer that calculation takes. A byte-level model has a vocabulary of just 256 values and is capable of representing anything; a character-level model for a restricted character set, like ASCII, will vary but likely will be in the hundreds or low thousands; full Unicode support would include over 1.1 million characters; and using word level tokens means a potential vocabulary size of many millions.

   In fact, working with words basically requires limiting the vocabulary to some subset of common words, and then you need to add in ways to deal with OOV tokens. There are some tricks you can implement to reduce the computations required at this softmax level to speed up training — look up terms like “adaptive softmax” and “hierarchical softmax” for some ideas — but it’s still easier to deal with fewer items.

2. The larger the vocabulary, the larger the model. That’s because it increases the width of your input and output layers (at least), and those layers end up contributing most of your model’s trainable parameters. Mobile devices don’t have the resources to deal with very large models, so it may not be feasible to support very large vocabularies on them.

3. The dreaded curse of dimensionality. You’ll hear this term a lot in machine learning. It refers to the fact that, as you increase the number of input features, the possible combinations of inputs can grow exponentially. (See the upcoming Note for an example.) As the possible combinations grow, each specific training sample covers a smaller percentage of those possibilities. The primary result: As you add features, you need to increase the size of your training set — possibly exponentially. Now consider that each token in the vocabulary is a unique input dimension, and you’ll see you need ever more training data as the size of your vocabulary grows.

Note: Here’s an example of the curse of dimensionality. Consider a model that takes only one input feature — an integer from 1 to 10. There are only 10 possible inputs, so each training sample essentially covers 10% of all possible inputs this model can ever see.

Adding a second input feature — again an integer from 1 to 10 — gives you a two-dimensional input with 100 possible combinations, so each training sample now only covers 1% of the possible inputs. And adding a similar third dimension brings the possible combinations up to 1,000, which would mean each sample then covers only one tenth of 1%.

As the number of dimensions goes up, a model must train on drastically more data in order to learn an accurate representation of the input space.

Secondly, working with subword tokens makes it easier to deal with OOV tokens. At the byte level, you can remove the problem entirely — there will only ever be 256 possible values; with characters you can delete OOV tokens and rarely lose much information. But with words, it becomes a difficult problem that’s still an open area of research. You’ll read more about it in the next section.

Even with those things going against it, using words still has one overwhelming advantage: Words convey meaning. Computers can’t really understand what they read — not yet, anyway — but they can definitely learn to identify important things like semantic relationships between words. The next section points out a few things you’ll need to do differently when dealing with word tokens instead of characters, and it introduces a popular way to represent them: embeddings.

Words as tokens and word embedding

Recall that neural networks require numerical inputs. So far, you’ve been one-hot encoding text prior to using it, but that essentially means your network sees mostly just zeros. What if you could provide more useful information?

It turns out, you can, with something called word embeddings, or word vectors; you’ll see these terms used interchangeably. These are vectors that represent words in some abstract n-dimensional space, and in the process, capture meaningful information about them. You can then use these vectors as inputs to your networks instead of one-hot encodings.

This essentially lets you provide information about a word instead of just a Boolean flag indicating the presence of a word. All that information makes the work easier for the rest of your model, and results in a better accuracy for your model as a whole.

This can all sound rather abstract. It’s easiest to understand by analogy.

For instance, to make a 2D map of Earth, you need to project 3D geographic positions into a 2D coordinate system. While that isn’t hard to imagine for positions, you can actually project any set of values into a different coordinate system. Consider the following graph, which projects Marvel characters onto the two alignment axes from Dungeons & Dragons — one represents their morality from good to evil, and one represents their behavior from lawful to chaotic:

The specific alignments given for each character are from the ComicsVerse blog post, “Good? Evil? The Alignment of MCU Characters” https://comicsverse.com/good-evil-mcu-alignment/.

Check it out if you’re curious about the arguments behind each assignment. It’s not important if you don’t know these characters or the details of D&D’s alignment system. What is important is how this shows words plotted in 2D space, where each axis represents a qualitative feature and the word’s position along that axis gives you a quantitative measure of how much that word represents or manifests that quality.

Note: This example uses discrete values of good-neutral-evil and lawful-neutral-chaotic, but there’s no reason we couldn’t plot these values in a continuous feature space instead. For example, it’s not difficult to imagine a character that is not totally good, but still leans toward being mostly good.

While this example was a bit contrived, it demonstrates a couple things. First, it show you can plot qualities along axes just like you plot quantities. That’s the essential idea behind word embeddings: Take a word and plot it into a n-dimensional space that describes how it relates to various qualities. This example only had two dimensions, but using a higher number lets you capture more features. In practice, word embeddings often use between 50 and several hundred dimensions, and we don’t specify what they represent. It may be that no single dimension represents any specific quality, but rather combinations of dimensions end up representing useful things.

The second thing shown is how embeddings are a matter of choice. The good vs evil and lawful vs chaotic word embedding is, obviously, fanciful. But you could, if you wanted, use this embedding to map out the relationships of all the world’s heads of state. However, you could also do it in more predictable ways. You could, for instance, map every head of state to the longitude and latitude coordinates of their country’s capital city.

From this perspective, even the one-hot encoding can be seen as a kind of embedding, a degenerate one. It’s the embedding you get when you don’t reduce the number of dimensions at all, and every single entity gets its own dimension, and the meaning of a dimension is just to identify that entity. The reason this is not particularly useful is that, because it does not make any choices about how it reduces the number of dimensions, it does not express relationships between the entities.

So what embedding should you choose? In general, you don’t do the choosing. In machine learning, you let a model learn the embedding from the data, rather than hand designing it based on prior insights.

There are several ways to learn such embeddings and we won’t go into the details here, but they all revolve around the same basic premise: Words are given random positions in some n-dimensional vector space, and their positions are adjusted a bit each time the word is used in the dataset. After processing a large text corpus, words that are used in similar ways end up closer to each other in vector space.

Because this process learns the embedding from the data, the embedding ends up discovering relationships between words that were implied by the data. For example, the following chart shows one such discovered relationship — that of gender — between the words “man” and “woman.” Other words that have a similar gender relationship, such as “king” and “queen” or “uncle” and “aunt,” display a similar mathematical up vs down relationship in the vector space:

You can learn more about word embeddings and the algorithms used to make them, as well as find many pre-trained vectors, by searching online for “word embeddings” or “word vectors.” Word2Vec, GloVe and fastText are common embedding options. You could train your own, but learning high-quality embeddings requires lots of data. For example, one set of GloVe embeddings you can download was trained on a corpus of 840 billion tokens.

Using word embeddings instead of one-hot encodings greatly increases performance on most NLP tasks. They are one of unsupervised learning’s greatest success stories.

Word embeddings in iOS

Apple provides support for word embeddings via the MLWordEmbedding and NLEmbedding types. That support makes certain uses of word embeddings straightforward.

For starters you don’t need to train your own embedding at all. NLEmbedding comes with built in word embedding for seven major languages. Apple does not say how these embeddings are trained except that they are trained over bodies of text with “billions of words.” This makes them good for general purposes uses.

However, if you do want to provide your own embedding then you can do that too. You can create an MLWordEmbedding by providing a dictionary mapping every word to its numerical vector, save the compiled embedding to disk, and then use that file to create a NLEmbedding. This would be a good idea if you want to experiment with other general purpose pretrained embeddings (like the GloVe embedding depicted in the figure above), to use a pretrained embedding targeted at a specific vocabulary domain, or to import an embedding that you trained yourself.

The saving to disk is not just a piece of busy work in the middle. As with text catalogs (mentioned in Chapter 14), with that step you are compiling the data into a highly efficient storage format. This will be critical if you are defining an embedding over a large vocabulary and do not want embedding data to increase your app’s download size unnecessarily.

Experimenting with this now will show you the benefits and the limits of the system. Open the starter playground projects/starter/playgrounds/WordEmbeddings.playground in the chapter resources. The code already in the playground imports modules and defines a URL for saving and loading the uncompiled embedding.

Now add the following code to define the Marvel character entities via an embedding:

let vectors = [
  "Captain America": [0.0, 1], "Rocket Raccoon": [1, 1],
  "Hulk": [1, 0],  "Loki": [1, -1],
  "Thanos": [0, -1], "Red Skull": [-1, -1],
  "Black Widow": [-1, 0], "Nova Corps": [-1, 1],
]

This defines vectors representing the characters exactly as depicted in the figure, using a positive x for chaotic and a positive y for good. Now add the following:

// 1
let embedding1 = try MLWordEmbedding(dictionary: vectors)
try embedding1.write(to: marvelModelUrl)
// 2
let compiledUrl = try MLModel.compileModel(at: marvelModelUrl)
// 3
let embedding2 = try NLEmbedding(contentsOf: compiledUrl)

This does the following:

1. Initializes an MLWordEmbedding and writes it to disk.
2. Compiles it as a Core ML model, for efficiency gains.
3. Load the compiled model as an NLEmbedding which can be used by the Natural Language framework.

So what can you do with this? Apple’s API makes it easy to use the embedding vectors to provide direct information about the relationships between entities.

For instance, since the embedding positions your entities into an embedding space, it becomes possible to talk about the “distance” between entities in that space. Suppose you asked a Marvel fan, “What’s the distance between Captain America and the Rocket Raccoon?” They might look at you funny, since if you’re not talking about where they are standing it’s not quite clear what the question means. But if then you asked, “Are they closer to each other than, say, Captain America and Loki?” you would probably get a quickly reply. Of course the captain and the raccoon are closer. They’re both good guys, they’re both trying to save the world, they seem to get along, while Loki is a chaotic god of disorder.

The point is, an embedding lets you take this intuitive notion of “distance” between entities and make it concrete. Add the following code to see this in action:

embedding2.distance(between: "Captain America",
                    and: "Rocket Raccoon",
                    distanceType: .cosine)
// => 1.414
embedding2.distance(between: "Captain America",
                    and: "Loki",
                    distanceType: .cosine)
// => 1.847

This asks the embedding object to calculate the distance between the two entities. It returns distances of 1.414 and 1.847, consistent with intuition (as of Xcode 11.3.1). The embedding object also provides API for efficiently searching for other entities within a given distance, or listing the entities nearest to another entity.

The parameter distanceType specifies how the distance is calculated; at present, cosine distance is the only available option. Let’s look at some other distances. Try the following:

embedding2.distance(between: "Captain America",
                    and: "Thanos",
                    distanceType: .cosine)
// => 1.414

Wait, what? That’s the same value we got for Captain America and Rocket Raccoon! Is Captain America as similar to Rocket Raccoon (another good guy) as he is to Thanos, who wants to destroy half the life in the universe? Thanos certainly looks farther away on the chart using the vectors that we ourselves defined. Something is puzzling here.

Apple’s documentation says that the .cosine represents cosine distance. But you can write your own definition of cosine distance as follows (following the definition offered in Mathematica and in Wikipedia):

func cosineDistance(v: [Double], w: [Double]) -> Double {
  let innerProduct = zip(v, w)
    .map { $0 * $1 }
    .reduce(0, +)

  func magnitude(_ x: [Double]) -> Double {
    sqrt(x
      .map { $0 * $0}
      .reduce(0,+))
  }

  let cos =  innerProduct / (magnitude(v) * magnitude(w))
  return 1 - cos
}

This computes one minus the cosine of the angle between two vectors. And if you calculate cosine distance directly, you get the following:

cosineDistance(v: vectors["Captain America"]!,
               w: vectors["Rocket Raccoon"]!)
// => 0.29
cosineDistance(v: vectors["Captain America"]!,
               w: vectors["Loki"]!)
// => 1.707
cosineDistance(v: vectors["Captain America"]!,
               w: vectors["Thanos"]!)
// => 2

These numbers are different from Apple’s not only in the magnitude of distances but even in their ordering.

So what is going on here? Hard to say. But one reasonable conclusion is that Apple’s embedding API is very new, not yet well documented, and may not behave in the way you expect. If you do want to use a custom embedding in order to use functionality like measuring distances or fast search for neighboring entities, you should take care to validate that the API is defining distance in the way you expect.

Whatever the explanation for these surprising values, the fact is that most of the time in machine learning models embeddings are not used in order to directly access information about entities, but as an early layer in a larger model, used in order to prepare an enriched representation of input values so that the rest of the model can do a better job.

The next section will discuss how this might work for our sequence to sequence translation model.

Building models with word embeddings

This section points out some changes you’d need to make to your existing seq2seq models in order to have them use word tokens instead of characters. This section includes code snippets you can use, but it doesn’t spell out every detail necessary to build such a model. Don’t worry! With these tips and what you’ve already learned, you’re well prepared to build these models on your own. Consider it a challenge!

We’ll assume you’re starting with pre-trained word embeddings. The first thing you’ll need to decide is how many words you want to include in your vocabulary. Just because you have an embedding for a word doesn’t mean you’ll want to use it in your model. Remember, vocabulary size affects model size, so you’ll need to keep things manageable. But there’s another reason you might not want to use all the embeddings you download: some of them may be garbage.

Word embeddings are trained in an unsupervised manner with huge datasets often scraped from the internet, so it’s not uncommon for bad tokens to slip through. For example, there are two million tokens in the Spanish embeddings you can download at https://fasttext.cc/docs/en/crawl-vectors.html. However, these include thousands of tokens like “112345678910111213141516173” and “PaísEnglishEspañolPortuguêsCanadaUnited” that would just take up space in your model without serving any useful purpose.

Important: The more embeddings you include, the larger your model will be. It’s common to choose a few tens of thousands of the most commonly used words, but whatever you do, try to limit it to a reasonable set for your task.

An important step when dealing with word tokens is… tokenizing your text into words. You’ve got different choices for how to do this. For example, the English contraction “don’t” could become the tokens don't, do and n't, don and 't, or don, ' and t.

The first two options are the ones you’re most likely to come across, but whatever you choose, you need to make sure of a couple things:

1. If you’re using pre-trained word embeddings, make sure you create tokens in the same way as the creators of the embeddings did, otherwise you’ll end up with more OOV tokens than you should because you won’t have embeddings for some of your tokens that you otherwise would have. For example, if you parse the word “don’t” as the token don't, but the pre-trained embeddings contain the two tokens do and n't, you’ll end up having to treat every “don’t” you encounter as an OOV token.

2. Tokenize text in your iOS code the same way you do in Python. The Natural Language framework’s NLTokenizer doesn’t let you configure its tokenization settings like you can with Python packages like nltk, so you may have to write your own tokenization logic to produce the tokens your model expects.

For your special START and STOP tokens, you should add some word to your vocabulary you know never appears in the data, e.g., <START> and <STOP>. You’ll need to deal with OOV tokens quite differently, too. You can’t just remove them like you did with the characters, so you’ll replace them with a special token, such as <UNK>.

Note: You learned in the first NLP chapter that it’s possible to tag text with its parts of speech. Instead of using a single <UNK> token for all OOV tokens, you may get better performance if you replace words with part-of-speech-specific tokens, such as <UNK_NOUN>, <UNK_VERB>, etc. This gives the model additional context that might help when translating sequences that contain OOV tokens.

Once you’ve loaded your vocabulary of pre-trained embeddings, you’ll need to add embeddings for the OOV token(s) you defined, too. Assuming your embeddings are in a NumPy array called es_token_vectors, and you have an OOV token in the variable unk_token, you could add a random embedding like this:

es_token_vectors[unk_token] =
  2 * np.random.rand(embedding_dim).astype(np.float32) - 1

This assigns an <UNK> token an embedding filled with embedding_dim random values in the range [-1,1). That’s not necessarily the best range to use — you might want to check the pre-trained embeddings you’ve got to see the range of values you’re already dealing with — but the point of using random values is to hopefully keep your OOV tokens from being too similar to any other words in the vocabulary.

Note: You could try various options for the embeddings for your unknown token(s). For example, take an average of all or some set of noun embeddings for an UNK_NOUN token, an average of verb embeddings for an UNK_VERB token, etc. It’s unclear whether that would be helpful, but trying out different ideas is part of the fun of machine learning, right?

You’d identify which tokens are OOV by checking the tokens in your training set against your word embeddings. That is, if your training set contains a token that does not exist in your pre-trained embeddings, you’d remove that term from your vocabulary.

Note: If you’re training your own embeddings, you would define your vocabulary differently. You’d most likely start by counting the uses of each token in your training data, and then choosing some number of most commonly used terms. Those terms then define the vocabulary for which you’d train embeddings.

Once you’ve settled on the vocabulary you’ll support — let’s assume it’s a set of tokens stored in in_vocab — you need to go through all your training, validation and test data and replace any OOV tokens with the appropriate <UNK> token(s). This is different from how you removed OOV from your datasets when you work with characters.

At this point your embeddings are likely in a dictionary keyed off of tokens, and it may contain more embeddings than you actually plan to use in your vocabulary. You’ll need to put the embeddings for these individual vocabulary words into a single NumPy array to use with your network, like this:

# 1
num_enc_embeddings = in_vocab_size + 1
# 2
pretrained_embeddings = np.zeros(
  (num_enc_embeddings, embedding_dim), dtype=np.float32)
# 3
for i, t in enumerate(in_vocab):
  pretrained_embeddings[i+1] = es_token_vectors[t]

Here’s how you’d grab the embeddings for the words in your vocabulary:

1. Your encoder will need to embed each token in your vocabulary, plus an additional embedding to act as a padding token during training.
2. Create a NumPy array of all zeros, shaped to hold the correct number of embeddings of size embedding_dim. You can find pre-trained embeddings in various dimensions, especially for English, but it seems 300 is the most common. Embeddings of that size for a reasonably sized vocabulary may give you models that are too big for mobile devices, so you may need to train your own if you cannot find smaller embeddings for the language you need.
3. Notice how each embedding is stored with the index i+1. This leaves index zero unassigned, which is the value reserved by Keras’s Embedding layer for the padding token. You’ll read about the Embedding layer next.

Next, you’d replace the Masking layers you were using in your earlier seq2seq models with Embedding layers. These can perform the same masking, but additionally they map scalar integers into higher dimensional vector coordinates. Here’s how you would define your encoder’s embedding layer:

# 1
enc_embeddings = Embedding(
  num_enc_embeddings, embedding_dim,
  weights=[pretrained_embeddings], trainable=False,
  mask_zero=True, name="encoder_embeddings")
# 2
enc_embedded_in = enc_embeddings(encoder_in)

Embedding layers usually go at the start of a network, like this:

1. When creating an Embedding layer, you specify the number of possible input tokens — defined here as num_enc_embeddings — and the number of dimensions to map those values into, defined here as embedding_dim. When using pre-trained embeddings, you provide them using the weights parameter and then set trainable=False so the model doesn’t modify their values while training.

2. Then you pass your Input layer into the Embedding layer, just like what you did before with the Masking layers. The Embedding layer will still perform masking because you created it with mask_zero=True. That tells it to treat the input value zero as a padding token, but you must include that extra token in your input token count. That’s why you had to add one to the size of the vocabulary when you declared num_enc_embeddings.

You could declare the decoder’s Embedding layer the same way, using pre-trained embeddings for your target language. However, Keras can also learn its own embedding values during training. To do that in your decoder, for example, you’d declare your Embedding layer like this:

num_dec_embeddings = out_vocab_size + 1
dec_embeddings = Embedding(
  num_dec_embeddings, embedding_dim,
  mask_zero=True, name="decoder_embeddings")

The difference here is that you don’t supply an argument for the weights parameter, and you rely on the default value of True for the trainable parameter. This Embedding layer will initialize its embeddings to random values and modifying them during training, just like it modifies the weights of other layers in the network.

Note: Another option for your Embedding layers is to start with pre-trained embeddings but set trainable=True. Or even train for a while with it set to False before training for additional epochs with it set to True. In either case, the idea is to take advantage of the information the embeddings contain while fine-tuning them to be more appropriate for your model’s specific task.

The Embedding layers reserve zero for the padding token, so you’ll need to avoid that ID for your real tokens. One option is to shift all your token IDs by one when making your conversion maps, like this:

in_token2int = {token : i + 1
                for i, token in enumerate(in_vocab)}
out_token2int = {token : i + 1
                 for i, token in enumerate(out_vocab)}

You’d need to make minor differences in make_batch_storage and encode_batch, too, because the input sequences are no longer one-hot encoded. For example, you would set tokens like this:

enc_in_seqs[i, time_step] = in_token2int[token]

Instead of:

enc_in_seqs[i, time_step, in_token2int[token]] = 1

You still one-hot encode the target sequence, since the decoder still outputs a probability distribution over the vocabulary. However, make sure the decoder’s output layer has the correct size.

You can use num_dec_embeddings, but that includes an extra padding token that your model should never output. Or you can use out_vocab_size, but then you’ll need to subtract 1 from its training output targets so it doesn’t try one-hot encoding values larger than its max.

Using word embeddings in iOS

When it comes to using your trained model in an app, it’s similar to what you did in the SMDB project. However, there are a few important caveats:

• You’ll have to include mappings for your tokens, just like you did with the character-based models. However, these will be quite a bit larger due to the larger vocabulary and the fact that the tokens aren’t single characters. You’ll also need to know the indices of any special tokens, like <START>, <STOP> and <UNK>.

• Pre-trained word embeddings are often produced for lower-case tokens. That means that you’ll need to convert your input sequences to lower case prior to translating them, then figure out what to capitalize in the output. You can use specific rules or heuristic-based approaches, or you might try training a separate neural network to capitalize text.

• As mentioned earlier, tokenize the input text the same way you did when training. You didn’t have to worry about that when working with characters, because there’s no ambiguity about what constitutes a token in that case.

• Be sure to replace any OOV tokens with the proper <UNK> token(s). You just dropped OOV tokens when working with characters, but you’ll lose too much information if you do that with whole words.

• After running your model —  maybe with a nice beam search — you’ll be left with a list of tokens that might look like this: ['i', 'do', "n't", 'like', '<UNK>', 'or', 'bugs', '.']. You’ll need to perform post-processing to properly capitalize words, insert spaces appropriately and connect contractions. You’ll also need a plan for dealing with any <UNK> tokens. One option is to align the sentence in some way and copy over words that seem to match. For example, if there is one <UNK> token in each of the input and output sequences, just copy the original unknown input token directly into the output. Translating between some language pairs requires a reordering of terms, and sometimes the numbers of missing terms won’t match up. Even when things seem to line up, that won’t always produce a good result. Dealing with OOV tokens is an open area of research and there isn’t any easy answer that always works. When in doubt, you always have the option of leaving them unknown.

To conclude this introduction to using word tokens, keep these tips in mind:

• Pre-trained embeddings learn the biases of the data they are trained on. For example, embeddings trained on internet data usually place the word “apple” closer to tech companies, like Microsoft, than to fruits, like banana.

• Each embedding represents the average of all uses for that word. That means words used in multiple different ways have diluted embeddings that don’t always represent any of their uses well. There have been attempts to deal with this issue using context-sensitive embeddings. These are more complex but may be worth looking into for some projects.

• Try running a spell checker on your sequences prior to looking for OOV tokens. That should help reduce the number of tokens you can’t find in your vocabulary.

• When you encounter an OOV token during preprocessing, you might want to consider lemmatizing or stemming it and checking for that instead. Sometimes, vocabularies only contain other variations of a word, but that may give you better results than using an <UNK> token.

• Similar words should have similar embeddings. That means you may get reasonable results for words in your vocabulary even if they weren’t present in your training data, as long as they are similar in usage to other words that were in the data. However, training with more data that fully covers your vocabulary is still preferred.

• There have been attempts to create vectors for OOV tokens on the fly, or to use hybrid models that treat OOV tokens at the character level instead of as whole words. You should research ideas like these if you have a project that needs to handle many rare or unknown words.

Key points

• Bidirectional recurrent networks produce better encodings in cases wherein context appears both before and after an item in a sequence.

• Use beam search with your decoder to produce better translations.

• Seq2seq models that include attention mechanisms generally outperform those that do not, because they learn to focus on the most relevant parts of sequences.

• Word embeddings encode information about the contexts in which words are used, capturing relationships between words in an unsupervised manner. Using them improves performance on most NLP tasks.

• OOV tokens are more difficult to handle when using word tokens, but you’ll need a plan for dealing with them because it’s much less likely that you’ll be able to fully support your source or target languages at the word level.

Where to go from here?

These past three chapters have only scratched the surface of the field of NLP. Hopefully, they’ve shown you how to accomplish some useful things in your apps, while sparking your interest to research other topics. So what’s next?

The past year has brought an important advance in the field of NLP that we didn’t cover: language models. While word embeddings are essentially a type of transfer learning for NLP, the latest state-of-the-art techniques extend this concept to use neural networks to create pre-trained language models. These essentially learn to predict the next word in a sentence, and in doing so they seem to learn useful, transferable knowledge about the language. Building models on top of them greatly improves performance on most NLP tasks, beyond what has been achieved with word embeddings, and is the basis for exciting efforts in language generation tasks, too.

The pre-trained language models currently available are still quite large, but they’ll make their way into smaller packages more suitable for mobile soon. They’re certainly a good candidate for inclusion as part of Core ML, the way Apple currently supplies common computer vision models, so who knows? Either way, I recommend reading up on the subject if NLP interests you.

Finally, rather than list specific projects to consider or topics to learn, here’s a great GitHub repo that tracks the current state-of-the-art solutions and useful datasets for various NLP tasks: nlpprogress.com. These aren’t necessarily all feasible on mobile devices, but it shows you the types of things people are doing with natural language — and how they do it. If you’re at all interested in the field and what’s currently possible, I recommend spending some time exploring it.