Text Generation Parameters

Using Parameters for Test Generation

Parameters allow you to adjust how the text is generated in various ways. The main parameters that are available include:

• Model: The model to use for this generation.
• Region: The region where you want to process your requests.
• Max Tokens: The maximum number of tokens for the response.
• Stop Sequences: A text string that flags the termination of the response.
• Temperature: The temperature controls the degree of randomness in token selection. The higher the temperature, the more varied the response.
• Top-K: The generation will only consider the most probable tokens. The higher the value, the more random the responses are.
• Top-P: The generation will consider the most probable tokens until their cumulative probability reaches Top-P. The higher the value, the more random the response.

Demystifying Top-K, Top-P and Temperature

Imagine you’re browsing an online store with hundreds of items. Top-K filtering narrows your choices to, say, the top 5 highest-rated products where Top-K equals 5.  This ensures you focus on the better options without being overwhelmed.

Now think of choosing paint colors. You want variety, but not every shade. Top-P sets a probability threshold. The AI picks colors until their combined probability reaches, say, 0.9. This guarantees a diverse selection within a controlled range, avoiding wild outliers.

Finally, consider writing a story. Low temperature like 0.2 makes the AI stick closely to predictable words, resulting in a safe, perhaps boring tale. A high temperature like 0.8 encourages the AI to take risks, leading to surprising, creative, but potentially nonsensical, outcomes.

In essence Top-K limits the number of choices. Top-P limits the probability range of choices. Temperature controls how predictable the AI’s choices are. These parameters help us fine-tune AI outputs to achieve the creativity we need.

A great way to familiarize yourself with these values is to give them a try. You’ll do that next!