Apple Foundation Models

Transcript

Foundation Model Options

You do have a few options to tune the output of Foundation Models. Open the Starter app for this project and look for the ConfigurationView.swift file. This view implements a new view that allows the user to add instructions, set a temperature for prompt responses and turn on greed mode. You will explore each of these options in this section.

Instructions

The first, and most important, of these options is the previously mentioned instructions. You can think of instructions as a super-prompt you give the model when creating a new LanguageModelSession. You will use the instructions to define the role and behavior of the model. Apple trained Foundation Models to obey instructions over any commands it receives in prompts. This makes it a critical place to provide guidance to the model on how you want prompts to be handled and to specify restrictions beyond those included by Apple. Good instructions:

1. Explain the model’s role, such as “you are a travel advisor” or “You are a helpful restaurant guide”.
2. Explain what the model should do, like “Help the user summarize text”.
3. Provide any desired style preferences, like “Provide as brief a response as possible”.
4. Add any instructions for tuning or handling problems, like “Respond with ‘I can’t help you with that request.’ if you’re asked to do something dangerous”.

You provide instructions when creating a LanguageModelSession. If you want to change the instructions, you must create a different LanguageModelSession. Open ChatView.swift and you will see several new properties that weren’t in the version at the end of the previous lesson. These properties will contain the possible adjustments you can make to the model. You will also see a showConfig property, which displays the ConfigurationView when tapped, along with a new gear icon button in the toolbar to toggle this property. The property in question is the optional promptInstructions, which can be set to nil for no instructions or hold text instructions for the model.

To use the instructions in your app, find the resetChatHistory() method. Change the line session = LanguageModelSession() to:

session = LanguageModelSession(instructions: promptInstructions)

This will pass the instructions to the model each time it creates a new LanguageModelSession. Note that you do not need to change the initial setting of the session property since the instructions will always be nil when the user first runs the app.

Run the app and tap the new gear icon to view the configuration sheet, where you can find the text area for the instructions. Anytime you change the instructions, the text will be stored in the promptInstructions of the parent ChatApp view through a binding.

Swipe down on the settings sheet to dismiss it. Enter the following prompt:

Solve the equation 8x - 4 = 2 step by step.

While LLMs are not always great at math, in this case, it does get the correct answer of 3/4.

Now, you’ll examine how instructions can influence responses. Tap the gear icon in the toolbar to bring up the configuration view. Enter the following into the Instructions:

Use decimals instead of fractions when presenting the solutions to math problems.

Swipe down to dismiss the sheet. Notice the chat clears. Since changing instructions requires creating a new session, the app will do this anytime the promptInstructions property changes. Now enter the prompt again.

Notice how literal it followed the instructions. When solving the equation, the model still involves fractions, but it provides the answer as a decimal. Let’s change the instructions to demonstrate the focus on instructions over other prompts. In the settings, change the instructions to:

Refuse to provide any assistance that solves equations step by step. Only provide the final answer without showing the intermediate steps.

Return to the app and re-enter the prompt using these new instructions. It will state that it cannot provide step-by-step solution. Sometimes it’ll provide the answer to the question. Sometimes it won’t. Sometimes the answer will be wrong. Anytime you provide instrucitons you risk this change in the results to the outcome.

Instructions provide a critical way to both guide the model in generating the desired results and protect your app from potentially malicious data. You should use them to guide the model to the desired responses for your use cases.

Temperature

The second parameter to tune model responses is the temperature. Temperature influences the randomness of the model’s responses and can be set to nil or a value between zero and one, inclusive. The default nil allows the system to choose a reasonable default. The temperature adjusts the probability distribution of responses before sampling. In other words, it adjusts the randomness of the model’s responses. A value of one causes no adjustment. Lower values shift the probability, causing the model to select the more likely tokens more frequently, resulting in more predictable responses. Higher values should be thought of as increasing the deviation from statistically probable responses that form LLM responses. Note that changing this value will not affect the weaknesses of LLMs, such as hallucinations. You cannot eliminate hallucinations by lowering the temperature.

To add the ability to adjust the temperature of responses, open ChatView.swift. Find the sendMessage() method. Look for let stream = session.streamResponse(to: messageText) and replace it with the following code:

let temperature = customTemperature ? modelTemperature : nil
let options = GenerationOptions(temperature: temperature)
let stream = session.streamResponse(to: messageText, options: options)

First, you create an optional Double temperature that will be nil if the user did not choose to use a custom temperature in the configuration view. Otherwise, it will contain the value selected on the slider on that page. It then creates a GenerationOptions with the appropriate temperature. The call to stream the model response now passes in the options to the options parameter. This will apply the chosen temperature to this response.

Note that, unlike instructions, you can provide a different temperature for each prompt in a session. Run the app and enter the following prompt:

Give me a one paragraph bedtime story.

Do the same prompt a few times, and you’ll get several short stories.

Now bring up the configuration view, toggle on Custom Temperature and set it to zero.

Swipe down to close the view and enter the same prompt. This time you’ll notice that the stories are very similar each time, often repeating the same words or events. That’s because a lower temperature makes the model more deterministic and consistent.

If you increase the temperature instead, the model introduces more randomness, leading to a wider variety of wording and events in its responses. You’ll need to experiment with your use case to see whether a lower temperature (more predictable) or a higher temperature (more diverse) produces better results.

Greedy Sampling

The inherent randomness in LLM output means that each answer remains distinct, even for the same prompt and setting. Apple Foundation Models allows you to specify how to sample values from the probability distribution. The most useful of these is applying the constant sampling .greedy. This value will always select the most likely token, providing consistency to the responses. To add this option to your app, find the section you just added when giving the app the ability to set the temperature and change the code to:

let temperature = customTemperature ? modelTemperature : nil
let samplingMode = useGreedy ? GenerationOptions.SamplingMode.greedy : nil
let options = GenerationOptions(sampling: samplingMode, temperature: temperature)
let stream = session.streamResponse(to: messageText, options: options)

This adds a new optional value that will either be GenerationOptions.SamplingMode.greedy or nil. You then add this to the GenerationOptions. A value of .greedy instructs the model to always choose the most likely token. A value of nil will use the default sampling method.

Now, run the app and access the configuration view. Turn on Use Greedy Sampling and dismiss the view.

Now enter the prompt:

Give me a one paragraph bedtime story.

You will get a one-paragraph story.

Now rerun the prompt. You might be surprised that you get two similar, but not identical, stories. Why does it not produce the same story since you’ve told it always to choose the most likely token?

The different stories occur because the response is determined by more than just the prompt. The entire context window, all prompts and responses, affect the output. Entering a prompt and getting a response changes the model in a way that means the following prompt will not produce the same output as the first one. Clear the chat and enter the prompt again. With nothing in the context window, you will see the same story as the first time. Using greedy sampling does not always produce the same output; it produces a consistent response. Given the same model state and the same prompt, you will get the same response.

Now that you’ve explored tweaking the output of the model, you will look at an approach to handling the small context window of Foundation Models in the next section.