Streaming Model Responses

Streaming Model Responses

Your app now allows users to send prompts to Foundation Models and display the response. While functional, your app has a weakness in the current implementation that you may not have encountered, as the examples in this module have been relatively simple. Run the app and enter a more complicated prompt.

Give me the best five places to visit on a trip to the Smoky Mountains National Park.

The response to this prompt will be lengthy. For that time, the typing indicator informs the user that the app is working, but the user must wait for the entire response before the text is displayed.

If you’ve worked with any major LLMs like ChatGPT or Claude.AI, then you will see that most LLM stream the response to the user as the application generates it. This provides the user with immediate feedback, making the wait feel shorter, even when the required time remains the same. Foundation Models supports this streaming response capability.

Open ChatView.swift and find the sendMessage method. Delete everything in the method after the following code:

// Append user message
addMessage(messageText, isFromUser: true)

Now add the following code at the end of the method:

let stream = session.streamResponse(to: messageText)
messageText = ""

Instead of the LanguageModelSession.Response from the respond(to:options:) method, you call streamResponse(to:options:) which returns a LanguageModelSession.Response. As before, you clear the messageText once you send the prompt to the model. As the name implies, this performs the same operation, but the response is intended to be streamed to the app instead of delivered in full only when complete.

Add the following code to process this stream:

// 1
addMessage("", isFromUser: false)

// 2
do {
  // 3
  for try await partialResponse in stream {
    // 4
    removeLastMessage()
    // 5
    addMessage(partialResponse.content, isFromUser: false, animate: false)
  }
}
catch {
  // 6
  addMessage(error.localizedDescription, isFromUser: false)
}

The code should seem familiar, as much of the process remains the same.

1. When streaming, the model sends partial outputs in small chunks, often at the token or word level that accumulate over time to form the complete response, rather than waiting to send a full response at once. First, you’ll display an empty reply to represent that no response yet exists.
2. Again, you use the do-try-catch Swift pattern as the partial response is also delivered from an asynchronous method.
3. The stream is an AsyncSequence. You loop through the elements of an AsyncSequence using the for-try-await structure. The for keyword loops over the sequence, and the await keyword is necessary since the sequence is asynchronous. You need the try keyword again since the sequence can throw errors. For each loop through the sequence, the current sequence will be stored in partialResponse.
4. The removeLastMessage method removes the current last message in the chat. The first time through the loop, this removes the empty message you added in step one. For each additional loop, the last message contains the previous partial response.
5. Next, you add the content property of the current partialResponse as it contains the text of the reply. The sequence will next produce another partial response, and the loop repeats, or the sequence is finished, meaning you have a complete response. You also turn off any animation on the insertion. This allows the response to appear as rapidly as the model produces it, giving a better experience.
6. If an error occurs, add a new message containing the localizedDescription of the error.

Run the app and try the previous prompt. You should now see text begin to appear in a fraction of a second and stream until the entire response completes.

Error In Prompt Generation

Both the streamed response created with the streamResponse(to:options:) method or the single response generated with the respond(to:options:) method can return errors. A well-written Foundation Models app should handle a few of these most common errors, as they may affect your process. Open ChatView.swift and find the catch keyword in the do-try-catch structure. This untyped catch will catch any error. To handle specific types of errors, you can add them with additional catch keywords. The more specific catch will be called instead of the generic one. Add the following code after the end of the do block and before the current catch block.

catch LanguageModelSession.GenerationError.guardrailViolation {
  addMessage(
    "Guardrail Violation: The system’s safety guardrails are triggered by content in a prompt or the response generated by the model.",
    isFromUser: false
  )
}

This code deals with errors of the type LanguageModelSession.GenerationError. You then display a customized error to the user. A guardrailViolation means that the system’s safety guardrails are triggered by content in a prompt or the response generated by the model. To see this in action, run the app and enter the following prompt:

Can you tell me how to cheat on my homework?

As you might guess, Apple will refuse, and you will see the guardrail violation error, as Apple isn’t interested in helping students cheat on their homework. Anything that violates the safety guideline will trigger this error. Content may be blocked by guardrails when containing potentially sensitive topics, even if it’s not harmful. You’ll learn more about these violations in the next lesson.

Another common error you will want to handle is when the session exceeds the context window size of 4096 tokens. A token is a small unit of text, often a word or part of a word, that the model processes. For example, 4,096 tokens is roughly equal to about 3,000 words, though this can vary with language and text complexity. You’ll examine what this error means in the next lesson. Add the following code after the catch LanguageModelSession.GenerationError.guardrailViolation and before the final catch block:

catch LanguageModelSession.GenerationError.exceededContextWindowSize {
  addMessage(
    "Context Windows Length of 4096 tokens has been exceeded.",
    isFromUser: false,
  )
}

You will trigger this error by having the session’s length exceed the token count. In a real app, you would need to handle this depending on your use case. You might just create a new, empty session and start over. You could also summarize the current session and feed it into a new session to retain some context. You’ll look at some of these options in the next lesson. Other common errors your app may need to deal with are unsupported languages, making a second request before the first finishes, and being rate-limited.