Building Tools for Foundation Models

Building Tools for Foundation Models

Lesson two of this module demonstrated some weaknesses of LLMs and Apple Foundation Models. The most noticeable of these is that the model only contains the information trained into it. It has no information beyond that. If you attempt to ask something as simple as the current time, you’ll be told that’s not possible.

This lack of knowledge extends to many things you might want to produce. It would be helpful to access other data from the iPhone, such as contacts, calendar events, and health data. There are also cases where you would like to access external data from systems beyond the consumer’s device.

Apple Foundation Models provide a way to accomplish this using tools. Tool calling lets the model call your functions to access additional data. This action occurs autonomously and can access anything that can be turned into a function call. You define a tool by implementing the Tool protocol in your struct or class. Open WeatherForecastTool.swift under the Models folder. This struct contains two methods: one to convert a location name into latitude and longitude, and another method that uses Open-Meteo to retrieve the weather forecast for a location when provided with the latitude and longitude. Open-Meteo provides a free access weather API for non-commercial use that works for most locations.

To add the ability for Foundation Models to load weather information, you’ll expand WeatherForecastTool to support the Tool protocol. The file already imports FoundationModels. Update the definition of the struct to:

struct WeatherForecastTool: Tool

This states that WeatherForecastTool will implement the Tool protocol. Several requirements must be met to implement this protocol. You must define a call(arguments:) method that accepts arguments of type ConvertibleFromGeneratedContent and returns a type conforming to PromptRepresentable. In practice, your return will usually be a String or a Generable object.

Add the following two properties to the top of the WeatherForecastTool struct:

let name = "weatherLookup"
let description = "Get the forecast temperatures and precipitation for the location provided as an argument."

These two properties set a name for the tool and provide a description. The description provides context for the tool to the model. Try to keep descriptions short, as they become part of the context and can introduce latency. Now you’ll need to provide the arguments expected by the tool. Add the following code after the description:

@Generable
struct Arguments {
  @Guide(description: "The name of the location to get the forecast for.")
  var location: String
}

Note that tools use the Generable macro, and everything you’ve already learned about guided generation applies here. The only argument for this tool expects a location name as a String. While this example only has one argument, you can provide more if needed for your tool. The final step in implementing the Tool protocol requires defining a call(arguments:) method. Enter the following code after the Arguments struct:

// 1
func call(arguments: Arguments) async throws -> WeatherForecast? {
  // 2
  var weatherForecast: WeatherForecast? = nil

  // 3
  if let coordinates = await getCoordinatesFor(arguments.location) {
    weatherForecast = try? await getForecastFor(coordinates: coordinates)
  }

  // 4
  return weatherForecast
}

Here’s how this code implements the call(arguments:) method:

1. The method takes the Arguments struct defined above. Note that these arguments will be provided by Foundation Models when it calls the tool. The method returns an optional WeatherForecast struct. You’ll examine this struct in a moment, but it implements a structure to hold the weather forecast.
2. You will create a weatherForecast variable and set it to nil.
3. You attempt to convert the location, passed as the location property of the arguments, to a latitude and longitude coordinate using the getCoordinatesFor(_:) helper method. This method uses MapKit to get the coordinates of the location. If this call succeeds, you use the getForecastFor(coordinates:) method to get a forecast as a WeatherForecast struct. The getForecastFor(coordinates:) method uses the Open-Meto SDK to get the forecast.
4. The method returns either a WeatherForecast struct on success or nil if anything went wrong.

If you attempt to build the app now, you will get an error message that’s not well worded. The error appears because the return value from the method must conform to PromptRepresentable. To fix this, you’ll change WeatherForecast and its properties to be Generable. Open WeatherForecast.swift and import FoundationModels at the top of the file. Then change the definition of ForecastElements to:

@Generable
struct ForecastElements {
  @Guide(description: "Time of the forecast.")
  var time: String
  @Guide(description: "Forecast temperature at time.")
  var temperature: Float
  @Guide(description: "Probability of precipitation at time.")
  var precipitationProbability: Float
}

This adds the @Generable macro along with using the @Guide description to provide context for the properties. Now update WeatherForecast to:

@Generable
struct WeatherForecast {
  @Guide(description: "The temperature and probability of precipitation element for a given time.")
  var forecasts: [ForecastElements]
}

Now that you have a working tool, you must tell LanguageModelSession about it. Open ChatView.swift and find the resetChatHistory() method. Change the method to:

messages = []
session = LanguageModelSession(
  tools: [WeatherForecastTool()],
  instructions: promptInstructions
)

The only change is adding an array of tools passed in to the tools parameter in the call to the LanguageModelSession initializer. Also, change the definition of the session property to:

@State private var session = LanguageModelSession(tools: [WeatherForecastTool()])

As tools can cause errors, add the following code in the sendMessage() method before the final catch in the do-try-catch section:

catch let error as LanguageModelSession.ToolCallError {
  addMessage(
    "Error occurred calling \(error.tool.name): \(error.localizedDescription)",
    isFromUser: false
  )
}

This will catch any errors caused by tool calls and print a message with the tool and a description of the error.

As with instructions, you can define tools at the session level. To change available tools, you must create a new session. As this is an array, you could pass in more than one tool. Foundation Models will call the appropriate tool when it feels appropriate. Calling multiple tools is supported. As this tool performs an intermediate step to convert a location name to geographic coordinates, you could also define two tools, one to perform each part, and the model could call them in sequence.

Run the app and enter the following prompt:

Give me the high temperature in Charlotte.

Your results will vary depending on the time of year, but you can see that August is the heart of summer in Charlotte (that’s 31.4 °C). To get a better idea of what’s going on, tap on the menu button and select the Session Transcript view. This will display the entire transcript, which lists tool calls and their corresponding responses. That’s very helpful for troubleshooting and refining your prompts.

Enter the following prompt:

What is the higher chance of rain for Atlanta tomorrow?

Tools offer a valuable way to extend Foundation Models with information beyond what is included in the model itself.