6. Building Tools for Foundation Models

Written by Bill Morefield

In Chapter Two, you worked through examples demonstrating some weaknesses of LLMs and Apple Foundation Models. One important limitation of any LLM is that it only contains information from its training data. It has no information about events after the training or information not included in the training. If you were to ask the earlier Foundation Explorer app something as simple as the current time, it would tell you that’s not possible.

This lack of knowledge provides a severe limitation to many things you might want to produce. For many apps, it would be helpful to access other data from the iPhone, such as contacts, calendar events, and health data. There are also cases where you would like to access external data from the web.

In some cases, you can collect the data and pass it to Foundation Models via instructions or a prompt. That requires you to anticipate the information and provide it in the best format. Apple Foundation Models provides a way to accomplish more through tool calling. Tools enable the model to plan and gather information to help it respond to the prompt. This action occurs autonomously and can access anything returned from a function call. You define a tool by implementing the Tool protocol in your struct or class. Then you include information on available tools when creating the LanguageModelSession.

Open and run the starter app for this chapter. You’ll see the start of a project to help the user pack for a trip occurring in the next few days. The most important information the model doesn’t have is the weather forecast for the travel destination. In this chapter, you’ll use Foundation Models to provide this packing advice and use Tools to provide this information to the model.

There are many ways to get weather forecast information, including Apple’s own WeatherKit. For this tutorial, you’ll use the United States National Weather Service to get forecast information. This service only works in the United States, but it is free and does not require an API key or any other sign-up. It requests that the user provide an agent name along with a valid email address for contacting the user in case of issues. Open NWSWeatherService.swift under the Services folder. Find the // Change Below to Your Email comment in the file and change email@example.com on the next line to your email address. Repeat the process with NWSWeatherService2.swift in the same folder.

The NWSWeatherService struct implements the service for retrieving the weather forecast via a web API. If you read through, you will notice that the API requires a latitude and longitude when requesting a forecast. You could ask the user to provide this information, but it would be better to convert their entered city name for them. Even better, you’ll write this as a Tool so the model can perform the city-to-coordinates conversion when needed.

There are a number of ways to convert a city name into a latitude and longitude, a process known as geocoding. Apple provides one as part of MapKit, and you’ll use that in this chapter. Create a new Swift file under the Tools folder named GeoLookupTool.swift. Replace the contents of the file with:

import MapKit
import FoundationModels

struct GeoLookupTool: Tool {
}

This code states that your GeoLookupTool will implement the Tool protocol. You must meet several requirements to implement this protocol. You must define a call(arguments:) method that accepts arguments of type ConvertibleFromGeneratedContent and returns a type conforming to PromptRepresentable. In practice, your return will be a String or a Generable object.

Add the following two properties to the start of the GeoLookupTool struct:

let name = "GeolocationTool"
let description = "This service returns the latitude and longitude for a city or location."

These two properties set the tool’s name and describe the tool’s purpose. The description provides context for the tool to the model. Try to keep descriptions short, as they become part of the context and can introduce latency. Now you’ll need to provide the arguments the tool expects. Add the following code after the description:

@Generable
struct Arguments {
  @Guide(description: "The name of the city to get the latitude and longitude for.")
  var location: String
}

Note that the argument uses the Generable macro. Everything you’ve already learned about guided generation in Chapter Five applies here. The only argument for this tool expects a location name as a String. While this example only has one argument, you can provide as many as your tool needs. It also uses the @Guide macro and description to better inform Foundation Models of this argument’s purpose and its relationship to the tool’s use.

Now you need to add data structures to contain the information returned by the tool, along with error handling. Add the following code to the top of the file after the import statements:

enum GeocodingError: Error {
  case invalidRequest
  case noMatchingLocation
}

struct GeocodedLocation {
  let name: String
  let latitude: Double
  let longitude: Double
}

The GeocodingError struct will contain information on any errors that might occur. The GeocodedLocation struct will contain the location name and the location’s latitude and longitude when the service succeeds. The final step in implementing the Tool protocol is to define the call(arguments:) method. Enter the following code after the Arguments struct:

func call(arguments: Arguments) async throws -> GeocodedLocation  {
  let placeName = arguments.location
}

The method begins by stating it is asynchronous and throws. A throwing method will propagate errors to the caller rather than handle them internally. In this case, all errors pass back to Foundation Models. You’ll look more into error handling later in the chapter.

You get the values from the Arguments struct, in this case, taking the location passed in by Foundation Models and storing it in a placeName variable. Any arguments sent to the tool, Foundation Models provides when it calls the tool. The method returns the GeocodedLocation struct.

Now add the following code to the end of the method to perform the geocoding:

// 1
guard let request = MKGeocodingRequest(addressString: placeName) else {
  throw GeocodingError.invalidRequest
}

// 2
let mapItems = try await request.mapItems
guard let item = mapItems.first else {
  throw GeocodingError.noMatchingLocation
}

// 3
let coordinate = item.location.coordinate
let name =
  item.name ??
  item.address?.shortAddress ??
  item.address?.fullAddress ??
  placeName

// 4
return GeocodedLocation(
  name: name,
  latitude: coordinate.latitude,
  longitude: coordinate.longitude
)

This code works by:

1. The MKGeocodingRequest class in MapKit facilitates looking up a geographic coordinate for a provided string. If creating the object fails, you return the GeocodingError.invalidRequest defined earlier in the file.
2. The request.mapItems call gets the array of MKMapItem objects that represent points of interest on the map that correspond to the location string passed during object instantiation. You then attempt to unwrap the first item in the array, and if that fails, return the GeocodingError.noMatchingLocation error.
3. The item.location.coordinate holds the coordinates of the location that you need to get the weather forecast. The MKMapItem object contains info on the point of interest. To get an official name, you first attempt to use the name property, then fall back to the address information, and finally to the string passed into the method if all else fails. This normalizes the name to ensure the coordinates and name match.
4. Finally, you return this information to Foundation models in a new GeocodedLocation structure.

If you attempt to build the app now, you will get a confusing error message. The error occurs because the return value of call(arguments:) must conform to PromptRepresentable. You can address this by making it a String, but for most return values, you’ll need to ensure the returned value conforms to the Generable protocol. Recall that the simple Bool, Int, Float, Double, Decimal, and Array types already implement this protocol. For other types, such as Date or custom data structures like GeocodedLocation, you must implement the protocol on the returned structure. Change the definition of GeocodedLocation to:

@Generable(description: "Contains a location name and the latitude and longitude for that location.")
struct GeocodedLocation {
  @Guide(description: "Location Name")
  let name: String
  @Guide(description: "Latitude of Location")
  let latitude: Double
  @Guide(description: "Longitude of Location")
  let longitude: Double
}

You do most of the work of implementing the Generable protocol by adding the @Generable macro before the class or struct. You also provide the description parameter, which provides Foundation Models with information about the structure’s purpose. You also use the @Guide macro to provide information on the meaning of the structure’s properties to Foundation Models. In some cases, the property names alone will suffice if they provide enough clarity, but that will require testing.

To see your new tool in action, open HelpMePackView.swift and add the following new method at the end of the packingSection subview:

func createNewSession() {
  // 1
  let instructions = """
    You are a tool to help users determine the latitude and longitude of locations provided to you.

    Use available tools to convert locations and city names into
    latitude and longitude
    """

  //2 
  session = LanguageModelSession(
    tools: [GeoLookupTool(),],
    instructions: instructions
  )
}

Here’s how this code sets up a session for tool use:

1. In earlier chapters, you’ve learned that instructions let you provide a prompt that guides the entire session. This prompt gives the model a purpose and tells it to use the tool you provided to find the coordinates for locations.
2. The new element here comes in the tools parameter. This parameter contains an array of tools available to the model. You pass in the GeoLookupTool you built in this section to the session.

Now it’s time to actually put the tool to use. Add the following new method after createNewSession():

func generatePackingList() {
  // 1
  let prompt = "Determine the latitude and longitude for \(information.destination).  Do not assume anything about the location from general knowledge."

  // 2
  Task {
    // 3
    isLoading = true
    defer {
      isLoading = false
    }
    // 4
    let stream = session.streamResponse(to: prompt)
    // 5
    do {
      for try await partialResponse in stream {
        information.packingRecommendation = partialResponse.content
      }
    // 6
    } catch {
      information.packingRecommendation = "Error: \(error.localizedDescription)"
    }
  }
}

1. The prompt is the most important new aspect of the app. As before, a good prompt requires time and testing to produce. This one provides a purpose for the prompt and includes the user-provided destination that the model should find a location for. It reinforces the use of the tool and not to rely on internal information.
2. The rest of the method should feel familiar from the early chapters. You start by creating a Task to wrap the asynchronous methods to come.
3. You repeat the pattern to set the isLoading property to provide visual feedback when the app is generating a response. The defer will ensure this is reset when the method ends, regardless of which method ends it.
4. You create an asynchronous streamed response using the streamResponse(to:options:) method.
5. The do loop will iterate over the stream as Foundation Models returns it, so you can provide faster feedback to the user. As you are returning a String, you simply set the Text to the latest response each time.
6. If anything goes wrong, for now, you’ll set the text to the error description.

To call these methods, find the // Add Button Action comment in the tripSection subview and replace it with:

createNewSession()
generatePackingList()

This will create a new session each time the user taps Generate Packing List. It will then call the generatePackingList() method to do the lookup. Run the app and enter a major United States city. After a pause, you will see the coordinates of that city.

To see how the tool integrated into the model, tap the page icon on the toolbar in the top-trailing corner. You will see that this simple prompt-and-response used a few hundred tokens. The transcript always begins with the instructions. The prompt to the session comes next. You provide these with the createNewSession() and generatePackingList() methods you created, respectively. You will then see the Tool call in the transcript, which shows Raleigh passed into the tool. The transcript then includes the information returned by the tool, which includes the coordinates. Finally, you see the response containing the coordinates.

Tool calls and outputs become part of the session’s context. This adds size to the already tight context length for Foundation Models.

Now that you see how to create a working tool, you need to add a second tool to look up the weather for the coordinates that this tool provides. You’ll do that in the next section.

Creating a Weather Forecast Tool

In the last section, you adapted an existing data structure to the Generable protocol. This was simple since all three properties of the GeocodedLocation struct were simple Swift types that already supported the protocol. Open NWSWeatherService.swift under the Services folder and look for the two structs that this service uses: WeatherSummary and WeatherPeriod. The WeatherSummary struct contains a String and an Array, both of which implement Generable. However, the WeatherPeriod struct contains two properties of type Date, which does not support the Generable protocol. You will need to provide a data structure that bridges the gap between the service’s data structure and the data structure required by Foundation Models.

Create a new Swift file called WeatherForecastTool.swift under the Tools folder. Replace the contents of the file with:

import FoundationModels

@Generable(description: "Weather Information for a location.")
struct WeatherInformation {
  let locationName: String
  let forecasts: [WeatherForecast]
}

This new struct matches the properties of WeatherSummary provided by the service, but uses the @Generable macro to implement the Generable protocol on it and provide a description. Notice you do not add a @Guide to either property since their names clearly explain their purpose to the model. Now add a replacement for WeatherPeriod better suited for Foundation Models:

@Generable(description: "Weather Forecast for a period.")
struct WeatherForecast {
  let name: String
  let startTime: String
  let endTime: String
  let temperature: Int
  let temperatureUnit: String
  let shortForecast: String
  let detailedForecast: String
  let windSpeed: String
  
  init(fromPeriod: WeatherPeriod) {
    self.name = fromPeriod.name
    self.startTime = fromPeriod.startTime.formatted(date: .numeric, time: .shortened)
    self.endTime = fromPeriod.endTime.formatted(date: .numeric, time: .shortened)
    self.temperature = fromPeriod.temperature
    self.temperatureUnit = fromPeriod.temperatureUnit
    self.shortForecast = fromPeriod.shortForecast
    self.detailedForecast = fromPeriod.detailedForecast
    self.windSpeed = fromPeriod.windSpeed
  }
}

Despite its length, this new WeatherForecast struct contains only two changes to the properties in the WeatherPeriod struct used by the service. It changes the start and end times for the forecast to Strings. You use the formatted(date:time:) instance method to convert the date into a localized string in the format 1/17/2021 4:03 PM. You also provide a convenience initializer that takes a WeatherPeriod struct, making it easier to convert the WeatherPeriod returned by the struct into a WeatherForecast used by the tool. You again let the property names provide the context.

With the data structure defined, continue with the following code to create the Tool.

struct WeatherForecastTool: Tool {
  let name: String = "WeatherForecastTool"
  let description = "This service returns the weather forecast for the next seven days for a given latitude and longitude."
}

This should look familiar, as the base attributes of implementing the Tool protocol are the same. Now add the following code to define the arguments to this tool at the end of the struct:

@Generable
struct Arguments {
  var latitude: Double
  var longitude: Double
}

This time, your tool expects a latitude and longitude of the location to get the weather forecast. As before, the Arguments must be Generable. Since Double has support for the Generable protocol by default, you don’t need to define anything for them. To finish the Tool, implement the call(arguments:) method by adding this code to the end of the struct:

func call(arguments: Arguments) async throws -> WeatherInformation {
  // 1
  let service = NWSWeatherService()
  // 2
  let forecast = try await service.forecast(latitude: arguments.latitude, longitude: arguments.longitude)
  // 3
  let convertedForecast = WeatherInformation(
    locationName: forecast.locationName,
    forecasts: forecast.periods.map { WeatherForecast(fromPeriod: $0) }
  )
  return convertedForecast
}

Here’s how the tool works:

1. You first create an NWSWeatherService object to get the forecast.
2. Now you call the service using the latitude and longitude passed in through Arguments. You then assign the result to forecast.
3. Next, you create an object of the WeatherInformation struct built to support the Generable protocol. The locationName property copies directly from the property of the same name inside WeatherSummary. To fill the forecasts array, you use the map(_:) method on the original array in the WeatherInformation struct. This method applies the closure to each element of the original array and returns a new array containing the results. You use that convenience initializer to convert the WeatherPeriod to a WeatherForecast, resulting in an array of WeatherForecast elements. You then return this new WeatherInformation struct.

Now that you’ve implemented this new Tool, you are ready to wire both tools into Foundation Models in the next section to help a user pack for their trip.

Integrating Tools into Foundation Models

Finally, it’s time to put all this work to use. Open HelpMePackView.swift and change the createNewSession() method to:

func createNewSession() {
  let instructions = """
    You are a packing assistant that creates practical packing lists for travelers.

    Use the available tools whenever current weather information is needed.
    Also use available tools to convert locations and city names into
    latitude and longitude
    Do not guess weather conditions, temperatures, or precipitation.

    When creating a packing list:
    - First determine the trip destination and dates.
    - Use tools to get the forecast for the destination and travel dates.
    - Base weather-related recommendations on tool results.
    - Recommend only items that are useful for the trip conditions.
    - Keep the list concise, realistic, and grouped by category.
    - Explain briefly why weather-specific items are included.
    """

  session = LanguageModelSession(
    tools: [GeoLookupTool(), WeatherForecastTool()],
    instructions: instructions
  )
}

This new prompt emphasizes that the model should not guess about weather information. Rather, it should use the Tool. The instructions also prompt the model to use the coordinate lookup tool. You also add the new WeatherForecastTool() to the tools parameter.

Now it’s time to actually generate the guidance on packing using Foundation Models. Change the prompt inside generatePackingList() to:

let prompt = """
Create a weather-aware packing list for this trip.

Destination: \(information.destination).
Travel dates: \(startDate.formatted(date: .numeric, time: .omitted)) through \(endDate.formatted(date: .numeric, time: .omitted)).

1. Retrieve the weather forecast for the travel dates.
2. Use the forecast to decide what clothing and accessories are needed.

Do not assume weather conditions from general knowledge. Provide a summary of the forecast when you use it to produce your suggestions.
"""

The new prompt provides a purpose and includes the destination and travel dates provided by the user. It then reinforces using the weather forecast to guide its suggestions and reminds it to use tools rather than relying on general knowledge, such as knowing that Phoenix, Arizona, is hot in the summer. It also tells the model to show where the weather forecast influences the suggestions. A good prompt requires time and testing, and this one is the result of several iterations to produce consistent results.

Now run the app and enter a city in the United States. Then tap the Generate Packing List button.

You will notice that it takes several seconds for the response to begin streaming. During this time, Foundation Models first uses the GeoLocation tool to convert the location name into coordinates. It then calls the weather forecast tool to get the forecast for those coordinates. Once it has all the necessary information, the model begins streaming the response. This delay depends heavily on the service calls. The geocoding lookup takes a fraction of a second, while the weather call takes several seconds. Make sure to factor in this extra time if you are building an app that calls external services.

Open the transcript for this session. You’ll first note the very large context length of almost 3/4 of the context in a single prompt and response. The weather forecast uses most of this context.

If you start seeing issues with the length, you can trim the tokens used by:

1. Reduce or eliminate @Guide descriptions by using self-explanatory property names or keeping the descriptions to a short phrase.
2. Include no more tools than you need in a session.

You can also reduce token usage by running a tool in advance for the model. In this app, you could implement geocoding as a service that your app calls. You then pass the coordinates directly in the prompt, rather than relying on the model to perform this conversion. The trade-off is that the model then has no access to that information itself. The correct balance will depend on your app’s needs.

Now that you have a working app that provides tools to Foundation Models, you’ll look at error handling for tools in the next section.

Error Handling

When you created the two tools in this chapter, you marked both using the throws keyword. This lets the function propagate any errors to the scope that called it, in this case, Foundation Models. If you did not mark the method as throws, then you would need to handle errors inside the function. This behavior can be desirable if you are returning a string, in which case you could return a string that describes what went wrong to the model.

In this app, you have a generic catch handle that shows any errors. To see this in action, enter a location outside the United States like Toronto, Ontario and tap Generate Packing List.

This generic error tells you the basics: that the exception occurred in the WeatherForecastTool and that the service threw the error. As with other errors, you can provide a specific handler to get more information about what went wrong. Open HelpMePackView.swift and find the generatePackingList() method. After comment six, before } catch {, add the following code:

// 1
} catch let error as LanguageModelSession.ToolCallError {
  var errorString: String
  // 2
  errorString = "Error occurred in \(error.tool.name)\n"
  // 3
  if case .invalidURL = error.underlyingError as? WeatherServiceError {
    errorString += "Service used an invalid URL.\n"
  }
  if case.invalidResponse = error.underlyingError as? WeatherServiceError {
    errorString += "Service returned an invalid response .\n"
  }
  // 4
  information.packingRecommendation = errorString

The lack of a closing brace is because the code that follows will begin with the closing brace for the final catch block. This code provides a more specific tool handler for the failure case.

1. You catch the LanguageModelSession.ToolCallError which Foundation Models will throw when it encounters an exception when running a Tool. The code stores the error information in the error variable.
2. The tool property on the error contains the information on the tool that threw the error. You show the name of the tool, which will match the name provided when implementing the protocol.
3. In this case, you’ll only work with the WeatherServiceError for simplicity, but you would handle all possible tools that could fail in a real app. These lines match one of the two possible errors and add a more readable error string to the errorString.
4. You again set the text of the packing recommendation according to the errorString.

Run the app, then enter a city outside the United States. You’ll see a cleaner error.

If you view the session transcript after the error, you will see that the call to the tool is not stored in the session transcript, nor is anything about the error in the response. While this keeps the transcript clean and conserves the limited tokens for successful interactions, it makes it more difficult to determine what went wrong.

You can only show information provided by the tool when the exception occurs. For a tool that does the work locally, it should return as much information as possible. When calling a service, such as these tools do, you should return all information the service provides to give the user better feedback.

In this case, you’ll improve the error handling by using an updated weather service that better explains error conditions. Open NWSWeatherService2.swift under the Services folder. Near the start of the file, you’ll see a new WeatherServiceError2 defined that provides both more error cases along with more detail about what went wrong. This is information you can then provide to the user as feedback. You will also see a errorDescription computed property that returns a string description customized to each error.

Open WeatherForecastTool.swift and find the call(arguments:) method. Change the line after comment one to:

let service = NWSWeatherService2()

This calls the new weather service, which provides richer error information in failure states. Return to HelpMePackView.swift and find the catch block for LanguageModelSession.ToolCallError. Delete the code between comments three and four and replace it with:

if let underlyingError = error.underlyingError as? WeatherServiceError2 {
  errorString += underlyingError.errorDescription ?? error.localizedDescription
}

This code attempts to cast the error to WeatherServiceError2 used by the new NWSWeatherService2. Instead of handling each case, you use the errorDescription on WeatherServiceError2 to provide a detailed error.

Run the app now, then enter an invalid location. You will see a much more useful error, though it’s more for developers than for end users of the app. The called URL shows the latitude and longitude of the failed tool call. And the description provides the real reason for the error: Data Unavailable For Requested Point.

This makes sense, as the requested location isn’t within the National Weather Service’s coverage area. Knowing this, you can provide a better error message to the user. Remove the code between comments three and four and replace it with:

// 1
if let underlyingError = error.underlyingError as? WeatherServiceError2 {
  // 2
  if case let .serverError(_, message) = underlyingError {
    // 3
    if message?.contains("Data Unavailable For Requested Point") ?? false {
    errorString += """
    The requested location is not covered by the National Weather Service.
    
    Please Check Your Location and Try Again.
    """
    } else {
      errorString += underlyingError.errorDescription ?? error.localizedDescription
    }
  // 4
  } else {
    errorString += underlyingError.errorDescription ?? error.localizedDescription
  }
}

Here’s how this code handles the invalid location while retaining previous functionality in other cases:

1. You first attempt to match the underlyingError property of the parent LanguageModelSession.ToolCallError error with a WeatherServiceError2. If this succeeds, the underlyingError variable will hold a WeatherServiceError2.
2. Since you only care about the single serverError(statusCode:message:) condition for this case, you again use pattern matching to match for that specific WeatherServiceError2 case. If successful, then you continue to step 3.
3. This code uses optional chaining on the optional message property of the serverError(statusCode:message:). If message is nil, then the nil coalescing operator sets the condition to false. If not nil, then it searches the message for the Data Unavailable For Requested Point string. If it finds this string, then it adds the more specific error message for the user. If the string is not found or message is nil, then it will use the errorDescription as before, falling back on the generic localizedDescription for the error if errorDescription is nil.
4. For any WeatherServiceError2 other than .serverError(statusCode:message:), the previous behavior of showing the errorDescription property on the error remains.

Run the app and enter a location outside the United States. You’ll see the new code handles this error and provides better feedback to the user.

Conclusion

In this chapter, you extended Foundation Models beyond its limited built-in knowledge by building tools to give the model access to real-world data. You first built a geocoding tool that converts city names into coordinates, then a weather forecast tool that uses those coordinates to retrieve live forecast data from a web API. The result is an app that can account for weather forecasts rather than relying on assumptions about general weather patterns. The ability to provide access to data outside of the trained model to Foundation Models gives you a powerful way to extend the capabilities of Foundation Models.

Key Points

• Foundation Models can only access information present in its training data. Adding Tools lets the model retrieve current or device-specific data.
• Implementing a tool requires conforming to the Tool protocol. This protocol expects a tool name and a description, along with implementing an Arguments struct marked @Generable and a call(arguments:) method that uses those arguments.
• Tool calls must return PromptRepresentable types, in most cases done by returning a String or a struct that implements the @Generable protocol.
• Tool calls and their results are part of the session context, which can consume a significant portion of the available token budget for large responses such as the weather forecast.
• Errors during tool execution result in a LanguageModelSession.ToolCallError in the session response. This error exposes information about the tool that failed and the underlying error for targeted handling and user feedback. This underlying error is where you include information so your app can provide actionable feedback to the user.