Advanced Chat Completion Techniques - Instruction

Fine-Tuning your Response

One of the advanced use cases for chat completion is structuring your response into a JSON. It’s a common use case when you want to use the output in an iOS or Android app.

What if you want to create a fact checker app that would verify if the user’s input statement is true or not, given a reference URL?

This is useful when arguing with your friends about a topic, and you can’t decide who is right or wrong. :]

Deterministic Output

One problem you might face in implementing this is that the output might say true one time and false one time for the same statement and source. You want to make this as deterministic as possible. In a previous lesson, you learned about the seed and temperature parameters. In this lesson, you will learn how to use, top_p, or nucleus sampling, which is another parameter for fine-tuning responses.

This parameter ranges from 0 to 1. It defaults to use all the tokens in the training data. It’s set to declare how much of the top tokens are to be used in the generated output. For example, using a value of 0.8, will use a 80% of the most popular tokens in the training data, while using a value of 0.05 will only use the top 5% of the tokens.

You can use this to reduce or increase the use of uncommon words in your output.

First Attempt

In Jupyter Lab, navigate to 04-advanced-chat-completion/Starter/lesson4.ipynb. You should see some code already there. Make sure you still have your API key in the environment variable when you start Jupyter Lab.

From your last lesson, you’ve learned how to do basic chat completion. You should see if that already accomplishes your task.

Create a new cell. Then add the following code:

# 1
user_statement = "White sand is made from fish poop https://oceanservice.noaa.gov/facts/sand.html"

messages = [
 {"role": "user", "content": user_statement}
]

# 2
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=messages
)
print(response.choices[0].message.content)

You declared the user’s statement that you want to fact-check. The statement includes a reference URL. Then, you did a basic chat completion call and printed the output.

Now run the cell. You should get an output like this:

The statement that "white sand is made from fish poop" is a simplification of a more complex process. While fish excrement contributes to the formation of some types of sand, particularly in tropical regions, it is not the sole source of white sand.

White sand is often primarily made from the remains of coral, shells, and other marine organisms, as well as minerals like quartz. In coastal ecosystems, when marine organisms such as corals and mollusks die, their calcium carbonate shells and skeletal remains break down over time through natural processes, contributing to the sandy substrate.

Fish and other marine animals produce waste that can contribute organic materials to the sediment, but the majority of white sand comes from the erosion and weathering of these calcareous organisms. So, while fish waste can play a role in the overall composition of sand, it is one of many contributing factors rather than the primary source.

It’s a pretty response, but it doesn’t directly tell you if it’s true or not. Also, it didn’t seem to use the reference. How can you further improve this to align better with the task?

JSON Output

The first thing to note about your previous output was that it was purely unstructured text. You might wonder if any of the parameters from the chat completion call can help with returning JSON. And you’re exactly right. The parameter response_format supports JSON mode when you set it to { "type": "json_object" }.

Go and try adding that to the chat completion by replacing the previous code with this:

...
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=messages,
  response_format={ "type": "json_object" }
)
...

Run the cell and see what error you get. It should say:

'messages' must contain the word 'json' in some form

You wouldn’t expect the user to input the word json anywhere. :]

System Prompt

Enter system prompts. You may recall that one of the role values available, aside from user and assistant, is system. You should be able to add the word json to the system prompt.

Also, since you would like a JSON output, you probably want it to have some structure. Say the structure would be:

{
  "isFactTrue": <true or false>
  "explanation": <explanation>
}

You should also be able to format your response like the above using the system prompt.

Try using it now. Replace the code in the last cell with this:

# 1
user_statement = "White sand is made from fish poop https://oceanservice.noaa.gov/facts/sand.html"


# 2
SYSTEM_PROMPT = (
  "You are a fact checker. Verify the validity of the sentence provided by the user, given a reference. You must return a response in JSON format:"
  "{'isFactTrue': <true or false>, 'explanation': <explanation> }"
)
messages = [
 {"role": "system", "content": SYSTEM_PROMPT},
 {"role": "user", "content": user_statement}
]

# 3
response = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=messages,
 response_format={ "type": "json_object" }
)
print(response.choices[0].message.content)

Here’s what you did:

1. Initialized the user statement variable with the same statement.
2. Declared a system prompt. It has instructions on what it should do about the user’s input. It also includes an instruction that it has to respond in JSON.
3. Added a message with the role system containing the SYSTEM_PROMPT. The rest of the chat completion call stayed the same.

Now, run the cell, and you should see a formatted response like below:

{
  "isFactTrue": false,
  "explanation": "White sand is primarily composed of tiny fragments of coral, shells, and calcium carbonate, rather than being made from fish poop. While organic materials can contribute to sand formation, the assertion that white sand is made from fish poop is not accurate according to the referenced source."
}

This is already a huge improvement, since it meets the requirement of having a JSON output. Although you might notice that it still didn’t seem to use the URL that the user gave to check if the fact is true.

Tool Calls

To fully use the reference URL, first your code has to notice the URL, and then decide that it needs to perform a web request in order to fetch the contents and pass the URL to that request.

For this use case, the chat completion call provides a parameter called tools, which used to be the now deprecated functions parameter. With tools, you can provide the name of your function, a description of what it does, together with its parameters, and the chat completion call will respond to you if any tools need to be executed in order to complete the response.

Defining a Tool

To define a tool, you should first define a Python function. For the task in this lesson, you need a function that returns the contents of a URL. Add a new cell and write the following:

def get_text_from_url(url):
  if url == "https://oceanservice.noaa.gov/facts/sand.html":
    return '''The famous white-sand beaches of Hawaii, for example, actually come from the poop of parrotfish. The fish bite and scrape algae off of rocks and dead corals with their parrot-like beaks, grind up the inedible calcium-carbonate reef material (made mostly of coral skeletons) in their guts, and then excrete it as sand.'''

Here, you’re defining a function get_text_from_url() that takes a URL. For now, you can hardcode that if the URL is the sand fact one, then you return a hardcoded text. You can, as an exercise, make it an actual URL request and fetch the data.

Once you have a function that returns the contents of the URL, you also need to define your tool. Add the following code to the same cell:

tools = [
  {
    # 1
    "type": "function",
    # 2
    "function": {
      # 3
      "name": "get_text_from_url",
      # 4
      "description": "Get the text contents of a url",
      # 5
      "parameters": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string",
            "description": "The url",
          },
        },
        "required": ["url"],
      },
    },
  }
]

Here you’re declaring:

1. type as function. Currently, the only supported type.
2. The function object.
3. name of the function. You can name it anything, but it’s easier to match the function that you would call later.
4. description of what this function does. It helps chat completion decide which tool is needed to be called.
5. parameters of the get_text_from_url function. Basically saying that URL is required and it’s a string of the URL.

That was plenty of lines of code. But now you’re ready to use it. Add these lines of code to call the tools you declared.

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=messages,
  tools=tools,
  response_format={"type": "json_object"},
)
print(response.choices[0].message.content)

Then, run the cell. You’ll see the output printed as None. This is because you need to process the response differently. Look at the response by replacing the print statement with the following:

print(response)

Then, run it again and see the output. You should see something like this:

ChatCompletion(
  choices=[
    Choice(
      finish_reason='tool_calls',
      index=0,
      logprobs=None,
      message=ChatCompletionMessage(
        content=None,
        role='assistant',
        function_call=None,
        tool_calls=[
          ChatCompletionMessageToolCall(
            id='call_m5CgudT097SnO6no43hoFmeb',
            function=Function(
              arguments='{"url":"https://oceanservice.noaa.gov/facts/sand.html"}',
              name='get_text_from_url'
            ),
            type='function'
          )
        ],
        refusal=None
      )
    )
  ],
  ...
)

Because you’ve now declared tools, the message in the choices response can now include a tool_calls object. This means you should perform the tool call in order to finish the response.

To extract it, create a new cell and write the following:

response_message = response.choices[0].message
tool_calls = response_message.tool_calls
print(tool_calls)

Run the cell, and you should see the single tool call from above like so:

[
  ChatCompletionMessageToolCall(
    id='call_m5CgudT097SnO6no43hoFmeb',
    function=Function(
      arguments='{"url":"https://oceanservice.noaa.gov/facts/sand.html"}',
      name='get_text_from_url'
    ),
    type='function'
  )
]

Now, this contains enough information about which function to call and with which arguments to use.

Create a new cell and write the following to perform the tool call.

# 1
import json

if tool_calls:
  # 2
  available_functions = {
    "get_text_from_url": get_text_from_url,
  }
  # 3
  messages.append(response_message)
  # 4
  for tool_call in tool_calls:
    # 5
    function_name = tool_call.function.name
    function_to_call = available_functions[function_name]

    # 6
    function_args = json.loads(tool_call.function.arguments)

    # 7
    function_response = function_to_call(
      url=function_args.get("url"),
    )
    # 8
    messages.append(
      {
        "tool_call_id": tool_call.id,
        "role": "tool",
        "name": function_name,
        "content": function_response,
      }
    )

  # 9
  second_response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    response_format={"type": "json_object"},
  )

  # 10
  print(second_response.choices[0].message.content)

You just did the following:

1. Import json, for parsing the tool_calls JSON string.
2. Create a dictionary to easily refer to functions by name.
3. Extend messages with assistant’s reply.
4. Loop over the tool_calls.
5. Get the function to call using the name in the tool call.
6. Parse the arguments from the JSON in tool_call.function.arguments.
7. Call the function with the arguments that you just got.
8. Add a message to your message history. The message has the special role: tool since this is the output of a tool call.
9. Perform another chat completion with the updated messages array but keeping the same parameters except for the tools parameter. You don’t want it to perform the tool calls again.
10. Print the new response.

Finally, you should run the cell and get a similar response to below.

{
  "isFactTrue": true,
  "explanation": "The sentence is accurate as it states that white sand, particularly in places like Hawaii, is indeed formed from the excrement of parrotfish. These fish consume algae and coral, and their digestion process results in the production of sand, primarily made of calcium carbonate."
}

Now, see that the fact is considered as true after checking the reference.