Multimodal Integration with OpenAI

This lesson provides coding examples for controlling image fidelity when making API requests to OpenAI’s GPT-4 Vision model and extracting structured information from the model’s responses.

Using the detail parameter helps to manage both the accuracy of the image analysis and the processing time. Here’s how you can set the fidelity to low for faster processing:

# Use the detail parameter when analyzing an image with GPT-4 Vision

# Text prompt
prompt = "How much calories are in this food?"

# Model
openai_model = "gpt-4o"

# Creating an API request
response = client.chat.completions.create(
  model=openai_model,
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": prompt},
        {
          "type": "image_url",
          "image_url": {
            "url": ramen_image_url,
            "detail": "low"
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

choice = response.choices[0]
print(choice.message.content)

Here, the "detail": "low" setting instructs the model to process the image more quickly, using fewer resources, which can be beneficial for faster responses and cost savings when precision is not the primary concern.

Run the code and you get the following result:

The dish shown in the image is a bowl of ramen. The caloric content of a
bowl of ramen can vary based on the ingredients and portion size, but on
average:

- A typical bowl of ramen (including broth, noodles, pork, vegetables,
and toppings) usually contains between 400 to 800 calories.

This estimate can vary significantly depending on factors such as the
type and amount of noodles, the richness of the broth, the size of the
serving, and additional toppings.

To efficiently use the results from GPT-4 Vision, it’s helpful to format the output into a structured JSON schema. This ensures that the relevant data is easily accessible and can be parsed programmatically.

Here’s an example of code that uses a schema to generate structured outputs when creating an API request:

# Extracting specific information when analyzing an image from GPT-4 Vision

from pydantic import BaseModel

class FoodCalories(BaseModel):
    total_calories: str
    analysis: str

# Use JSON format to make extracting information easier

# Text prompt
prompt = "How much calories are in this food?"

# Model
openai_model = "gpt-4o-2024-08-06"

# Creating an API request
response = client.beta.chat.completions.parse(
  model=openai_model,
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": prompt},
        {
          "type": "image_url",
          "image_url": {
            "url": ramen_image_url,
            "detail": "low"
          },
        },
      ],
    }
  ],
  response_format=FoodCalories,
  max_tokens=300,
)

choice = response.choices[0]
print(choice.message.content)

Running this code would generate a structured output like the following:

{
  "total_calories":"Approximately 400-500 calories",
  "analysis":"This bowl of ramen likely contains noodles, broth, pork,
  green onions, bamboo shoots, seaweed, and fish cake. The broth and
  noodles contribute the most to the calorie count, while the toppings
   like pork and the egg add additional calories."
}

By defining this schema, you ensure that the model’s output fits into the expected structure, making it easier to extract specific information (e.g., the calorie count and the analysis provided by the model).

The model gpt-4o-2024-08-06 should be used when working with structured outputs. The schema is passed to the response_format parameter.