Multimodal Integration with OpenAI

Transcript

This lesson explores how to use the OpenAI API in your Python projects. The OpenAI API allows you to integrate powerful language models, such as GPT-4, into your applications. You’ll learn the process of making API requests, handling responses, and using structured outputs with the help of Pydantic.

To set up your development environment for using the OpenAI GPT-4 Vision API, please refer to Lesson 1: Introduction to Multimodal AI. This lesson covers the installation of necessary libraries and the configuration of your environment.

To interact with the OpenAI API, you need an API key. If you’ve already followed the instructions in the previous lesson, you should have an API key stored in a .env file. If not, please follow the instructions in the previous lesson to obtain an API key.

Once you have your API key, you can authenticate your API requests as follows:

# Load the OpenAI library
from openai import OpenAI

# Set up relevant environment variables
from dotenv import load_dotenv

load_dotenv()

# Create the OpenAI connection object
client = OpenAI()

You send images to the GPT-4 Vision API endpoint in one of two ways: using URLs and uploading base64 encoded images. You’ll start with image URLs.

Before analyzing an image with the API, it’s often helpful to visually inspect it yourself. To do this in Jupyter Lab, you must download the image, create an image object, and display it.

Start by importing the necessary libraries:

# Show images in Jupyter Lab

# Import necessary libraries
import requests
from PIL import Image
from io import BytesIO
import matplotlib.pyplot as plt

• requests for downloading the image
• Image from PIL (Pillow library) for creating the image object
• BytesIO for handling the image data
• matplotlib.pyplot for displaying the image

Now, you can fetch the image from a URL, create an image object, and display it in your Jupyter Lab:

# Set image URL
ramen_image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/
  e/ec/Shoyu_ramen%2C_at_Kasukabe_Station_%282014.05.05%29_1.jpg/
  1280px-Shoyu_ramen%2C_at_Kasukabe_Station_%282014.05.05%29_1.jpg"

# Fetch the image from the URL
response = requests.get(ramen_image_url)
img = Image.open(BytesIO(response.content))

# Display the image
plt.figure(figsize=(12, 8))
plt.imshow(img)
plt.axis('off')
plt.show()

• First, you define the URL of the image you want to analyze.
• Then, you use requests.get() to download the image data.
• After that, you create an image object using the Python Imaging Library (PIL) by opening the downloaded data with Image.open() and BytesIO().
• Then, you set up a figure using plt.figure(), specifying the size if necessary.
• Afterward, you display the image with plt.imshow().
• Then, you remove the axes using plt.axis('off').
• Finally, you render the image in your Jupyter Lab with plt.show().

This process lets you visually confirm the image you’re about to analyze with the GPT-4 Vision API, ensuring you’re working with the correct image and giving you a chance to visually inspect it before processing.

You’d see the ramen image.

This ramen looks delicious, but because you’re on a diet, you need to know the calorie count before deciding whether to eat it. You’ll use the GPT-4 Vision API to analyze it. It’s important to note that the GPT-4 Vision API uses the same endpoint as the standard OpenAI text generation API. The key difference is that instead of sending only text, you include an image URL in your request as well.

To make an API request, you first define your prompt as a question about the image. In this case, you’re asking about the calorie content of the food in the image. Then, you specify the model you want to use, which is “gpt-4o” for GPT-4 with vision capabilities.

Add the following code to make an API request using the image URL:

# Use an image URL when analyzing an image with GPT-4 Vision

# Text prompt
prompt = "How much calories are in this food?"

# Model
openai_model = "gpt-4o"

# Creating an API request
response = client.chat.completions.create(
  model=openai_model,
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": prompt},
        {
          "type": "image_url",
          "image_url": {
            "url": ramen_image_url,
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

choice = response.choices[0]
print(choice)

• First, you define the prompt you want to send to the model. In this case, you’re asking about the calorie content of the food in the image.
• Then, you specify the model you want to use, which is “gpt-4o” for GPT-4 with vision capabilities.
• Next, you create an API request using the client.chat.completions.create() method. This method takes several parameters, including the model, messages, and max_tokens.
• Finally, you print the Choice object to see the response from the model.

client.chat.completions.create() is designed to interact with GPT models capable of processing both text and image inputs. This function requires a messages parameter, which is a list containing a single dictionary. This dictionary has two key-value pairs:

• "role": "user" indicates that the input is coming from the user.
• "content": This is a list containing two dictionaries: one for text and one for the image.

The text input is provided as a dictionary with "type": "text" and the actual prompt in the text field.

The image input is provided as a dictionary with "type": "image_url". The image URL is nested in another dictionary under the image_url key. This makes the difference between the text generation and the image analysis.

The max_tokens parameter is set to 300, limiting the generated response’s length.

From the code above, you get the Choice object printed:

Choice(finish_reason='stop', index=0, logprobs=None,
  message=ChatCompletionMessage(content="The image
  shows a bowl of ramen, ... and serving sizes.",
  refusal=None, role='assistant', function_call=None,
  tool_calls=None))

If you want to get the content only, add and run the following code:

# Extract the content
print(choice.message.content)

With this, you get the calories:

The image shows a bowl of ramen, which typically includes noodles, broth,
slices of pork, vegetables, and garnishes. The calorie content can vary
significantly based on the specific ingredients and portion sizes. On
average, a typical serving of ramen similar to the one in the image might
contain approximately 400-600 calories. Here is a rough breakdown:

- Noodles: 200-300 calories
- Broth (depending on type and amount): 50-150 calories
- Pork slices: 100-150 calories
- Vegetables and garnishes (scallions, seaweed, narutomaki): 20-50
  calories

Please note that these values are approximate and can vary based on
specific recipes and serving sizes.

This structure allows you to send both text and image inputs to the model in a single API call, enabling the model to generate responses based on both textual and visual information.

After making the request, you receive a response object. The actual content of the response is contained in the choice.message.content field. You can print this to see the model’s analysis.

The output from the GPT-4 Vision API can vary. Even when given the same image and prompt multiple times, you might receive slightly different responses. You can adjust this randomness level using the temperature parameter. The value is between 0 and 1, with lower values producing more deterministic results and higher values introducing more randomness.

In the example output provided, the model estimates the calorie content of the ramen bowl to be about 400-600 calories, breaking down the estimate by components like noodles, broth, pork slices, and vegetables.

It’s too bad that you can’t eat the ramen because it already crosses your calories threshold today. What about eating fried rice? You have to calculate the calories again. This time, you want to use a base64 encoded image.

Now, you’ll learn how to convert an image into a base64 encoded format, display the image in Jupyter Lab, and use that base64 encoded image as part of an API request.

To send an image in a request, you first must convert it into a base64 encoded string. This encoding ensures the image can be easily embedded in a text-based format like JSON. You can do this using Python’s base64 library.

Here’s how you can convert an image into base64 encoding:

# Convert an image to a base64 encoded image
import base64

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

image_path = "images/fried_rice.png"
base64_image = encode_image(image_path)

This function reads the image in binary mode and then encodes it to a base64 string. The .decode('utf-8') method ensures that the encoded data is returned as a string rather than as bytes.

As you’ve done with the first image, before you proceed with the API request, it’s a good practice to view the image directly in Jupyter Lab to ensure it’s the correct one. Add the following code to end of the previous code snippet:

# Show the image in Jupyter Lab
img = Image.open(image_path)

plt.imshow(img)
plt.axis('off')
plt.show()

This code snippet opens and displays the image in your notebook, without any distracting axes or labels. You’d see this image:

Now that the image is successfully converted to a base64 string, you can send it as part of an API request. In this example, you’ll send the image along with a text prompt to an AI model.

To include the image in the API request, use the image_url field, formatted as a data: URL with the base64-encoded image prefixed by "data:image/png;base64,".

Add the following code to call the API request using the base64-encoded image:

# Upload the base64 encoded image to the OpenAI API server

# Text prompt
prompt = "How much calories are in this food?"

# Model
openai_model = "gpt-4o"

# Creating an API request
response = client.chat.completions.create(
  model=openai_model,
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": prompt},
        {
          "type": "image_url",
          "image_url": {
            "url": f"data:image/jpeg;base64,{base64_image}",
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

choice = response.choices[0]
print(choice)

You get the Choice object printed:

Choice(finish_reason='stop', index=0, logprobs=None,
  message=ChatCompletionMessage(content='It is
  difficult to provide an...calorie content.', refusal=None,
  role='assistant', function_call=None, tool_calls=None))

Like before, if you want to get the content only, add and run the following code:

# Extract the content
print(choice.message.content)

You get the calories:

It is difficult to provide an exact calorie count for the food in the
image without knowing the specific quantities and exact ingredients
used. However, I can provide an approximate calorie breakdown based
on typical portions of the ingredients shown:

- **Fried rice (1 cup):** Approximately 200-250 calories
- **Fried egg (1 large):** Approximately 90-100 calories
- **Sausage slices (1 sausage):** Approximately 100-150 calories
- **Lettuce and cucumbers (small amount):** Approximately 10-20
  calories
- **Lime wedges (2 slices):** Approximately 5 calories

Summing these estimates together, the total calorie count is approximately
 405-525 calories for the plate shown. Please note that cooking methods
 and variations in portion size can significantly affect the actual calorie
 content.

This response gives you a general idea of the calorie content of the food in the image.

Being hungry is natural, and eating is an essential part of life. Sometimes, you might want to eat food with less calories. Fortunately, you can compare calories between those two food servings with GPT-4 Vision.

To send more than one image in a request, you can send two dictionaries with the "image_url" field. Add and run the following code that sends an image with a URL and a base64-encoded image:

# Creating an API request consisting of two images

# Text prompt
prompt = "Which food has less calories?"

# Model
openai_model = "gpt-4o"

# Creating an API request
response = client.chat.completions.create(
  model=openai_model,
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": prompt},
        {
          "type": "image_url",
          "image_url": {
            "url": f"data:image/jpeg;base64,{base64_image}",
          },
        },
        {
          "type": "image_url",
          "image_url": {
            "url": ramen_image_url,
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

choice = response.choices[0]
print(choice)

You get the Choice object printed:

Choice(finish_reason='stop', index=0, logprobs=None,
  message=ChatCompletionMessage(content="It's
  difficult to determine...portion sizes.", refusal=None,
  role='assistant', function_call=None, tool_calls=None))

If you want to get the content only, add and run the following code:

# Extract the content
print(choice.message.content)

You get which food serving has less calories:

It's difficult to determine the exact calorie count without specific
measurements and detailed nutritional information, but generally
speaking:

1. The dish with rice, fried egg, sausage, and vegetables is likely
to be higher in calories, especially due to the presence of sausages
 and fried egg, which are typically calorie-dense.

2. The bowl of ramen may have fewer overall calories, but this can
 vary widely depending on the ingredients used. Ramen can have a
  high calorie count as well, particularly if it contains fatty
   pork, rich broth, and noodles.

Given the images and typical ingredient usage, the rice dish
(first image) is likely to have a higher calorie count than the
 ramen (second image). However, actual calorie content can vary
  based on the specific preparation methods and portion sizes.

Then, you proceed to eat a bowl of ramen.