Multimodal Integration with OpenAI

Transcript

This lesson explores the capabilities and applications of DALL-E for image generation. You will learn how to generate images based on text prompts and use various parameters to adjust the image generation.

To set up your development environment for using the DALL-E API, please refer to Lesson 1: Introduction to Multimodal AI. This lesson covers the installation of necessary libraries and the configuration of your environment.

Having said that, you also need to install another library, Pillow:

# Install dependencies
!pip install Pillow

The Pillow library is used for creating image objects.

Similar to previous lessons, you can authenticate your API requests as follows:

# Load the OpenAI library
from openai import OpenAI

# Set up relevant environment variables
from dotenv import load_dotenv

load_dotenv()

# Create the OpenAI connection object
client = OpenAI()

Then, import necessary libraries.

# Import necessary libraries
import requests
from PIL import Image
from io import BytesIO
import matplotlib.pyplot as plt
import base64

To streamline the process of generating and displaying images, you can define some helper functions.

First, define a function to generate an image using DALL-E:

# Define a function to generate an image using DALL-E
def generate_image(client, model, prompt, size, quality=None,
  style=None, response_format='url', n=1):
    params = {
        'model': model,
        'prompt': prompt,
        'size': size,
        'n': n,
        'response_format': response_format
    }
    if style:
        params['style'] = style
    if quality:
        params['quality'] = quality
    response = client.images.generate(**params)
    return response

Here, you defined a function generate_image that takes several parameters to generate an image using the DALL-E model. This function allows you to specify the model, prompt, size, quality, style, response format, and the number of images to generate. This function uses the client.images.generate method to generate the image(s).

Next, define a function to display an image from a URL:

# Define a function to display an image from a URL
def display_image_from_url(image_url):
    response = requests.get(image_url)
    img = Image.open(BytesIO(response.content))
    plt.imshow(img)
    plt.axis('off')
    plt.show()

Here, you defined a function display_image_from_url that takes an image URL as input. This function downloads the image using the requests library, opens it with PIL, and displays it using matplotlib.pyplot.

Next, define a function to display an image from a base64 string:

# Define a function to display an image from base64
def display_image_from_base64(b64_string):
    img_data = base64.b64decode(b64_string)
    img = Image.open(BytesIO(img_data))
    plt.imshow(img)
    plt.axis('off')
    plt.show()

Here, you defined a function display_image_from_base64 that takes a base64-encoded string as input. This function decodes the base64 string using the base64 module, opens the image with PIL, and displays it using matplotlib.pyplot.

Define a function to save an image to local storage:

# Define a function to save an image to local storage
def save_image_to_local(image_url, filename):
    response = requests.get(image_url)
    img = Image.open(BytesIO(response.content))
    img.save(filename)

Here, you defined a function save_image_to_local that takes an image URL and a filename as input. This function downloads the image using the requests library, opens it with PIL, and saves it to the specified filename.

Define a function to display multiple images in a grid:

# Define a function to display multiple images in a grid
def display_images_in_grid(image_urls):
    num_images = len(image_urls)
    grid_size = int(num_images**0.5)
    fig, axes = plt.subplots(grid_size, grid_size, figsize=(10, 10))
    for i, image_url in enumerate(image_urls):
        response = requests.get(image_url)
        img = Image.open(BytesIO(response.content))
        row, col = divmod(i, grid_size)
        axes[row, col].imshow(img)
        axes[row, col].axis('off')
    plt.show()

Here, you defined a function display_images_in_grid that takes a list of image URLs as input. This function calculates the grid size based on the number of images, downloads each image using the requests library, opens it with PIL, and displays all images in a grid using matplotlib.pyplot.

DALL-E can generate images from textual descriptions. Here’s how you generate an image using DALL-E 3:

# Generate and display an image with DALL-E 3
dalle_model = "dall-e-3"
dalle_prompt = "a samurai cat is eating ramen"
image_size = "1024x1792"
image_quality = "standard"

response = generate_image(client, dalle_model, dalle_prompt,
  image_size, image_quality)
image_url = response.data[0].url
display_image_from_url(image_url)

To generate images, use the generate_image function. The parameters include the model, prompt, size, quality, and n. The n parameter refers to the number of images to generate. But for DALL-E 3, the value for the n can only be 1. The URL of the generated image is retrieved from the url field in response.data[0].

To display the image, you use the display_image_from_url function. You should see an image of samurai cat eating ramen!

You can experiment with different qualities and sizes. For instance, to use the hd quality and change the image size, write the code like so:

# Change the image quality and size
image_quality = "hd"
image_size = "1792x1024"

response = generate_image(client, dalle_model, dalle_prompt,
  image_size, image_quality)
image_url = response.data[0].url
display_image_from_url(image_url)

You would see a landscape image!

You can also change the image style to natural:

# Change image quality and style
image_quality = "standard"
image_style = "natural"

response = generate_image(client, dalle_model, dalle_prompt,
  image_size, image_quality, image_style)
image_url = response.data[0].url
display_image_from_url(image_url)

This time, the generated image doesn’t have high-definition quality.

DALL-E 2 supports generating multiple images in a single API call. The style and quality parameters are not available, and only square sizes are supported.

To this, enter the following code:

# Generate multiple images with DALL-E 2
dalle_model = "dall-e-2"
image_size = "512x512"
dalle_prompt = "a samurai cat is singing on a stage"

response = generate_image(client, dalle_model, dalle_prompt,
  image_size, n=4)
image_urls = [img.url for img in response.data]
display_images_in_grid(image_urls)

You would see 4 images of cat singing on a stage!

To receive the image in base64 format, set the response_format parameter to b64_json in the generate_image function:

# Generate an image in base64 format
image_response_format = "b64_json"

response = generate_image(client, dalle_model, dalle_prompt,
  image_size, response_format=image_response_format)
b64_string = response.data[0].b64_json
display_image_from_base64(b64_string)

Here, you specify the response format as b64_json to get the image in base64 format. You then use the generate_image function to create the image and obtain the base64 string. Finally, you use the display_image_from_base64 function to display the image.

To save the image to storage, use the save_image_to_local method. Add the following code:

# Save the image to a local file
file_path = "samurai_cat_singing_on_stage.png"

response = generate_image(client, dalle_model, dalle_prompt,
  image_size, image_quality)
image_url = response.data[0].url
save_image_to_local(image_url, file_path)

You’ve saved the image as samurai_cat_singing_on_stage.png in the current directory