Multimodal Integration with OpenAI

You’ll start by building several simple Gradio apps, which will prepare you to build a multimodal AI app later. You’ll begin by building a simple Gradio app that takes a name and a time of day as inputs and returns a greeting message.

First, ensure you’ve installed the necessary libraries, including Gradio, and set up the environment. Run the following code:

# Install the required libraries
!pip install openai requests python-dotenv matplotlib librosa
  ipyaudioworklet gradio Pillow

# Load the OpenAI library
from openai import OpenAI

# Set up relevant environment variables
# Make sure OPENAI_API_KEY=... exists in .env
from dotenv import load_dotenv

load_dotenv()

# Create the OpenAI connection object
client = OpenAI()

Then, start by importing the Gradio library and creating a simple app that takes a name and a time of day as inputs and returns a greeting message.

# Import the Gradio library
import gradio as gr

# Define a simple function that takes a name and a time of day as inputs
def greet(name, greeting_time):
    return "Good " +  greeting_time + ", " + name + "!"

# Create a Gradio interface for the function
demo = gr.Interface(
    fn=greet,  # The function to wrap a UI around
    inputs=[ # Define input components
        gr.Text(), # Input field for name
        # Dropdown for time of day
        gr.Dropdown(["morning", "evening", "night"])
    ],
    outputs=[
        gr.Text() # Define text output
    ], # Define output components
)

# Launch the Gradio app
demo.launch()

You’ll be presented with an app that can take inputs and give an output. You can open this app in a dedicated mode by clicking this link. You define the function to process the inputs with the fn argument. You can see that the inputs argument defines the input fields and the outputs argument defines the output field. The Gradio library provides many components such as gr.Text(), gr.Dropdown(), and so on. The number of the arguments to the greet function must match the number of the elements of an array passed to the inputs argument.

Next, you’ll modify the greet function to return both a greeting message and an image URL. You’ll also update the outputs arguments to return an array consisting of the text element and an additional image element.

Update your code to the following:

# Define a function that returns a greeting message and a
# hard-coded image URL
def greet(name, greeting_time):
    greeting = "Good " +  greeting_time + ", " + name + "!"
    image_url = "https://upload.wikimedia.org/wikipedia/commons/d/d6
      /An_Oberoi_Hotel_employee_doing_Namaste%2C_New_Delhi.jpg"
    return (greeting, image_url)

# Create a Gradio interface for the function
demo = gr.Interface(
    fn=greet,
    inputs=[ # Define input components
        gr.Text(), # Input field for name
        # Dropdown for time of day
        gr.Dropdown(["morning", "evening", "night"])
    ],
    outputs=[
        gr.Text(), # Define text output
        gr.Image() # Define image output
    ],
)

# Launch the Gradio app
demo.launch()

As you can see, you can have multiple output fields. You define them in the outputs argument of the gr.Interface method. Make sure the greet function returns a tuple consisting of two output elements. To create the image field, you use the gr.Image() component.

You can also add audio components either as the input field or the output field.

# Define a function that returns a greeting message,
# an image URL, and an audio file path
def greet(name, greeting_time, audio_path):
    greeting = "Good " +  greeting_time + ", " + name + "!"
    image_url = "https://upload.wikimedia.org/wikipedia/commons/d/d6
      /An_Oberoi_Hotel_employee_doing_Namaste%2C_New_Delhi.jpg"
    return (greeting, image_url, audio_path)

# Create a Gradio interface for the function
demo = gr.Interface(
    fn=greet,
    inputs=[
        gr.Text(), # Define input components
        # Input field for name
        gr.Dropdown(["morning", "evening", "night"]),
        # Audio input field
        gr.Audio(sources=["microphone"], type="filepath")
    ],
    outputs=[
        gr.Text(), # Define text output
        gr.Image(), # Define image output
        gr.Audio(type="filepath") # Define audio output
    ],
)

# Launch the Gradio app
demo.launch()

In this example, the app is further extended to include audio input and output components. The gr.Audio() component lets users provide audio input through a microphone, and the function returns a greeting message, an image URL, and an audio file path. The gr.Audio() output field doesn’t need the sources argument because you play the audio only in the output field.

You can also make your app more informative using the title and description arguments in the gr.Interface method.

# Define a function that returns a greeting message, an image URL,
# and an audio file path
def greet(name, greeting_time, audio_path):
    greeting = "Good " +  greeting_time + ", " + name + "!"
    image_url = "https://upload.wikimedia.org/wikipedia/commons/d/d6
      /An_Oberoi_Hotel_employee_doing_Namaste%2C_New_Delhi.jpg"
    return (greeting, image_url, audio_path)

# Create a Gradio interface for the function with a title and description
demo = gr.Interface(
    fn=greet,
    inputs=[
        gr.Text(), # Define input components
        # Input field for name
        gr.Dropdown(["morning", "evening", "night"]),
        # Audio input field
        gr.Audio(sources=["microphone"], type="filepath")
    ],
    outputs=[
        gr.Text(), # Define text output
        gr.Image(), # Define image output
        gr.Audio(type="filepath") # Define audio output
    ],
    title="Greeting App",
    description="This is a billion-dollar greeting app."
)

# Launch the Gradio app
demo.launch()

The Gradio interface is enhanced by adding a title and description to provide context and make the app more user-friendly.