Multimodal Integration with OpenAI

In this demo, you’ll create a multimodal language tutor app using Gradio. The app will simulate conversational scenarios, allowing users to practice their English skills interactively. The app will display images, play audio prompts, and let users respond via recorded speech. It will then update the conversation, generate new images, and provide audio feedback based on the user’s input.

Start by defining the seed prompt for the initial situational context. Generate the initial situational description and corresponding image using the generate_situational_prompt function from the previous demo. Remember that the Jupyter Lab file that you see now is the same Jupyter Lab file you worked on in the last demo.

# Build the multimodal language tutor app using Gradio

# Initial seed prompt for generating the initial situational context
seed_prompt = "cafe near beach" # or "comics exhibition",
  "meeting parents-in-law for the first time", etc

# Generate an initial situational description based on the seed prompt
initial_situation = generate_situational_prompt(seed_prompt)

# Generate an initial image based on the initial situational description
img = generate_situation_image(initial_situation)

# Flags to manage the state of the app
first_time = True
combined_history = ""

Define a helper function to extract the first and last segments of the conversation history. This ensures the prompt for DALL-E does not exceed the maximum character limit. Add the function to the code to the end of the code cell:

# Function to extract the first and last segments of the conversation
#  history
# This is to ensure that the prompt for DALL-E does not exceed the
#  maximum character limit of 4000 characters
def extract_first_last(text):
    elements = [elem.strip() for elem in text.split('====')
      if elem.strip()]

    if len(elements) >= 2:
        return elements[0] + elements[-1]
    elif len(elements) == 1:
        return elements[0]
    else:
        return ""

Define the main function conversation_generation to handle the conversation logic. This function will transcribe the user’s speech, update the conversation history, generate a new conversation response, and update the visual and audio outputs. Add the function to the code cell:

# Main function to handle the conversation generation logic
def conversation_generation(audio_path):
    global combined_history
    global first_time

    # Transcribe the user's speech from the provided audio file path
    transcripted_text = transcript_speech(audio_path)

    # Create conversation history based on whether it is the first
    # interaction or not
    if first_time:
        history = creating_conversation_history(initial_situation,
          transcripted_text)
        first_time = False
    else:
        history = creating_conversation_history(combined_history,
          transcripted_text)

    # Generate a new conversation based on the updated history
    conversation = generate_conversation_from_history(history)

    # Update the combined history with the new conversation
    combined_history = history + "\n====\n" + conversation

    # Extract a suitable prompt for DALL-E by combining the first
    # and last parts of the conversation history
    dalle_prompt = extract_first_last(combined_history)

    # Generate a new image based on the updated combined history
    img = generate_situation_image(combined_history)

    # Generate speech for the new conversation and save it to an
    # audio file
    output_audio_file = "speak_speech.mp3"
    speak_prompt(conversation, False, output_audio_file)

    # Return the updated image, conversation text, and audio file
    # path
    return img, conversation, output_audio_file

This function, conversation_generation, manages the conversation logic for the app. It starts by transcribing the user’s speech from the provided audio file path. Based on whether it’s the first interaction, it creates the conversation history accordingly. It then generates a new conversation response using the updated history and updates the combined history. The function extracts a suitable prompt for generating a new image based on the conversation history, generates the image, and produces speech for the new conversation, saving it to an audio file. Finally, it returns the updated image, conversation text, and audio file path.

Create the Gradio interface for the language tutor app. This interface will handle user interactions and update the visual and audio outputs accordingly. Launch the Gradio app to start practicing conversational English. Add the following code to the end of the code cell:

# Create the Gradio interface for the language tutor app
tutor_app = gr.Interface(
    conversation_generation,
    gr.Audio(sources=["microphone"], type="filepath"),
    outputs=[gr.Image(value=img), gr.Text(), gr.Audio(type="filepath")],
    title="Speaking Language Tutor App",
    description=initial_situation
)

# Launch the Gradio app
tutor_app.launch()

This code sets up the Gradio interface for the language tutor app. The gr.Interface function takes conversation_generation as the main function to handle the conversation logic. It specifies that the user will provide audio input via a microphone, and the outputs will include an image, text, and an audio file. The interface is titled “Speaking Language Tutor App” and includes a description based on the initial situation. Finally, tutor_app.launch() starts the Gradio app, enabling users to practice conversational English interactively.

Once the app is ready, you can use it to practice conversational English.

As you can see, the initial situation is you being in the cafe near the beach and striking up a conversation with a stranger. You can presss the Record button, and say something like, “Yes, I do. What about you?” Then click the Stop button to finish the recording. You can hear your voice by clicking the Play button. If you’re happy, you can click the Submit button.

Wait for a while, like 20-30 seconds. Then you’ll get a generated image and a response from AI. You can read the response or play the audio response. In this case, the response is, “Oh, absolutely! There’s something magical about the ocean waves, don’t you think?”

To continue this conversation, you can click the small x button in the audio input. Then click the Record button again. You can say something like, “Yes, even my favorite hobby is surfing.” Then you can click the Submit button. You’ll get another generated image representing the latest situation and another response from AI. In this case, the response is, “That’s awesome! I’ve always wanted to learn how to surf. Maybe you could give me some pointers sometime?”

This process continues, with the app dynamically updating the conversation, images, and audio prompts based on your responses, creating an engaging and interactive language learning experience.

Now, it’s time to proceed to this lesson’s conclusion.