Multimodal Integration with OpenAI

Transcript

To set up your development environment for using the OpenAI API, please refer to Lesson 1: Introduction to Multimodal AI. This lesson covers installing necessary libraries and configuring your environment.

You also need to install additional libraries for this project. Add the following code to your notebook:

# Install additional dependencies for this lesson
!pip install librosa

The librosa library is for handling audio files.

Like previous lessons, you need to authenticate your API requests and the code for that is already included in the Starter notebook for this lesson:

# Load the OpenAI library
from openai import OpenAI

# Set up relevant environment variables
# Make sure OPENAI_API_KEY=... exists in .env
from dotenv import load_dotenv

load_dotenv()

# Create the OpenAI connection object
client = OpenAI()

OpenAI’s Whisper model is a powerful tool for speech recognition. First, you need to prepare the audio files. You can either record audio directly using your computer’s microphone or download free sample audio files from Pixabay.

Add the following code to download and load an audio file using the librosa library:

# Download and load an audio file using librosa

# Import libraries
import requests
import io
import librosa
from IPython.display import Audio, display

# URL of the sample audio file
speech_download_link = "https://cdn.pixabay.com/download/audio/2022/03/10/
  audio_a8e603753c.mp3?filename=self-destruct-sequence-31505.mp3"

# Local path where the audio file will be saved
save_path = "audio/self-destruct-sequence.mp3"

# Download the audio file
response = requests.get(speech_download_link)
if response.status_code == 200:
    audio_data = io.BytesIO(response.content)

    # Save the audio file locally
    with open(save_path, 'wb') as file:
        file.write(response.content)

    # Load the audio file using librosa
    y, sr = librosa.load(audio_data)

    # Display the audio file so it can be played
    audio = Audio(data=y, rate=sr, autoplay=True)
    display(audio)

Here’s a breakdown of the code step by step:

1. Import Libraries:

import requests
import io
import librosa
from IPython.display import Audio, display

You start by importing the necessary libraries:

• requests for downloading the audio file.
• io for handling byte streams.
• librosa for audio processing.
• IPython.display for displaying the audio player in a Jupyter Lab.

2. Specify the URL of the Audio File:

speech_download_link = "https://cdn.pixabay.com/download/audio/2022/03/10/
  audio_a8e603753c.mp3?filename=self-destruct-sequence-31505.mp3"

This variable holds the URL of the audio file you want to download.

3. Specify the Local Path to Save the Audio File:

save_path = "audio/self-destruct-sequence.mp3"

This variable defines the local path where the downloaded audio file will be saved.

4. Download the Audio File:

response = requests.get(speech_download_link)
if response.status_code == 200:
    audio_data = io.BytesIO(response.content)

You send a GET request to the URL and check if the download was successful (status code 200). If successful, you store the audio data in a byte stream.

5. Save the Audio File Locally:

with open(save_path, 'wb') as file:
    file.write(response.content)

This step writes the downloaded audio data to a file on your local system.

6. Load the Audio File Using Librosa:

y, sr = librosa.load(audio_data)

The librosa.load function loads the audio file from the byte stream, returning the audio time series (y) and the sampling rate (sr).

7. Display the Audio File:

audio = Audio(data=y, rate=sr, autoplay=True)
display(audio)

Finally, you create an audio player using the loaded audio data and display it, allowing you to play the audio directly in a Jupyter Lab.

Next, extract the logic to play the audio file into a separate function because you’ll use it multiple times:

# Function to play the audio file

def play_speech(file_path):
    # Load the audio file using librosa
    y, sr = librosa.load(file_path)

    # Create an Audio object for playback
    audio = Audio(data=y, rate=sr, autoplay=True)

    # Display the audio player
    display(audio)

Now, it’s time to transcribe the audio file using the Whisper model. Add the following code to your Jupyter Lab:

# Transcribe the audio file using the Whisper model

with open(save_path, "rb") as audio_file:
    # Transcribe the audio file using the Whisper model
    transcription = client.audio.transcriptions.create(
      model="whisper-1",
      file=audio_file,
      response_format="json"
    )
# Print the transcription result in JSON format
print(transcription.json())
# Print only the transcribed text
print(transcription.text)

You can also get a more detailed transcription with time stamps for each word:

# Retrieve the detailed information with timestamps

with open(save_path, "rb") as audio_file:
    # Transcribe the audio file with word-level timestamps
    transcription = client.audio.transcriptions.create(
      model="whisper-1",
      file=audio_file,
      response_format="verbose_json",
      timestamp_granularities=["word"]
    )

Then, you can look at the verbose JSON result.

# Print the detailed information for each word timestamp

import json

json_result = transcription.json()
print(json_result)

json_object = json.loads(json_result)
print(json_object["text"])

To print the detailed information for each word, add the following code:

# Print the detailed information for words

# Print the detailed information for each word
print(transcription.words)
# Print the detailed information for the first two words
print(transcription.words[0])
print(transcription.words[1])

You can also obtain segment-level time stamps for the transcription. Pass the segment value to the timestamp_granularities parameter:

# Retrieve the detailed information with segment-level timestamps

with open(save_path, "rb") as audio_file:
    # Transcribe the audio file with segment-level timestamps
    transcription = client.audio.transcriptions.create(
      model="whisper-1",
      file=audio_file,
      response_format="verbose_json",
      timestamp_granularities=["segment"]
    )

To print the detailed information for the first two segments, use the following code:

# Print the detailed information for the first two segments
print(transcription.segments[0])
print(transcription.segments[1])

Now, load and play another audio file:

# Load & play kodeco-speech.mp3 audio file

# Path to another audio file
ai_programming_audio_path = "audio/kodeco-speech.mp3"
# Play the audio file
play_speech(ai_programming_audio_path)

You would hear Kodeco and RayWenderlich being mentioned. Next, transcribe the speech again. This time, use the text response format, which is simpler than the JSON response format. The returned result is just the transcription text.

# Transcribe the audio file with `text` response format

with open(ai_programming_audio_path, "rb") as audio_file:
    # Transcribe the audio file to text
    transcription = client.audio.transcriptions.create(
      model="whisper-1",
      file=audio_file,
      response_format="text"
    )
# Print the transcribed text
print(transcription)

Notice that the transcription is not perfect. Kodeco and RayWenderlich are misspelled. You can guide the transcription process with the prompt parameter to improve accuracy.

# Transcribe the audio file with a prompt to improve accuracy

with open(ai_programming_audio_path, "rb") as audio_file:
    # Transcribe the audio file with a prompt to improve accuracy
    transcription = client.audio.transcriptions.create(
      model="whisper-1",
      file=audio_file,
      response_format="text",
      prompt="Kodeco,RayWenderlich"
    )
# Print the transcribed text
print(transcription)

Now, the transcription should be more accurate. The prompt parameter helps guide the transcription, making it particularly useful for correcting specific words or continuing a previous segment. In this case, the prompt ensures that names like Kodeco and RayWenderlich are transcribed correctly.

Other than transcription, you can also translate the audio file directly to English. Currently, only English is supported.

First, listen to the Japanese audio file:

# Load & play japanese-speech.mp3 audio file

# The speech in Japanese: いらっしゃいませ。ラーメン屋へようこそ。
  何をご注文なさいますか？
# Path to the Japanese audio file
japanese_audio_path = "audio/japanese-speech.mp3"
# Play the Japanese audio file
play_speech(japanese_audio_path)

To translate, use the client.audio.translations.create method. The model, file, and response_format parameters work the same as in the client.audio.transcriptions.create method. Add the following code to your Jupyter Lab:

# Translate the Japanese audio to English text

with open(japanese_audio_path, "rb") as audio_file:
    # Translate the Japanese audio to English text
    translation = client.audio.translations.create(
      model="whisper-1",
      file=audio_file,
      response_format="text"
    )
# Print the translated text
print(translation)

The translated text should be: “Welcome. Welcome to the ramen shop. What would you like to order?”. The Whisper model can translate audio in any supported language into English text, making it a versatile tool for multilingual apps.

To create synthesized speech, you can use the client.audio.speech.with_streaming_response.create method with the context manager, as shown below:

# Generate speech from text using OpenAI's TTS model

# Path to save the synthesized speech
speech_file_path = "audio/learn-ai.mp3"

# Generate speech from text using OpenAI's TTS model
with client.audio.speech.with_streaming_response.create(
  model="tts-1",
  voice="alloy",
  input="Would you like to learn AI programming? We have many AI
    programming courses that you can choose."
) as response:
  # Save the synthesized speech to the specified path
  response.stream_to_file(speech_file_path)

The model parameter is set to tts-1, specifying the text-to-speech model to be used. This model is optimized for speed. You can use another model, tts-1-hd, if you care more about the quality. The voice parameter is set to alloy, which determines the voice characteristics such as tone and accent. You have other choices, like echo, fable, onyx, nova, and shimmer. Finally, the input parameter contains the text that you want to convert to speech: “Would you like to learn AI programming? We have many AI programming courses that you can choose.”

Now, play the synthesized speech:

# Play the synthesized speech
play_speech(speech_file_path)

Nice! You’ve created synthesized speech.

If you don’t want to use the context manager, you can use the client.audio.speech.create method to create synthesized speech. Generate speech again. This time, you experiment with another voice and speed:

# Generate speech with a different voice and slower speed
response = client.audio.speech.create(
  model="tts-1",
  voice="echo",
  speed=0.6,
  input="Would you like to learn AI programming? We have many
    AI programming courses that you can choose."
)

# Save the synthesized speech to the specified path
response.stream_to_file(speech_file_path)

# Play the synthesized speech
play_speech(speech_file_path)

Notice that the voice is now echo, which has a different tone than alloy. Also, the speed is set to 0.6, making the speech slower. If you want to make the speech faster, you can set the speed to a value greater than 1.

However, if you use client.audio.speech.create method, you’ll get the warning:

DeprecationWarning: Due to a bug, this method doesn't actually stream the
  response content, `.with_streaming_response.method()` should be used
  instead response.stream_to_file(speech_file_path)

Therefore, it’s better to use the client.audio.speech.with_streaming_response.create method with the context manager to avoid this warning.