Multimodal Integration with OpenAI

Now, you want to combine speech recognition and synthesis to create a simple language tutor app. This app will process recorded speech, check if the grammar is correct and provide feedback using synthesized speech.

Define a function to transcribe the recorded speech using the Whisper model:

# Define a function to transcribe the recorded speech

def transcript_speech(speech_filename="my_speech.m4a"):
    with open(speech_filename, "rb") as audio_file:
        # Open the audio file and transcribe using the Whisper model
        transcription = client.audio.transcriptions.create(
          model="whisper-1",
          file=audio_file,
          response_format="json",
          language="en"
        )
    # Return the transcribed text
    return transcription.text

Then, define a function to check the grammar of the transcribed text using OpenAI’s GPT model:

# Check the grammar of the transcribed text

def check_grammar(english_text):
    # Use GPT to check and correct the grammar of the input text
    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {"role": "system", "content": "You are an English grammar
          expert."},
        {"role": "user", "content": f"Fix the grammar: {english_text}"}
      ]
    )
    # Extract and return the corrected grammar message
    message = response.choices[0].message.content
    return message

In this function, you use the GPT model to check and correct the grammar of the input text. The client.chat.completions.create method sends the input text to the GPT model along with a prompt that instructs the model to act as an English grammar expert. The response from GPT contains the corrected text, which is then extracted and returned by the function.

After that, define a function to generate spoken feedback using the text-to-speech capability:

# Provide spoken feedback using TTS

def tell_feedback(grammar_feedback, speech_file_path="
  feedback_speech.mp3"):
    # Generate speech from the grammar feedback using TTS
    response = client.audio.speech.create(
      model="tts-1",
      voice="alloy",
      input=grammar_feedback
    )

    # Save the synthesized speech to the specified path
    response.stream_to_file(speech_file_path)
    # Play the synthesized speech
    play_speech(speech_file_path)

Finally, put everything together in a function that handles the entire process from recording audio to providing spoken feedback:

# Implement the grammar feedback application

def grammar_feedback_app(speech_filename):
    # Transcribe the recorded speech
    transcription = transcript_speech(speech_filename)
    print(transcription)
    # Check and correct the grammar of the transcription
    feedback = check_grammar(transcription)
    print(feedback)
    # Provide spoken feedback using TTS
    tell_feedback(feedback)

In this function, you:

1. Transcribe the Recorded Speech: The transcript_speech function is called with speech_filename to transcribe the speech from the audio file.
2. Check and Correct the Grammar: The transcribed text is passed to the check_grammar function to check and correct its grammar.
3. Provide Spoken Feedback: The corrected text is then passed to the tell_feedback function to create and play a spoken version of the feedback using text-to-speech.

To test the grammar feedback app, you have to give the grammatically incorrect audio file to the app. To create an audio file for speech input, you can use the Sound Recorder app on Windows, QuickTime on MacOS, or a similar recording app on Linux. You can refer back to the instructions segment on how to do this if you need help.

Once recorded, place the audio file in the audio folder and update the wrong_grammar_audio variable accordingly. Alternatively, you can use a provided audio sample containing a grammatically incorrect sentence — “My sister don’t like to eat on night” — for testing purposes.

# Set the audio file. Use the audio sample or record the
# audio yourself and place the file here.
wrong_grammar_audio = "audio/grammar-wrong.mp3"

You can play it first to confirm this audio file has a grammatically incorrect sentence.

# Play the grammatically wrong audio file
play_speech(wrong_grammar_audio)

Run the application and get the grammar feedback:

# Run the grammar feedback application
grammar_feedback_app(wrong_grammar_audio)

You’ve now seen how to use Whisper for speech recognition and synthesis in an app. Move on to the next segment for this lesson’s conclusion.