Voice Transcription and Synthesis with Whisper & TTS

OpenAI provides two key models for handling audio: Whisper for transcription and TTS for speech synthesis. Each model has unique features that makes it suitable for different tasks.

Whisper

The Whisper model is designed for speech-to-text transcription and translation. It supports multiple languages and can transcribe audio into the original language or translate it into English.

Here are its key features:

• Transcriptions: Convert audio into text in the same language as the audio.
• Translations: Convert audio in any supported language into English text.
• Timestamps: Provide word- or segment-level time stamps in the transcription.
• Prompts: Guide the transcription process with specific instructions to improve accuracy.

Text-to-Speech (TTS)

The TTS model converts text into natural-sounding speech. It supports multiple voices and can produce high-quality audio suitable for various apps.

Here are its key features:

• Multiple Voices: Choose from six built-in voices (alloy, echo, fable, onyx, nova, and shimmer) to match your desired tone and audience.
• Audio Quality: Use the tts-1 model for low latency or tts-1-hd for higher-quality audio.
• Speed Control: Adjust the speed of the generated speech.

Whisper can transcribe audio files into text in the same language or translate them into English. This is useful for creating transcripts of meetings, lectures, or any audio content where a text version is needed.

Steps for using transcription:

1. Prepare Your Audio File: Ensure your audio file is in one of the supported formats (flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, or webm).
2. Transcribe the Audio: Use the Whisper model to convert the audio into text. You can specify the response format (such as json, text, srt, verbose_json, or vtt).
3. Optional Parameters: Use additional parameters like prompt to guide the transcription and timestamp_granularities to get word or segment-level time stamps.

Steps for using translation:

1. Prepare Your Audio File: Ensure your audio file is in one of the supported formats.
2. Translate the Audio: Use the Whisper model to translate the audio into English text. This is similar to transcription but the output is always in English.
3. Optional Parameters: Use additional parameters like prompt to guide the translation process.

OpenAI’s Text-to-Speech (TTS) capabilities let you generate natural-sounding speech from text. This can be used to narrate blog posts, produce spoken audio in multiple languages, or provide real-time audio output.

Steps for using TTS:

1. Formulate Your Text: Prepare the text that you want to convert into speech.
2. Choose Your Model: Decide whether to use tts-1 for lower latency or tts-1-hd for higher-quality audio.
3. Select a Voice: Choose from the available voices (alloy, echo, fable, onyx, nova, shimmer) to match the desired tone and audience.
4. Set Parameters: Adjust the speed of the speech if needed. The default speed is 1.0, but you can make it slower or faster.
5. Generate Speech: Use the TTS model to convert the text into audio. You can save the audio in various formats such as mp3, opus, aac, flac, wav, or pcm.

The combination of Whisper and TTS models opens up a wide range of applications, from accessibility tools to interactive, voice-based apps.

Accessibility

To improve accessibility, you can offer transcription services that convert spoken content into text for the deaf and hard of hearing. Additionally, real-time translation of spoken content into English enables a broader audience to access the information.

Interactive Apps

In interactive apps, you can create voice assistants that understand spoken commands and respond with natural-sounding speech. Language tutors can be developed to provide spoken feedback and corrections based on the user’s spoken input. Further, you can automate the narration of written content, such as blogs or articles, in a natural and engaging voice.

If you need to record audio for use with the Whisper model, you can use the Sound Recorder app on Windows, the QuickTime app on MacOS, or a similar app on Linux.

Recording Audio on Windows

Sound Recorder provides a straightforward way to capture high-quality audio. To use this app on Windows:

1. Open the Sound Recorder app from the Start menu.
2. Click the Settings menu to choose the recording format. It’s recommended to choose mp3.
3. Click the Record button to start recording your voice or any other audio.
4. Click the Stop button when you’re done.
5. Move the audio file to the designated folder.

Recording Audio on MacOS

QuickTime Player is a built-in app on MacOS that you can use to record audio. To record audio using QuickTime Player:

1. Open QuickTime Player from the Applications folder.
2. Click File in the menu bar and select New Audio Recording.
3. Click the Record button to start recording your audio.
4. Click the Stop button when you’re done.
5. Save the audio file to the desired location.

Because QuickTime Player supports recording only in m4a format, you might receive following warning when playing the audio file:

/var/folders/3f/719ptbxs55s757lhsc6677380000gn/T/ipykernel_83224/541254894.py:12: DeprecationWarning: Due to a bug, this method doesn’t actually stream the response content, .with_streaming_response.method() should be used instead response.stream_to_file(speech_file_path)

Make sure to install the ffmpeg package. You can install this through brew install ffmpeg on a Mac if you have Homebrew installed.