8. Building Interactive App with Gemini Live

Written by Zahidur Rahman Faisal

Have you checked out the Gemini Live API? It’s a total game-changer for building real-time, interactive experiences in Android. Forget managing a whole backend just to stream audio or video to an LLM — Gemini Live makes it effortless.

Imagine building an app where the user can talk to a chatbot and get instant responses, just like a real conversation. That’s what the Live API enables.

What Makes an App ‘Interactive’ ?

When we talk about an interactive app in this context, especially with the Gemini Live API, we’re talking about an application that doesn’t just listen and reply — it actually acts on the user’s instructions. It goes beyond a simple question-and-answer chatbot.

Think of it this way:

• Standard Chatbot App: You say, “What’s the weather like?” The model figures out the answer and replies. That’s a back-and-forth conversation.

• Interactive App (with Function Calling): You say, “Please add coffee to my shopping list.”

  • The model doesn’t just say, “Okay, I’ve added coffee.”

  • It recognizes that “add to shopping list” is an action this app can perform.

  • It executes the function call that triggers the addListItem function in the actual Android code.

  • The app’s internal state (the shopping list), actually changes.

  • Then, the model gets confirmation and tells you: “Done. I’ve added coffee to your shopping list.”

The key is that the user’s voice prompt is translated directly into app-logic execution. The app is no longer just a passive interface; it’s an agent that can manipulate its own data and features based on a natural language command. It creates a seamless, hands-free experience where the AI is integrated directly into the core functionality of the app — that’s what makes it truly ‘interactive’ in the most powerful sense.

The Gemini Live API

When I first worked with the Gemini Live API, I realized it’s a major leap for mobile generative AI. Instead of the old request–response model, it now supports real-time, two-way streaming. That means the client and model can send and receive data simultaneously — creating a live conversation rather than a sequence of turns.

It provides an optimized low-latency stream for both the audio you send to the model (your speech as requests) and the audio/text the model sends back (its response). Since you’ve used Firebase AI Logic in earlier chapters, you can invoke this directly from your Android app — with no need for a custom server. It’s essentially a bidirectional real-time audio channel connecting straight to a Gemini model.

With this, you can build conversational apps that feel more like talking to another person than interacting with a bot. Users can even interrupt Gemini mid-sentence, and the model smoothly adjusts. The API handles continuous multimodal input (currently audio + text) and outputs immediate speech, which is perfect for voice-first or hands-free interfaces.

Gemini Live has built-in support for:

• Robust session management for long conversations.

• Function calling or tool use

• Voice Activity Detection (VAD) to detect pauses in speech

• Multimodal streaming in both directions

Note: Firebase AI Logic for Gemini Live API is currently in developer preview, meaning that non-backward compatible changes may occur in future releases.

Hands On Gemini Live

Let’s extend the Firebase AI Logic app from the previous chapter with Gemini Live bidirectional streaming.

Download the starter project, and open it in Android Studio.

Project Setup and Dependencies

First things first, ensure you’re targeting Android API level 23+ and the app is connected to Firebase.

Open the app level build.gradle file, add the Firebase AI Logic at the end of dependencies block.

// Firebase AI Logic: Gemini Live Dependency
var firebaseAiLogicVersion = "17.6.0"
implementation "com.google.firebase:firebase-ai:$firebaseAiLogicVersion"

Since you’ll be interacting with Gemini Live through audio, add this permission in AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO"/>

Now, sync your project with the Gradle files and verify that it compiles without any errors.

Build and run the app, and select an item from the list. On the item-detail screen, you’ll see button at the bottom showing:

"WARNING: Gemini Live Not Initialized"

Model Initialization and Configuration

The first step in using Gemini Live is initializing the backend service and creating a LiveGenerativeModel instance. The Live API configuration is handled through the liveGenerationConfig object, which determines the model’s behavior and the nature of the streaming output.

To handle initialization cleanly, the best practice is to create a utility/manager class. The starter project already has that for your convenience.

Go to the com.kodeco.android.aam.live package and open the LiveModelManager file. This handles interacting with the Gemini Live model and updates UI states. These variables below are declared within the class for the same reason:

// The core Gemini Live model instance.
private lateinit var liveModel: LiveGenerativeModel

// Mutable state flow holding the current state of the live session.
private val _liveSessionState = MutableStateFlow<LiveSessionState>(LiveSessionState.Unknown())
val liveSessionState = _liveSessionState.asStateFlow()

Now, add the initializeGeminiLive() function as follows:

fun initializeGeminiLive(activity: Activity) {
  requestAudioPermissionIfNeeded(activity)

  coroutineScope.launch {
    try {
      val liveGenerationConfig = liveGenerationConfig {
        speechConfig = SpeechConfig(voice = Voice("FENRIR"))
        responseModality = ResponseModality.AUDIO
      }

      liveModel = Firebase.ai(backend = googleAI()).liveModel(
        modelName = "gemini-live-2.5-flash-preview",
        generationConfig = liveGenerationConfig,
      )

      _liveSessionState.value = LiveSessionState.Ready()
    } catch (e: Exception) {
      _liveSessionState.value = LiveSessionState.Error(message = e.localizedMessage)
    }
  }
}

This function requests audio permissions, sets up the configuration, and prepares the model for use.

When configuring the model, several parameters are mandatory for building with Gemini Live:

• Model Selection: The modelName parameter must explicitly reference a Live API compatible model, such as gemini-live-2.5-flash-preview.

  Here’s the list of models that supports the Live API.

• Output Modality: The responseModality parameter must be set to ResponseModality.AUDIO to generate and stream Text-to-Speech (TTS) output back to the client.

• Voice Configuration: The speechConfig parameter allows for specifying a preferred voice, for example, Voice("FENRIR").

Pro Tip: You can also provide an additional parameter, systemInstruction for defining the model’s persona, its role (e.g., “You are a helpful assistant, an expert about domestic pets.”), and establishing clear operational constraints.

The initializeGeminiLive() function is relaying the ‘Ready’ state after initializing the liveModel.

_liveSessionState.value = LiveSessionState.Ready()

It also throw an ‘Error’ with a message in case of failure:

_liveSessionState.value = LiveSessionState.Error(message = e.localizedMessage)

The LiveSessionState is a sealed interface, containing data classes that reflects current state of the Gemini Live session. The LiveSessionState is located in the same package and defined as below:

sealed interface LiveSessionState {
  data class Unknown(val message: String = "UNKNOWN: Gemini Live Not Initialized") : LiveSessionState

  data class Ready(val message: String = "READY: Ask Gemini Live") : LiveSessionState

  data class Running(val message: String = "RUNNING: Gemini Live Speaking...") : LiveSessionState

  data class Error(val message: String = "ERROR: Failed to initiate lGemini Live") : LiveSessionState
}

This is used to notify updates of the session in the screen.

The UI interactions are managed by MainViewModel class, which has a LiveModelManager instance as shown below:

private val liveModelManager = LiveModelManager(
  context = application,
  coroutineScope = viewModelScope,
)

It also contains liveSessionState to observe any state changes in liveModelManager.

val liveSessionState = liveModelManager.liveSessionState

Add the below function that’ll initialize the Gemini Live model:

fun initializeGeminiLive(activity: Activity) {
  liveModelManager.initializeGeminiLive(activity)
}

You need to call this function from the MainActivity when your app launches.

Now go to the MainActivity.kt, add the below code inside the onCreate(savedInstanceState: Bundle?) block:

viewModel.initializeGeminiLive(activity = this@MainActivity)

Build and run again, you’ll see a permission dialog for recording audio:

Approve the permission, and then select an item from the list to go to the next screen. You’ll see the Gemini Live button is now displaying the READY state!

Real-Time Connection: Starting The Live Session

At this point, the app can connect to Gemini and start the live session. You need to use LiveModelManager for that.

Open the LiveModelManager class, declare a LiveSession instance.

private var session: LiveSession? = null

The LiveSession object is the core abstraction for continuous interaction. It represents the persistent connection established with the model and manages all input/output streaming.

The Live API is bidirectional, and you’ll use it for:

1. Sending Voice commands.
2. Generating speech from text.
3. Transcribing audio to text.

Eventually, you will also be able to send images and a live video stream to the model. In the sample app, you’ll implement text -> speech in this flow:

List of Cat Breeds > Select a breed > Gemini Live Describes it

To do so, add the startSessionFromText() function in the LiveModelManager:

@RequiresPermission(Manifest.permission.RECORD_AUDIO)
fun startSessionFromText(catBreed: String) {
  val text = "Tell me about $catBreed cats in maximum 80 words."

  coroutineScope.launch(Dispatchers.IO) {
    try {
      // Start the conversation
      session = liveModel.connect()
      session?.send(text)
      session?.startAudioConversation()
        
      // Update State
      _liveSessionState.value = LiveSessionState.Running()
      } catch (e: Exception) {
        _liveSessionState.value = LiveSessionState.Error(message = e.localizedMessage)
    }
  }
}

This function accepts a single string argument, catBreed that you have selected, and then executes as described below:

1. val text = "Tell me about $catBreed cats.": It constructs a text prompt using the input catBreed. For example, if “Siamese” is passed to the function, then text will be “Tell me about Siamese cats in maximum 80 words.”

2. session = liveModel.connect(): It establishes a persistent WebSocket connection to the Gemini model to start a new session. This LiveSession object allows for real-time, low-latency streaming of input and output.

3. session?.send(text): The function then sends the initial text prompt to the Gemini Live model through the established session.

4. session?.startAudioConversation(): This is a key step that begins the audio part of the conversation. Following connection establishment, the core audio interaction is initiated by calling it. This command signals to the model that the client is ready to begin streaming microphone data.

5. _liveSessionState.value = LiveSessionState.Running(): This updates the live session’s state to “Running” as soon as the session successfully starts. This value is exposed as a StateFlow (backed by MutableStateFlow) so that the UI layer can observe it and react in real time.

6. _liveSessionState.value = LiveSessionState.Error(message = e.localizedMessage): Handles errors by updating the _liveSessionState to “Error” state and including the relevant message.

Note that, the @RequiresPermission(Manifest.permission.RECORD_AUDIO) annotation at the top ensures this function only runs when the RECORD_AUDIO permission is granted - necessary for bidirectional audio streaming.

Lifecycle Management: Toggling Session Start/Stop

You learned how to start a session, but you also need to know how to stop the session. The session should be explicitly closed when the microphone is deactivated or when the user navigates away from the screen. Even when you start a new session, the right approach is to stop any ongoing session before starting a new one.

Stopping a session is simple. You can do that by adding this function to LiveModelManager:

fun stopSession() {
    session?.apply {
      stopAudioConversation()
      stopReceiving()
    }
    _liveSessionState.value = LiveSessionState.Ready()
}

The function simply checks if there’s a session initiated. If so, it stops the audio conversation with the model.

The stopReceiving() method ensures the live session object will no longer receive data from the server, effectively closing the socket connection.

Then it resets the UI state executing _liveSessionState.value = LiveSessionState.Ready(). This indicates to the user that there’s no ongoing session and you’re ready to start a new one.

This will be handy when you tap on the Mic button:

1. In the Ready state, the app will start a new session.

2. Once the session is Running, it will stop the session if you tap the same button.

As MainViewModel is responsible for handling user interactions, update the askAbout() function in MainViewModel.kt to implement the session start/stop toggle:

@RequiresPermission(Manifest.permission.RECORD_AUDIO)
fun askAbout(catBreed: String) {
  when (val state = liveSessionState.value) {
    is LiveSessionState.Ready -> {
      liveModelManager.startSessionFromText(catBreed)
    }

    is LiveSessionState.Running -> {
      liveModelManager.stopSession()
    }

    else -> {
      Log.d(TAG, "Live session state: $state")
    }
  }
}

The above function starts or stops a live audio session to ask Gemini Live about a specific cat breed. The catBreed parameter is the name of the cat breed to ask about. This is used as the initial prompt when starting a new session.

Then it checks the current state of the liveSessionState and performs the following:

• If the state is Ready, it starts a new live session.

• If the state is Running, it stops the currently active session.

• In other states (like Error or Loading), it logs the current state.

Ready to try this out? Build and run the app now. Navigate to the detail screen by selecting a cat breed and then tap the Mic button at the bottom.

You’ll see the state change to RUNNING, and Gemini Live will start talking about the cat breed you selected!

Function Calling: Making Gemini Your App’s Agent

Now you know how to turn your app into a voice assistant using the Gemini Live API. The next big step is Function Calling - the superpower that lets the model actually interact with the logic and functionality of an Android app. It’s what makes the voice assistant an agent for your app.

Function calling allows the model to determine when an action is necessary to satisfy a user request. The implementation follows a standard multi-step process.

Step 1: Define the App Function and its Declaration

First, you need the actual function in your app that you want the model to be able to call. In the sample app, you may want the user to ask for pictures of a specific cat breed - which means opening a Google Image search.

Here’s the function in LiveModelManager.kt that handles launching the Google search Intent:

fun showPicture(catBreed: String) {
  coroutineScope.launch(Dispatchers.Default) {
    val query = Uri.encode("$catBreed cat pictures")
    val url = "https://www.google.com/search?q=$query&tbm=isch"

    val intent = Intent(Intent.ACTION_VIEW)
    intent.data = Uri.parse(url)
    intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)

    try {
      context.startActivity(intent)
    } catch (e: Exception) {
      Log.e(TAG, "Error opening Google Images", e)
    }
  }
}

Next, create a FunctionDeclaration to describe this function to the Gemini model. This is like creating an API reference for the model. It needs a name, a plain-English description (which is crucial for the model to understand when to use it), and the required parameters.

// The FunctionDeclaration for the model
val showPictureFunctionDeclaration = FunctionDeclaration(
    name = "showPicture",
    description = "Function to show picture of cat breed",
    parameters = mapOf(
        "catBreed" to Schema.string(
            description = "A short string describing the cat breed to show picture"
        )
    )
)

Step 2: Pass the Tool to the LiveModel

The Gemini model needs to know what tools (functions) it has available before the conversation even starts. You need to package the FunctionDeclaration into a Tool object and pass it to the liveModel initialization.

So, define functionHandlerTool below showPictureFunctionDeclaration as follows:

// Packaging the declaration into a Tool
val functionHandlerTool = Tool.functionDeclarations(listOf(showPictureFunctionDeclaration))

Then update the liveModel initialization inside the initializeGeminiLive() function:

// Initializing the LiveGenerativeModel
liveModel = Firebase.ai(backend = googleAI()).liveModel(
    modelName = "gemini-live-2.5-flash-preview",
    generationConfig = liveGenerationConfig,
    tools = listOf(functionHandlerTool), // Passing the tool here!
)

Now, the model knows that if a user asks something like “Can you show me pictures of a Siamese cat?”, it has a tool named showPicture that can handle that request.

Step 3: Implement the Handler Function

When the user says something that triggers the function, the model sends a FunctionCallPart to the app. You need a special function — a handler, to intercept this call, execute the app logic, and send the result back to the model.

Below functionHandlerTool, add the following function to act as the handler:

fun functionCallHandler(functionCall: FunctionCallPart): FunctionResponsePart {
  return when (functionCall.name) {
    "showPicture" -> {
      val catBreed = functionCall.args["catBreed"]!!.jsonPrimitive.content
      showPicture(catBreed = catBreed)
      val response = JsonObject(
        mapOf(
          "success" to JsonPrimitive(true),
          "message" to JsonPrimitive("Showing pictures of $catBreed")
        )
      )
      FunctionResponsePart(functionCall.name, response)
    }

    else -> {
      val response = JsonObject(
        mapOf(
          "error" to JsonPrimitive("Unknown function: ${functionCall.name}")
        )
      )
      FunctionResponsePart(functionCall.name, response)
    }
  }
}

The crucial parts here are:

1. Extracting the parameters: The model sends the arguments it inferred from the user’s speech (catBreed in this case).

2. Calling the local app logic: This is where showPicture() is executed, opening the browser.

3. Sending the response back: You return a FunctionResponsePart confirming the execution. The model uses this confirmation to finalize its spoken reply (e.g., “I’m on it! Showing pictures of Siamese cats now”).

Step 4: Start the Conversation with a Function Handler

Finally, when you start or continue the live session, pass the handler function to the startAudioConversation() call. This tells the LiveSession which function to invoke when the model decides to use a tool.

Update the startSessionFromText() function as follows:

// Start the conversation
session = liveModel.connect()
session?.send(text)
session?.startAudioConversation(::functionCallHandler) // Pass the handler here!

That’s it! Build and run the app, and ask about your selected cat breed using Gemini Live.

You’ll be amazed to hear Gemini ending the description with follow-up questions like: “… Do you want me to show pictures of Siamese cats?”

Reply something affirmative, like “Yes, I’m curious!”

Then the magic happens – Gemini Live handles all the real-time voice streaming and the function calling handshake, resulting in the image search opening right on your phone.

Conclusion

To wrap this up, what you’ve done with the Gemini Live API and Function Calling isn’t just an evolutionary step; it’s a massive leap forward in how we build mobile AI experiences.

The chapter started with the core idea: using the Gemini Live API to achieve low-latency, real-time voice streaming without needing a complex backend. That alone moves us beyond the clunky “wait-and-reply” experience of older chatbots. Setting up a “live” model that can manage a live session made real-time interaction possible!

But Function Calling is where the true power of an interactive app shines. You’ve turned Gemini into a genuine agent for your app just by taking just a few steps:

• Defining a function.
• Packaging it as a tool to be used with a model.
• Implementing a simple function call handler.

Now, when a user says, “Show me pictures of a Maine Coon cat,” it’s not just a conversation; it’s a seamless command that triggers native Android code, opening the search Intent directly!

This ability to blend natural language with your app’s specific logic is what truly unlocks next-generation, hands-free voice control on Android. It’s time to stop building apps that just talk - and start building apps that act.