The previous lesson built a complete voice agent against OpenAI’s Realtime API. This lesson builds the same category of system against Google’s Gemini Live API, using the native-audio model gemini-2.5-flash-native-audio-preview-09-2025, and the point isn’t just “here’s a second vendor’s syntax.” Gemini Live’s session model extends past audio into genuine multimodal streaming, accepting video and screen-share input alongside speech, which opens up product surfaces a purely audio-native API can’t reach, and it’s worth understanding both the similarities (tool calling, session configuration) and the real differences before choosing between them.
Establishing the session
Gemini Live is also a persistent, bidirectional streaming session, conceptually parallel to the Realtime API’s WebSocket but exposed through Google’s google-genai SDK, which handles the underlying transport for you rather than requiring you to hand-roll WebSocket framing.
import asyncio
from google import genai
from google.genai import types
client = genai.Client(api_key="YOUR_API_KEY")
MODEL = "gemini-2.5-flash-native-audio-preview-09-2025"
config = types.LiveConnectConfig(
response_modalities=["AUDIO"],
speech_config=types.SpeechConfig(
voice_config=types.VoiceConfig(
prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Kore")
)
),
tools=[
types.Tool(function_declarations=[
types.FunctionDeclaration(
name="check_order_status",
description="Look up the current status of a customer order by order ID.",
parameters={
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"],
},
)
])
],
)
async def run_session():
async with client.aio.live.connect(model=MODEL, config=config) as session:
return sessionresponse_modalities=["AUDIO"] is the direct analogue of the Realtime API’s modalities setting from the previous lesson: without it declared explicitly, you can’t assume the session defaults to spoken audio output over text. The voice_config and tools blocks play the same role as session.update’s voice and tools fields did for gpt-realtime, configured once at connection time rather than per-turn.
Streaming audio and consuming responses
The send/receive shape mirrors the pattern from the previous lesson: push audio chunks in as they’re captured, consume audio (and optionally text) deltas as they’re generated.
async def voice_turn(session, mic_audio_chunks, playback):
async def send_audio():
async for chunk in mic_audio_chunks:
await session.send_realtime_input(
audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000")
)
async def receive_responses():
async for response in session.receive():
if response.data is not None:
await playback.enqueue(response.data) # raw audio bytes
if response.text is not None:
log.debug("assistant_partial_transcript", text=response.text)
if response.tool_call is not None:
await handle_tool_call(session, response.tool_call)
await asyncio.gather(send_audio(), receive_responses())Voice activity detection and turn-taking are handled server-side by default here as well, comparable to the Realtime API’s server_vad mode, and the same barge-in principles from Module 5 apply: when the session signals the user has started speaking, the client should stop playback of any in-flight response audio immediately rather than letting it finish.
Tool calling: the same shape, different names
Function calling inside a Gemini Live session follows the identical request-execute-respond pattern used by the OpenAI Realtime API, which is worth noting explicitly: despite different vendors and different SDKs, the industry has converged on essentially one shape for tool use inside a live audio session, which means the mental model from the previous lesson transfers directly even though the exact event and field names differ.
async def handle_tool_call(session, tool_call):
responses = []
for fc in tool_call.function_calls:
fn = tool_registry[fc.name]
result = await fn(**fc.args)
responses.append(
types.FunctionResponse(id=fc.id, name=fc.name, response={"result": result})
)
await session.send_tool_response(function_responses=responses)As with gpt-realtime, the API never executes your business logic itself; it only signals intent and arguments, and your application is responsible for actually querying the order database, calling the internal service, or whatever the function represents, then reporting the result back into the same live session.
Where Gemini Live pulls ahead: true multimodal input
The clearest differentiator is scope of input modality. The OpenAI Realtime API’s live session is built around audio and text; genuine visual understanding requires a separate call outside the realtime session. Gemini Live’s session accepts video frames and screen-share content as streaming input alongside audio within the same connection, which means a single session can process “the user is describing a problem with their router while pointing their camera at the blinking lights on it” as one coherent multimodal context, rather than requiring your application to stitch together a live audio session and a separate frame-by-frame vision API call and somehow keep the two synchronized in time.
That capability matters concretely for product categories a pure audio-native model can’t serve well: remote visual troubleshooting, guided physical setup (“show me the cable you’re holding”), or any assistant that needs to reason jointly over what it’s hearing and what it’s currently seeing. If your product is a phone-based voice agent with no visual component, this advantage is largely irrelevant and the choice between the two APIs comes down to the same latency, voice quality, and ecosystem factors covered for gpt-realtime. If visual grounding is part of the product vision even on the roadmap, Gemini Live’s native multimodal session is worth weighing heavily, since retrofitting vision onto an audio-only architecture later is exactly the kind of synchronization problem a natively multimodal session avoids by design.