Function Calling in Voice Agents

A support agent that can only talk, and can’t actually check an order status or update an address, is a chatbot wearing a phone number. Real utility means calling real backend systems mid-conversation, and that’s exactly where voice diverges hardest from text-based tool use: a backend call that takes two seconds is a non-event in a chat UI with a typing indicator, and a conversation-breaking silence on a live call. This lesson builds the pattern that makes tool calling work on a phone line.

Why tool latency hits differently in voice

In a text chat interface, waiting for a tool call to resolve has an entire visual vocabulary built around it: spinners, “thinking…” indicators, progressively-appearing text. None of that exists on a phone call. The only channel available is audio, and the only way to signal “I’m working on this” is to actually say something. Left unhandled, a caller-facing tool call of even 1-2 seconds produces literal silence, and silence on a phone line reads as one thing to nearly everyone: the call dropped. Callers hang up, or worse, start talking again (“hello? are you there?”) which can trigger the pipeline’s barge-in handling and interrupt the very tool call meant to help them.

This isn’t a hypothetical latency number either. A typical backend lookup, an order-status API, a CRM query, a payment processor call, realistically takes anywhere from 300ms for a fast, well-indexed internal service to 2-3 seconds for a third-party API under load. All of that sits squarely inside the range where an unmasked silence is noticeable and unnatural.

The filler-acknowledgment pattern

The fix is to make the acknowledgment itself part of the dialogue manager’s output, triggered the moment a tool call is decided on, spoken by TTS while the tool call runs concurrently in the background, rather than sequentially after it resolves.

flowchart TB
    CALLER["Caller: 'What's the status\nof my order 4471?'"] --> DM["Dialogue manager\ndecides tool call needed"]
    DM --> ACK["Emit filler phrase immediately\n'Let me check that for you'"]
    DM --> TOOL["Start tool call concurrently\nget_order_status(4471)"]
    ACK --> TTS1["TTS speaks filler\nwhile tool runs"]
    TOOL -->|success| RESULT["Tool result available"]
    TOOL -->|timeout/error| FAIL["Tool call failed"]
    RESULT --> FINAL["Generate final response\nwith real data"]
    FAIL --> FALLBACK["Generate honest fallback\n+ offer alternative/transfer"]
    FINAL --> TTS2["TTS speaks final response"]
    FALLBACK --> TTS2

    style CALLER fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style DM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style ACK fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TOOL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS1 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style RESULT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style FAIL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style FINAL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style FALLBACK fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS2 fill:#fff7ed,stroke:#f59e0b,color:#0F172A

The key architectural point: the acknowledgment and the tool call run concurrently, not sequentially. Speaking “let me check that” and then starting the tool call afterward only delays the problem by the length of the phrase; the acknowledgment has to fire and the tool call has to start in the same breath.

Implementing the loop

Here’s a working pattern for this, using claude-sonnet-5 as the reasoning backend with a simple tool schema:

import asyncio

FILLER_PHRASES = {
    "get_order_status": "Let me check that for you, one second.",
    "update_address": "Sure, let me pull up your account.",
    "check_appointment": "One moment while I look that up.",
}

async def handle_turn(transcript: str, tools: list, tts_stream):
    # Ask the model whether this turn needs a tool call.
    response = await llm.messages.create(
        model="claude-sonnet-5",
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": transcript}],
        tools=tools,
    )

    tool_call = extract_tool_call(response)  # None if no tool needed

    if tool_call is None:
        # No tool needed: stream the direct response straight to TTS.
        await tts_stream.speak(response.text)
        return

    # Fire the filler phrase and the tool call concurrently.
    filler = FILLER_PHRASES.get(tool_call.name, "Let me look into that.")
    filler_task = asyncio.create_task(tts_stream.speak(filler))
    tool_task = asyncio.create_task(
        run_tool_with_timeout(tool_call, timeout=2.5)
    )

    await filler_task  # filler is short; usually finishes well before the tool

    try:
        tool_result = await tool_task
        final = await llm.messages.create(
            model="claude-sonnet-5",
            system=SYSTEM_PROMPT,
            messages=[
                {"role": "user", "content": transcript},
                {"role": "tool_result", "content": tool_result},
            ],
        )
        await tts_stream.speak(final.text)
    except asyncio.TimeoutError:
        # Graceful degradation: honest fallback, never silence.
        await tts_stream.speak(
            "I'm having trouble pulling that up right now. "
            "Would you like me to connect you with a specialist?"
        )


async def run_tool_with_timeout(tool_call, timeout: float):
    return await asyncio.wait_for(execute_tool(tool_call), timeout=timeout)

Two details here matter more than they look. The asyncio.wait_for timeout is deliberately tighter (2.5 seconds) than you might allow a backend call in a text-based system, because even with a filler phrase playing, a caller’s patience for continued silence afterward is limited; a filler buys time, it doesn’t buy unlimited time. And the timeout failure path is a real, spoken fallback with a next step, not a silent retry or a dropped call, applying the same graceful-degradation principle used for the ASR and turn-detection stages elsewhere in this pipeline.

Choosing filler phrases that don’t feel canned

A dictionary of static filler phrases, as in the example above, works well for a first version but tends to feel repetitive on longer calls with several tool calls in a row. Production systems often vary phrasing (a small rotating set, or a fast, cheap LLM call to generate a one-off acknowledgment tailored to the specific request) and, for tool calls expected to take longer than roughly 3-4 seconds, insert a second, later filler (“still working on that, thanks for your patience”) rather than a single phrase followed by continued silence. The underlying principle stays the same throughout: a caller should never experience more than a second or so of unexplained silence, no matter what’s happening on the backend.

Module 3’s final lesson builds on this same conversational loop to handle something tool calls make harder: keeping track of everything that’s happened so far, across a call that might run many turns and several tool calls long.