Everything upstream of this lesson, streaming ASR, turn detection, was built to deliver one thing cleanly to the dialogue manager: a finalized, well-timed transcript of what the caller just said. What happens next is where most of the perceived intelligence of a voice agent actually lives, and it’s also where a straight port of chatbot prompting habits quietly breaks the whole pipeline. This lesson is about prompting and model choice for a brain that has to think and speak in real time, not write an essay.
Spoken language is a different medium, not a shorter version of text
Text chat responses can afford to be long, structured, and visually organized, because a reader controls their own pace, can re-scan a paragraph, and can visually parse a bulleted list in a glance. None of that applies on a phone call. A caller hears words in a fixed order at a fixed pace, cannot re-read anything, and loses track of a response the moment it exceeds roughly two sentences of density. A markdown bulleted list, harmless in a chat UI, becomes either literal spoken punctuation (“asterisk, first item…”) or, if the TTS layer silently strips markdown, a wall of run-on sentences with none of the structure the list was meant to provide.
The practical rules that follow are specific and non-negotiable for a voice system prompt: no markdown, ever. Short sentences, generally under 20 words, because that’s roughly what a listener can hold in working memory without needing repetition. One idea per sentence, since spoken language doesn’t support the dense subordinate clauses written prose tolerates. Numbers, dates, and currency should be phrased the way a person would say them aloud (“four forty-five” rather than “$4.45” as a literal string, though smart_format on the ASR side and the TTS engine’s own normalization typically handle the reverse direction). And responses should default shorter than feels natural for a text model, since a caller will always ask a follow-up if they want more, but they can’t skim past a response that’s too long.
Latency-aware prompting
Response length isn’t just a comprehension concern, it’s directly a latency concern, because time-to-first-audio is bounded below by how long the LLM takes to produce enough tokens for TTS to start synthesizing. A system prompt that encourages long, thorough answers is also, silently, a system prompt that increases the latency budget from Lesson 2’s pipeline diagram. Two techniques address this directly. First, structure prompts so the model’s natural first sentence is a direct, useful answer, not a preamble (“Great question! Let me look into that for you…”) that adds tokens before anything TTS-worthy exists. Second, stream the LLM’s output token by token into TTS rather than waiting for the full response, so synthesis of the first sentence begins while the model is still generating the third.
The other latency lever is filler-phrase strategy, covered in full in the next lesson on function calling: when the dialogue manager needs to call a tool that will take real time, acknowledging that immediately (“let me pull that up”) gives the caller something to hear while the tool runs, rather than silence that reads as a dropped call.
Choosing the reasoning backend
claude-sonnet-5 and GPT-5 are both credible 2026 choices for a voice dialogue manager, and neither is universally faster or better; the right choice depends on your specific tool schema, system prompt length, and the task distribution of real conversations. What matters more than picking a “winner” from general benchmarks is measuring both under your actual conditions: time-to-first-token with streaming enabled, using your real system prompt and tool definitions (longer tool schemas measurably slow first-token latency on both providers), and task success rate against a representative sample of your own conversations, not a generic leaderboard. A model that’s marginally slower but meaningfully more reliable at correctly invoking the right tool is usually the better trade for a support agent, where a wrong tool call costs more than 100ms of latency; a model that’s faster but occasionally verbose may be the better trade for a casual in-app assistant where speed is what users actually notice.
flowchart TB
TRANSCRIPT["Final transcript\nfrom turn detection"] --> DM["Dialogue manager\nclaude-sonnet-5 or GPT-5"]
DM --> DECIDE{"Needs a tool call?"}
DECIDE -->|yes| FILLER["Emit filler phrase\n'let me check that'"]
FILLER --> TOOL["Execute tool call"]
TOOL --> DM2["Generate final response\nwith tool result"]
DECIDE -->|no| DM2
DM2 --> STREAM["Stream tokens\nto TTS as generated"]
style TRANSCRIPT fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style DM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style DECIDE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style FILLER fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style TOOL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style DM2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style STREAM fill:#fff7ed,stroke:#f59e0b,color:#0F172AA production system prompt
Here’s a realistic starting point for a support agent’s system prompt, encoding the rules above directly rather than hoping the model infers them:
SYSTEM_PROMPT = """
You are a voice support agent for Aria Bank, speaking on a live phone call.
Your response is converted directly to speech, so follow these rules exactly:
- Never use markdown, bullet points, numbered lists, or any visual formatting.
Speak in plain, natural sentences only.
- Keep responses to 1-3 short sentences unless the caller explicitly asks
for more detail. You can always continue in the next turn.
- Use contractions and a warm, conversational tone, the way a helpful
human agent would talk, not a written notice.
- Say numbers, dates, and amounts the way a person would say them aloud
("your balance is two hundred and forty dollars", not "$240.00").
- If you need to call a tool that might take a moment, say a short
acknowledgment first ("let me pull that up") before the tool result
arrives, so the caller isn't left in silence.
- If you're not confident you understood the caller correctly, ask a
short clarifying question rather than guessing.
- If the caller's request is outside what you can help with, offer to
transfer them to a human agent rather than refusing outright.
"""Every rule in that prompt maps directly to a real failure mode covered in this course: the no-markdown rule prevents TTS from reading punctuation aloud, the length limit protects the latency budget, the acknowledgment rule is the filler-phrase strategy the next lesson builds out in code, and the transfer-to-human rule is the escalation path Lesson 1 flagged as the difference between a demo and a system real users can call.
On a managed platform this same prompt is a config field, not application code. NextNeural’s campaign object accepts a persona_prompt string directly (POST /api/voice/flows), and the platform wires it into every call the campaign dials without you owning the dialogue-manager loop that assembles it. The rules above still apply word for word, since a persona_prompt is still text handed to the same class of reasoning model; what changes is where that prompt lives and who’s responsible for iterating on it.