The earlier lessons in this module measured whether a voice agent works. This one is about whether it’s safe to let it talk to strangers on the phone. Voice introduces risks a text chatbot doesn’t have, because the input arrives as raw audio in real time, the output is spoken to a caller who may be vulnerable or upset, and there’s no chat window to scroll back and check before words leave the speaker.
Prompt injection via spoken content
Anything a caller says becomes text the LLM core reads as part of its context, exactly like a text-based agent, but with one difference: there’s no UI surface for a human to review the transcript before the model acts on it. A caller reading “ignore your previous instructions” is functionally identical to a text injection attempt, except the agent’s response, or worse, its tool call, happens live.
flowchart TB
CALL["Caller audio"] --> ASR["Nova-3 transcription"]
ASR --> FILTER["Injection screen\npattern + classifier check"]
FILTER -->|flagged| SAFE["Sanitized context,\nlog + alert"]
FILTER -->|clean| LLM["claude-sonnet-5 / GPT-5\ndialogue core"]
SAFE --> LLM
LLM --> TOOLCHECK["Tool-call allowlist\n+ confirmation gate"]
TOOLCHECK --> TTS["TTS response"]
style CALL fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style ASR fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style FILTER fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style LLM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style TOOLCHECK fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style SAFE fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style TTS fill:#fff7ed,stroke:#f59e0b,color:#0F172AThe screen doesn’t need to be perfect; it needs to catch the obvious cases and, more importantly, ensure nothing downstream trusts caller text enough to bypass the tool-call allowlist on its own.
PII spoken aloud
Callers volunteer account numbers, SSNs, and card numbers constantly, regardless of what the agent asks for. Redaction has to be automatic at the pipeline level, applied to both the transcript stored for QA and the retained audio itself.
# pii_redact.py - runs on every finalized transcript segment
PII_PATTERNS = {
"card_number": r"\b(?:\d[ -]*?){13,16}\b",
"ssn": r"\b\d{3}-\d{2}-\d{4}\b",
}
def redact_transcript(segment: str) -> str:
for label, pattern in PII_PATTERNS.items():
segment = re.sub(pattern, f"[REDACTED:{label}]", segment)
return segment
def redact_audio(audio_chunk, start_ms: int, end_ms: int):
# Replace the matched span with a tone or silence before
# the recording is written to durable storage.
return mute_span(audio_chunk, start_ms, end_ms)Redact before persistence, not after. Once raw audio with a spoken card number lands in long-term storage, deleting it later is a much harder operation than never writing it that way in the first place.
Abuse and toxic callers
Voice agents will encounter abusive callers more often than text agents, because the barrier to shouting at a phone system is lower than typing. Define a graduated response: a calm redirect on the first instance, a firm boundary statement on repetition, and an escalation or call-termination path if abuse continues, logged for the human team reviewing the queue.
Hallucinated tool calls
A hallucinated fact in a spoken response is wrong information the caller can question. A hallucinated tool call is a wrong action that executes immediately, especially dangerous for destructive operations. Treat consequential tools (cancellations, refunds, account changes) differently from read-only tools (balance lookups): require an explicit verbal confirmation turn before executing, and keep a hard allowlist of which tools the dialogue core can invoke at all.
Disclosure and escalation
Most jurisdictions now require telling callers they’re speaking with an AI system, typically at call start and again on request. Build this as a scripted, non-negotiable opening line rather than something the LLM core generates fresh each time, so it can’t be prompt-engineered away mid-conversation. Pair it with a reliable escalation path: a caller saying “let me talk to a person” should route to a human agent within the same turn, not after another round of the agent trying to resolve the issue itself.
Some platforms make a slice of this declarative rather than something you implement. NextNeural’s campaign config accepts a recording_disclosure string that’s played automatically at call start, and a guardrails array (values like no_price_fabrication, no_stock_fabrication) that constrains what the agent will assert. That’s a narrower surface than the injection screening and tool-call allowlist built above, worth layering on top of rather than relying on alone for anything with a real compliance requirement, but it’s a real example of guardrails-as-config rather than guardrails-as-code.