Nothing in a voice agent is more visible to a caller than getting this lesson’s topic wrong. A caller who gets talked over, or who sits through three seconds of silence wondering if the call dropped, forms an opinion about the whole system in that one moment, regardless of how accurate the transcription was or how smart the underlying model is. Endpointing and turn detection are a small amount of code with an outsized effect on perceived quality, which is exactly why they get their own lesson instead of being folded into the ASR stage.
VAD endpointing is necessary but not sufficient
Voice activity detection answers a narrow question: is there speech energy in this audio right now. Endpointing, in its simplest form, is VAD plus a timer: if there’s been N milliseconds of silence since the last detected speech, assume the caller is done and hand off to the dialogue manager. Deepgram’s endpointing=300 parameter from Lesson 4 is exactly this: a silence-duration heuristic, cheap and fast to compute, with zero understanding of what was actually said.
The problem is that human speech is full of pauses that aren’t turn endings: recalling a number, deciding how to phrase something, taking a breath mid-sentence. A fixed silence threshold treats all of these identically to an actual completed turn, which is why systems relying on VAD-only endpointing develop a reputation for interrupting people, especially on anything involving spelled-out information like account numbers, addresses, or confirmation codes.
Semantic turn detection
The fix is layering a semantic signal on top of the silence signal. Instead of asking only “how long has it been quiet,” the system also asks “does the transcript so far sound complete.” A partial transcript ending in “so my account number is” reads as obviously incomplete regardless of how long the silence has lasted; a partial transcript ending in “…and that’s everything, thanks” reads as complete even after a shorter pause. Production systems typically implement this as a lightweight classifier or a fast LLM call scoring transcript completeness, run only on partial transcripts near the silence threshold rather than continuously, to keep the added latency small.
flowchart TB
SIL["Silence detected\nby VAD"] --> T{"Silence duration\n>= threshold?"}
T -->|no| WAIT["Keep listening"]
T -->|yes| SEM{"Is partial transcript\nsemantically complete?"}
SEM -->|incomplete\ne.g. trailing 'and...'| EXTEND["Extend wait window\n+ keep listening"]
SEM -->|complete| PROS{"Trailing intonation\nsuggests continuation?"}
PROS -->|yes, rising/hanging| EXTEND
PROS -->|no, falling/final| END["Turn ends\ntrigger dialogue manager"]
WAIT --> SIL
style SIL fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style T fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style WAIT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style SEM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style EXTEND fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style PROS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style END fill:#fff7ed,stroke:#f59e0b,color:#0F172AThis decision logic runs on every silence gap near the threshold, not on every audio frame, so its cost is bounded. A minimal version can ship as a rules-based completeness check (does the transcript end mid-clause, does it end with a conjunction, is it a bare number sequence); a more sophisticated version uses a small, fast classifier or a low-latency LLM call, trading a modest amount of latency (typically 50-150ms) for meaningfully fewer false interruptions.
The latency-vs-interruption tradeoff
Every endpointing threshold is a dial with two failure modes on opposite ends. Set it too short (150-200ms) and the agent feels snappy in clean, single-clause utterances but interrupts constantly on anything with a natural pause. Set it too long (600-800ms+) and interruptions nearly disappear, but every response now carries that much dead air before the agent starts speaking, which reads as slow or unresponsive even when the underlying pipeline is fast. There is no single correct value; production systems tune this per use case. A quick account-lookup IVR replacement can run a shorter threshold since utterances are typically short and simple. A support call handling complex, multi-clause explanations benefits from a longer threshold or, better, from leaning on semantic completeness so the raw silence threshold can stay short without causing false endpoints.
Barge-in: handling interruption gracefully
Turn detection isn’t just about the caller’s silence, it’s also about what happens when the caller speaks while the agent is speaking. A natural conversation is always interruptible: real people cut each other off constantly, and a voice agent that can’t be interrupted reads as robotic and, worse, becomes actively frustrating when the caller is trying to correct a mistake the agent is in the middle of repeating back.
Handling barge-in correctly requires full-duplex audio, listening for caller speech continuously, even while the agent’s own TTS output is playing, which Module 5 covers as its own transport-layer topic. The turn-detection responsibility here is narrower but critical: as soon as VAD detects genuine caller speech during agent playback (filtering out the agent’s own audio bleeding into the microphone, a real problem on speakerphone calls), the pipeline must immediately cancel the in-flight TTS stream, stop sending further audio, and route the new caller speech into ASR as a fresh turn. Getting the cancellation fast (under roughly 100-150ms from detected speech) is what makes barge-in feel like a natural interruption rather than the agent stubbornly finishing its sentence over the caller.
Combined, VAD-based endpointing, semantic completeness scoring, and responsive barge-in handling are what separate a voice agent that feels like it’s listening from one that’s just waiting for its turn to talk. Module 3 builds directly on top of this: everything the dialogue manager does assumes turn detection has already told it, correctly, when to start.