The WER numbers in the last lesson, 5-8% on clean audio, describe a best case that a real production system rarely lives in. Real callers phone in from moving cars, warehouse floors, and crowded kitchens; they carry accents Nova-3’s training data underrepresents; and a meaningful fraction of calls, especially in multinational deployments, aren’t monolingual at all. This lesson is about closing the gap between benchmark WER and the WER your actual traffic will produce.
The four things that actually break transcription
In practice, four distinct problems account for nearly all production WER degradation, and each has a different fix, which is why lumping them together as “bad audio” leads teams to the wrong mitigation. Background noise (traffic, crosstalk, HVAC hum) raises the noise floor and masks phonetic detail; the fix is signal-level, not model-level. Accented or non-native speech shifts pronunciation away from the model’s dominant training distribution; the fix is often a configuration parameter, not a different model. Code-switching, speakers alternating languages mid-utterance, breaks any pipeline that assumes one fixed language per call. Overlapping or multi-speaker audio (crosstalk between caller and agent, or a caller talking to someone else in the room) confuses both VAD and transcription unless speakers are separated, physically or logically, before or during transcription.
flowchart TB
RAW["Raw caller audio"] --> N{"Noise floor\nhigh?"}
N -->|yes| DENOISE["Upstream noise\nsuppression"]
N -->|no| LANG{"Fixed language\nor multilingual?"}
DENOISE --> LANG
LANG -->|code-switching\nexpected| MULTI["Nova-3 multilingual\nconfig, no fixed language"]
LANG -->|single language| PIN["Nova-3 with language\npinned + smart_format"]
MULTI --> DIAR{"Multiple\nspeakers?"}
PIN --> DIAR
DIAR -->|yes| DIARIZE["diarize=true or\nseparate channels"]
DIAR -->|no| ASR["Final ASR call"]
DIARIZE --> ASR
style RAW fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style N fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style DENOISE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style LANG fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style MULTI fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style PIN fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style DIAR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style DIARIZE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style ASR fill:#fff7ed,stroke:#f59e0b,color:#0F172AConfiguring Nova-3 for real conditions
Most of the mitigation for accent, code-switching, and multi-speaker problems is a configuration choice, not a model swap. Here’s a realistic production configuration for a multinational support line where callers may switch between English and Spanish and where diarization matters for downstream tool-call safety:
DEEPGRAM_URL = (
"wss://api.deepgram.com/v1/listen"
"?model=nova-3"
"&encoding=linear16"
"&sample_rate=16000"
"&interim_results=true"
"&smart_format=true" # normalizes numbers, dates, currency in transcript
"&diarize=true" # labels segments by speaker
"&language=multi" # enables code-switch-aware multilingual detection
"&endpointing=300"
)A few of these are easy to get backwards. smart_format=true is a near-free win for anything with numbers in it, order IDs, phone numbers, dollar amounts get formatted correctly rather than spelled out as words, which matters a lot when the dialogue manager needs to parse an account number out of the transcript reliably. diarize=true is the right call when caller and agent audio genuinely share a channel; if your transport already gives you separate channels per party (common with Twilio Media Streams’ dual-channel mode), diarization is redundant and channel separation upstream is both cheaper and more reliable. Pinning language=en when you expect single-language calls generally improves accuracy over multilingual mode, because the model isn’t spending capacity hedging against a language switch that won’t happen, so don’t default to multilingual mode everywhere; reserve it for traffic that actually needs it.
Fixing the signal vs asking the model to compensate
There’s a recurring temptation to treat every accuracy problem as an ASR configuration problem, when a chunk of them are better solved before the audio ever reaches Deepgram. A basic noise suppression pass, spectral gating or a lightweight learned denoiser running in the audio ingress stage, measurably improves both WER and VAD accuracy on noisy lines, and it’s a component you can tune independently of your ASR vendor. Similarly, tuning your VAD sensitivity for the specific noise floor of your traffic (a call center with constant background chatter needs a higher speech-confidence threshold than a quiet mobile app) prevents noise from being misclassified as speech and sent to ASR at all, which both saves cost and removes a source of spurious partial transcripts.
The order of operations that tends to work best in practice: clean the signal first (noise suppression, correct sample rate, mono channel per speaker if possible), then configure the ASR model for the actual language and speaker conditions of your traffic, and only after both of those are genuinely exhausted consider a custom fine-tuned model. Fine-tuning is a real option for narrow, high-volume domains (a call center that only ever discusses insurance claims, for instance) but it’s slow to iterate on and expensive relative to the configuration and signal fixes above, so it should be the last lever pulled, not the first.
Measuring whether it worked
None of this is worth doing blind. Pull a stratified sample of real production audio across your worst-performing segments (by accent, by call center, by noise condition) and score WER against human-verified transcripts before and after each change, the same golden-set methodology Module 7 formalizes for the whole pipeline. A change that helps clean audio can hurt noisy audio and vice versa, language=multi being a good example, so segment-level measurement, not one aggregate number, is what tells you whether a mitigation actually worked for the callers who needed it.