Voice Agent Turn-Taking: Stop Live AI Calls From Talking Over Users

Voice agents do not usually fail because the model is “not smart enough.” They fail in the awkward half-second where the user pauses, breathes, corrects themselves, or interrupts while the AI is still talking. That tiny moment decides whether the product feels useful or robotic. If your live AI call cuts people off, talks over them, ignores barge-in, or waits so long that users repeat themselves, no prompt will save the experience. The fix is not one magic model. It is a turn-taking system: audio signals, semantic checks, interruption rules, streaming, and metrics that work together. This guide walks through a practical voice agent turn-taking design you can ship in a real product. Why turn-taking is the real voice agent bottleneck Text chat is forgiving. A user types. The model answers. If the response takes two seconds, the user may still wait. Voice is different. Humans expect conversation to move quickly. A delay feels like confusion. An early response feels rude. Talking over the user feels broken. A production voice agent has to answer three questions again and again: Is the user still speaking? Is the user finished enough for the agent to respond? If the user interrupts, should the agent stop, listen, or continue? Most teams start with a simple pipeline: Microphone -> Speech-to-text -> LLM -> Text-to-speech -> Speaker Enter fullscreen mode Exit fullscreen mode That is enough for a demo. It is not enough for a live workflow where the caller changes their mind, uses filler words, speaks in a noisy room, or interrupts because the AI misunderstood them. The practical goal is not “lowest latency at any cost.” The goal is comfortable turn timing: fast enough to feel alive, patient enough to avoid cutting people off, and interruptible enough to recover when the user takes control. Research signals behind this topic Recent AI platform activity points toward live agents moving from demos into production workflows: Product launches are emphasizing embedded live agents that can see, speak, and operate inside software. Voice agent platforms are highlighting same-day deployment, multilingual calls, and modular conversation blocks. Developer discussions keep returning to latency, context loss, governance, evaluation, and whether agents can be trusted without constant babysitting. Search results for voice agent latency are getting crowded, but practical implementation content around turn-taking, barge-in tuning, and interruption policy is thinner and often scattered across vendor docs. That creates a useful content gap: builders do not only need “make it faster.” They need a concrete way to decide when the AI should speak, wait, pause, resume, or hand off. The turn-taking stack Think of turn-taking as a small control plane beside your voice pipeline. Audio stream -> Voice activity detection -> Partial transcript stream -> End-of-turn detector -> Interruption policy -> Agent state machine -> Response planner -> Streaming TTS Enter fullscreen mode Exit fullscreen mode Each layer answers a different question. Layer Job Common failure VAD Detect speech vs silence Background noise triggers false speech Endpointing Decide when a turn may be over Cuts off slow speakers Semantic end-of-turn Check whether the thought is complete Waits too long on short answers Barge-in Let user interrupt AI speech User speaks but AI keeps talking State machine Track listen/respond/pause states Race conditions between audio and tools Metrics Measure timing and recovery Team optimizes averages while p95 is bad You can buy pieces of this stack from providers, but you still need product-specific policy. A banking workflow, medical triage flow, coding assistant, and onboarding agent should not share the same interruption behavior. A simple state machine for live calls Start with explicit states. Do not let every async callback mutate the session freely. LISTENING user speech starts -> USER_SPEAKING USER_SPEAKING possible end detected -> THINKING noise rejected -> LISTENING THINKING response ready -> AGENT_SPEAKING user resumes -> USER_SPEAKING AGENT_SPEAKING user barge-in -> INTERRUPTED speech complete -> LISTENING INTERRUPTED stop TTS -> USER_SPEAKING Enter fullscreen mode Exit fullscreen mode A basic TypeScript sketch: type CallState = | 'LISTENING' | 'USER_SPEAKING' | 'THINKING' | 'AGENT_SPEAKING' | 'INTERRUPTED'; type Event = | { type: 'speech_started'; at: number; confidence: number } | { type: 'speech_ended'; at: number; transcript: string } | { type: 'partial_transcript'; text: string; at: number } | { type: 'barge_in'; at: number; confidence: number } | { type: 'response_ready'; responseId: string } | { type: 'tts_done'; responseId: string }; function transition(state: CallState, event: Event): CallState { if (state === 'LISTENING' && event.type === 'speech_started') { return event.confidence > 0.65 ? 'USER_SPEAKING' : 'LISTENING'; } if (state === 'USER_SPEAKING' && event.type === 'speech_ended') { return 'THINKING'; } if (state === 'THINKING' && event.type === 'speech_started') { return 'USER_SPEAKING'; } if (state === 'THINKING' && event.type === 'response_ready') { return 'AGENT_SPEAKING'; } if (state === 'AGENT_SPEAKING' && event.type === 'barge_in') { return event.confidence > 0.75 ? 'INTERRUPTED' : 'AGENT_SPEAKING'; } if (state === 'AGENT_SPEAKING' && event.type === 'tts_done') { return 'LISTENING'; } if (state === 'INTERRUPTED') { return 'USER_SPEAKING'; } return state; } Enter fullscreen mode Exit fullscreen mode The exact states can change, but the discipline matters. Voice systems are streaming systems. Without a state machine, you will eventually play stale audio after the user has already corrected the agent. Use both acoustic and semantic end-of-turn detection Silence alone is a weak signal. A user might pause because they are thinking. They might say “I need to book a flight from Delhi to…” and pause before giving the city. If your agent jumps in, the call feels clumsy. A better end-of-turn detector combines: Voice activity detection: did speech stop? Silence duration: how long has the user been quiet? Partial transcript: does the text look complete? Intent confidence: do we have enough slots to act? Conversation context: did the agent ask a yes/no question or an open-ended question? Example policy: type TurnSignal = { silenceMs: number; transcript: string; intentConfidence: number; requiredSlotsFilled: boolean; lastAgentQuestionType: 'yes_no' | 'slot_fill' | 'open'; }; function shouldEndTurn(signal: TurnSignal): boolean { const text = signal.transcript.trim().toLowerCase(); if (signal.lastAgentQuestionType === 'yes_no') { return signal.silenceMs > 250 && /^(yes|no|yeah|nope|correct|right)\b/.test(text); } if (signal.lastAgentQuestionType === 'slot_fill') { return signal.silenceMs > 450 && signal.requiredSlotsFilled; } const looksIncomplete = /\b(and|or|from|to|because|with|for)$/i.test(text); if (looksIncomplete) return false; return signal.silenceMs > 700 && signal.intentConfidence > 0.7; } Enter fullscreen mode Exit fullscreen mode This is intentionally simple. You can replace the regex with a classifier later. The key point is that end-of-turn is not only an audio problem. It is a conversation problem. Design barge-in as a policy, not a toggle Barge-in means the user can interrupt while the AI is speaking. Many teams treat it as a boolean setting: on or off. That is too crude. A production system should decide what kind of interruption happened: Correction: “No, I meant the other account.” Cancellation: “Stop.” Clarification: “Wait, what does that mean?” Background noise: another person talks nearby. Backchannel: “mm-hmm,” “okay,” “yeah.” These should not all behave the same. type BargeInDecision = 'ignore' | 'duck_audio' | 'pause_and_listen' | 'stop_and_reset'; function classifyBargeIn(text: string, confidence: number): BargeInDecision { const clean = text.trim().toLowerCase(); if (confidence ; reversible: boolean; }; function canExecuteTool(currentIntentVersion: number, req: ToolRequest) { if (req.intentVersion !== currentIntentVersion) return false; if (!req.reversible) return false; // require confirmation elsewhere return true; } Enter fullscreen mode Exit fullscreen mode This is especially important for builders shipping AI workflows into customer-facing products. Users speak casually. They revise themselves. Your system has to treat speech as evolving input until the turn is stable. Measure the moments users actually feel Average latency is not enough. Track these metrics per conversation, per environment, and per user segment: Time to first agent audio: from detected end-of-turn to first audible response. False cutoff rate: user resumes within 500 ms after agent starts speaking. Barge-in success rate: user interrupts and AI stops within target time. Ignored interruption rate: user speaks over AI but the system continues. Dead-air p95: long silence before response. Repeat rate: user repeats the same intent after a bad turn. Correction rate: user says “no,” “actually,” or “I meant.” Tool-after-interruption incidents: tool results spoken after intent changed. A simple event log helps: { "call_id": "call_123", "turn_id": "turn_009", "events": [ { "type": "speech_started", "t": 1200 }, { "type": "speech_ended", "t": 4100 }, { "type": "agent_audio_started", "t": 4620 }, { "type": "barge_in", "t": 5300 }, { "type": "agent_audio_stopped", "t": 5410 } ], "metrics": { "end_to_audio_ms": 520, "barge_in_stop_ms": 110 } } Enter fullscreen mode Exit fullscreen mode Review bad calls weekly. The fastest way to improve turn-taking is to listen to the exact moments where the user had to repeat, correct, or wait. Tune by conversation type Not every turn deserves the same silence threshold. Use different settings for different moments: Conversation moment Better behavior Yes/no question Respond quickly after short answer Address, email, or ID capture Wait longer; users speak in chunks Emotional complaint Leave more space; avoid rushing Confirmation before action Require complete answer and explicit consent Long explanation by agent Enable barge-in aggressively Background-noise environment Raise speech confidence threshold A voice agent that handles an angry support call should not interrupt like a fast command palette. Context matters. Common mistakes to avoid Mistake 1: Optimizing only for speed A faster agent that interrupts users is worse than a slightly slower agent that listens well. Optimize for completed turns, not benchmark bragging rights. Mistake 2: Using one silence threshold everywhere A 300 ms pause may be enough after “yes.” It is not enough after “my account number is…” Use adaptive thresholds. Mistake 3: Letting TTS queues keep playing When the user interrupts, old audio must stop. Cancel queued chunks, tool summaries, and delayed follow-ups tied to the previous intent. Mistake 4: Treating backchannels as full interruptions People say “yeah,” “right,” and “mm-hmm” while listening. Do not reset the whole conversation every time. A practical implementation checklist Use this before you ship a live voice agent: [ ] Define explicit call states. [ ] Combine VAD with semantic end-of-turn detection. [ ] Add adaptive silence thresholds by question type. [ ] Make every streamed response cancellable. [ ] Drop stale TTS chunks after interruption. [ ] Classify barge-ins: ignore, duck, pause, or reset. [ ] Version user intent before tool calls. [ ] Require confirmation for irreversible actions. [ ] Track false cutoffs, repeat rate, and ignored interruptions. [ ] Review real call traces, not only aggregate dashboards. FAQ What is voice agent turn-taking? Voice agent turn-taking is the system that decides when the user is speaking, when the user is done, when the AI should respond, and how the AI should behave if the user interrupts. What is barge-in for AI voice agents? Barge-in lets a user interrupt an AI voice agent while it is speaking. A good implementation stops or lowers the AI audio, listens to the user, and updates the conversation state without losing context. Is latency the same as turn-taking? No. Latency is about speed. Turn-taking is about timing and control. A low-latency agent can still feel bad if it cuts users off or ignores interruptions. How long should a voice agent wait before responding? It depends on the conversation. A yes/no answer may need only a short pause. A form field, address, or emotional explanation needs more patience. Use adaptive thresholds instead of one global silence value. Should AI voice agents always allow interruption? Usually yes for long spoken responses, but interruption should be classified. Backchannels like “mm-hmm” may only require audio ducking. Corrections and cancellation should pause or stop the response. How do you test voice agent turn-taking? Test with noisy audio, slow speakers, interruptions, corrections, accents, long tool calls, and users who change their mind mid-sentence. Measure false cutoffs, ignored interruptions, repeat rate, and barge-in stop time.