NVIDIA VoiceChat-11B Brings Full-Duplex AI Speech to Real-Time Agents

OverviewNVIDIA-NemotronLabs-VoiceChat-11B is an 11-billion parameter end-to-end full duplex speech model developed by nvidia that performs streaming speech understanding and generation in a single unified architecture. Unlike traditional cascaded pipelines that chain ASR, LLM, and TTS models separately, this model operates directly on audio signals with 16 kHz input and produces 22.05 kHz output, achieving ~450 milliseconds turn-taking latency while supporting real-time interruption handling and tool calling. The architecture combines a Fast Conformer speech encoder, a Nemotron Nano V2 9B LLM backbone, and a proprietary NVIDIA TTS decoder with a separate output channel for tool-calling scripts. Training involved approximately 550,000 hours of audio across a hybrid blend of real speech datasets (Fisher, LibriVox, LibriTTS, VCTK) and synthetic data generated from text corpora including Nemotron 5.5, Ultrachat, and PromptTTS. The model runs on NVIDIA GPU-accelerated systems via the vLLM inference engine, supporting A100, H100, H200, B100, B200, and RTX-6000 hardware on Linux. This model is ready for research purposes only under the OpenMDW License Agreement version 1.1.Best use casesReal-time customer service voice agents with natural conversation flow. This model excels at handling live customer interactions where millisecond-level response latency matters. The ~450ms turn-taking latency means customers experience natural back-and-forth conversation without the mechanical delays of traditional cascaded ASR-LLM-TTS systems. The barge-in capability allows customers to interrupt and the model yields immediately, creating the natural interruption patterns humans expect. Deployment as a streaming WebSocket service via the NVIDIA inference container enables production-grade voice agent infrastructure.Voice applications requiring tool execution during conversation. This is the first open full-duplex model supporting tool calling while maintaining conversational naturalness. When the LLM generates text triggering a tool call (such as database queries, API calls, or calculations), a custom "on-hold" message plays to the user while the tool executes, then the conversation resumes seamlessly. This eliminates awkward silences or repeated user interactions common in simpler voice systems.Multilingual or speaker-diverse applications benefiting from robust pre-training. The training data includes multiple speaker datasets (VCTK, LibriVox, Voxmovies, JL-Corpus) and phonetically diverse content. The 550,000 hours of training audio—combining studio recordings, internet-scale data, and synthetic examples from various TTS systems—provides robustness to accents, background noise variation, and speech quality diversity that specialized models might struggle with.Research into full-duplex speech interaction and spoken dialogue systems. The model ranks #2 on VoiceBench among open full-duplex models and #2 on Full-Duplex-Bench 1.0, demonstrating strong performance on pause handling, smooth turn-taking, and user interruption management. Researchers studying natural conversation dynamics, barge-in behavior, or end-to-end speech understanding have a competitive open baseline with published benchmark results to build upon.Voice applications where end-to-end latency reduction directly impacts user experience. By eliminating multiple model handoffs, inference calls, and serialization overhead of cascaded systems, the unified architecture reduces total system latency. This matters for interactive use cases where cumulative delays from ASR completion + LLM inference + TTS synthesis become noticeable. The model operates in a single forward pass, reducing the opportunities for latency accumulation.LimitationsResearch-only deployment status limits production use. The model documentation explicitly states "This model is ready for research purposes only," meaning commercial deployment, integration into enterprise voice systems, or service offerings require verification of compliance and licensing terms that may restrict practical use cases beyond evaluation and research.Benchmark performance gaps on certain tool-calling tasks. While this model pioneered open full-duplex tool calling, its performance varies significantly by complexity: Simple tool calls reach 58.5%, multiple tools 62.5%, but parallel tool execution drops to 42.5% and parallel multiple tools to 27.5%. This indicates the model struggles with complex multi-step tool interactions that more capable text-based LLMs handle reliably, limiting its use for applications requiring sophisticated workflow orchestration via voice.Argument accuracy limitations in tool calling (44.2%). Beyond tool selection, the model achieves only 44.2% accuracy on argument accuracy and 33% pass@1 rate on Full-Duplex-Bench v3. This means the model frequently extracts incorrect parameters or values from user speech for tool invocation, making it unsuitable for applications where precision in argument passing is critical (e.g., financial transactions, medical directives).Hardware requirements restrict deployment flexibility. The model requires NVIDIA GPUs (A100, H100, H200, B100, B200, or RTX-6000) and runs exclusively on Linux. No CPU inference, ARM support, or Windows deployment paths are documented. Inference acceleration via vLLM is mandatory, not optional. This hardware lock-in means significant cloud costs or on-premises GPU infrastructure investment for any production deployment.Audio quality constraints inherited from training data bias. Training heavily on synthetic TTS data from multiple systems may create robustness to specific TTS artifacts but weakness on naturally-spoken edge cases not well-represented in the 550,000-hour training corpus. The model's behavior on heavily accented speech, singing, whispered speech, or audio with severe background noise remains unquantified in published benchmarks.No quantization or smaller variant mentioned. The 11B parameter count in a single unified model cannot be split into separate deployable components, and no quantized (4-bit, 8-bit) versions, smaller distilled models, or specialized variants are documented, constraining deployment to systems with sufficient VRAM and compute availability.Limited English scope, unclear multilingual capability. The model is trained primarily on English audio datasets with English text corpora. Support for other languages or code-switching scenarios is not documented, making it unsuitable for non-English voice assistants without retraining.How it comparesnemotron-speech-streaming-en-0.6b is a 600M parameter streaming ASR encoder-only model, whereas VoiceChat-11B is an 11B full-duplex speech model combining speech understanding and generation. Choose VoiceChat when you need end-to-end speech conversation with natural turn-taking and tool calling; use nemotron-speech-streaming for lower-latency ASR-only transcription pipelines where you handle the LLM and TTS separately. VoiceChat trades model size and complexity for unified architecture advantages, while the speech streaming model offers minimal footprint and flexibility to plug into existing LLM/TTS stacks.NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 is a massive 550B parameter text-only LLM, whereas VoiceChat is 11B and multimodal (audio-to-audio). The Ultra model offers vastly superior text intelligence and reasoning for complex tasks; VoiceChat optimizes for real-time voice interaction with native speech I/O. If your application is text-based, the Ultra model provides dramatically better capability. If your application must handle voice input and output with low latency, VoiceChat is the only option here and was built specifically for that constraint.NVIDIA-Nemotron-3-Super-120B-A12B-BF16 is a 120B text-based LLM, still 10 times larger than VoiceChat but with pure text I/O. For voice applications, you would cascade the Super model with separate ASR and TTS systems, incurring multiple inference calls and higher latency than VoiceChat's unified approach. Use the Super model if text reasoning quality is paramount and you can tolerate increased total latency; use VoiceChat if you need real-time voice interaction and acceptable reasoning depth at low latency.Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 is a 30B multimodal model supporting video, audio, image, and text understanding for enterprise Q&A and transcription. VoiceChat is smaller (11B) and specialized for real-time voice conversation with turn-taking and tool calling; Nano Omni is larger and broader, covering video analysis and complex document workflows. Choose VoiceChat for real-time voice agents; choose Nano Omni for batch processing of enterprise content like meeting recordings or document intelligence where lower latency is not critical.nemotron-3-nano-omni/audio is a 30B audio reasoning variant accepting audio plus prompts and returning text, whereas VoiceChat is 11B and returns both text and audio. Nano Omni audio is suited for analyzing or transcribing audio content; VoiceChat is suited for generating natural spoken responses in real-time conversation. If you need to understand audio but not speak, Nano Omni works. If you need bidirectional voice conversation, VoiceChat is the choice.Technical specificationsArchitecture and Parameters:Architecture type: Hybrid Mamba/TransformerTotal parameters: 11 billionLLM backbone: NVIDIA Nemotron Nano v2 (9B)Speech encoder: Fast Conformer module from nemotron-speech-streaming-en-0.6bSeparate output channel for tool-calling script generationModel version: v1.0Training data:Audio training size: Approximately 550,000 hoursData modality: Audio (speech) and text combinedReal speech datasets: Fisher, LibriVox, LibriTTS, VCTK, Voxmovies, JL-Corpus, Blackwell studio recordings, Riva Speakers internal datasetSynthetic speech datasets: Generated using PromptTTS, HiFi-TTS, and various TTS systems applied to text corporaText training data: Nemotron 5.5 pre-training and SFT data, Brainy-mantis, Greteal AI v1/v2, Ultrachat 200k, Nemotron Nano v3 function calling data, Persona Plex training datasets, publicly available internet-scale dataData collection: Hybrid (human, synthetic, automated)Labeling: AutomatedInput specifications:Input types: Text (prompt) and audio (user speech)Input format: String and WAV/WebAudioSample rate: 16 kHzFull context window and maximum sequence length not specifiedOutput specifications:Output types: Text (agent response), audio (agent speech), text (user transcription)Output format: String and WAV/WebAudioSample rate: 22.05 kHzResponse latency: ~450 milliseconds for turn-takingInference and runtime:Inference engine: vLLMSupported GPU hardware: NVIDIA A100, H100, H200, B100, B200, RTX-6000Operating system: Linux (required)Testing hardware: NVIDIA H100Quantization options: Not documentedModel file format: Hugging Face checkpointPerformance benchmarks:VoiceBench (open full-duplex models): Ranked #2Full-Duplex-Bench 1.0: Ranked #2Pause handling (Synthetic, TOR): 0.153Pause handling (Candor, TOR): 0.255Smooth turn-taking (TOR): 0.82Smooth turn-taking latency: 448 msUser interruption (TOR): 1.0User interruption latency: 480 msUser interruption (GPT-4o comparison): 4.33Tool calling simple: 58.5%Tool calling multiple: 62.5%Tool calling parallel: 42.5%Tool calling parallel multiple: 27.5%Tool calling irrelevance detection: 89.6%Tool calling average: 56.1%Full-Duplex-Bench v3 tool selection: 82.5%Full-Duplex-Bench v3 argument accuracy: 44.2%Full-Duplex-Bench v3 pass@1: 33%Model inputs and outputsInputsAudio waveforms: WAV or WebAudio format at 16 kHz sample rateText prompts: String-based prompts to condition the agent's behaviorSystem instructions: Custom on-hold messages can be defined per toolUser speech: Real-time streaming audio from microphone or file inputOutputsAgent speech: WAV or WebAudio format at 22.05 kHz sample rate, naturally paced with appropriate pausingAgent text response: String transcription of generated speech contentUser transcription: String transcription of recognized user speechTool calling signals: Text representation of tool invocation scripts with parametersGetting startedThe model supports two inference modes. For non-interactive batch testing, load the Hugging Face checkpoint from a conda environment with the Speech repository:git clone https://github.com/NVIDIA-NeMo/Speech.git cd Speech git switch nemotron-labs-voicechat export NEMO_DIR="$(pwd)" conda create -y -n voicechat python=3.12 conda activate voicechat pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 pip install -e ".[all]" pip uninstall -y nvidia-resiliency-ext pip install transformers==4.56.0 tokenizers==0.22.0 lhotse==1.32.2 For real-time interactive conversations, deploy the optimized NVIDIA inference container with WebSocket support. The model accepts audio input and produces audio output in a streaming fashion, with response latency around 450 milliseconds for natural turn-taking. Detailed deployment instructions are available in the Github repository.Frequently asked questionsQ: Can I use this model for commercial applications?A: This model is designated for research purposes only. Commercial deployment would require verification of the OpenMDW License Agreement v1.1 terms and likely needs permission from NVIDIA. Review the license carefully before any production use.Q: What GPU hardware do I need to run this model?A: The model requires NVIDIA GPUs: A100, H100, H200, B100, B200, or RTX-6000. Specific VRAM requirements are not documented, but 11B parameter full-duplex models typically require 20-40 GB depending on batch size and inference precision. CPU inference is not supported.Q: How does VoiceChat compare to cascading separate ASR, LLM, and TTS models?A: VoiceChat eliminates API handoffs between components, reducing total end-to-end latency to ~450 ms while supporting simultaneous speech understanding and generation. Cascaded systems introduce cumulative latency from multiple inference calls and serialization overhead. However, cascaded systems allow swapping components for specialized use cases (e.g., domain-specific ASR), while VoiceChat is a fixed unified model.Q: What are the failure modes I should expect?A: Complex tool calling (parallel multiple tools) succeeds only 27.5% of the time. Argument extraction accuracy is 44.2%, so parameter values extracted from speech are frequently incorrect. The model struggles with complex multi-step workflows. Tool calling performance is significantly weaker than text-based LLMs on the same tasks.Q: Does this model support languages other than English?A: Training was English-focused with datasets like Fisher, LibriVox, VCTK, and English text corpora. Multilingual capability or code-switching support is not documented or benchmarked. This model is intended for English voice assistants.Q: What is the maximum input audio length I can provide?A: Maximum sequence length and context window are not specified in the documentation. The model is designed for streaming real-time conversation, suggesting it processes audio in chunks, but exact chunk sizes and maximum utterance lengths are not published.Q: How fast is inference? What batch sizes are practical?A: Turn-taking latency is ~450 milliseconds on NVIDIA H100 hardware. The model is optimized for streaming real-time inference via vLLM, not batch processing. Batch processing capabilities and throughput metrics are not documented.Q: Is this model actively maintained and will it receive updates?A: The model was released August 3, 2026, and is available as v1.0. Active maintenance status and update roadmap are not specified in the documentation. Community contributions are possible via the Github repository, but official NVIDIA support level is unclear beyond research use.Q: Can I fine-tune this model on my own data?A: The documentation does not address fine-tuning procedures, data preparation for custom training, or framework support for adaptation. The model appears designed for use as-is rather than as a foundation for fine-tuning. Inquire with NVIDIA or check the Github repository for any fine-tuning guidance.This is a simplified guide to an AI model called NVIDIA-NemotronLabs-VoiceChat-11B maintained by nvidia. If you like these kinds of analysis, join AIModels.fyi or follow us on Twitter.