How Audio8 TTS 0.1B Brings Voice Cloning to Smaller GPUs

OverviewAudio8-TTS-Preview-0.1b is a compact zero-shot text-to-speech model with voice cloning capabilities, built by Audio8. The model uses an Audio8 Falcon H1 architecture with dual autoregressive branches: a slow branch (24 layers, 512 width, 8 attention heads, 2 KV heads) that predicts semantic tokens and a fast branch (4 layers) that predicts codec codebooks. The main generative model contains approximately 170M parameters, with a separate 120M-parameter codec decoder bundled in the checkpoint. The complete audio generation stack operates at 44.1 kHz with 2,048 samples per model frame (~21.5 frames/s), supporting up to 2,048 packed text/audio positions for context. This model supports primary languages in Chinese and English, with experimental multilingual evaluation on German, Spanish, French, Italian, Japanese, and Korean. You load it through Hugging Face Transformers with trust_remote_code=True and require Python 3.11 or newer with a CUDA-capable GPU. The critical distinction of this preview release is its extreme compactness—it delivers zero-shot voice cloning at a fraction of the parameter count of contemporary multilingual TTS systems, making it practical for resource-constrained deployments while trading some quality against larger competitors.Best use casesZero-shot voice cloning for applications with speaker diversity. The model excels at cloning voices from short audio samples without requiring speaker-specific training. Provide a reference audio clip with a matching transcript, and the model generates speech in that voice for new text. This is ideal for interactive applications, customer service bots, or audiobook platforms where you need consistent speaker characteristics across generated content without maintaining separate speaker models. The dual-branch architecture conditions the fast codebook predictions on slow semantic representations, enabling robust voice preservation even with limited reference material.Low-resource deployment scenarios requiring multilingual synthesis. At 170M parameters for the main model, this checkpoint fits in consumer-grade GPU memory and can run on edge devices compared to 4.6B-8.5B parameter alternatives. Organizations with constrained infrastructure—mobile applications, on-device processing, or inference on older hardware—can deploy this model where larger systems are impractical. The 120M codec decoder is bundled, eliminating dependency resolution issues.Chinese and English speech synthesis with acceptable quality trade-offs. The model achieves 3.619 WER in Chinese and 3.307 WER in English on CV3 benchmarks. For applications where these languages are primary and some quality loss is acceptable compared to larger models, the 4-5x reduction in model size makes this viable. Chinese speaker similarity reaches 68.2% and English 56.7% on Seed-TTS evaluation—workable for many production scenarios despite being below larger alternatives.Rapid prototyping and research with compact multilingual architectures. Researchers and developers building TTS pipelines can experiment with the dual-branch autoregressive design at manageable computational cost. The model's architecture patterns scale—the 0.6B variant shares the same design—making this useful for understanding how parameter efficiency affects quality gradients across languages.LimitationsDegraded quality on non-primary languages. Japanese reaches 12.322 WER (compared to 7.205 for the 0.6B variant), Korean 7.653 WER (vs 4.223), and Italian 14.480 WER (vs 4.790). These are reference comparisons from CV3 evaluation rather than strictly matched re-evaluation, but the trend is clear: this model shows "weaker and more variable quality" on languages outside Chinese and English. Do not expect production-grade multilingual performance beyond the primary languages.Lower speaker similarity scores than larger competitors. On Seed-TTS evaluation, similarity reaches 56.7% in English and 68.2% in Chinese. The 0.6B variant achieves 63.2% and 73.1% respectively; Higgs Audio v2 (4.7B) reaches 66.4% and 72.1%. If precise speaker matching is critical, this model's smaller capacity limits how well it preserves voice characteristics, particularly for English.Reference audio quality significantly affects output stability. Very long, noisy, or incorrectly transcribed reference clips reduce generation stability and speaker similarity. You must provide clean reference audio with accurate transcripts matching the spoken content. Mismatched or poor-quality reference material will degrade results in ways that larger models might tolerate better through their additional capacity.Non-commercial license restricts deployment. The model uses Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0) licensing, prohibiting commercial use without explicit permission from Audio8. Research, internal prototyping, and non-commercial applications are permitted, but productization for revenue-generating services requires separate licensing agreements.Computational requirements limit true "edge" deployment. While compact compared to larger TTS systems, the model still requires a CUDA-capable GPU and PyTorch 2.5.0 with torchaudio 2.5.0. The codec decoder alone is 120M parameters. Running on pure CPU is possible but slow; running on mobile or embedded devices without GPU acceleration is not practical without quantization (which the base checkpoint does not provide).Inference speed not documented. The README does not specify tokens-per-second, latency, or practical throughput for real-time applications. Batch inference is supported with pre-encoded reference codes, but without published benchmarks you cannot assess whether this meets your latency requirements.How it comparesvs Audio8-TTS-Preview-0.6b by Audio8: The 0.6B variant is 3.5x larger (0.6B vs 0.17B main model) but delivers substantially better quality across all languages and both evaluation metrics. Choose the 0.1B model only if model size and deployment constraints are the primary driver; the 0.6B model is the better default for quality-conscious projects. The 0.6B achieves 3.205 WER in Chinese vs 3.619, and 63.2% similarity in English vs 56.7%, making it the superior choice unless you have strict model size limits.vs Audio8-TTS-Preview-0.6B-ONNX-INT4 by Audio8: The ONNX INT4 quantized version of the 0.6B model targets CPU inference on low-resource systems, compressing the larger model for edge deployment. If you need CPU-only inference without GPU, choose the ONNX INT4 version despite its higher base parameter count. The 0.1B native PyTorch version runs on GPU but still requires CUDA; the ONNX version trades some quality for actual CPU viability.vs Audio8-ASR-0.1B by Audio8: This is a different task entirely—Audio8-ASR-0.1B is an autoregressive speech recognition model, not text-to-speech. They complement each other in a speech processing pipeline (recognize audio, then synthesize it), but this TTS model serves no purpose for speech-to-text tasks.vs Raon-OpenTTS-1B by KRAFTON: Raon-OpenTTS is 6x larger (1B vs 0.17B) and under Apache 2.0 license, permitting commercial use. Choose Raon-OpenTTS if you need commercial deployment rights and higher quality; choose Audio8 0.1B if you must minimize model size and accept non-commercial restrictions. Raon likely delivers better multilingual quality given its larger capacity, but specific benchmark comparisons are not provided in the available materials.vs kitten-tts-mini-0.8 by KittenML: Kitten TTS Mini is 2x smaller (80M parameters vs 170M main model) and outputs at 24 kHz vs 44.1 kHz. Kitten targets ultra-compact English-only synthesis with fixed voices; Audio8 0.1B supports voice cloning and multiple languages. Choose Kitten only if you need the absolute smallest footprint for fixed-speaker English synthesis; choose Audio8 for zero-shot voice cloning capability.Technical specificationsArchitecture: Audio8 Falcon H1 with dual autoregressive branches. Slow autoregressive branch: 24 transformer layers, 512 hidden width, 8 attention heads, 2 KV heads, predicting semantic tokens. Fast autoregressive branch: 4 transformer layers, 512 hidden width, 8 attention heads, 2 KV heads, predicting codec codebooks conditioned on slow branch hidden states.Model scale:Main generative model: ~170M parametersCodec decoder: ~120M parameters (bundled, no separate checkpoint required)Total stack: ~290M parameters end-to-endAudio codec:Sample rate: 44.1 kHzFrame size: 2,048 samples per model frame (~21.5 frames/second)Codebooks: 10 codebooks with 4,096 entries per codebook (total acoustic token vocabulary: 40,960 tokens)Bundled as codec.pth in repositoryContext and processing:Context window: Up to 2,048 packed text/audio positionsProcessor: Custom AutoProcessor with trust_remote_code=TrueReference audio transcription: Must match spoken content exactlySupported languages:Primary: Chinese, EnglishExperimental/multilingual: German, Spanish, French, Italian, Japanese, KoreanFramework and dependencies:Library: Hugging Face Transformers (≥4.57.0, <5)PyTorch: ≥2.5.0torchaudio: ≥2.5.0soundfile: ≥0.12safetensors: ≥0.4Python: 3.11 or newerHardware: CUDA-capable GPU recommended (CPU inference possible but slow)Model format: SafeTensorsInference configuration options:temperature: 0.7 (default, adjustable)top_p: 0.9 (nucleus sampling)top_k: 50 (top-k filtering)max_new_tokens: 512 (maximum generation length)Sampling: Enabled via do_sample=TrueLicense: Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0). Commercial use prohibited without explicit licensing agreement.Model inputs and outputsInputsText: String or list of strings containing the speech synthesis target (required)Reference audio: WAV file path or list of paths for zero-shot voice cloning (optional for synthesis without cloning)Reference text: String or list of strings with exact transcription of reference audio content (must match spoken content if reference audio provided)Tensors: Processed inputs returned from AutoProcessor with return_tensors="pt" for batch inferenceGeneration parameters: max_new_tokens (max 512), temperature, top_p, top_k, do_sample booleanOutputsAudio codes: Discrete acoustic token sequences (10 codebooks × 4,096 vocabulary size)Waveforms: Raw audio arrays via model.decode_audio(output.codes) returning (waveforms, waveform_lengths)Format: NumPy float32 array, 44.1 kHz sample rate, mono audioSaving: Via soundfile.write() using model.config.codec_sample_rateGetting startedimport soundfile as sf import torch from transformers import AutoModel, AutoProcessor model_id = "Audio8/Audio8-TTS-Preview-0.1b" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = torch.bfloat16 if device == "cuda" else torch.float32 # Load processor and model processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, ) model = AutoModel.from_pretrained( model_id, trust_remote_code=True, dtype=dtype, ).eval().to(device) # Prepare inputs for voice cloning inputs = processor( text=["Hello, this is a test of the voice cloning system."], reference_audio=["reference.wav"], reference_text=["The transcription of your reference audio."], return_tensors="pt", ) inputs = {name: value.to(device) for name, value in inputs.items()} # Generate speech with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, top_k=50, do_sample=True, return_dict_in_generate=True, ) waveforms, waveform_lengths = model.decode_audio(output.codes) # Save output audio = waveforms[0, : int(waveform_lengths[0])].float().cpu().numpy() sf.write("output.wav", audio, model.config.codec_sample_rate) For synthesis without voice cloning, omit reference_audio and reference_text from the processor call.Frequently asked questionsQ: Can I use this model commercially?A: No. The model is licensed under CC-BY-NC-4.0 (Creative Commons Attribution-NonCommercial 4.0), which explicitly prohibits commercial use. You may use it for research, internal prototyping, and non-revenue applications. Contact Audio8 for commercial licensing options.Q: What GPU memory do I need to run this model?A: The README does not specify VRAM requirements. The main model is 170M parameters and codec decoder is 120M parameters. In bfloat16 precision on GPU, expect approximately 1.2-1.5 GB for the model alone, plus additional memory for input tensors and intermediate activations during inference. A GPU with 4GB+ VRAM should handle this comfortably; 2GB might work with careful optimization.Q: How does this model compare to the 0.6B variant for multilingual use?A: The 0.6B variant outperforms this model on all supported languages. On CV3 benchmarks, 0.6B achieves 3.205 WER in Chinese vs 3.619 for 0.1B, and 3.128 WER in English vs 3.307. The gap widens significantly for non-primary languages—Japanese performance is 7.205 WER vs 12.322 for this model. Choose 0.6B unless model size is your absolute constraint.Q: What happens if my reference audio is noisy or the transcript doesn't match exactly?A: Generation stability and speaker similarity will degrade. The model documentation explicitly warns that very long, noisy, or incorrectly transcribed reference clips reduce performance. Provide clean reference audio with accurate transcriptions matching the spoken content. Mismatches between audio and text hurt voice cloning fidelity significantly.Q: Can I fine-tune this model or use it with LoRA?A: The README does not mention fine-tuning or LoRA support. The model is released as a preview checkpoint with evaluation-focused design. Contact Audio8 or check their GitHub repository for training and adaptation documentation if this is needed for your application.Q: What input audio formats does the reference audio accept?A: The processor accepts WAV files (specified as reference_audio=["reference.wav"] in the example code). The documentation does not list other formats explicitly. Stick to WAV to avoid compatibility issues; convert other formats to WAV before processing.Q: Does this model support real-time inference or streaming audio generation?A: The README does not specify inference latency or streaming capabilities. The model supports batch inference with max_new_tokens=512 and sampling-based generation, but without published timing benchmarks you cannot assume real-time performance. Test with your target hardware before committing to latency-sensitive applications.Q: Is this model actively maintained?A: The repository includes a GitHub link and demo page, indicating active development. However, this is labeled a "Preview" release (0.1B), suggesting it is not production-stable. Check the GitHub repository for the latest updates, issue status, and community feedback before using this in production systems.This is a simplified guide to an AI model called Audio8-TTS-Preview-0.1b maintained by Audio8. If you like these kinds of analysis, join AIModels.fyi or follow us on Twitter.