Colibri: Running a 744B AI Model on Your Laptop

GLM-5.2 is a frontier AI model with 744 billion parameters. It normally requires H100 GPUs, hundreds of gigabytes of VRAM, and a cloud bill larger than a car payment. Colibri runs it on a laptop with 25 GB of RAM. The secret? It never loads the whole model. What Is Colibri? Colibri is a pure C inference engine for GLM-5.2, a Mixture-of-Experts (MoE) model released by Z.ai. It streams model experts from disk instead of keeping them in memory, trading disk I/O for RAM. "This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan." — Colibri README How It Works: The MoE Architecture GLM-5.2 uses Mixture-of-Experts. For each token, only a small subset of "experts" are active. The model has: 21,504 routed experts (19 MB each at int4) — stored on disk (~370 GB) Dense parts (attention, shared experts, embeddings) — resident in RAM (~10 GB) The key insight: only ~11 GB of parameters change token-to-token. Those 11 GB are the routed experts. The remaining 34 GB are dense layers that stay loaded. Colibri keeps the dense parts in RAM and streams experts from NVMe SSD on demand. It uses per-layer LRU caching, optional pinned hot-store for frequently used experts, and relies on the OS page cache as a free L2 cache. Technical Specifications Metric Value Model on disk (int4) ~370 GB Resident RAM 9.9 GB Minimum RAM 25 GB Load time ~30 seconds Peak RSS during chat ~20 GB (auto-capped) Cold decode cost ~11 GB disk reads/token MTP speculation (int8 head) 2.2–2.8 tok/forward Key Features Faithful GLM-5.2 forward — token-exact validation against transformers oracle MLA attention with compressed KV-cache (576 floats/token vs. 32,768 — 57× smaller) DeepSeek-V3-style sigmoid router with shared expert Native MTP speculative decoding — GLM-5.2's multi-token-prediction head drafts tokens verified in batched forward Grammar-forced speculative drafts for constrained JSON/NDJSON output Integer-dot kernels — Q8_0-style int8 activations, AVX2 maddubs MLA weight absorption for decode Async expert readahead Quantization kernels: int8 / packed int4 / packed int2 DSA sparse attention — GLM-5.2's lightning indexer KV-cache persistence — conversations reopen warm across engine restarts Router-lookahead prefetch (experimental) — 71.6% predictable routing Platform Support Platform Status Notes Linux ✅ Full Primary platform macOS ✅ Full Metal backend option Windows 11 ✅ Native MinGW-w64 with _WIN32 compatibility layer Requirements: gcc with OpenMP, AVX2, ≥16 GB RAM, ~370 GB model on local NVMe. How to Use Quick start: cd c ./setup.sh # Build and self-tests COLI_MODEL=/path/to/glm52_i4 ./coli chat Enter fullscreen mode Exit fullscreen mode OpenAI-compatible API: COLI_MODEL=/nvme/glm52_i4 ./coli serve \ --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri Enter fullscreen mode Exit fullscreen mode Quality benchmarks: ./coli bench # hellaswag, arc_challenge, mmlu Enter fullscreen mode Exit fullscreen mode Performance: Slow but Correct This is the trade-off. Colibri is not fast. Performance depends heavily on disk speed, RAM capacity, and expert caching: Hardware Tokens/second Apple M5 Max (128 GB RAM, Metal) 1.06–1.83 tok/s Ryzen AI 9 HX 370 (128 GB RAM) 0.37 tok/s Ryzen 9 9950X (PCIe 5.0 NVMe) 0.28 tok/s For context, a cloud H100 inference might run at 30–50 tok/s. Colibri is 10–100× slower. But it costs nothing after setup and runs entirely locally. Why This Matters Frontier AI models are becoming inaccessible. GPT-4, Claude 3.5, GLM-5.2 — these require proprietary APIs, monthly subscriptions, and internet access. Your data leaves your machine. Your conversations are logged. Colibri proves that with clever engineering, you can run a frontier model on consumer hardware. You need: A laptop with ≥25 GB RAM A fast NVMe SSD with ~370 GB free space Patience (0.3–2 tok/s) The model answers correctly. The quality is there. The trade-off is speed. Real-World Use Cases Offline coding assistant: Run GLM-5.2 locally without API keys or quotas. Your code never leaves your machine. Privacy-first knowledge base: Build a RAG system with local retrieval and local generation. Zero data exfiltration. Benchmarking research: Test frontier model quality without cloud costs. Run hellaswag, arc_challenge, mmlu locally. Edge deployment: Install on a rugged laptop for field deployment where internet is unreliable. What Makes This Possible Colibri is a one-person project built on a 12-core laptop with 25 GB of RAM. The author (JustVugg) exploited the Mixture-of-Experts architecture: MoE models only activate ~40B parameters per token Only ~11 GB change token-to-token The rest is dense, shared layers that stay resident Streaming experts from disk is viable with NVMe Quantization (int4) reduces the model from ~1.5 TB to ~370 GB on disk. MLA attention compresses KV-cache by 57×. Speculative decoding with MTP head drafts 2–3 tokens per forward pass. These techniques are not new. Colibri combines them into a single engine optimized for streaming experts. What Is Missing Colibri does not try to be fast. It does not support LoRA fine-tuning, RLHF alignment loading, or custom quantization recipes. It is purpose-built for one thing: running GLM-5.2 faithfully on minimal hardware. For BD developers with cheap VPS or older laptops, this is more theoretical than practical. You need 370 GB of NVMe storage and a modern CPU with AVX2. But it shows the direction: frontier models can run locally if we stop loading them entirely into RAM. The Future of Local AI Colibri is a proof of concept. It proves that streaming experts from disk works. It proves that int4 quantization preserves quality. It proves that MLA attention compression is viable. The next step is making it fast. Better caching, smarter prefetching, GPU acceleration for matmul, multi-threaded expert loading. Each improvement inches closer to "fast enough" for real work. But even at 0.3 tok/s, Colibri is useful. It runs a 744B model on a machine that costs less than a single H100 GPU fan. That is remarkable. How to Try It If you have the hardware, here is the path: Clone the repo: git clone https://github.com/JustVugg/colibri Get GLM-5.2 weights: Download from Z.ai (released under MIT) Quantize to int4: Use the tools in tools/ directory Run setup: cd c && ./setup.sh Chat: COLI_MODEL=/path/to/glm52_i4 ./coli chat You will need patience. The first token takes 30–60 seconds. Subsequent tokens are faster. But the answers are correct. The quality is there. Conclusion Colibri is not for everyone. It requires 370 GB of NVMe storage, a modern CPU, and patience. But it proves that frontier AI models do not require cloud infrastructure. A one-person project, a 12-core laptop, and clever engineering can run a 744B model that would otherwise cost tens of thousands in GPU time. That is the story of local AI. The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. Colibri keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience. Sources GitHub: JustVugg/colibri GLM-5.2 weights released by Z.ai under MIT license Apache 2.0 license for Colibri engine