We Gave an AI Agent a Long Context Caching Idea. Here's what happened next!

A few days ago, Han Xiao (VP AI @ Elastic) shared an experiment on Linkedin that asked a provocative question: what happens if you stop treating retrieval as a separate system and instead use the model’s own KV cache as the document store? The setup was ambitious: Qwen3.5-35B-A3B LLM, 1M token context, A single 24 GB L4 GPU, and A pipeline that avoids embeddings, vector databases, and chunking entirely. The core idea was simple. Prefill the document once, save the KV cache to disk, rest...