KV Caching in LLMs

You must have seen it every time you use ChatGPT or Claude that the first token takes noticeably longer to appear. Then the rest stream out almost instantly. Behind the scenes, it's a deliberate engineering decision called KV caching, and the purpose is to make LLM inference faster. Before we get into the technical details, here's a side-by-side comparison of LLM inference with and without KV caching: 0:01 / 0:47 Now let's understand how it works, from first principles. Part 1: How LLMs gene...