Why Local LLMs Suddenly Slow Down at Long Context

Local large language models (LLMs) often perform well until the context length exceeds a critical threshold, causing a significant slowdown, sometimes by as much as 50%. This issue arises because the KV cache overflows from the faster VRAM into slower shared memory, which is less efficient. Notably, on Windows systems, this happens without any out-of-memory warnings, leaving users unaware of the performance degradation. Understanding this helps users manage expectations and optimize their use of local LLMs, especially for tasks requiring extensive context processing.