How much VRAM do you actually need to run Llama 3 or Gemma locally?

Running large language models like Llama 3 or Gemma locally often leads to questions about VRAM requirements, with many users unsure if their 3060 or similar GPUs can handle the load. Despite general advice, precise calculations are rarely provided, resulting in users downloading models only to find they exceed their GPU's memory limits. The real challenge lies not just in fitting model weights but also in accommodating the KV cache during inference, which can cause out-of-memory errors. Understanding these nuances is crucial for anyone looking to leverage local AI models without running into unexpected technical hurdles.