LLM on EKS: Serving with vLLM

Last year, I mentioned that I'm interested in learning how to serve LLMs in production. At first it was just curiosity, but over time I wanted to actually try building something—not just reading about it. This post is a small step in that direction: serving an LLM using vLLM, deployed on Amazon EKS, provisioned the infra using AWS CDK, and wrapped into a simple chatbot using Streamlit. TL;DR Exploring LLM serving on a Kubernetes cluster (EKS) Using vLLM as the inference engine Prov...