We Cut Our LLM API Bill 30% With Four Lines of YAML

Our gateway handles a few thousand LLM calls per hour. Mostly internal tools, some customer-facing agents. We noticed something in the logs: a lot of prompts were basically the same question worded differently. "Summarize this quarterly report" and "give me a summary of the Q2 report" hitting the same model, getting nearly identical responses, costing us twice. Multiply that across a few hundred users and it adds up fast. The math on duplicate calls Quick back-of-envelope. GPT-4o ru...