LLM Gateways: Routing, Fallbacks, And Semantic Caching

Here's a line of code that's quietly running in production at a surprising number of companies: const response = await openai.chat.completions.create({ model: "gpt-4o", messages }); It looks harmless. It's also why your AI bill is whatever it is this month, why your app goes down the moment OpenAI has a bad afternoon, and why the same question typed by ten thousand users costs you ten thousand inference calls. That one line hardcodes a vendor, a model, a pricing tier, and a single point...