When Prompt Batching Made My LLM App More Expensive

I was working on cost optimization for an LLM-based document translation pipeline. At that point, the LLM translation flow was still very direct: one extracted text segment became one API call. It worked, but it was not ideal for cost. For a document with many text segments, the number of API calls grew linearly. So the optimization idea was straightforward: batch multiple text segments into one prompt. In simpler terms: Instead of sending one API call for every text segment, we group multi...