Streaming an LLM response, in 4 GIFs

We have watched tokens stream in from an LLM before where they appeared one at a time, like the model was typing. If you used the Anthropic SDK's .stream() method, it just worked and you probably never saw what was on the wire. This post will majorly focus on how a stream response works and how bugs are handled by SDK behind the hood. 1. Why Streaming exists To enable the streaming option we would need to make one change in the post request that is a single field "stream": tru...