Streaming

What is Streaming

Streaming is sending a model's output token by token as it's produced, rather than waiting for the entire response to finish generating before showing any of it. In a chat interface, this is the difference between text appearing word by word as the model writes it, versus a blank screen followed by the full answer appearing all at once.

Streaming changes perceived latency, not actual total generation time. The full response still takes exactly as long to finish either way, but a user watching text appear immediately feels like the system is working, while a user staring at nothing until the whole thing is done feels like it's stalled, even if the total time is identical.

Streaming matters most for the human-facing side of an interaction, a long chat reply benefits visibly from it. It matters much less for a turn that's mostly tool calls happening behind the scenes, where there's no continuous text being generated for a human to watch arrive, and the useful signal is the final result, not the incremental drafting of it.