Streaming (LLM Responses)

What is streaming?

LLMs generate tokens one at a time, and streaming forwards each one to the client as it’s produced — usually over server-sent events (SSE). A 30-second generation feels instant because the first words appear in under a second. Every serious chat UX streams; the “wait for the spinner” pattern died with GPT-3.5.

Why it’s a UX decision and an architecture decision

The metric streaming optimizes is time-to-first-token (TTFT) — perceived responsiveness — decoupling it from total generation time. But it ripples through your stack: every hop (gateway, backend, CDN) must pass chunks through without buffering (proxy buffering silently un-streams you), timeouts must accommodate minutes-long connections, and billing/logging happens at stream end, not request start.

The hard parts nobody mentions

• Structured output — you can’t JSON.parse half a JSON object; use incremental parsers or stream freeform text and emit structure at the end.
• Tool calls mid-stream — the stream pauses while your code executes tools, then resumes; UIs need states for “thinking”, “acting”, and “writing”.
• Guardrails on partial output — content you’ve already streamed can’t be unshown; moderation must run on the stream (with small buffers) or accept post-hoc retraction UX.
• Error mid-stream — a failure at token 500 needs retry-with-resume or a graceful “regenerate”, not a blank bubble.

What people get wrong

• Streaming everything. Machine-to-machine calls (pipelines, batch jobs) gain nothing; non-streaming is simpler and easier to retry.
• Judging speed by tokens/sec alone. Users perceive TTFT far more than throughput; optimizing the wrong one wastes money.
• Forgetting mobile networks. Long-lived SSE connections through flaky networks need reconnect logic; test on a train, not localhost.

Try it in code

Consuming a server-sent-events stream and measuring what matters (TTFT):

import time

def stream_chat(prompt):
    start = time.monotonic()
    first_token_at = None
    full = []
    for event in llm_stream(prompt):          # SSE iterator from any provider SDK
        if event.type == "token":
            if first_token_at is None:
                first_token_at = time.monotonic() - start   # <-- TTFT
            full.append(event.text)
            print(event.text, end="", flush=True)
    total = time.monotonic() - start
    print(f"\nTTFT: {first_token_at:.2f}s | total: {total:.2f}s "
          f"| {len(full)/max(total,0.001):.0f} tok/s")
    return "".join(full)

Log TTFT and total separately — they are optimized by entirely different levers.