Prefill Phase

What is the prefill phase?

When an LLM receives a request, it doesn’t start generating immediately. First it runs the prefill phase: every prompt token is pushed through the model in one large parallel pass, computing attention keys and values for the whole input and storing them in the KV cache. Only when prefill completes does the first output token appear — which is why prefill time largely is your TTFT.

Why prefill behaves differently from generation

Prefill is compute-bound: thousands of tokens processed simultaneously saturate the GPU’s matrix units. The decoding phase that follows is the opposite — memory-bandwidth-bound, one token at a time. This asymmetry drives real engineering decisions: a GPU that’s fast at prefill isn’t automatically fast at decode, and serving frameworks schedule the two differently (vLLM’s chunked prefill splits long prompts into pieces so ongoing decodes aren’t stalled behind a 100k-token newcomer).

The numbers that matter

Prefill cost scales roughly linearly with prompt length for compute, and attention scales quadratically in theory — in practice, a 100k-token prompt on a modern GPU takes seconds of pure prefill. That is why prompt caching exists: a cached prefix skips its share of prefill entirely, which is where the “up to 90% off and much faster TTFT” numbers come from.

What people get wrong

• Blaming the model for slow first tokens when the real cause is a bloated prompt: halving input length roughly halves prefill time — the cheapest TTFT optimization there is.
• Ignoring queueing: on busy servers, requests wait before prefill even starts; measured TTFT includes that wait, so p99 TTFT is often a capacity problem, not a model problem.
• Assuming prefill parallelism is free: one giant prefill can monopolize the GPU and freeze every in-flight generation — the exact problem chunked prefill was invented to fix.

Primary source: vLLM documentation — chunked prefill

Why Prefill Latency Matters for User Experience

The prefill phase’s duration is the primary driver of time-to-first-token (TTFT), the metric that determines how long a user waits before seeing any response at all after submitting a request. Because prefill scales with prompt length and is compute-bound, applications with very long prompts (a large system prompt, a long conversation history, or a substantial retrieved document set in a RAG pipeline) will generally have noticeably longer TTFT than the same request with a shorter prompt, independent of how fast the subsequent token-by-token decoding phase runs. This is why prompt caching — reusing the already-computed KV cache for a repeated prompt prefix rather than reprocessing it from scratch — provides such a meaningful latency improvement specifically for applications with long, frequently repeated prompt prefixes like a stable system prompt.