PagedAttention
vLLM's virtual-memory trick for the KV cache: store attention state in fixed-size pages instead of contiguous blocks, eliminating the fragmentation that wasted most GPU memory.
What is PagedAttention?
PagedAttention applies an operating-system idea — paged virtual memory — to the LLM KV cache. Before it, serving frameworks reserved one contiguous memory block per request, sized for the maximum possible output. Requests rarely used their full reservation, so 60–80% of KV memory sat allocated-but-empty. PagedAttention stores the cache in small fixed-size pages allocated on demand, with a block table mapping each sequence to its scattered pages.
Why it mattered enough to launch vLLM
Wasted KV memory is wasted batch size, and batch size is throughput (see decoding phase). By cutting waste to under 4%, PagedAttention let the same GPU hold 2–4× more concurrent sequences — the headline throughput gains that made vLLM the default open serving engine. It also enabled prefix sharing: identical prompt prefixes (a common system prompt across thousands of requests) map to the same physical pages, copy-on-write style, making shared-prefix serving nearly free.
The practical read for builders
If you use APIs, PagedAttention is why provider prompt caching is cheap to offer and why long system prompts don’t linearly destroy provider economics. If you self-host, it’s the reason “just use vLLM” became the standard advice — the naive HuggingFace generate() loop leaves most of your GPU’s serving capacity unrealized.
What people get wrong
- Treating it as a speed trick. It’s a memory trick; the speed comes indirectly, from bigger batches.
- Assuming all engines have it. TensorRT-LLM, SGLang, and others now implement paged KV, but toy servers and older stacks don’t — a 3× throughput gap between “runs the model” and “serves the model” is normal.
- Forgetting page granularity in capacity math: memory-per-request is quantized to page size; thousands of short sequences still pay per-page overhead.
Primary source: Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.