KV Cache
The attention key/value tensors an LLM stores per token during generation — the memory bottleneck of inference and the mechanism behind prompt caching discounts.
What is the KV cache?
During generation, a transformer computes attention keys and values for every token in context. Rather than recomputing them for each new token, it caches them — the KV cache. This makes generation fast, but the cache grows linearly with context length and batch size, and at long contexts it rivals or exceeds the model weights in GPU memory.
Why developers should care (even API users)
The KV cache is why prompt caching exists: providers keep the cache for a repeated prompt prefix (your system prompt, few-shot examples, a large document) and skip recomputing it — billed at up to 90% discount and with much faster time-to-first-token. Structuring prompts as stable prefix + variable suffix is the single easiest large cost win in production LLM apps. Cache breaks on any prefix change, so put timestamps and user data after the static content.
For self-hosters
KV cache management is what serving engines actually compete on: vLLM’s PagedAttention allocates cache in pages (like OS virtual memory) to eliminate fragmentation and share common prefixes across requests, multiplying throughput. Long-context serving is usually KV-memory-bound, not weight-bound — KV cache quantization (FP8/INT8) and grouped-query attention are the standard mitigations.
Cost reality
Rule-of-thumb: KV cache per token = 2 × layers × kv-heads × head-dim × bytes. For a 70B-class model that’s roughly 100k tokens ≈ tens of GB at FP16 per sequence — why long-context requests cost more and batch fewer.
What people get wrong
- Putting dynamic content first in the prompt, silently breaking the provider’s cache on every call.
- Assuming caching is automatic everywhere. Some APIs require explicit cache markers; check your provider.
- Benchmarking self-hosted throughput at short contexts then hitting OOM in production at real conversation lengths.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.