FP8 Quantization
Compressing model weights to lower-precision numbers (8-bit, 4-bit) so models need less memory and run faster with minimal quality loss.
What is quantization?
Quantization stores model weights in fewer bits — FP16 → INT8 → INT4 — shrinking memory roughly proportionally. A 70B-parameter model needs ~140 GB at FP16 (multiple GPUs) but ~40 GB at 4-bit (one big GPU). Since LLM inference is memory-bandwidth-bound, smaller weights also mean faster token generation.
Why it matters
Quantization is the difference between “runs on a GPU cluster” and “runs on one GPU” — or a laptop. It’s the enabling technology for local/edge LLMs (llama.cpp, Ollama, GGUF files) and a primary cost lever for self-hosted serving. FP8 is now standard on serving hardware; 4-bit (GPTQ, AWQ, GGUF Q4) is the common self-hosting sweet spot.
Quality trade-off, honestly
8-bit is essentially lossless. Good 4-bit methods lose a little — often unnoticeable in chat, but measurable on reasoning, math, and code, and worse for small models (quantizing a 7B hurts more than a 70B). Below 4-bit degrades sharply. The rule: a bigger model quantized to 4-bit usually beats a smaller model at full precision in the same memory footprint.
What people get wrong
- Trusting perplexity alone. Run your own task evals on the quantized model; aggregate metrics hide task-specific damage.
- Confusing weight-only and activation quantization. Weight-only (GGUF, GPTQ) saves memory; FP8/INT8 activation quantization is what unlocks faster compute on modern GPUs.
- Forgetting the KV cache. At long contexts the attention cache, not weights, dominates memory — it can be quantized too.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.