Parameter-Efficient Fine-Tuning (PEFT)
A collection of fine-tuning techniques that adapt pre-trained models by modifying only a tiny fraction of parameters.
Last reviewed: July 25, 2026
Parameter-Efficient Fine-Tuning (PEFT) is an umbrella term for a family of techniques that adapt a pretrained model to a new task or dataset by training only a small subset of its parameters — or a small number of newly added parameters — rather than updating every weight in the model, as full fine-tuning does.
Why It Exists
Full fine-tuning of a large language model requires enough GPU memory to hold gradients and optimizer states for every one of the model’s parameters, which for a modern LLM with tens or hundreds of billions of parameters can mean needing memory many times larger than the model’s own weights just to train it. This puts full fine-tuning out of reach for most individuals and smaller teams, and even for well-resourced organizations, it’s often unnecessarily expensive when adapting a model to a narrower task doesn’t require changing most of its learned knowledge.
Common PEFT Techniques
LoRA (Low-Rank Adaptation) freezes the original weights and trains small low-rank matrices injected alongside specific layers, a technique that has become the dominant PEFT approach due to its strong results and simplicity. QLoRA extends LoRA by additionally quantizing the frozen base weights to 4-bit precision, further reducing memory needs. Other, less commonly used PEFT methods include prompt tuning (learning a small set of continuous “soft prompt” embeddings prepended to the input) and adapter layers (small trainable modules inserted between a model’s existing frozen layers).
Practical Impact
PEFT techniques, and LoRA-family methods in particular, have made fine-tuning large models routine for individual developers and small teams working on consumer GPUs, rather than a capability limited to organizations with large GPU clusters — a shift widely credited with much of the explosion in specialized, fine-tuned open-weight models available today.
PEFT and Catastrophic Forgetting
An additional benefit of PEFT techniques, beyond compute and memory efficiency, is that freezing the base model’s original weights naturally mitigates catastrophic forgetting — the tendency of full fine-tuning to degrade a model’s general capabilities while it learns a new, narrower task, since every weight is subject to modification and can drift away from behaviors useful for other, untrained-for tasks. Because LoRA and similar PEFT methods leave the original weights untouched and only add a small trainable delta on top, the model’s original broad capabilities remain largely intact even after fine-tuning for a specific narrow purpose, which is part of why PEFT-fine-tuned models tend to retain more general usefulness than fully fine-tuned counterparts trained on the same narrow dataset.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.