Fine-Tuning

What is fine-tuning?

Fine-tuning continues training a pretrained LLM on hundreds-to-thousands of your own input/output examples. The dominant technique is LoRA (low-rank adaptation), which trains small adapter matrices instead of all weights — cheap enough to run on a single GPU for small models.

What it’s actually for

Fine-tuning teaches behavior, not facts:

• Consistent output format (strict JSON schemas, house style, tone)
• Domain-specific classification/extraction at high volume
• Distilling a large model’s skill into a small, cheap one
• Teaching a niche language, DSL, or codebase conventions

For knowledge — things that change or must be cited — use RAG. A fine-tuned model still hallucinates facts it saw once in training; retrieval grounds answers in live data.

When it’s overkill

Try in order: prompting → few-shot examples → RAG → fine-tuning. Frontier models follow instructions so well that fine-tuning is a last resort except for one economically compelling case: replacing an expensive frontier model with a fine-tuned small model on a narrow, high-volume task — 10–100× cheaper inference at comparable quality on that task.

Cost reality

Training is cheap (a LoRA run costs dollars to low hundreds). The real costs are dataset construction — curating and cleaning thousands of quality examples is weeks of work, and quality beats quantity — and maintenance: every base-model upgrade means re-running and re-evaluating, so you’re now shipping a model artifact, not just a prompt.

What people get wrong

• Fine-tuning to add knowledge. It doesn’t reliably work; that’s RAG’s job.
• Skipping evals. Without a held-out test set you can’t tell whether tuning helped or just overfit your examples.
• Catastrophic forgetting. Narrow tuning degrades general ability; keep some diverse data in the mix.

When Fine-Tuning Is (and Isn’t) the Right Tool

Fine-tuning is most valuable when a task requires consistent behavior that’s difficult to reliably achieve through prompting alone — a specific output format that needs to be followed with very high consistency, a specialized domain vocabulary or writing style, or a narrow task where the efficiency of a smaller fine-tuned model matters more than the flexibility of a large general-purpose one. It’s generally the wrong tool for injecting new factual knowledge into a model, since fine-tuning is much better at teaching behavior and style than at reliably teaching facts — retrieval-augmented generation is typically a better fit for that goal, since it grounds responses in retrievable documents rather than hoping specific facts were successfully absorbed into the model’s weights during training.