Aliases: RLHF
Reinforcement Learning from Human Feedback
The post-training technique that turns a raw next-token predictor into a helpful assistant by optimizing against human preference judgments.
Last reviewed: July 25, 2026
What is RLHF?
RLHF is why chatbots chat. A pretrained LLM only continues text; RLHF post-training makes it helpful: humans rank pairs of model responses, a reward model learns those preferences, and reinforcement learning optimizes the LLM to score highly. The result answers questions, follows instructions, and declines harmful requests — behaviors nowhere in the pretraining objective.
The family tree
- RLHF (PPO) — the original InstructGPT/ChatGPT recipe: reward model + policy optimization.
- RLAIF / Constitutional AI — AI feedback graded against written principles replaces most human labeling; scales far cheaper.
- DPO (Direct Preference Optimization) — skips the reward model, optimizing directly on preference pairs; simpler and popular for open-weight models.
- RLVR (verifiable rewards) — RL on checkable outcomes (tests pass, math verifies) rather than taste; the engine behind reasoning models.
Why practitioners should care
Post-training explains model personality — why models hedge, over-apologize, or refuse oddly: those are learned preferences, not knowledge limits. It also explains sycophancy (agreeing with users because agreement rated well) and historically contributed to hallucination — confident answers out-rated honest “I don’t know”s. When two models with similar benchmarks feel different to use, you’re feeling their post-training.
What people get wrong
- “RLHF makes models truthful.” It makes them preferred — aligned to what raters rewarded, which correlates imperfectly with truth.
- Confusing it with fine-tuning you can buy. API fine-tuning products are supervised tuning; preference-based post-training at quality requires data and infrastructure few teams have — though DPO on open models has narrowed the gap.
- Fighting post-training with prompts. If a model’s trained-in behavior (refusal style, verbosity) conflicts with your product, switching models usually beats prompt-wrestling.
RLHF’s Legacy Even as Newer Methods Emerge
Even as simpler alignment methods like DPO and KTO have grown popular for their reduced implementation complexity, RLHF’s underlying conceptual framework — collect human preference judgments, train a signal that captures those preferences, optimize a model against that signal — remains the foundational paradigm the newer methods are variations on rather than complete departures from. Understanding classic RLHF, including its reward-model-plus-PPO implementation, remains valuable context for understanding why the newer, simpler alternatives were developed in the first place and what specific pain points (reward model training cost, RL training instability, the number of interacting moving parts) they were each designed to address.
This is why most technical discussions of modern alignment techniques still introduce RLHF first, even when describing a system that ultimately uses DPO or KTO in production, since RLHF supplies the shared vocabulary (reward models, preference data, the KL-divergence penalty keeping a fine-tuned model close to its reference) that the newer methods are explained relative to.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.