Understanding RLHF Alignment Pipelines

Introduction

Pre-trained language models are trained to predict the next token — they’re skilled at producing plausible text, but this objective doesn’t inherently produce helpful, honest, or harmless outputs. A model trained purely on internet text learns to reproduce all of the internet’s biases, toxicity, and misinformation. Reinforcement Learning from Human Feedback (RLHF) is the technique used by OpenAI (InstructGPT, GPT-4), Anthropic (Claude), and Meta (Llama 2/3) to align pre-trained models with human preferences. It involves three sequential phases: supervised fine-tuning, reward model training, and PPO optimization.

flowchart TD
    P1["Phase 1: Supervised fine-tuning (SFT)"] --> P2["Phase 2: Reward model training"]
    P2 --> P3["Phase 3: PPO optimization (RL)"]
    P3 -.->|"alternative"| CAI["Constitutional AI (Anthropic)"]

Phase 1: Supervised Fine-Tuning (SFT)

A dataset of high-quality prompt-response pairs, written by expert annotators, is used to fine-tune the base model:

Dataset format:
  Prompt: "Explain quantum entanglement to a 10-year-old"
  Response: [Expert-written clear, age-appropriate explanation]

Prompt: "Write a Python function to reverse a string"  
  Response: [Well-commented, idiomatic Python code]

# SFT training loop (simplified)
from transformers import AutoModelForCausalLM, Trainer, TrainingArguments

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

training_args = TrainingArguments(
    output_dir="./sft-checkpoint",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-5,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=sft_dataset,  # Curated prompt-response pairs
)
trainer.train()

Result: A model that follows instructions — but may still produce harmful or unhelpful outputs. SFT alone is insufficient.

Phase 2: Reward Model Training

Human annotators compare multiple model responses to the same prompt and rank them by preference. A separate reward model (RM) is trained to predict which response humans prefer:

For prompt "Summarize this article":
  Response A: [Accurate 3-sentence summary]
  Response B: [Verbose, includes irrelevant details]
  Response C: [Contains factual errors]
  
Annotator ranking: A > B > C

Reward model learns: R(prompt, response_A) > R(prompt, response_B) > R(prompt, response_C)

The reward model uses the Bradley-Terry preference model:

P(A preferred over B) = σ(R_θ(prompt, A) - R_θ(prompt, B))

Loss: -E[log P(preferred response)] = -E[log σ(R_θ(preferred) - R_θ(rejected))]

Phase 3: PPO Optimization (Reinforcement Learning)

The SFT model is further fine-tuned using Proximal Policy Optimization (PPO) to maximize the reward model score while staying close to the SFT model (preventing mode collapse):

Objective = E[R_reward_model(response)] - β × KL[π_PPO || π_SFT]

Where:
  R_reward_model: Score from the reward model (what humans prefer)
  β: KL divergence coefficient (prevents the model drifting too far from SFT)
  π_PPO: Current policy being optimized
  π_SFT: Reference SFT policy

# Conceptual PPO training loop
from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
    learning_rate=1.41e-5,
    batch_size=32,
    kl_penalty="kl",
    init_kl_coef=0.2,    # β: higher = stay closer to SFT model
    adap_kl_ctrl=True,   # Adaptive KL coefficient
)

ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=sft_model,          # Policy to optimize
    ref_model=sft_model_ref,  # Reference (frozen SFT model for KL)
    reward_model=reward_model,
    tokenizer=tokenizer,
)

for batch in training_data:
    queries = batch["prompt_ids"]
    responses = ppo_trainer.generate(queries)
    rewards = reward_model.score(queries, responses)
    
    # PPO update: maximize rewards while minimizing KL divergence from reference
    stats = ppo_trainer.step(queries, responses, rewards)

Constitutional AI (Anthropic’s Alternative)

Anthropic’s Claude uses Constitutional AI (CAI) as an alternative/complement to RLHF:

1. Generate initial response to harmful prompt
2. Critique the response against a constitution (principles like "don't assist with violence")
3. Revise the response based on critique
4. Use revised response as training data (RLAIF — RL from AI Feedback)

Advantage: Reduces reliance on human annotators, scales better, more consistent principles

Key Takeaways

• Phase 1 (SFT) fine-tunes on curated expert demonstrations — produces an instruction-following model but not necessarily a safe or helpful one.
• Phase 2 (Reward Model) learns human preferences from comparative rankings — the RM is the proxy for “what humans find helpful/harmless.”
• Phase 3 (PPO) optimizes the policy to maximize RM scores, constrained by a KL divergence penalty that prevents the model from drifting too far from the SFT reference.
• The KL coefficient β is critical: too low → reward hacking (model finds adversarial inputs that fool the RM); too high → insufficient alignment improvement.
• Constitutional AI replaces human annotators with AI-generated critiques and revisions — Anthropic uses it alongside RLHF for scalable alignment.