Hallucination

What is hallucination?

Hallucination is an LLM producing fluent, confident, wrong output: invented citations, nonexistent API parameters, fabricated case law, plausible-but-false statistics. It happens because LLMs are trained to predict likely text, not to verify truth — when the model lacks knowledge, the most probable-sounding answer is still generated with the same confident tone.

Why it won’t fully go away

Training objectives historically rewarded guessing over admitting uncertainty (a test-taker who never leaves blanks scores better). Newer models hallucinate less and abstain more, but the failure mode is inherent to next-token prediction. Engineering conclusion: design systems assuming some rate of confident falsehood, the way distributed systems assume some rate of network failure.

The mitigation stack (in order of impact)

1. Grounding — RAG or tool calls put verifiable facts in context; instruct the model to answer only from them and say “not found” otherwise.
2. Verification — check claims against sources (guardrails), validate generated code by running it, validate citations by resolving them.
3. Abstention prompting — explicitly permitting “I don’t know” measurably reduces fabrication.
4. Measurement — track hallucination rate on your own domain with evals; rates vary wildly by topic and model.

What people get wrong

• Treating low hallucination benchmarks as safety. A 2% rate at a million queries/day is 20,000 wrong answers daily; what matters is consequence per error in your domain.
• Thinking fine-tuning fixes it. Tuning shapes style; ungrounded factual recall stays unreliable.
• Trusting confidence. Fluency and certainty of tone carry no signal about correctness — that’s precisely what makes hallucination dangerous.