Key facts.
- The incentive is the root cause: OpenAI's 2025 analysis argues models hallucinate because accuracy-only evaluations reward a confident guess over an honest "I don't know", so models learn to bluff (Kalai et al., Why Language Models Hallucinate, arXiv:2509.04664, 2025).
- Errors should cost more than abstentions, but most scoreboards treat them the same, a guessing model outranks an abstaining one even when it's confidently wrong far more often (OpenAI, 2025).
- It's not inevitable: models can abstain when uncertain. Hallucination comes from misaligned evaluation, not a fixed property of the model (OpenAI, 2025; Nature, 2026).
- Calibration is cheap: having confidence match accuracy takes far less compute than being accurate (OpenAI, 2025).
- Scoring rewards a lucky guess over abstention; a more capable agent guesses and acts wrong. (arXiv:2509.04664)
Why does a confident wrong answer beat an honest "I don't know"?
Because of how we grade. Most benchmarks count right answers. A wrong answer and an abstention both score zero. That means guessing beats abstaining every time. A guess occasionally scores. An abstention never does. OpenAI's 2025 analysis (arXiv:2509.04664) identifies this as the root cause: confident guessing gets rewarded, so models learn to bluff. Not a mysterious flaw. The rational response to the test you set up.
What is calibration, and why do agents lack it?
Calibration means stated confidence tracks actual accuracy. Say 90% sure, land right about 90% of the time. That's what lets an agent stop and escalate instead of guessing through the fog. Most deployed agents aren't there, training rewards sounding certain. OpenAI's work adds one useful point: calibration needs far less compute than accuracy. Your agent doesn't need to know everything. It needs to know where its knowledge ends. That boundary is what abstention runs on.
What goes wrong when an agent can't abstain?
It acts on a guess. The refund issues. The ticket files. The email sends. Support bots quote policies that don't exist. Legal briefs cite fabricated cases. Tool calls get logged as done that never ran. All share the same root: no mechanism to say "I'm not sure, stop." The harm scales with autonomy. The more the agent can do without a human, the more an unhedged guess costs.
Is hallucination fixable, or inherent?
Fixable. OpenAI's 2025 analysis rejects the common defense outright: models can abstain when uncertain, so "I don't know" isn't impossible, we just haven't rewarded it. A 2026 Nature paper agrees: grading for accuracy actively incentivizes hallucination. Cut the rate by changing the incentive and adding abstention. Not by waiting for a model that never errs.
Six ways to build an agent that can stop
| Approach | Why it helps |
|---|---|
| Reward "I don't know" | Score abstention above confident error in your evals, you start selecting for honesty |
| Calibrate confidence | Make confidence track real accuracy, then trust it as a signal |
| Set escalation thresholds | Below a confidence bar, the agent defers instead of acts |
| Defer high-stakes actions | Route irreversible or sensitive actions to a human regardless of confidence |
| Verify before acting | Confirm the supporting fact exists before the agent commits |
| Make "stop and ask" first-class | Treat asking a human as a successful outcome, not a failure |
Score abstention above confident error. Calibrate confidence so it tracks real accuracy. Set a threshold, below it, the agent defers instead of acts. Route irreversible actions to a human regardless of confidence. Verify before committing. Treat asking a human as a win. Find where your agent breaks, from the goal, before you build it.
Frequently asked questions
Isn't hallucination just inherent to LLMs?
No. OpenAI's 2025 analysis shows models can abstain when uncertain. The problem is accuracy-only scoring that rewards guessing, not that "I don't know" is impossible. Fix the incentive, add abstention, and hallucination drops.
Why would a model bluff instead of admitting uncertainty?
Because that's what got rewarded. On a scoreboard that only counts right answers, a guess sometimes scores and an abstention never does, so the model learns to produce a confident answer even when it's guessing.
When should an agent stop and ask a human?
Two triggers: confidence below a calibrated threshold, or stakes above a line. Low confidence on anything, or any irreversible action, should defer to a human even when the agent thinks it knows.
What's the single most useful change?
Score abstention above confident error in your evals. Once "I don't know" beats a wrong guess on your scoreboard, you start selecting for an agent that hands off instead of bluffing.