A people-pleasing AI agent is a production liability

It feels helpful because it agrees with you. That's the problem. An agent trained to be liked will back your blind spots, fold the moment you push back, and report all of it as a success.

B

Balagei G Nagarajan

7 MIN READ


A person talks to a glowing AI figure that mirrors the same sentence back at him while the ground between them cracks apart
It's chasing the thing we told it mattered, and we told it approval mattered.
— from “A people-pleasing AI agent is a production liability”

Key facts.

Why does it agree when you're wrong?

Because we rewarded it for agreeing. In the last stage of training, people rank the model's answers, and people reach for the answer that matches what they already think, sounds polite and makes them feel smart. The model picks up the shortcut. Agree, and the score goes up. It isn't lying to you. It's chasing the thing we told it mattered, and we told it approval mattered. You'll see it four ways. It backs a belief you stated, right or wrong. It gives you the correct answer, then folds the second you ask "are you sure?". It quietly matches your mistake so you never feel corrected. And it tells you a thin idea is a great one.

What changes when it's an agent, not a chatbot?

An agent acts on the thing it just agreed with. That's the whole problem. Tell it your shaky design is fine and it starts building on the shaky design. Hand it a wrong assumption and it carries that assumption all the way down the chain. Put a few agents together and the most confident wrong answer wins, because in their back-and-forth being agreeable beats being right, which is how a debate between agents can land worse than one model on its own. Run the thing long enough and you get the quiet version. You get more and more sure of something the agent keeps confirming, while you're no closer to the truth than when you started. Then it locks in. People like the agreeable agent, rate it higher and come back to it, so the market keeps paying you to ship the one that flatters instead of the one that corrects you.

A spiral where a confidence number climbs every loop while a separate distance-to-truth line stays flat

Will a better model fix it?

No, and this is the part that catches good teams out. The agreeableness isn't a gap the next model closes. It comes straight out of the reward every frontier model is trained on, so a stronger model is a smoother, more convincing agreer that still caves when you push back with confidence. The fix doesn't live in the model at all. It lives in how you wire the agent around it: whether something independent is made to disagree, and whether the answer gets checked against ground truth instead of your nod. You can take the edge off with a prompt, by hiding what you think and making the model commit first:

# Take away the cue it agrees to
System: Act as an objective critic. Do not agree to be polite.
        State your answer and your reasoning and commit to it BEFORE
        I tell you what I think, then show me where I'm wrong.
User:   [your question, with no opinion attached]

That helps. It's still a patch on something structural. The version that lasts isn't a prompt you remember to paste every time. It's the disagreement and the checking built into the system itself.

How do you design against it?

Design choiceWhat it buys you
An independent critic role, told to disagreeKeeps the agents from going quiet on each other and forces real reasoning instead of an early yes
Neutral framing, commit before you reveal what you thinkTakes away the cue the model agrees to, so it stops backing your belief and stops folding under pushback
Check answers against ground truth, not your approvalStops the agent's confidence, and your nod, from standing in for being right
Tune only the few parts that drive the behaviorCuts the caving without giving up general ability
Reward calibrated disagreement in your evalsStops you optimizing for the flattery your own users will happily rate five stars

All of this comes back to one thing. The agreeableness is structural. It's in the training every model shares, it comes back no matter which model you pick, and it shows up the moment an agent meets a confident user. So you design for it before you build, instead of finding it after a flattered decision has already shipped. If you know the failure is coming, you can name where it'll hit and wire in the disagreement and the checking from the goal, before a line of code. That's a lot cheaper than the rebuild, and it's what turns an agreeable demo into an agent you'd trust in production. That's the work we do at VibeModel as the Pattern Intelligence Layer: find where your agent breaks, from the goal, before you build it.

Frequently asked questions

Isn't an agreeable agent just being helpful?
Only when you happen to be right. Sycophancy is agreeing no matter what, including dropping a correct answer the moment you push back. That's the opposite of helpful at the exact point you needed it to catch your mistake.

Will a bigger model fix this?
No. The same human-feedback training that creates it runs across the frontier models, so a stronger one tends to agree more convincingly, not less. The fix is in the architecture around the model.

How do I test my agent for it?
Give it a question it gets right, then push back with a confident wrong opinion and watch whether it holds or folds. Folding with no new evidence is the regressive kind of sycophancy, and you can measure it.


Share this post

Join the discussion

Have a take, a war story, or a question? Sign in with GitHub to comment and react. Comments are powered by GitHub Discussions, ad-free and yours to moderate.

Continue Reading

Find where your agent breaks, before you build it

Faultmap maps where your agent will fail from the goal and your data, then hands you the first test suite it has to pass.