Key facts.
- Among deployed agents, 70% prompt off-the-shelf models instead of tuning, so the agent is not learning from its own production outcomes.source
- 74% depend primarily on human evaluation, which catches failures but rarely feeds them back into the planning process automatically.source
- Without an outcome-to-planning loop, a plan that failed in a recognizable situation will fail the same way the next time that situation appears.source
- In production 70% prompt off-the-shelf and few learn from failed plans; a frontier model adds no feedback loop, rework repeats. (arXiv:2512.04123)
Why is the loop so rarely built?
Because building it requires capturing outcomes in a structured way, connecting them back to the plans that produced them and changing future planning based on what you learn and most deployments stop at the first step or skip it entirely. The production data is blunt about this: the dominant pattern is an off-the-shelf model prompted to plan, with a human checking the result. That setup ships fast, which is why it dominates, but it has no mechanism to improve. The human catches a bad plan today, the same prompt produces the same bad plan tomorrow and the lesson lives in the reviewer's head instead of in the system. The agent is not getting better at planning. It is repeating a fixed behavior, with a human absorbing the failures one at a time.
This is a missed compounding opportunity more than an acute failure. Each failed plan is information about where the planning is weak and discarding it means paying for the same mistake repeatedly. The teams that pull ahead are the ones that turn outcomes into a signal the planning actually consumes.
What does closing the loop look like?
Capture outcomes against the plans that caused them, in a form you can analyze rather than a pile of logs. Identify the situations where plans fail and feed that back, through better prompts, retrieved examples of past failures or targeted tuning, so the next plan in a similar situation starts from the lesson instead of the blank slate. You do not need to retrain the model. You need a memory of what failed and a mechanism that lets it shape the next plan. That is the difference between an agent that holds steady and one that gets reliably better at the job it is doing.
| Loop state | What happens to a failed plan's lesson |
|---|---|
| Open loop | Discarded, repeated next time |
| Closed loop | Captured and fed into future planning |
Building that memory is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of plans that failed and feed them forward, so the agent's planning improves from its own outcomes instead of repeating them.
Frequently asked questions
Do I have to fine-tune to close the loop?
No. Retrieved past-failure examples and improved prompts close much of it without tuning. The point is that outcomes inform future plans.
Why isn't human review enough?
It catches failures but stores the lesson in a person, not the system, so the same plan recurs. The loop has to feed the system.
What do I capture?
The plan, the situation and the outcome, structured enough to find patterns in what fails and feed them back.