Key facts.
- MIT NANDA found about 5% of GenAI initiatives achieved real operational or financial impact, with most showing no measurable business return. source
- The successful projects shared deep integration into specific processes, continuous learning capability and evaluation on business outcomes rather than benchmarks. source
- The core barrier was learning, not infrastructure or talent: most systems did not retain feedback or improve over time. source
Why do most support pilots stall?
Because they are built to demo, not to operate and the gap between those is the three things MIT found the scaled projects had. A pilot that answers questions in a sandbox is not integrated into the real support workflow, the CRM, the ticketing, the escalation paths, so it cannot actually resolve anything end to end. A pilot with no learning loop repeats its mistakes, which MIT identifies as the core barrier, because the system never improves from the failures the team keeps catching. And a pilot evaluated on a benchmark score rather than on resolved tickets and customer outcomes optimizes the wrong thing, looking good while delivering no measurable return. Stack those three gaps and you get the common outcome: a promising demo that never becomes a dependable operation and quietly gets shelved.
The reason capability does not save these pilots is that none of the three gaps is a capability problem. A more powerful model still is not integrated, still does not learn and still is graded on the wrong metric. The pilots that scaled closed the gaps; the ones that died waited for a model that would make the gaps not matter and it never came.
What should you build for scale?
Integrate deeply into the actual support workflow so the agent can resolve issues end to end, not just answer in a sandbox. Build the learning loop so failures feed back and the agent improves over time, which MIT flags as the decisive missing piece. And evaluate on business outcomes, resolved tickets, customer satisfaction, real cost per resolution, not on benchmark accuracy that does not predict value. These are the three the scaled deployments shared and they are system choices, available to you now regardless of which model you run.
| Pilot foundation | Outcome |
|---|---|
| Sandbox demo, no learning, benchmark-graded | Stalls with no measurable return |
| Integrated, learning loop, outcome-evaluated | Scales to real impact |
MIT found ~5% of GenAI pilots reached impact via workflow depth and outcome evals; the dead ones lacked the system, not a bigger model. (source)
Building those three into the system is the core of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of a support agent that integrates, learns and is judged on outcomes, so your pilot is built like the ones that scaled rather than the ones that quietly died.
Frequently asked questions
Is the 5% about weak models?
No. MIT points to the system around the model, especially the missing learning loop, not raw capability. The three features are system choices.
Which of the three matters most?
MIT identifies learning as the core barrier, but integration and outcome evaluation are what let the learning translate into value.
Why not wait for a better model?
Because none of the three gaps is a capability gap. A stronger model still is not integrated, still does not learn and still is graded wrong.