Key facts.
- SWE-Bench Pro: frontier models score under 25% on enterprise long-horizon tasks (GPT-5 ~23%) and below 20% on the commercial set. source
- Some repositories show resolve rates under 10% across all models, indicating codebase-specific complexity dominates. source
- Documentation quality, problem type and codebase complexity strongly affect whether any model can succeed in a given repo. source
Why does enterprise complexity break the agent?
Because a large, legacy or domain-specific codebase encodes context the agent does not have and cannot infer. The benchmark issue is self-contained in a clean, modern, well-documented repo, which is the friendly case. Your codebase is the opposite: a decade of accumulated decisions, undocumented conventions, a legacy framework with its own quirks and a domain-specific language or pattern the model rarely saw in training. SWE-Bench Pro deliberately captures this harder distribution and the scores fall off a cliff, under 25% for frontier models and into single digits on the hardest repos. The agent is not failing because it got dumber. It is failing because solving a real change in a complex system requires understanding the system and that understanding is exactly what a large unfamiliar codebase withholds.
The repo-to-repo variance is the tell. Some codebases let models reach decent rates while others stay under 10% across every model, which means the bottleneck is the specific complexity of the codebase, not a fixed model limitation. A more capable model helps a little; the codebase's documentation, structure and conventions decide the rest.
What makes agents useful on real codebases?
Context and scope. Give the agent deep, accurate context about your system, the conventions, the architecture, the domain patterns, so it is not inferring a complex codebase from fragments. Scope it to the kinds of changes it can make safely in your repo, the well-bounded, well-understood ones and route the deep, cross-cutting, legacy-tangled work to humans. The SWE-Bench Pro numbers say an agent let loose on enterprise complexity will fail most of the time, so the win is matching the agent to the changes where your context makes it reliable, not expecting it to absorb a decade of system knowledge on its own.
| Deployment | Outcome on a complex codebase |
|---|---|
| Agent let loose, benchmark expectations | Fails most changes, low resolve rate |
| Deep context plus scoped changes | Reliable on the changes it can own |
Providing that system context is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of your specific codebase and the changes an agent can make safely in it, so a coding agent contributes where your context makes it reliable instead of stalling on complexity it cannot infer.
Frequently asked questions
If the model gets stronger, will it handle our legacy code?
On SWE-Bench Pro frontier models score under 25%; an agent clearing a toy issue meets your legacy repo and cannot, costing rework. (arXiv:2509.16941)
Why is SWE-Bench Pro so much harder than SWE-bench Verified?
It uses enterprise-grade, long-horizon tasks closer to real codebases, where context and complexity matter, so frontier models drop under 25%.
Will a better model fix legacy-codebase performance?
Only partly. Repo-specific factors like documentation and structure dominate, with some repos under 10% across all models.
How do I make the agent useful here?
Give it deep context about your system and scope it to safe, well-understood changes, routing the legacy-tangled work to humans.