Key facts.
- Top agents resolve roughly 76 to 79% of SWE-bench Verified issues, but the benchmark provides isolated, well-specified tasks not live repo state.source
- Differential patch testing found reported SWE-bench Verified resolve rates can be inflated by up to about 6 percentage points, so the headline overstates real capability.source
- Real repos add dependency conflicts, stale state, test flakiness, and CI rules the benchmark never models.source
- Agents hit ~79% on curated SWE-bench, a number shown inflated; your repo is not, so the upgrade stumbles on CI, growing rework. (arXiv:2310.06770)
Why does the benchmark overstate real performance?
Because it removes everything that makes your repo hard. A SWE-bench task is a well-specified issue in a known state with a clear test of success. Is the easy version of software engineering. Your repository is the hard version: branches that have drifted, dependencies that conflict, tests that flake. Build steps that fail for reasons unrelated to the change and a CI pipeline that enforces rules the agent does not know. The agent that resolves four in five curated issues meets this and the success rate drops. That happens because the skill the benchmark measured, fixing a clean issue, is only part of the job. The other part is operating inside a messy, stateful system and the benchmark deliberately excluded it. The inflation finding sharpens the point: even the curated number is a few points generous. The gap between the score and your reality is wider than it looks.
This is why "the agent scores 79%" does not translate to "the agent will resolve 79% of my tickets." The benchmark and your repo are different difficulties and the agent was measured on the easier one.
What closes the gap?
Ground the agent in the real context and gate it on the real CI. Give it accurate repo state, the actual dependency graph and the project's conventions. It is not reasoning about an idealized version of your code. Run its changes through your real test suite and CI pipeline as a hard gate. A change that breaks the build or fails a check never merges, regardless of how confident the agent is. The benchmark measured the agent solving a clean problem. Production requires the agent to solve your problem and pass your pipeline and only the second one ships working software.
| What you rely on | Result |
|---|---|
| The benchmark score | Overstated, ignores repo state and CI |
| Real context plus CI gate | Changes that actually work in your repo |
Grounding the agent in real repo context is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of your repo's actual state and pipeline. A coding agent is judged by whether it works in your codebase, not by a score it earned on someone's clean benchmark.
Frequently asked questions
Is SWE-bench useless then?
No, it is a useful capability signal. It just measures isolated issue-solving, not operating in your live repo, so do not read it as a production success rate.
What is the inflation about?
Differential testing found some reported resolve rates overstated by a few points, so even the curated number is slightly generous.
How do I ground the agent?
Give it real repo state and conventions and gate every change on your actual tests and CI before merge.