Key facts.
- On CUAD, strong models achieve high balanced accuracy on contract clause identification overall, but performance degrades on longer text sequences and multi-class classification. source
- ContractEval found open models often return "no related clause" even when a relevant clause is present, a laziness or low-confidence failure on harder extraction. source
- Real legal documents are long, dense and multi-issue, which is the regime where the benchmarks show degradation. source
Why does complexity degrade the agent?
Because the benchmark's friendly cases and the real documents are different difficulties and legal work lives in the hard one. CUAD shows that on clean, bounded clause-identification tasks, strong models do well, which is the easy regime, but performance falls on longer text and multi-class settings, which is the regime real contracts, briefs and agreements occupy. A genuine legal document is long, internally cross-referential and carries multiple interacting issues and that is exactly where the agent's extraction and reasoning degrade, missing a clause buried deep in a long document, mishandling a provision that interacts with another, or, as ContractEval found, simply returning that no relevant clause exists when one does. So the agent that looks capable on a short clause excerpt meets a fifty-page agreement and starts missing things and the things it misses on the complex documents are precisely the consequential ones, the buried indemnity, the cross-referenced limitation, the issue that spans clauses. The clean-clause performance gave false confidence about the hard documents where the agent actually fails.
The "no related clause" failure is especially insidious in legal work, because a missed clause looks like a clean result. The agent returns nothing, which reads as "no issue found," when the truth is the agent failed to find the issue that was there and in legal review a missed clause is a missed risk.
What makes legal document work reliable?
Scope and verification matched to document complexity. Use the agent where it is reliable, short, well-bounded extraction and clean clauses and apply rigorous human verification on the long, dense, multi-issue documents where the benchmarks show it degrades. Treat a "no relevant clause" result on a complex document with suspicion rather than relief, because the agent may have missed the clause rather than confirmed its absence. As legal agents integrate with document management, e-signature and matter management systems, ensure the integration does not let an agent's degraded extraction on a complex document flow into a filed or executed instrument unverified. The agent helps most on the simple documents; the complex ones, which are the substance of legal practice, need the human.
| Document type | Agent reliability |
|---|---|
| Short, clean clauses | High; agent is useful |
| Long, dense, multi-issue | Degrades; verification required |
On CUAD, accuracy drops on long, multi-issue contracts and the rework lands late; a more capable model lifts easy clauses, hard ones keep failing. (arXiv:2508.03080)
Matching the agent to document complexity is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns of the long, multi-issue documents where legal extraction degrades, so the agent is trusted on the simple cases and verified on the complex ones that carry the real risk.
Frequently asked questions
Why does clean-clause performance mislead?
Because real documents are long and multi-issue, the regime where CUAD shows degradation. The easy-case number does not predict the hard-document result.
Why distrust a "no relevant clause" result?
Because models sometimes return that even when a clause is present. On a complex document, it may be a missed clause, not a confirmed absence.
Where should humans focus?
On the long, dense, multi-issue documents, where the agent degrades and a missed clause is a missed legal risk.