Key facts.
- Large Legal Fictions found LLMs hallucinate on specific legal questions 58% (GPT-4) to 88% (Llama 2), with rates varying across jurisdictions, courts and time periods. source
- The models also failed to correct users' incorrect legal premises, compounding errors in a contra-factual setup. source
- Legal rules differ by jurisdiction and change over time, so a model trained on a snapshot applies outdated or wrong-jurisdiction rules confidently. source
Why does jurisdiction and change break the agent?
Because the model holds a single, averaged, dated representation of the law and the law is the opposite of that. A legal question has a correct answer that depends on the jurisdiction, the court and the current state of the law, which differs across places and shifts as statutes and precedents change. The model, trained on a broad corpus frozen at a point in time, blends all of this into a general sense of what the law says, which is wrong whenever the specific jurisdiction or the current moment diverges from that blend, and Large Legal Fictions shows that divergence is the norm, with hallucination rates that vary precisely by jurisdiction and time period. So the agent confidently applies a federal rule to a state question, an outdated standard to a changed area or one state's law to another's and the variation in its error rate across jurisdictions is the measurable signature of this failure. The contra-factual finding makes it worse: the model does not even reliably correct a user's wrong legal premise, so it will confirm an error rather than catch it.
This is why general legal capability is the wrong thing to optimize. A model that is better on average is still applying an average to a domain that punishes averages, because the right answer is jurisdiction-specific and time-specific and the agent's confidence is uniform across cases where the correct answer is not.
What makes a legal agent jurisdiction-aware?
Grounding in the specific, current authority rather than the model's general knowledge. Retrieve and apply the law of the actual jurisdiction, in its current form, for every answer, so the agent is reasoning from the rule that governs rather than a blended average. Make the agent identify the jurisdiction and the temporal validity of any authority it relies on and verify that the cited law is current and applicable to the place in question. Where the law has changed or differs across jurisdictions, the grounding catches it; the model's general knowledge does not. The agent's value is in finding and applying the right specific authority, not in recalling an average that is wrong wherever the specifics matter, which is everywhere in law.
| Source of the legal answer | Reliability across jurisdiction and time |
|---|---|
| The model's general knowledge | A dated average, wrong where specifics diverge |
| Grounded in current, jurisdiction-specific authority | The rule that actually governs |
Grounding answers in the governing authority is part of what VibeModel does as the Pattern Intelligence Layer. We model the patterns that tie a legal answer to its jurisdiction and its current validity, so a legal agent applies the law that governs rather than a blurred average that is wrong wherever the specifics matter.
Frequently asked questions
A sharper model knows more law, so won't it get jurisdiction right?
LLMs hallucinate 58% to 88% on jurisdiction-specific questions, caught late; a more capable model lifts the average, but law is not an average. (arXiv:2401.01301)
Why is a more capable model not enough?
Because it improves the average and law is not an average. The right answer is jurisdiction-specific and current, which grounding provides and general knowledge does not.
What does the error-rate variation tell us?
That the agent is least reliable exactly where the law is most specific, which is the signature of applying a blended average to a specific question.
Does the agent catch a user's wrong legal premise?
Often not. Large Legal Fictions found models fail to correct incorrect premises, so they confirm errors rather than flag them.