Key facts.
- Traditional monitoring can confirm a 200 OK but cannot detect when an agent picks the wrong tool, loops in reasoning, or returns confidently wrong content. source
- Agents fail in semantic ways: right API with wrong parameters, plausible but subtly wrong output, or a workflow that completes while skipping a critical validation step. source
- Detecting this needs a different category of tool, one that understands the semantics of the output, not just the shape of the metrics. source
- Frontier models still fabricate on short-answer factual benchmarks, and reasoning training does not reliably reduce it. source
- A semantic error returns 200 with the wrong meaning, so nothing pages, and a stronger model scores worse on factual recall, not better. (source)
Why does standard monitoring miss semantic errors entirely?
Your monitoring stack was built for technical truth. Did the call return? Was the status 200? Was latency within bounds? Those questions catch crashes, timeouts, and 500s. They are blind to meaning. An agent that confidently books the wrong flight, summarizes a document with an inverted conclusion, or calls the correct API with a swapped parameter returns a clean 200 every time. The dashboard stays green because, technically, nothing failed. The business outcome is wrong anyway.
This is why teams so often learn about agent failures from a customer complaint or a dropped KPI rather than an alert. The signal that something is wrong never reached the monitoring layer, because the monitoring layer only measures technical success.
How do you catch an error the status code says doesn't exist?
You add a layer that judges meaning, not just shape. Check whether the retrieved context was actually relevant, whether the tool arguments matched intent, whether the output satisfied the goal, and whether required validation steps actually ran. Some of this is a verifier model, some is a rules check on outputs, some is comparing the agent's stated plan against what it actually did. The point is that semantic correctness has to be measured deliberately, because it never shows up as a failed request.
| Property | Technical error | Semantic error |
|---|---|---|
| Status code | 500 / timeout | 200 OK |
| Shows in dashboards | Yes, immediately | No, looks healthy |
| How you find it | Alert fires | Customer complaint, KPI drop |
| What catches it | Standard monitoring | Semantic verification layer |
A Pattern Intelligence Layer treats semantic correctness as a first-class signal. Reliability at the pattern level means meaning is checked around every run (context relevance, argument fidelity, goal satisfaction) so a confidently wrong 200 is caught as a failure, not filed as a success. The status code measures the plumbing. The pattern measures whether the agent was actually right.
Frequently asked questions
Why can't my existing APM catch these?
APM measures technical health: status, latency, errors. A semantic error returns a healthy 200. You need a layer that evaluates meaning, which APM was never built to do.
Will a better model just stop making semantic errors?
No. Leading models still fabricate on basic factual benchmarks, and reasoning tuning can make factual recall worse. The format improves faster than the meaning.
What is the cheapest first check to add?
Verify that the agent's final output actually satisfies the stated goal, and that any required validation step ran. Both catch a large share of semantic failures.