Key facts.
- Technical failures throw and show up in standard monitoring; semantic failures return HTTP 200 with confidently wrong content and don't.source
- Semantic failures include the right API with wrong parameters and a workflow that completes while skipping a critical validation step.source
- Because identical inputs can yield different outputs, fixed-threshold alerting on a non-deterministic agent either misses signal or floods you with noise, so you baseline behavior instead.source
- Key agent alert signals include hallucination rate, policy violations, escalation frequency, and audit-trail completeness, not just error rate.source
- A clean 200 hides a wrong answer that needs output-eval alerts, not status codes, and a stronger model keeps producing them, so rework is real. (source)
Why does one alerting rule fail both jobs?
A rule tuned for technical failures keys on status codes and latency. Point it at semantic failures and it sees nothing, because the failing requests all return 200. A rule loose enough to flag anything unusual on a non-deterministic agent fires constantly, because unusual is normal when the same input can produce different valid outputs. So a single rule either misses the silent failures or buries the real ones in noise. The two failure classes have different signatures and need different detectors.
The fix starts with classifying the failure before you alert on it. A technical failure has a clear signal: an error, a timeout, a schema violation. A semantic failure has to be evaluated: did the output satisfy the goal, was the context relevant, did every required step run. Different inputs, different alerts, different responders.
What does the semantic alert key on?
Outputs, not status. Run an evaluation on the result (goal satisfaction, context relevance, policy adherence) and alert when it fails, even though the call returned 200. Baseline normal behavior and alert on drift rather than fixed thresholds, so non-determinism doesn't drown you. Track signals like hallucination rate, policy violations, and skipped validations as first-class alert sources. The technical lane keeps your existing rules. The semantic lane is the new build, and it's the one that catches the failures customers feel.
| Dimension | Technical alert | Semantic / business alert |
|---|---|---|
| Trigger | Error, timeout, schema violation | Failed output evaluation |
| Status code | 500 / 4xx | 200 OK |
| Detection | Fixed thresholds | Baselines and drift |
| Catches | Crashes | Confident-wrong, skipped steps |
A Pattern Intelligence Layer is where the two lanes live side by side. Reliability at the pattern level means every run is checked for both technical health and semantic correctness, so a crash and a confident wrong answer each trip the alert built for it. You stop choosing between noise and blindness, because the pattern routes each failure to the detector that understands it.
Frequently asked questions
Can I just tighten my existing alerts?
No. Existing alerts key on status codes, which semantic failures never trip. You need a separate evaluation-based lane, not a tighter version of the technical one.
How do I keep semantic alerts from flooding me?
Baseline behavior and alert on drift instead of fixed thresholds. Non-determinism makes static thresholds noisy, so measure against the agent's own normal.
what's the highest-value semantic alert?
Goal-satisfaction failure on a 200 response. It catches the confident wrong answer that no technical alert will ever see.