Key facts.
- A run can return success while its trace is incomplete (a missing retrieval span behind a lucky guess), so outcome alone is a weak health metric and completeness catches what it misses. source
- Orphan spans (no valid parent) and broken parent-child links fragment the trace tree, which makes it impossible to reconstruct the reasoning path and is a known red flag in distributed tracing. source
- The TRAIL benchmark (148 human-annotated agent traces, 841 errors across reasoning, planning-coordination, and system execution) shows trace-based error localization is hard and that outcome metrics miss internal execution fidelity. source
- TRAIL, a benchmark of 148 real agent traces with 841 annotated errors, shows that failures hide inside otherwise-normal-looking runs and that even the best model localizes the faulty step only about 11% of the time, the direct argument for monitoring trace completeness over pass/fail. source
Why is a missing span worth an alert on its own?
An agent can return success while half its reasoning never reached the trace, so completeness is a health signal a frontier model cannot self-flag, hard for it on TRAIL. (arXiv:2505.08638)
In a well-instrumented agent, a run has a predictable shape: a root invoke_agent span, a planning step, one or more chat spans for the model calls, retrieval spans, execute_tool spans, and a verification step. When a span that should be there is absent, or the trace truncates partway, that absence has a small set of causes, and every one of them is something you would want paged about. Context propagation broke between services or in-process. Sampling dropped a critical part of the run. A step crashed without emitting an error span. The agent looped silently with no telemetry. None of those show up in a pass/fail metric. All of them show up as a hole in the trace.
Orphan spans and broken links are the same story at the structure level. A span with no valid parent means the tree cannot be reassembled, so even if every step ran, you have lost the ability to reconstruct why a decision was made. For an agent, where the whole point of observability is to see the reasoning chain, a fragmented tree is a failure of the observability itself.
How do you turn that into a metric you can alert on?
Define the expected shape for each agent type, then score against it. A trace completeness score is observed spans over expected spans, weighted toward the span types that matter (the GenAI operation names: invoke_agent, chat, execute_tool, and your verification step). Track an orphan-span ratio (spans with no valid parent over total) and a link-integrity check (continuous trace id, sane hierarchy, no unexplained timestamp gaps). Aggregate at p95 and p99 over a window, and alert when completeness drops below your threshold or orphan rate spikes, weighting low completeness on high-volume agents as the more urgent case. Be honest about vendor framing: specific "AI-powered completeness scoring" thresholds are marketing, but the underlying pieces (orphan spans, propagation integrity, observed-versus-expected) are grounded in OpenTelemetry and ordinary distributed-systems practice.
| Signal | What it measures | What a drop means |
|---|---|---|
| Completeness score | Observed vs expected spans per run | Steps skipped, crashed, or never instrumented |
| Orphan-span ratio | Spans with no valid parent | Context propagation is breaking |
| Link integrity | Hierarchy and trace-id continuity | The trace tree cannot be reconstructed |
| Success rate (for contrast) | Did the task pass | Silent on all of the above |
Completeness as a first-class signal is exactly what a Pattern Intelligence Layer needs. Reliability at the pattern level depends on a trustworthy record of how each run behaved, so the health of that record becomes a property you monitor, not an afterthought you discover during an incident. Alert on the gaps, and you protect the very traces your reliability work depends on, run after run, model after model.
Frequently asked questions
Isn't an incomplete trace just a logging problem?
No. It can mean the agent skipped a step, a call crashed silently, or propagation broke. Those are runtime health problems, and the gap is the earliest visible sign.
What threshold should I alert at?
That depends on your agent and baseline. Track p95/p99 completeness over a window and alert on a drop from your own normal, not a number a vendor picked.
Why not just trust success and failure?
Because runs return success with incomplete reasoning, and benchmarks like TRAIL show the faulty step is hard to localize even with the full trace. Completeness catches what outcome alone hides.