Key facts.
- AI agent observability means capturing every model call, tool execution, and reasoning step as structured spans, not just log lines and counters. source
- What you actually want is the reasoning trace, the tools considered, the tools invoked, the arguments, the responses, tokens, and latency, stitched into one replayable trace. source
- OpenTelemetry's GenAI semantic conventions (late 2025) define span kinds and attributes for LLM calls, agent invocations, and tool executions, with Datadog, Honeycomb, and New Relic supporting them. source
- Frameworks like LangChain, CrewAI, and AutoGen emit OpenTelemetry-compliant spans natively or via instrumentation. source
What can a log line actually tell you about an agent decision?
Logs show what ran, not why the agent chose it, and a frontier model raises the stakes, since GPT-4.1 completes only 62% of multi-step goals you cannot debug unrecorded. (source)
Almost nothing about the decision itself. A log says "called search_orders with id 4821." It does not say why the agent chose search_orders over refund_lookup, what the planning step concluded, or which retrieved snippet tipped the call. When the agent does the wrong thing, the log shows the wrong thing happening. It does not show the reasoning that produced it, so you are left guessing at cause from a record of effect.
Metrics are worse for this. A 4% tool-error rate tells you something is off at the fleet level and nothing about any single failure. You cannot reconstruct one agent's bad decision from an aggregate count. Both tools are useful, but neither was designed to capture reasoning, and reasoning is the thing that decides what the agent does next.
What replaces guesswork?
A reasoning trace. Capture the planning step, the candidate tools, the chosen tool and its arguments, the tool response, and the next decision, all as nested spans you can replay in order. With that, a failure stops being a mystery. You see the agent weigh the options, pick the wrong one, and act, and you see exactly which input pushed it there. OpenTelemetry's GenAI conventions make this portable, so the trace plugs into the monitoring stack you already run.
| Question | Logs / metrics answer | Reasoning trace answer |
|---|---|---|
| What ran? | Yes | Yes |
| How often does it fail? | Yes (metrics) | Per-run, with context |
| Why this tool? | No | Yes, candidates and choice |
| What input caused it? | No | Yes, stitched to the decision |
The reasoning trace is the backbone of a Pattern Intelligence Layer. Reliability at the pattern level means every decision the agent makes is captured and replayable, so root cause is something you read off the trace instead of reconstruct from effects. Logs and metrics still earn their keep. They just answer a different question than the one an agent failure asks.
Frequently asked questions
Do I throw out logs and metrics?
No. Keep them for technical health and fleet-level trends. Add reasoning traces for the per-decision "why" they were never built to answer.
Is this a proprietary format I get locked into?
It does not have to be. OpenTelemetry's GenAI conventions are vendor-neutral and supported by major backends, so traces stay portable.
Why does non-determinism make this urgent?
Identical inputs can produce different decisions, so you cannot reproduce a failure on demand. The recorded trace of the actual run is often your only path to root cause.