The Logging and Observability Infrastructure That Makes Agent Verification Actually Work

Key facts.

• OpenTelemetry GenAI semantic conventions (CNCF, 2025): the first standard telemetry schema built for AI agent observability. Covers model invocations, tool calls, and correlation IDs that link decisions to effects.
• The most common observability gap in agent systems: absence of a correlation ID that links a model decision (in the LLM layer) to its downstream tool calls (in the integration layer) and their system effects (in the target system layer).
• Without correlation IDs, post-incident analysis can't trace from a customer-visible outcome back to the agent decision that caused it - making root cause analysis slow, expensive, and often inconclusive.
• Three data requirements: event completeness (every action emits a loggable event), timing precision (millisecond-accurate timestamps), context propagation (correlation ID crosses every system boundary in the execution graph).

What no observability actually costs

Payment agent, 10,000 transactions a day. Day 15: the fraud team spots a chargeback cluster in one merchant category. They need to know which sessions processed those transactions, what decision the agent made, what tool calls ran, what the system state looked like at decision time. Without correlation IDs, each of these questions requires querying three separate systems (LLM logs, tool call logs, payment system logs) and manually correlating the results by timestamp and session ID. The investigation takes three analysts three days.

With correlation IDs propagated through the full execution graph, the same investigation runs in 20 minutes: query for all sessions with chargebacks in the merchant category, follow the correlation ID to the LLM decision log, pull the tool call sequence, cross-reference with the payment system state at decision time. The correlation ID makes the execution graph navigable instead of opaque.

The three data requirements for effective verification

Event completeness means that every action the agent takes emits a structured log event. Not a print statement - a structured event with defined fields: event type, correlation ID, session ID, timestamp, action parameters, and action result. Completeness is a design requirement, not an operational afterthought. An action that doesn't emit an event is an action that verification can't observe.

Timing precision means that event timestamps are accurate to milliseconds and synchronized across all systems in the execution graph. When a verification check queries the state of multiple systems within a defined time window, millisecond-precision timestamps are required to determine which events occurred before the verification window and which occurred after. Clock skew between systems is a known source of false verification results.

Context propagation means that the correlation ID travels with every downstream call the agent makes. When the agent calls a tool, the correlation ID is in the tool call header. When the tool calls a database, the correlation ID is in the database query. When the database result returns to the tool, and the tool result returns to the agent, the correlation ID is in the response. The full execution graph is navigable from any point to any other point via the correlation ID.

# OpenTelemetry GenAI convention implementation
from opentelemetry import trace
from opentelemetry.semconv.ai import SpanAttributes
tracer = trace.get_tracer("agent.core")
def traced_llm_call(prompt, model, session_id, correlation_id):
 with tracer.start_as_current_span("llm.invoke") as span:
 span.set_attribute(SpanAttributes.LLM_REQUEST_MODEL, model)
 span.set_attribute("gen_ai.session.id", session_id)
 span.set_attribute("gen_ai.correlation.id", correlation_id)
 result = llm.invoke(prompt, model=model)
 span.set_attribute(SpanAttributes.LLM_RESPONSE_MODEL, result.model)
 span.set_attribute("gen_ai.usage.prompt_tokens", result.usage.prompt_tokens)
 span.set_attribute("gen_ai.usage.completion_tokens", result.usage.completion_tokens)
 return result
def traced_tool_call(tool_name, params, correlation_id):
 with tracer.start_as_current_span(f"tool.{tool_name}") as span:
 span.set_attribute("gen_ai.tool.name", tool_name)
 span.set_attribute("gen_ai.correlation.id", correlation_id) # propagated
 span.set_attribute("gen_ai.tool.params", str(params))
 result = tools[tool_name](**params)
 span.set_attribute("gen_ai.tool.result.status", result.status)
 return result

Observability requirements by use

The Pattern Intelligence Layer requires OpenTelemetry GenAI-compliant instrumentation as a pre-condition for any pattern marked production-ready. A pattern that can't emit observable events for each action type can't be verified. The observability contract is part of the pattern specification, not a separate infrastructure concern - because without the data, verification isn't a check, it's a hope.

Frequently asked questions

Will better telemetry arrive on its own with a stronger model?
Without correlation IDs the output outran the trace as a class; a bigger model inherits the blind spot and its rework.