Key facts.
- The OpenTelemetry GenAI semantic conventions define structured attributes for model, agent, and tool spans (model name, input and output token counts, finish reasons), giving agent traces a vendor-neutral shape. Most of the conventions were still marked experimental in early 2026. source
- Storing full prompt and completion text inside span attributes is a documented anti-pattern: attributes are size-limited, always indexed, and leak sensitive content into your backend. The conventions put that content in span events instead, so it can be filtered or dropped at the Collector. source
- High-cardinality dimensions (session and trace IDs, prompt-template variants, model versions, tool names, per-user context) multiply the unique series an observability backend must index, which is the classic driver of storage and query cost. source
- Tail-based sampling keeps every error, slow, or costly trace and samples the rest, but it requires buffering full traces before deciding. Head-based sampling is cheaper and can throw away the failing trace before anyone sees it. The OpenTelemetry Collector supports both. source
Why does an agent generate so much more telemetry than a normal service?
A traditional request fans out into a few spans: receive, query a database, return. An agent run fans out into a plan, several reasoning steps, multiple tool calls, retrievals, and sometimes sub-agent handoffs, each of which you want as its own span with its own context. Vendors report agent and LLM workloads producing on the order of 10x to 100x the telemetry of a comparable traditional app (treat those multiples as reported vendor figures, not an independent benchmark). The payloads are the second problem. A span that records a prompt, the documents it retrieved, and the model's full output is not a few bytes of numbers. It is kilobytes of text, repeated for every decision, for every run.
Multiply that by production traffic and the cost is not theoretical. Several practitioner and vendor accounts describe observability bills that rival the compute bill for the agent itself. The data you most want for debugging (the full reasoning, not just the outcome) is precisely the data that is most expensive to store.
So how do teams keep the context that matters without keeping all of it?
The working answer is selective, not exhaustive. Keep numbers and identifiers as attributes so they stay queryable, and push large free text (prompts, retrieved documents, raw outputs) into events that can be dropped or masked at the Collector before they ever hit storage. Use tail-based sampling so every failed, slow, or expensive run is retained in full while routine successes are sampled down. Mask sensitive fields at capture so you are not paying to store, and exposing, customer data you never needed. The OpenTelemetry GenAI conventions exist so you can do all of this on a standard schema instead of a bespoke one per backend.
| Decision | Capture everything | Capture selectively |
|---|---|---|
| Prompts and outputs | Stored as indexed attributes, full text, always | Stored as events, droppable and maskable at the Collector |
| Which runs kept in full | All of them, until the bill forces a cut | Every error, slow, or costly run, plus a sample of the rest |
| High-cardinality fields | Indexed unbounded, cost grows with traffic | Bounded and chosen on purpose |
| Sensitive context | Lands in the backend by default | Masked at capture, never stored raw |
An agent run emits hundreds of spans, so the hard part is keeping the trace that explains a failure, and a stronger model does not rescue you, since MAKER shows 99% per-step accuracy derails. (source)
This is where the Pattern Intelligence Layer changes the question. Reliability at the pattern level means you decide, once and consistently, which decisions are worth capturing in full and which can be summarized, based on the failure patterns that actually recur. You are not trying to store every token forever. You are keeping the semantic context that explains the failures you keep seeing, enforced the same way on every run, so the bill stays sane and the trace you need is still there when an incident review asks for it.
Frequently asked questions
Should I just store full prompts and outputs as span attributes?
No. That is a documented anti-pattern: attributes are size-limited, indexed, and expose sensitive text. Put large content in events you can filter or drop at the Collector.
Is head-based or tail-based sampling better for agents?
Tail-based, in most cases. A single failed sub-step is the thing you most need, and head-based sampling can discard it before completion. Tail-based keeps every error and slow run.
Are the 10x-100x telemetry figures reliable?
Treat them as reported vendor numbers, not an independent benchmark. The direction (agents produce far more telemetry per request) is well supported; the exact multiple depends on your workload.