Key facts.
- An agent's internal deliberation (the "cognitive surface") includes its plan, extracted chain-of-thought, and confidence estimates, which raw input/output logging does not preserve in a usable form. source
- OpenInference semantic conventions, built on OpenTelemetry, standardize these trace attributes across providers and frameworks and support evaluation at both the per-step level and the goal-achievement level. source
- Confidence-aware behavior is a real mechanism: FLARE (Forward-Looking Active Retrieval) triggers a retrieval whenever the model generates low-confidence tokens, instead of pressing on with a shaky guess. source
- Logging the alternatives an agent considered and rejected exposes the decision rationale, which is the difference between knowing the agent was wrong and knowing why it chose wrong. source
What does semantic logging capture that raw logs miss?
A raw log shows what the agent said, not the plan and alternatives it rejected, and a stronger model does not align confidence with correctness, since OSWorld puts the best at 12%. (arXiv:2305.06983)
Four things, and they are the four you reach for during an incident. The plan: the agent's intended decomposition of the task, so you can see whether it set out down the right path. The assumptions: what it believed about the user and the world, which is usually where a silent failure starts. The confidence: how sure it was at each step, so a low-confidence decision that turned out wrong is something you can spot and a low-confidence decision that turned out fine is something you can trust less. And the alternatives: the branches it weighed and rejected, which turn "it gave a bad answer" into "it chose this over that, for this reason." Raw logs give you the final text and maybe a wall of chain-of-thought you have to read by hand. Semantic logging gives you those four as structured fields you can query and alert on.
{
"plan": ["look up account", "check balance", "issue refund"],
"assumptions": {"account_id": "from session, not confirmed"},
"confidence": 0.42,
"alternatives_rejected": [
{"action": "ask user to confirm account", "why_not": "assumed session was enough"}
]
}
That record is worth more than the raw transcript, because the confidence of 0.42 on a refund and the rejected "ask user to confirm" are exactly the signals a reviewer or an automated check would want to catch before the refund went out.
How does confidence-aware logging improve detection?
Once confidence is a logged signal rather than something buried in prose, you can act on it. Low confidence on a high-stakes step is a reason to branch: ask the user, trigger a retrieval, or route to a human, which is exactly what confidence-triggered methods like FLARE do at the token level. As a monitoring signal, a cluster of low-confidence decisions on a particular input or model is an early warning of trouble, well before it shows up as a wrong outcome. And in review, the alternatives field tells you whether the agent even considered the right option, which is the question a postmortem actually wants answered. None of this works if all you logged was the final string. Be clear-eyed about vendor framing: some platforms market "AI-graded confidence" as magic, but the durable idea is standardizing these fields on OpenInference or OpenTelemetry so they are queryable wherever your traces live.
| Question in an incident | Raw logs answer | Semantic logs answer |
|---|---|---|
| What did the agent do? | Yes | Yes |
| Why did it choose that? | No | Yes, from plan and assumptions |
| Was it sure? | No | Yes, from the confidence field |
| Did it consider the right option? | No | Yes, from rejected alternatives |
Semantic logging is the raw material a Pattern Intelligence Layer runs on. Reliability at the pattern level means the agent's plans, assumptions, confidence, and rejected options are captured as structured signals on every run, so the recurring mistakes (the assumption it keeps making, the low-confidence step it keeps committing to anyway) surface as patterns you can fix once. The text of the conversation never gave you that. The reasoning, logged with structure, does.
Frequently asked questions
Isn't logging the chain-of-thought enough?
Not in raw form. A wall of reasoning text is hard to query or alert on. Semantic logging extracts the plan, assumptions, confidence, and alternatives into structured fields you can actually use.
How does confidence help if the model is overconfident?
Even imperfect confidence is a signal. Clusters of low confidence flag risk early, and you can pair it with verification rather than trusting it alone.
Do I need a special standard for this?
Use OpenInference or the OpenTelemetry GenAI conventions so these fields are consistent across providers and queryable wherever your traces are stored.