In a regulated industry, an agent you cannot audit is an agent you cannot deploy

Finance, healthcare, and legal do not ask whether the agent worked, they ask you to reconstruct why it did what it did. An agent that cannot produce that record is a compliance problem, not a product.

B

Balagei G Nagarajan

4 MIN READ


A regulator's magnifying glass over an agent decision, with a complete reconstructable trail visible underneath

Key facts.

  • EU AI Act (Regulation 2024/1689) Article 12 requires high-risk AI systems to technically allow automatic recording of events (logs) over the system's lifetime, with deployers retaining logs, making reconstructability a legal obligation. source
  • ISO/IEC 42001:2023 sets management-system controls including documentation, traceability and human oversight for AI, the operational backbone of an auditable system. source
  • Moffatt v Air Canada (2024 BCCRT 149) confirmed an organization is liable for its agent's outputs, so a deficient audit trail is the organization's exposure, not the model's. source

Why does regulated work need the why, not just the what?

Because regulation governs decisions and a decision you cannot explain is one you cannot defend. In finance, healthcare and legal work, the obligation is not only that the outcome was correct but that the process can be reconstructed: what data the agent saw, what it concluded, what action it took and why. That is what an auditor, a regulator or a court asks for after the fact and it is what the EU AI Act's Article 12 logging requirement codifies for high-risk systems, recording over the lifetime so the trail exists when it is needed. An agent that emits a final answer with no record of the reasoning and inputs behind it fails this on its face: there is nothing to reconstruct. The output might be right, but in a regulated setting an unexplainable right answer is still a compliance gap, because the obligation is the explanation, not just the result.

A more capable model does not produce the trail. It might reason better, but unless the system captures the inputs, the reasoning and the actions as durable records, none of that is auditable after the run. The trail is instrumentation: structured logging of decisions and their context, retained and reconstructable, which ISO/IEC 42001 frames as documentation and traceability controls. Upgrading the model improves the decision and leaves the record exactly as thin as it was, which is why regulated deployments stall on auditability rather than on capability.

Timeline of an agent decision with capture points for inputs, reasoning, action, and outcome forming a reconstructable trail

What makes an agent auditable?

A record that lets someone reconstruct the decision without being there. The inputs the agent acted on, captured at decision time, so the basis is known. The reasoning or decision factors, logged as the agent's account of why, so the path is visible. The actions taken and their results, so the effect is traceable. And retention that meets the regulation's window, so the trail is still there when the audit comes, which under Article 12 means logs kept over the system's lifetime. Built this way, the agent can answer the regulator's real question and the deployment clears the bar that capability alone never could. Built without it, every correct decision is still a record the organization cannot produce and in a regulated industry that is the deployment blocker.

Audit elementOutput-only agentAuditable agent
InputsNot capturedLogged at decision time
ReasoningDiscardedRecorded as decision factors
ActionsUntracedTraceable with results
RetentionAd hocMeets the regulatory window

The Pattern Intelligence Layer is where the audit trail is captured by default. Inputs, decision factors and actions are recorded at the pattern level, so each decision is reconstructable on demand and the retention meets the obligation rather than being bolted on after a finding. Reliability at the pattern level is what turns a regulated deployment from a compliance risk into one that can answer the why.

Frequently asked questions

Isn't a correct outcome enough for a regulator?
No. In regulated work the obligation includes explaining the decision. The EU AI Act's Article 12 requires logging that lets you reconstruct it, so an unexplainable correct answer is still a gap.

Whose problem is a missing audit trail?
The deploying organization's. Moffatt v Air Canada confirmed the organization is liable for its agent's outputs, so the inability to reconstruct a decision is your exposure, not the vendor's.

Can a better model make the agent auditable?
No. Auditability is instrumentation: capturing inputs, reasoning and actions as retained records. A stronger model improves the decision but does not create the trail.


Share this post

Join the discussion

Have a take, a war story, or a question? Sign in with GitHub to comment and react. Comments are powered by GitHub Discussions, ad-free and yours to moderate.

Continue Reading

Find where your agent breaks, before you build it

Faultmap maps where your agent will fail from the goal and your data, then hands you the first test suite it has to pass.