The agent you build in development has never seen your production data, your production users, or your production edge cases. The accuracy you measure in testing is accuracy on test data. The accuracy in production — before you measure it — is unknown.
Deploying a full-capability agent into production on day one is betting that your test coverage captured everything that matters. It usually didn't.
The alternative: deploy the agent with no ability to act, measure its accuracy on real production data, and expand its capabilities progressively as the measurements justify the trust.
The deployment stages
Stage 0: Shadow mode
The agent runs against real production inputs. It produces recommendations. It executes no actions. All recommendations are logged but not applied.
The purpose of shadow mode is measurement. How often does the agent recommend the right action? How often does it identify the wrong customer, call the wrong tool, produce the wrong output? You're building a ground truth dataset on real production inputs.
Run shadow mode for two to four weeks before proceeding. Measure the accuracy on at least a few hundred real inputs. Establish baseline metrics: recommendation accuracy rate, tool selection accuracy rate, edge case rate, hallucination rate.
If accuracy is below your threshold for Stage 1, fix the agent. Don't proceed until shadow mode accuracy meets the bar.
Stage 1: Assisted mode
The agent runs against real inputs and makes recommendations. A human reviews and approves each recommendation before it's applied. The agent does not execute actions autonomously.
The purpose of assisted mode is: catching errors before they become real effects, building the human reviewer's intuition about the agent's failure modes, and generating labeled data on where the agent is right and where it's wrong.
Assisted mode has a cost: the human review time. This cost should be modeled. If the agent is accurate enough, review is a rubber stamp and throughput is limited by review time. If the agent is not accurate enough, review finds too many errors and the human value-add justifies the cost. Either way, you're learning.
Stage 2: Supervised autonomy
The agent acts autonomously on low-risk operations. High-risk operations still require human approval. The threshold is a product decision: what's low-risk enough to automate?
Measure error rates in supervised autonomy separately for low-risk and high-risk operations. Track how often human reviewers override the agent. Track error rates for approved autonomous actions.
Stage 3: Full autonomy
The agent acts autonomously on all operations within its permission scope. Human review is triggered only by anomalies (detected by your observability stack), by user escalations, or by periodic audit sampling.
Reach Stage 3 only for operations where Stage 2 error rates are consistently below your threshold for an extended period.
The metrics that govern progression
Define your progression thresholds before you start. Not after you've collected data and then decide what threshold justifies moving forward.
From Stage 0 to Stage 1 (shadow to assisted):
- Recommendation accuracy ≥ X% on real production data
- Hallucination rate ≤ Y% on real production data
- Tool selection accuracy ≥ Z%
From Stage 1 to Stage 2 (assisted to supervised autonomy):
- Human reviewer override rate ≤ N% on low-risk operations
- Error rate for approved autonomous actions ≤ M%
From Stage 2 to Stage 3 (supervised to full autonomy):
- Sustained low error rate over 30 days
- No high-severity errors in Stage 2
- Human review confirms anomaly detection catches errors before user impact
These thresholds are yours to define. The point is to define them prospectively, not retrospectively. Retroactive threshold-setting allows motivated reasoning: "the accuracy is 87%, which is good enough for Stage 2" — because 87% already exists and 90% would require more work.
Regression gates on new capabilities
Every time you add a new tool, change the agent's permissions, or modify the system prompt, you've potentially broken the accuracy you measured in previous stages. Treat capability changes as new deployments.
When a capability changes, regress to the previous stage for that capability and measure again. A new write permission that was never in Stage 0 should run in shadow mode before it's used autonomously, even if the rest of the agent has reached Stage 3.
The practical implementation:
Maintain a capability registry: a list of every action the agent can take, what stage each is in, and what the accuracy metrics are for each. New capabilities start at Stage 0. Capabilities advance through stages independently.
capabilities:
- name: lookup_customer_record
stage: 3
accuracy: 0.97
last_reviewed: 2026-06-01
- name: initiate_refund
stage: 2
accuracy: 0.91
human_override_rate: 0.08
last_reviewed: 2026-06-20
- name: cancel_subscription
stage: 1 # New capability, still in assisted mode
shadow_accuracy: 0.89
started: 2026-06-22
The business case
Progressive deployment costs time. The agent that could be autonomous in week one is running in shadow mode for the first month.
The cost is real. The benefit is that you don't deploy a full-capability agent that errors on 12% of production inputs and spends two months firefighting the consequences before you've measured what the actual error rate is.
The teams that skip progressive deployment because it feels slow are usually the teams running incident postmortems six weeks later, trying to explain to customers why the automated system made the wrong decision on their account.
Go slow before you go autonomous. The measurements you collect in the early stages are the trust you've earned for every subsequent stage.