Key facts.
- RLSR (arXiv:2505.08827, 2025): verification signals that transfer well across coding tasks fail to generalize to dialogue or orchestration - cross-domain verification architecture isn't valid.
- Coding agents have the most reliable verification signals: test execution is deterministic, the success condition is unambiguous, and the oracle (test suite) is external and independent.
- Support agents have the least reliable automated verification: policy compliance is rule-checkable, but tone, empathy quality, and resolution adequacy require semantic judgment that automated checks approximate poorly.
- Orchestration agents have verification challenges unique to multi-system state: completeness (did all downstream systems receive the instruction?), ordering (did the effects occur in the correct sequence?), and consistency (are all downstream states mutually consistent?).
Coding agent verification
Coding agents have the clearest verification signal in production AI: run the tests. If the tests pass, the code change achieves its intended effect (within the scope of what the tests cover). If the tests fail, the change didn't. this is deterministic, automatable, and provides precise failure localization.
The limits of test-suite verification for coding agents: tests cover specified behavior but not unspecified behavior. A change that passes all tests but introduces a security vulnerability, adds a performance regression, or violates a style convention isn't caught by functional tests. Coding agent verification needs supplementary checks: static analysis for security and style, performance benchmarks for critical paths, and diff review for changes that affect system boundaries.
Support agent verification
Support agents need three verification layers: policy compliance (did the response adhere to the company's stated policies?), factual accuracy (did the response contain correct product, pricing, and policy information?), and resolution confirmation (did the customer's issue get resolved, as confirmed by a follow-up check or explicit customer confirmation?).
Policy compliance is automatable with a rule engine if policies are formally encoded. Factual accuracy is automatable with ground truth lookups against authoritative product and policy databases. Resolution confirmation is the hardest to automate - it requires either a follow-up interaction with the customer or a proxy metric (customer didn't recontact within 48 hours for the same issue).
Orchestration agent verification
Orchestration agents coordinate work across multiple downstream agents or systems. Their verification requirements are: completeness (all intended downstream actions were dispatched and confirmed), ordering (the sequence of downstream actions followed the required dependency order), and consistency (all downstream systems reflect a mutually consistent state after the orchestration run).
Completeness verification requires a manifest: before the orchestration run, enumerate all intended downstream actions. After the run, confirm each manifest item was dispatched and acknowledged. Ordering verification requires sequence logging with timestamps: each downstream action logs its receipt and execution time; the orchestrator confirms the log sequence matches the intended dependency order. Consistency verification is the distributed verification problem - confirm that all downstream system states are mutually consistent after allowing for propagation delays.
# Domain-specific verification dispatcher
VERIFICATION_BY_DOMAIN = {
"coding": CodingVerifier(
run_tests=True, static_analysis=True, perf_check=True
),
"support": SupportVerifier(
policy_engine=policy_db, factual_oracle=product_db,
resolution_check="48h_no_recontact"
),
"orchestration": OrchestrationVerifier(
manifest_check=True, sequence_log=True,
consistency_window_s=300, saga_coordinator=coordinator
),
}def verify(agent_type, output, context):
verifier = VERIFICATION_BY_DOMAIN.get(agent_type)
if verifier is None:
raise ConfigError(f"No verification config for agent type: {agent_type}")
return verifier.evaluate(output, context)
Verification signal comparison by domain
| Agent domain | Primary verification signal | Determinism | Human judgment required |
|---|---|---|---|
| Coding | Test suite execution | High | For coverage gaps only |
| Support/customer service | Policy + factual + resolution | Medium (policy, facts); Low (resolution) | For tone and resolution quality |
| Orchestration | Completeness + ordering + consistency | High (completeness); Medium (consistency) | For cross-system edge cases |
| Data pipeline | Row count + checksum + schema | High | Rarely |
The Pattern Intelligence Layer classifies every pattern by domain and specifies the corresponding verification architecture. A coding pattern ships with test execution and static analysis contracts. A support pattern ships with policy, factual accuracy, and resolution check specifications. An orchestration pattern ships with completeness manifest, sequence logging, and consistency window specifications. The right verification architecture is a function of the domain, and the Pattern Intelligence Layer encodes that function so teams don't have to derive it from first principles for every deployment.