Key facts.
- Constitutional Classifiers (arXiv:2501.18837, 2025): combined deterministic plus probabilistic verification reduced violation rates by 90% vs 45% for either approach alone.
- The complementarity is structural: deterministic checks have 100% recall for in-distribution violations but 0% for novel cases; confidence checks have broad coverage but higher false-positive rates on known cases.
- Confidence score thresholds without deterministic rules create a tuning treadmill: adjusting thresholds to reduce false positives for one class of output increases false negatives for another, with no stable operating point.
- The dual-layer design decouples the two optimization problems: deterministic rules are maintained by policy teams; confidence thresholds are tuned by ML engineers. Separation of concerns produces a more maintainable system.
What deterministic checks cover
Deterministic checks evaluate output against explicit, encoded criteria with binary results. They do not use model inference. Schema validation: does the output conform to the required structure? Business rule evaluation: does the output comply with encoded policies? Range checks: are numeric values within approved bounds? Reference validation: do the IDs in the output correspond to real records in the system of record?
For any violation type that can be precisely specified, deterministic checks are the right tool. They are fast, cheap, predictable, and independently auditable. A compliance team can read the rule specification and verify that it captures the intended policy. A regulator can inspect the check and confirm it implements the requirement. These properties are not available from model-based checks.
The limit of deterministic checks is the limit of what can be precisely specified. Novel input combinations, semantic errors that violate the intent of a policy without violating its letter, and failure modes that were not anticipated during rule design are invisible to deterministic checks.
What confidence thresholds cover
Confidence thresholds flag outputs where the agent's own uncertainty is high, even when no specific rule flags a problem. A well-calibrated model's confidence score is a meaningful signal: when the model is highly uncertain, the output is more likely to be wrong, novel, or in a distribution the model has not been trained to handle reliably.
The threshold is a policy decision: below what confidence level do we require human review or additional verification? Higher thresholds flag more outputs for review (higher sensitivity, lower specificity). Lower thresholds pass more outputs automatically (lower sensitivity, higher specificity). The right threshold depends on the consequence of a missed error vs the cost of an unnecessary escalation.
The limit of confidence thresholds is calibration. Models are not reliably calibrated across all task types - they can be overconfident on novel inputs precisely when uncertainty-based routing is most needed. Deterministic checks provide the backstop for cases where confidence is high but the output violates an explicit rule.
# Dual-layer verification implementation
def dual_layer_verify(output, task_spec, confidence_threshold=0.85):
# Layer 1: deterministic rule checks
rule_result = business_rules_engine.evaluate(output, task_spec.rules)
if not rule_result.ok:
return VerificationResult.fail("rule", rule_result.violations)Layer 2: confidence threshold check
confidence = output.model_confidence
if confidence < confidence_threshold:
return VerificationResult.escalate(
reason=f"Confidence {confidence:.2f} below threshold {confidence_threshold}",
output=output
)
return VerificationResult.PASS
Coverage comparison
| Verification approach | Known violations | Novel violations | Semantic errors | False positive rate |
|---|---|---|---|---|
| Deterministic rules only | 100% recall | 0% recall | 0% recall | Very low |
| Confidence threshold only | Variable (calibration-dependent) | Good recall | Good recall | Higher |
| Dual-layer combined | 100% recall | Good recall | Good recall | Low (rules filter known cases) |
The Pattern Intelligence Layer specifies both layers for each pattern: the deterministic rule set relevant to the pattern's domain, and the confidence threshold appropriate to the pattern's consequence level. High-consequence patterns use tighter thresholds and more comprehensive rule sets. The dual-layer specification ships with the pattern - not as a downstream integration task.