How to Measure Whether Your Agent's Verification Is Actually Working

Key facts.

• BFCL (2025): frontier models below 60% on multi-turn function-calling evaluation even under full-visibility harnesses.
• Verification coverage is typically not tracked as a production metric - teams monitor model accuracy or task completion rate, neither of which measures how much of the outcome space is verified.
• The three coverage gaps: decision coverage (which decision paths have a check?), state coverage (which world states can your checks detect as wrong?), and timing coverage (which time windows after action does your check run in?).
• A system with 90% step-completion coverage can have less than 30% outcome coverage if checks run only on the happy path and within the synchronous response window.
• BFCL puts even frontier models below 60% on multi-turn calls; a stronger model scores lower where coverage runs thin.

Three dimensions of verification coverage

Decision coverage asks: for every decision the agent can make, is there a verification check that runs after it? Decision trees in complex agents can have dozens of branches. Verification is typically written for the expected path. Edge-case branches - rare input combinations, unusual tool responses, partial completion states - go unchecked because they were not anticipated during development.

State coverage asks: if the world ends up in a wrong state, can your checks detect it? A check that reads back a single field from a database after a write has low state coverage if the write affects multiple tables, triggers downstream processes, or has delayed effects. High state coverage means your checks can observe all the ways the outcome can be wrong, not just the most obvious one.

Timing coverage asks: when does the check run relative to when the failure manifests? Synchronous checks that run immediately after an API call miss async failures that appear seconds or minutes later. End-of-task checks miss intermediate state corruption that resolves before the task ends. Checks need to cover the right time windows, not just the convenient ones.

Measuring coverage in practice

Decision coverage can be approximated by running the agent against a test suite that exercises all documented branches, then logging which post-action checks executed. Branches where no check ran are uncovered. The ratio of covered branches to total branches is your decision coverage score.

State coverage requires a fault injection approach: deliberately introduce known failure states (wrong database value, failed async job, partial write) and measure whether the verification layer detects them. The fraction of injected faults detected is your state coverage score.

Timing coverage requires injecting failures at different time offsets and confirming your checks catch them. A failure that appears 30 seconds after action completion should be caught by a check that runs 60 seconds post-action. If your only check runs synchronously, timing coverage for async failures is zero.

# Coverage measurement scaffold
class VerificationCoverageTracker:
 def record_check_run(self, decision_branch, check_name, time_offset_s):
 self.runs.append({
 "branch": decision_branch,
 "check": check_name,
 "time_offset": time_offset_s
 })
 def decision_coverage(self, all_branches):
 covered = {r["branch"] for r in self.runs}
 return len(covered) / len(all_branches)
 def timing_coverage(self, known_async_windows):
 covered_windows = {r["time_offset"] for r in self.runs}
 return len(covered_windows & set(known_async_windows)) / len(known_async_windows)

Coverage metric targets

The Pattern Intelligence Layer encodes expected verification coverage requirements as part of each pattern's production-readiness criteria. A pattern is not marked production-ready until its coverage scores meet the thresholds for its risk tier. This turns verification coverage from a post-incident retrospective finding into a pre-deployment gate.

Frequently asked questions