Tested as agent tools, real REST APIs fail across a whole taxonomy of error modes.A framework that generated over 2,400 test cases found schema mismatches, input misinterpretation, and output-handling failures are recurring, not rare (A Framework for Testing and Adapting REST APIs as LLM Tools, arXiv:2504.15546, 2025). A tool call that returns 200 isn't proof the contract held.
Key facts.
- Real REST APIs exposed as agent tools fail in patterned ways: a framework that generated over 2,400 test cases across domains produced a taxonomy of errors including input misinterpretation, output failures, and schema mismatches (A Framework for Testing and Adapting REST APIs as LLM Tools, arXiv:2504.15546, 2025).
- Consumer-driven contract testing, the agent publishes the shape it depends on and the provider verifies it in CI, catches a breaking change in the provider's build instead of in your agent (consumer-driven contract testing, e.g. Pact, established practice).
- Breaking-change detection at the boundary, an OpenAPI or schema diff run on every pull request, turns a silent provider change into a failed check before release (API change-management practice).
Why isn't a passing call proof the contract holds?
Because a call can succeed at the transport level while violating the shape the agent expects. The model emits arguments it believes are right, the tool returns a 200, and the agent proceeds, but a required field may be missing, a type may have changed, or an error may be hiding in a success body. Nothing checked the actual structure against an agreed contract. The taxonomy from testing real APIs as tools names this directly: input misinterpretation, output failures, and schema mismatches are recurring modes, not rare ones (arXiv:2504.15546). A passing call tells you the wire worked. It doesn't tell you the data matched what the agent was built to handle.
What does contract testing add over a schema check?
It catches the breaking change before deploy, not after. A runtime JSON Schema check validates a single call as it happens, which is essential, but it only fires once production traffic hits the drift. Consumer-driven contract testing flips that: the agent, as the consumer, publishes the exact shape it depends on, and the tool provider verifies that contract in its own CI, so a field rename or a removed property fails the provider's build before it reaches you. Standardizing the boundary as enforceable schemas makes the contract a real artifact both sides hold rather than an assumption. Runtime validation guards the call; contract tests guard the release.
# Validate every tool response against the agreed contract, at the boundary from jsonschema import validate INVOICE = {"type":"object", "required":["id","status","currency"], # status + currency are NOT optional "properties":{"status":{"enum":["paid","open","declined"]}}} validate(tool_response, INVOICE) # raises on drift -> caught here, not three steps later
How do you put contracts at every boundary?
Make the shape explicit and check it on both clocks. Define each tool's inputs and outputs as a JSON Schema, validate every call and response against it at the wrapper before the model sees a result or the tool runs, and reject drift loudly instead of letting it through. Publish consumer-driven contracts, with a framework like Pact or bidirectional contract testing, so providers verify the agent's expectations in CI and a breaking change fails their pipeline. Keep a schema registry, version your tools, and run breaking-change detection, an OpenAPI diff plus contract tests, on every pull request. Adopt an enforceable schema standard so the contract is enforced, not documented. The agent then runs against a contract that something actually checks.
Where the boundary breaks and how a contract holds it
| Boundary failure | Without a contract | With contract testing |
|---|---|---|
| Provider renames a field | Agent misparses silently | Contract test fails in provider CI |
| Required field removed | Downstream null/garbage | Schema validation rejects at the boundary |
| Type changed | Coercion error later | Runtime validation catches the call |
| Error in a 200 body | Treated as success | Error shape is part of the schema |
| New required input | Calls start failing | Breaking-change detection in PR |
The pattern is that an agent and its tools agree on a shape only by convention, so a quiet provider change becomes a silent agent failure. Make the agreement an explicit schema, validate it at every call, and verify consumer contracts in CI, and drift fails a build instead of a customer. Treating the agent-tool boundary as a checked contract rather than a shared assumption is reliability at the pattern level, which is what VibeModel builds as the Pattern Intelligence Layer.
Frequently asked questions
Is runtime validation enough on its own?
it's necessary but reactive, it catches drift only once production traffic hits it. Pair it with consumer-driven contract tests so the breaking change fails in the provider's CI before it ever deploys.
Does adopting MCP give me this for free?
It standardizes the boundary as enforceable schemas, which is most of the value, but you still want consumer contracts and breaking-change detection in CI to catch provider changes ahead of release.
what's the cheapest first step?
A JSON Schema validator on every tool response at the wrapper. It converts silent shape drift into a loud, catchable error at the exact boundary where it happens.