The demo worked because you made it work.
You picked a real use case, cleaned the input data, ran it once, captured the successful output, and showed it on a slide. The agent handled it perfectly. Everyone was impressed.
Production is different. Production has users who type things no one anticipated. APIs that rate-limit on Tuesday afternoons. Records with null fields that your demo dataset never had. Concurrent sessions that modify shared state at the same time. Inputs that are slightly outside the distribution the model was optimized for.
The gap between the demo environment and the production environment is where agents go to fail.
The five conditions the demo didn't test
1. Dirty data. Demo datasets are curated. Someone cleaned them, formatted them, removed the edge cases. Production databases have records created by twelve different teams over five years, imported from three different legacy systems, with inconsistent field naming, null values in required fields, and dates formatted four different ways.
The agent that parsed your clean demo data confidently will stall, hallucinate, or silently skip records when it hits real production data.
2. Concurrent users. Your demo ran for one user at a time. Production has fifty. When two agents read the same record, both decide to update it, and both write back — one of those writes gets lost. The agent doesn't know. It reported success.
3. Rate limits. The API your agent calls has a rate limit. At one user, you never hit it. At fifty concurrent users, you hit it constantly. The agent receives 429 Too Many Requests, doesn't know what to do with it, and either crashes, retries infinitely, or reports a fabricated result.
4. Partial failures. In the demo, every step succeeded. In production, step three sometimes fails. Your agent was never tested on a half-completed task. It doesn't know how to recover, so it either starts over from the beginning or tries to continue from a corrupted intermediate state.
5. Adversarial inputs. Your demo used inputs you created. Production users will type things you didn't anticipate: questions outside the agent's intended scope, attempts to override its behavior, inputs in different languages, inputs with special characters, inputs that are twenty thousand words long.
Why this keeps happening
Most teams test the happy path. The demo is the happy path. They build confidence in the happy path and then deploy into a world that is mostly edge cases.
Agents are harder to test than traditional software because their behavior is non-deterministic and their failure modes are subtle. A failed SQL query raises an exception. An agent that misunderstood the task reports success.
The other factor: agents are usually evaluated in isolation. Each tool call is tested independently. But in production, the agent is making ten tool calls in sequence, each one depending on the output of the previous one. Errors compound. A slightly wrong output from step two becomes a significantly wrong output by step eight.
What to test before you ship
Test on real production data. Take a sample of your actual database — dirty fields, nulls, inconsistencies and all — and run the agent against it. Measure the error rate. Fix the handling before you ship.
Test failure conditions explicitly. Make the API return a 500. Make the database return null. Make a required field missing. Watch how the agent handles each case. If it handles them wrong, fix the handling.
Test at scale. Run twenty concurrent sessions. Measure what happens to shared state. Measure what happens to rate limits. Measure whether the agent degrades gracefully or catastrophically.
Test recovery. Force a failure at step three of a ten-step task. Does the agent recover? Does it resume from step three or restart from step one? Does it report the partial failure honestly?
Faultmap runs these tests systematically before deployment. It maps the full failure space of your agent against your actual data and your actual APIs, and surfaces the gaps while you still have time to close them.
The demo environment is an optimistic fiction. Production is the truth. The gap between them is the risk. Measure it before you ship.