Key facts.
- Concept drift and data drift are well-studied. A widely cited review covers 130-plus papers on detection and adaptation, using tests like Kolmogorov-Smirnov and Chi-squared with window-based monitoring. source
- Embedding drift hits RAG and retrieval specifically: when the embedding space shifts from a model update or evolving data, retrieval quality drops before accuracy visibly fails. You detect it with cosine distance, clustering, and nearest-neighbor consistency. source
- Frontier models drift between versions. GPT-4's prime-identification accuracy dropped from 84% to 51% over three months, a model update can silently change agent behavior. source
- AWS now publishes prescriptive guidance on detecting drift in production generative-AI applications. source
Why does an unchanged agent still degrade?
An agent depends on things you don't fully control. The inputs are live production data, that shifts as users and the business change. Retrieval depends on an embedding space that moves when the model or corpus changes. Reasoning depends on a base model the provider can update under you. Actions depend on tool APIs that change schemas and responses. Any one of these can move while your code stays frozen. The agent's behavior moves with it. The wrong answer isn't a bug you introduced, it's drift you didn't catch.
The agent-specific flavors have names now: goal drift as task distributions shift, context drift as retrieval and memory change, reasoning drift as models and prompts move, collaboration drift as tools and other agents change. The labels are newer than the classic ML literature, but the detection logic is the same: baseline normal, watch for statistically meaningful divergence.
What do you actually monitor for drift?
Behavioral baselines, not just accuracy. At deployment, establish what normal looks like: typical tool-call distributions, reasoning-path stability, output entropy and tone, task success rates, embedding similarity against a reference set. Then compute divergence on a schedule using statistical tests, embedding distances, and LLM-as-judge evals. Correlate any shift with deployment events so you can tell a model upgrade from a risky drift. Agent-specific drift benchmarks are still maturing, so you'll tune thresholds against your own baselines rather than an industry standard. But the practice is real and shipping, it catches the slide before a customer files the ticket.
| Drift type | What moved | Signal to watch |
|---|---|---|
| Data / concept | Input distribution, intent mix | Feature distributions, KS / Chi-squared |
| Embedding | Embedding space, corpus | Cosine distance, retrieval relevance |
| Reasoning | Base model or prompt update | Tool-call patterns, success rate, judge scores |
| Tool / API | Downstream schema or response | Tool error rate, output validity |
An agent degrades untouched when its data or a tool API shifts, and the newer model is no fixed point, since GPT-4's prime accuracy fell from 84% to 51% between versions. (arXiv:2004.05785)
Drift detection is a pattern-level property. The Pattern Intelligence Layer holds and watches a behavioral baseline on every run, a shift in data, embeddings, model, or tools trips an alert instead of a customer complaint. The model can be upgraded, the data can evolve, and the pattern still catches when agent behavior leaves its baseline. That's drift detection before it becomes an incident.
Frequently asked questions
Isn't drift just a model-accuracy problem?
No. For agents it shows up as changed tool choices, reasoning paths, and outputs, often before any accuracy metric moves. You baseline behavior, not only correctness.
What's the sneakiest source of drift?
Embedding drift in retrieval. The embedding space shifts on a model or data change and quietly degrades what the agent retrieves, well before the final answer looks wrong.
Can a stronger model remove the need to watch?
No. Frontier models drift between versions on their own, as GPT-4's accuracy swing showed. Drift detection has to live around the agent, independent of the model.