Key facts.
- "Extracting Training Data from Large Language Models" (Carlini et al., arXiv 2012.07805, USENIX Security 2021) showed black-box extraction of verbatim training sequences from GPT-2, including names, addresses, and phone numbers. source
- "Scalable Extraction of Training Data from (Production) Language Models" (Nasr et al., arXiv 2311.17035, November 2023) used a divergence attack to extract memorized data from aligned ChatGPT at about 150x the normal rate. source
- OWASP's LLM02:2025 Sensitive Information Disclosure covers PII, credentials, and proprietary data surfacing in outputs; LLM07 System Prompt Leakage covers hidden instructions and secrets being extracted. source
- In agentic systems, sensitive context from inputs, retrieved documents, or memory can pass into a tool's arguments, so a leak becomes an action (an API call carrying data it should not). source
How does an agent leak without anyone attacking it?
The memorization path is a property of the model. During training the model absorbs its data, and rare or duplicated text gets memorized closely enough to reproduce. Under the right prompt, including ordinary ones, it surfaces. The Carlini work proved the principle on GPT-2, and the Nasr work proved it still holds on an aligned production model, which is the uncomfortable part. Alignment reduces the rate, it does not remove the capability.
The context-bleed path is a property of the agent. A production agent holds a lot in context: the system prompt, the user's input, documents pulled by retrieval, and memory from prior turns. When it plans a tool call, any of that can end up in the arguments. An agent that queries a database might place a piece of PII into the query. An agent that calls an external API might include a snippet of a confidential document. No one asked it to leak. The data was simply in reach, and the agent reached for it. The blast radius is larger than a chat leak because the tool call is an action with real effects.
What actually stops both leaks?
Different controls, because they are different problems. For memorization, you cannot un-train the model, so you filter and monitor outputs for sensitive patterns and avoid putting secrets where a model could memorize them. For context bleed, you scope what the agent can place into a tool call, validate tool arguments against what the task needs, and keep untrusted or sensitive context out of reach of tools that send data outward. Least privilege on tools limits how far any leaked value can travel, and an audit trail shows you when it happened.
| Leak path | Where it lives | What controls it |
|---|---|---|
| Memorized training data | The model's parameters | Output filtering, monitoring, no secrets in training |
| Context bleed to output | The agent's context | Output validation, sensitive-data detection |
| Context bleed to tool call | The agent's tool arguments | Argument validation, least-privilege tools |
This is a pattern-level problem, which is why a Pattern Intelligence Layer is where it gets solved. Reliability and security at the pattern level mean outputs and tool arguments are checked for sensitive data around every run, tools are scoped so a leaked value cannot travel far, and the data flow is traced so a leak is visible rather than silent. The model will keep some memorization and the agent will keep a rich context. The pattern is what keeps either from quietly becoming a breach, and it holds when you change the model.
Frequently asked questions
Does a newer, aligned model fix memorization?
It reduces it, not eliminates it. The divergence attack pulled memorized data from aligned production ChatGPT. Treat output filtering as a standing control, not a temporary patch.
Which leak is more dangerous in an agent?
Context bleed into a tool call, usually. It turns a leak into an action with side effects and a larger blast radius than a chat response. Validate tool arguments and scope tools tightly.
We do not train models. Are we safe from the memorization path?
Partly. You inherit the base model's memorization, so you still filter outputs. But your bigger exposure is context bleed from your own prompts, documents, and memory, which is fully yours to control.