Key facts.
- Anthropic prompt caching: cached reads cost about 0.1x standard input, roughly 90% off. Cache writes run ~1.25x for a 5-minute TTL or 2x for a 1-hour TTL. After the first call, repeated context is nearly free.source
- Microsoft's LLMLingua (arXiv 2310.05736) compresses prompts up to ~20x with minimal performance loss (about a 1.5 point drop at high ratios on tested tasks), turning long context into far fewer billed tokens.source
- "Lost in the Middle" (arXiv 2307.03172) found a U-shaped accuracy curve: models use information at the start and end of a long context well but degrade sharply when it sits in the middle, even for explicitly long-context models.source
Why are these cost techniques also reliability techniques?
Shorter, better-organized context costs less and performs better. Caching stops you from re-sending the same system prompt or document on every turn. Pure savings, zero downside. Compression and summarization go further: by cutting a long prompt down to its load-bearing content, they reduce the bill and, at the same time, remove the filler that buries the important fact in the middle where "Lost in the Middle" says accuracy collapses. The team that compresses context isn't trading reliability for cost. they're buying both, because the failure modes of bloated context and expensive context are the same bloat.
this is why treating these as optional polish is a mistake. An agent that re-sends its full history uncached and uncompressed on every step pays an ever-growing token premium, while feeding the model the exact kind of long, unstructured context that degrades its answers. Cost problem and reliability problem: same root cause. Caching, compression, and summarization fix both.
How do the three techniques fit together?
They stack. Caching handles the parts of the context that repeat unchanged, so you pay full price once and a tenth after that. Compression handles the parts that vary but are verbose, squeezing them to their essential tokens before they're billed. Summarization handles accumulated history, replacing a growing transcript with a compact running state so the context doesn't balloon as the task runs long. Used together, they keep the per-step token cost flat instead of climbing, which is the difference between an agent whose economics hold over a long task and one whose bill grows quadratically until it's shut off.
| Technique | Cost effect | Reliability effect |
|---|---|---|
| Prompt caching | ~90% off repeated context | Neutral (same content, cheaper) |
| Compression (LLMLingua) | Up to ~20x fewer tokens | Less filler to get lost in |
| Summarization | Flat history cost over time | Keeps key facts out of the middle |
Bury the key fact in a long prompt and accuracy drops; a bigger model doesn't exempt you, the same degradation holds for long-context models too. (arXiv:2310.05736)
The Pattern Intelligence Layer is where these techniques are applied by pattern instead of by hand. Which context repeats, which is verbose, and which history can be summarized are measurable properties of each workflow, so caching, compression, and summarization land where they pay without guesswork. Reliability at the pattern level means the agent's token cost stays flat and its accuracy stays high as the task runs long, which is the combination that keeps it in production.
Frequently asked questions
Is token optimization just about saving money?
No. The same moves that cut cost also cut the long, unstructured context that degrades accuracy. "Lost in the Middle" shows buried facts get missed, so trimming context helps reliability too.
How much can caching actually save?
Anthropic prices cached reads at about 0.1x standard input, a roughly 90% cut on the repeated portion of every prompt. After the first call, stable context is nearly free.
Does compression hurt the answer?
Minimally when done well. LLMLingua reports up to ~20x compression with only a small accuracy drop, and by removing filler it can even help the model focus on what matters.