Even GPT-4o falls from 99.3% to 69.7% as its context fills toward 32k tokens.Long-context accuracy degrades sharply as the window fills (NoLiMa, arXiv:2502.05167, 2025), and a base64-encoded file is exactly the kind of bulk that fills it. Push a PDF through a tool call and you spend context, money, and accuracy on bytes the model can't even use.
Key facts.
- Base64 inflates by ~33%. Three bytes become four characters, 24 bits mapped to four 6-bit symbols (RFC 4648). Every encoded file is a third bigger before it reaches the model.
- That blob lands in context as tokens. Long-context accuracy falls as the window fills, even for frontier models. The file you stuffed in degrades the reasoning you needed (NoLiMa, arXiv:2502.05167, 2025).
- Even with perfect retrieval, longer context hurts performance. More tokens is not a safe way to move a payload (Context Length Alone Hurts LLM Performance, arXiv:2510.05381, 2025).
- The fix is structural: pass references or handles instead of bytes, the pattern vendor agent frameworks adopt with artifact and file-reference services so binary never transits the model's context (Google ADK Artifacts; OpenAI/Anthropic file handling, 2025).
Why does binary data cost so much in an agent?
Because the channel was built for text. Function arguments and tool results are JSON, and JSON can't hold raw bytes, so a file becomes a base64 string. That string is about a third bigger than the original by the math of the encoding, four characters per three bytes (RFC 4648). Then it gets tokenized and placed in the context window, where it competes with the actual task for space and is billed per token. A two-megabyte attachment isn't a two-megabyte problem, it's a larger, token-priced one. And the model gains nothing from seeing the bytes. It can't meaningfully read an encoded PDF. It just pays to carry it.
What actually fails when payloads get large?
Three things, in order. Cost: the encoded blob is billed as input tokens on every turn it stays in context, so a few large tool results can multiply the bill of an otherwise cheap run. Capacity: the blob crowds out the instructions, history, and tool definitions the agent needs, so quality drops or the call overflows the window. Behavior: handed a giant encoded string as an observation, the model often tries to process it, which stalls the loop or produces nonsense. The symptom looks like a slow or confused agent. The cause is that you asked a text reasoner to hold a binary file in working memory.
How do you handle files without paying the tax?
Keep the bytes out of the model. Store the file once and pass a stable reference, an id, URL, or handle, as the tool argument or result, and let the runtime resolve the actual bytes when a tool needs them. Route genuinely multimodal work to a vision or document model through a dedicated path rather than stuffing base64 into a text tool. When a tool must return data, return metadata plus a short summary plus an optional fetch tool, not the raw payload. For uploads, write to storage first and hand the agent the location. The agent then reasons over compact references, and the file moves through infrastructure built to carry it.
Four ways to move a file through an agent
| Approach | What the model holds | Cost and context impact |
|---|---|---|
| Base64 in arguments/results | The whole encoded file | +33% size, billed per token, crowds the window |
| Inline blob in every turn | Payload persists in history | Repeated token cost each turn |
| Reference id or handle | A short string | Negligible; bytes resolved by the runtime |
| Metadata + summary + fetch tool | The gist, fetch on demand | Small; full payload only when needed |
| Dedicated multimodal path | Nothing in the text channel | No base64 tax |
The pattern is that the tool-calling channel is text, so binary pushed through it's taxed by encoding, by tokens, and by the context it displaces, none of which buys the agent any understanding. Pass references and resolve bytes in infrastructure, and the agent reasons over meaning while the file travels a path built for it. Designing the agent-to-tool boundary so it carries the right thing, a handle rather than a megabyte, is reliability at the pattern level, which is what VibeModel builds as the Pattern Intelligence Layer.
Frequently asked questions
Why not just raise the context window?
A bigger window still bills the encoded blob per token and still wastes capacity on bytes the model can't use. References remove the cost instead of paying more for it.
What about images the model genuinely needs to see?
Send those through the dedicated multimodal path the provider offers, not as base64 inside a text tool result. Keep the text tool channel for references and summaries.
How big is too big?
Any payload large enough to be a noticeable fraction of your context budget. When in doubt, return a reference plus a summary and let a tool fetch the full bytes only if a later step needs them.