Every time someone asks "how do I give my agent memory?" they're actually asking three different questions. The answer to each is different infrastructure.
Working memory is the context window: what the agent is actively thinking about right now. Episodic memory is what happened in past sessions: the specific events, decisions, and outcomes from previous interactions. Semantic memory is factual knowledge: the product catalog, the customer profile, the domain ontology.
These serve different purposes. They have different retrieval patterns. They live on different infrastructure. Building them all the same way — typically, "put it in a vector store" — produces an agent that's good at none of the three.
Working memory
Working memory is the context window. It's finite, fast, and volatile.
The working memory architecture problem is context budget management. Everything in working memory competes for the same token budget. The system prompt, the current user request, retrieved episodic memories, retrieved semantic knowledge, tool outputs — all of it.
The design decisions:
What goes in automatically: System prompt, current user message, the most recent N turns of conversation. These are always in context.
What gets retrieved on demand: Episodic memories relevant to the current request. Semantic facts needed for the current reasoning step. Retrieved on similarity match, not loaded wholesale.
What gets compressed out: Tool outputs beyond the current step. Older conversation turns beyond the sliding window. Replaced by summaries.
Context budget enforcement: Hard budget per component. If tool output exceeds its allocated budget, compress it before it enters context. Track budget utilization per execution. Alert when executions regularly exceed 80% of budget (headroom is eroding).
Episodic memory
Episodic memory is the record of past events. For an agent, that means: what did this user ask before, what did the agent do, what was the outcome, what decisions were made.
The episodic store should be:
- Structured: Each episode has schema'd fields (timestamp, user_id, task_type, summary, outcome, key_decisions)
- Queryable by relevance: Given a current user request, retrieve past episodes that are semantically similar
- Queryable by recency: Retrieve the last N interactions with this user
- Summarized, not verbatim: Episodes are summaries of what happened, not transcripts. The full conversation is archived separately if needed for audit, but the episodic store holds compact, information-dense summaries
Infrastructure: a relational database (PostgreSQL with pgvector extension) is usually enough. Store summaries as text plus embeddings in the same row. Query by user_id for recency; query by embedding similarity for relevance.
When to retrieve: at the start of every user session, retrieve the last N episodes for this user and any past episodes semantically similar to the current request. Inject a compact summary into context.
When to write: at the end of every session, generate a summary of what happened and store it. What was asked, what the agent did, what the outcome was, what key decisions were made.
class EpisodicMemory:
def retrieve_for_session(self, user_id: str, current_request: str, n_recent: int = 5) -> list[Episode]:
recent = self.db.query(
"SELECT * FROM episodes WHERE user_id = ? ORDER BY created_at DESC LIMIT ?",
user_id, n_recent
)
similar = self.vector_search(
embedding=self.embed(current_request),
filter={"user_id": user_id},
limit=3
)
return deduplicate(recent + similar)
def store_episode(self, user_id: str, session: Session) -> None:
summary = self.summarize(session)
embedding = self.embed(summary.text)
self.db.insert("episodes", {
"user_id": user_id,
"summary": summary.text,
"outcome": session.outcome,
"key_decisions": summary.decisions,
"embedding": embedding,
"created_at": now()
})
Semantic memory
Semantic memory is factual knowledge: the things that are true about the domain. Product catalog. Policy documents. Customer records. FAQs. API documentation.
This is what most people mean when they say "RAG." The difference in a well-designed memory architecture is that semantic memory is separate from episodic memory and retrieved differently.
Retrieval pattern: Chunk the knowledge base into semantically coherent units (not arbitrary character counts). Embed each chunk. At query time, embed the current question and retrieve the top-K most similar chunks.
The chunking problem matters. A chunk that cuts across a section boundary — ending mid-sentence in the middle of a definition — is worse than no chunk. Chunk at natural boundaries: paragraphs, sections, list items. Chunk size should match the typical query's scope: too small and you miss context, too large and you dilute the relevant signal.
Reranking: After top-K retrieval by similarity, rerank the results with a cross-encoder model or an LLM-based reranker. Initial embedding similarity is a coarse signal. Reranking improves precision.
Freshness. Semantic memory needs to be updated when the underlying knowledge changes. A product catalog that's updated daily needs daily re-embedding of changed documents. Implement a change detection pipeline rather than full re-embedding every time.
Don't use semantic memory for structured data. If the question can be answered by a database query (what is the price of product X?), query the database. Don't embed the database into a vector store and retrieve it by similarity. Structured data has exact queries. Use them.
The integration: memory triage
At session start, a memory triage step decides what to pull into working memory:
async def prepare_session_context(user_id: str, request: str) -> Context:
# Working memory starts with system prompt + current request
context = Context(system_prompt=SYSTEM_PROMPT, request=request)
# Episodic: recent history + relevant past episodes
episodes = await episodic_memory.retrieve_for_session(user_id, request)
context.add(episodes.to_compact_summary(), budget=2000) # Token budget
# Semantic: relevant domain knowledge
knowledge = await semantic_memory.retrieve(request, top_k=5)
context.add(knowledge, budget=3000)
return context # Total budget consumed, remainder for tool outputs
The triage step is where your budget enforcement lives. It decides what's important enough to be in context and what stays in storage. Get this right and the agent has what it needs without context pressure. Get it wrong and you're back to the problem of an agent reasoning from an overloaded, distorted context.
Memory architecture isn't about giving the agent more to remember. It's about giving it the right things at the right time.