Agent Memory Design: Working, Episodic, Semantic and Procedural Memory

Prashant Dhingra

Quick answer

Agent memory design The layered system provides an AI agent with extended, retrievable context that goes beyond a single prompt, typically structured around four cognitive-inspired types: working memory (current session state), episodic memory (specific past events and decisions), semantic memory (durable facts and preferences), and procedural memory The architecture is influenced by four decisions, each tailored to the specific backend for storing and retrieving learned workflows and skills. what to store, how to store it, how to retrieve it, and when Not every agent requires every layer to function effectively; the key is aligning the architecture with the specific use case instead of striving for theoretical completeness that will never be utilized.

SLIDE 02 : WHY STATELESS ISN'T ENOUGH Why stateless language models need designed memory systems

Forgetting is expensive, not just inconvenient

Without memory, every session starts from zero

A big language model possesses great reasoning skills and a wealth of embedded knowledge, yet lacks its own lasting memory : requiring a fresh start with each session unless external context is provided. If ignored, this necessitates continual context injection during each interaction, leading to increased token consumption, delays, and expenses as the conversation or task extends.

As interactions extend, failures such as memory drift, context degradation, and hallucination are more prone to occur due to weakening attention. This underscores the importance of a intentionally designed memory system to combat these challenges.

Four cognitive-inspired layers

Working, episodic, semantic, procedural

Modern memory architectures directly adopt this structure from human cognition as it aligns with how agents truly utilize various types of context, rather than for its elegance.

SLIDE 03 : THE MEMORY HIERARCHY The four-layer agent memory hierarchy: working, episodic, semantic, procedural

Different speed, different lifespan

Recent and fast on top, durable underneath

Working memory functions within the model's context window at the pace of conversation, however it is transient - important information needs to be intentionally transferred to a more enduring layer before the session concludes and it disappears. Episodic, semantic, and procedural memory collectively create long-term memory, with each tailored for distinct types of information and retrieval methods.

Working

Session state includes recent messages and ongoing reasoning, offering fast, temporary storage that expires with the session.

Episodic

Detailed historical occurrences - what transpired, what resolutions were made, what resulted.

Semantic

Solid information and preferences, condensed from various experiences into common knowledge.

Procedural

Acquired workflows and techniques - systematic procedures honed by the agent through experience.

SLIDE 04 : SHORT-TERM / WORKING MEMORY Short-term working memory: session state, low-latency cache, TTL expiry

The agent's immediate workspace

Fast, ephemeral, and gone by design

The current session state, including active task variables, in-progress tool outputs, loop counters, and recent messages, is stored in working memory for quick access within milliseconds. This allows an agent to refer back to previous steps and efficiently coordinate its actions without constantly re-establishing context.

It is intentionally short-lived. A typical production pattern clears working memory based on a time-to-live policy at the end of a session, utilizing a storage system such as Redis for the necessary speed of real-time tasks : any data that should persist beyond this point must be explicitly transferred to a longer-term storage layer.

SLIDE 05 : EPISODIC, SEMANTIC & PROCEDURAL MEMORY The three long-term memory types with concrete examples

Three kinds of "remembering," in practice

What happened, what's true, and how to do it

Episodic memory - 'The user's last session involved updating Artifact X and choosing approach Y.' Semantic memory This user's preferences and insights are gathered from numerous interactions - 'this user favors dark mode and is employed in fintech.' Procedural memory Improves workflows and expertise – a systematic approach to invoice approval, honing processes like a seasoned employee would refine their tasks.

A customer support agent relies heavily on episodic memory to recall ticket history; a product recommendation agent relies heavily on semantic memory to remember specs and relationships; a coding assistant relies heavily on procedural memory to apply learned debugging strategies effectively.

Where the layers actually live

No single database is right for every layer

In production memory architectures, two or three storage backends are often combined, with each matched to the retrieval pattern required by its corresponding layer.

SLIDE 06 : CHOOSING A STORAGE BACKEND Vector, graph, and SQL storage backends compared for agent memory

Similarity search, relationships, or hard facts

Match the backend to the query it needs to answer

Vector databases excel in semantic similarity but struggle with multi-hop reasoning. Graph databases excel at quickly traversing relationships, making them well-suited for episodic and procedural memory tasks. SQL and Postgres remain the dependable choice for long-term, auditable, ACID-compliant data that must be provably correct.

Backend	Strongest at	Weakest at	Typical layer
Vector database	Semantic similarity search	Multi-hop relationships	Semantic
Graph database	Fast relationship traversal	Fuzzy similarity matching	Episodic / procedural
SQL / Postgres	Auditable, ACID-compliant facts	Unstructured similarity search	Semantic (facts)
Key-value cache	Sub-millisecond hot access	Complex querying	Working

SLIDE 07 : FOUR DECISIONS THAT SHAPE THE ARCHITECTURE Four design decisions: what to store, how to store, how to retrieve, when to forget

The questions that come before any code

What, how, how, and when

What to store : determining which details are worthy of preservation is a crucial design decision, not an afterthought, as not every detail is deserving of persistence.
How to store it - unprocessed content, an organized overview, or an encoded representation, selected for future searchability
How to retrieve it : fuzzy relevance measured through vector similarity, relationships traversed via graph traversal, known facts retrieved directly through lookup
When to forget it : memories that linger indefinitely eventually tarnish all recollections stored beside them

Keeping the archive usable

A memory system that only grows becomes a liability

The balance between storage and inference cost is at the core of this issue: a complete, unfiltered history increases both token cost and retrieval noise as it expands.

SLIDE 08 : SUMMARIZATION, FORGETTING & CONFLICT RESOLUTION Memory management techniques: summarization, consolidation, forgetting, conflict resolution

Memory management techniques: summarization, consolidation, forgetting, conflict resolution

Forgetting isn't a bug : it's a feature

Three techniques that keep memory from becoming noise

Summarization summarizes discussions with an LLM by continuously updating a summary with new information instead of keeping every single conversation. Forgetting mechanisms Allowing low-value memories to naturally fade through temporal decay, relevance scoring, or user-defined policies can actually enhance the quality of signals by preventing the retention of unnecessary information. Conflict resolution manages situations where new data contradicts existing information, determining which version to trust moving forward instead of simply retrieving both.

SLIDE 09 : MATCH THE ARCHITECTURE TO THE USE CASE Summary: matching memory architecture to the actual use case, with proper scoping

Not every agent needs every layer

Build for the queries you'll actually run

Not every agent requires all four memory types. A customer support agent uses episodic memory for ticket history, while a product recommendation agent relies on semantic memory for product details and connections. Introducing unnecessary complexity for a hypothetical need creates technical debt.

The importance of scoping is equal to that of structure: memory designated for a user, agent, session, or organization allows for true personalization while maintaining proper data segregation : a crucial aspect when multiple agents or customers access the same memory store.

Frequently asked

Agent memory design FAQ

Working memory is made up of the current session state and temporary information, while episodic memory consists of specific past events and decisions. Semantic memory encompasses durable facts and preferences, and procedural memory involves learned workflows and skills. Together, episodic, semantic, and procedural memory make up an agent's long-term memory, contrasting with the short-term nature of working memory.

The effectiveness of database systems varies based on the query pattern. Vector databases are proficient in semantic similarity search but struggle with multi-hop relationships. In contrast, graph databases quickly navigate relationships and are optimal for episodic and procedural memory. To ensure accuracy and auditability, many production systems utilize a combination of both databases, along with a SQL store for verifiable facts.

Unfiltered memories that continuously accumulate can drive up token cost and diminish retrieval quality, as outdated memories can overshadow the relevance of current information. Implementing deliberate forgetting techniques, like temporal decay or relevance scoring, helps maintain a high signal-to-noise ratio in the memory store as it expands.

Episodic memory captures individual moments in time - a specific encounter, choice, or result, connected to a particular time and setting. Semantic memory, on the other hand, contains the overarching information gleaned from numerous episodes - enduring facts and preferences that no longer rely on recalling the exact circumstance in which they were learned.

The appropriate architecture aligns with the specific needs of each user: a customer support agent relies on episodic memory for ticket history, a recommendation agent relies on semantic memory for product knowledge, and a coding assistant relies on procedural memory for debugging strategies. Including memory types that will not be accessed by the agent only adds unnecessary complexity.

By explicitly scoping memory to individual users, agents, sessions, or organizations instead of using a single undifferentiated store, personalization and cross-agent context sharing can be facilitated as needed, while also preventing context bloat and privacy breaches.

Get started

Ready to design your agent's memory system?

Before selecting a storage backend, begin by addressing the four design questions - what, how, how, and when to forget - and only build the memory layers necessary for your specific use case.

A memory-type selection worksheet mapped to your agent's use case
A storage-backend decision guide: vector, graph, SQL, and cache
A forgetting-policy template using decay, relevance, and conflict resolution

Revisit storage backends → Back to memory management