Agent Systems · 21-c04
26 June 2026 · Memory architecture
Agent memory is not endless chat history. When designed well, it is a read/write system for active tasks, past experiences, world knowledge, user preferences, and safety boundaries.
- Misread
- history != memory
- Core
- write / read / update
- Value
- continuity + personalization
- Risk
- stale + poisoned memory
Key terms in this post
- working memory
- episodic memory
- semantic memory
- procedural memory
- preference memory
- memory retrieval
- forgetting
- memory poisoning
A memory-enabled agent sounds simple: store conversations and bring them back later. In production, the questions multiply quickly: what should be stored, who may write, when should it be forgotten, and what happens when two memories conflict?
This post treats memory as engineering surface: storage, retrieval, update, forgetting, permission, and evaluation.
What is memory?
Agent memory is selected information carried from the past into future decisions.
Chat history is raw material; memory is the policy that decides what is stored, summarized, retrieved, and excluded.
long-term stores
working memory
Active goals, observations, temporary variables, and retrieved context.
The current desk. Only this reaches the model.
retrieve → relevant, permitted records move onto the desk.
← write new experience, inference, and preference flow back.
Working memory
Working memory is the agent's current desk: active goal, latest tool result, temporary plan, and parsed fields.
The prompt is only the slice shown to the model; working memory can persist across LLM calls.
Bad working memory can make good long-term memory dangerous.
working memory · full state
active_goal
Resolve the missing shipment request.
latest_observation
Shipping API: package stuck for three days.
retrieved_context
Customer prefers concise answers.
pending_decision
Offer refund instead of coupon?
guardrail
Refund requires human approval.
slice sent to model (prompt)
Resolve the customer's missing-shipment request. Per the shipping API the package has been stuck for three days. The customer prefers short answers. A refund needs human approval.
The controller moves only what this step needs into the prompt, not every field. Ids and raw logs stay in state.
Episodic memory
Episodic memory is the agent's experience log.
Generative Agents uses a memory stream retrieved by relevance, recency, and importance.
Reflection can make memory useful, but it can also fossilize wrong interpretations.
score = relevance × recency × importance
Yesterday: same test broke on timezone
high relevance + recency
Last month: a refund was denied
high importance
6 months ago: a routine shipping delay
low recency
Routine delivery notification
low importance
Semantic memory
Semantic memory stores knowledge rather than events.
Read-only corpora and agent-learned facts should be separated because they do not have the same trust level.
Good semantic memory can cite where the fact came from.
read-only corpus
Catalogs, policies, runbooks.
high trust · cite sourcelearned facts
Inferences from episodes with evidence pointers.
hypothesis · verify with evidencetenant scope · applies to both
Retrieval must respect authorization.
freshness · applies to both
Old facts may be stale even when well formatted.
Preference memory
Preference memory stores how a user wants to interact.
Preferences drift; recent shifts and long-term tendencies should both matter.
Preference is not authorization.
t1–t3: short, bulleted · t4: wants detail and examples
response policy
Explain deeply, keep sections scannable.
Memory pipeline
Read memory as memory writing, memory reading, and memory management.
MemGPT separates main context from external context; stored information does not automatically enter the prompt.
The best memory systems are selective.
Message, tool result, error, approval, or event arrives.
Choose memory type.
Add embedding and metadata.
Select by relevance, recency, importance, and policy.
Move only permitted fragments into working memory.
Summarize, decay, merge, resolve, or delete.
Stored does not mean in the prompt; only retrieve + inject cross the boundary.
Memory poisoning
Memory poisoning changes future behavior by inserting malicious records into long-term memory or a RAG knowledge base.
The base model need not be modified; retrieval can bring the malicious record back later.
Defense needs write-time validation, source trust, tenant isolation, retrieval monitoring, human review, and deletion paths.
t0 · attacker writes
inject
Attacker adds a targeted record.
the record waits silently
index
The record enters the normal index.
t1 · innocent query triggers
trigger
A later query retrieves it.
condition
The model trusts the context.
effect
Wrong answer or risky action follows.
The danger is not in today's prompt; it was written yesterday and detonates via retrieval tomorrow.
Design contract
The right question is not how much the agent should remember, but what kind of memory lives under which purpose, permission, duration, confidence, deletion right, and eval.
Memory makes the product more capable and more accountable.
My production rule: type memory first, retrieve second, inject third.
type
Is every record classified?
scope
Are tenant and user boundaries stored?
evidence
Do learned facts point to raw records?
freshness
Do old records decay or expire?
consent
Can users inspect and delete memory?
security
Are writes validated by source trust?
eval
Are recall, precision, stale harm, and poisoning tested?
Sources
- S1Sumers et al., Cognitive Architectures for Language Agents, TMLR 2024
- S2Park et al., Generative Agents, UIST 2023
- S3Zhong et al., MemoryBank, arXiv 2023
- S4Packer et al., MemGPT, arXiv 2023
- S5Zhang et al., Survey on Memory Mechanisms of LLM-based Agents, arXiv 2024
- S6Sun et al., Preference-Aware Memory Update, arXiv 2025
- S7Chen et al., AGENTPOISON, NeurIPS 2024
- S8Zou et al., PoisonedRAG, USENIX Security 2025