AI Memory

Without memory, an LLM starts every interaction from a blank slate — no recollection of prior turns, no user preferences, no accumulated facts. AI Memory is the umbrella term for the mechanisms that fix this: everything from the transformer’s in-context attention over the current prompt, to external vector and graph databases that let an agent recall something from months ago. As agents have moved from single-turn chat toward long-running, autonomous work, memory architecture has become one of the highest-leverage design decisions in the stack.

Short-Term vs. Long-Term Memory

The most basic split is between what fits inside the model’s context window and what lives outside it.

• Short-term (in-context) memory — the tokens the model can directly attend to: the system prompt, recent turns, and any retrieved snippets injected into the current call. It’s fast and precise, but strictly bounded by the context window, and everything is dropped once the window fills or the session ends.
• Long-term (externalized) memory — information persisted outside the model, typically in a vector database or knowledge graph, and pulled back into context only when relevant. This is what lets an agent remember a user across sessions without paying to re-process the entire history on every call — the same retrieve-then-inject pattern used in Retrieval-Augmented Generation.

Simply expanding the context window (some models now support millions of tokens) doesn’t replace long-term memory: attention cost scales with sequence length, and models reliably show worse recall for facts buried in the middle of a very long context — the “lost in the middle” effect. Externalized memory keeps the working context small and relevant instead.

A Finer-Grained Taxonomy

Borrowing loosely from cognitive science, agent memory is often broken into four functional categories:

How a Retrieval-Based Memory Pipeline Works

flowchart LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef ai fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;
    classDef store fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef result fill:#0D9488,stroke:#0D9488,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    A([New Interaction]) --> B(Chunk + Embed):::ai
    B --> C[(Vector Store)]:::store
    D([Current Query]) --> E(Embed Query):::ai
    E --> F{HNSW Search}:::store
    C -.-> F
    F -->|top-k candidates| G(Rerank):::ai
    G -->|most relevant| H[Context Window]:::result
    H --> I(LLM Reasoning):::ai

1. Ingest & chunk — new text (chat turns, documents) is split into manageable pieces.
2. Embed — each chunk becomes a high-dimensional vector via an embedding model.
3. Index — vectors are stored using an approximate nearest-neighbor structure like HNSW, so search stays fast at scale.
4. Retrieve & rerank — at query time, the closest vectors are pulled back and reordered by relevance before being injected into the prompt.

This is the same pipeline that powers RAG; agent memory is essentially RAG applied to an agent’s own history instead of a static document set.

Frameworks Worth Knowing

• MemGPT — treats the LLM like an OS, with the model itself issuing commands to page information between “RAM” (context) and “disk” (external storage).
• LangChain / LlamaIndex — general-purpose building blocks for chunking, embedding, and memory buffers.
• Zep / Mem0 — dedicated memory-as-a-service layers that handle summarization, embedding, and hybrid search server-side.
• Hermes Agent — bundles persistent, cross-session memory out of the box using full-text search plus LLM summarization.

Key Challenges

• Context pollution — overly broad retrieval pulls in loosely related noise that distracts the model. Tight filtering and reranking mitigate this.
• Staleness and contradiction — facts change over time (“I live in New York” → “I live in London”); naive retrieval can surface both. Systems address this with temporal weighting and active conflict resolution.
• Latency and cost — every retrieval adds an embedding call, a search, and often a rerank step before generation even starts.
• Privacy — persistent memory means persistent storage of potentially sensitive data, requiring strict per-user isolation, encryption at rest, and a real answer for “right to be forgotten” deletion requests.

Forgetting Is a Feature

Not every interaction deserves to be remembered forever. Two common techniques keep memory useful instead of bloated:

• Decay — older or rarely-accessed memories are scored lower over time (e.g. multiplying relevance by an exponential decay factor), so they naturally fall out of retrieval unless uniquely relevant.
• Consolidation — periodically, many small episodic memories are summarized by an LLM into a single dense semantic fact, then archived or discarded — trading granularity for a much smaller, higher-signal memory footprint.

Where It’s Heading

Two directions are gaining traction: continual learning, where lightweight adapters (e.g. LoRA) are updated from a user’s interactions so memory is baked into weights rather than retrieved as text; and active curation, where agents periodically review and reorganize their own memory during idle time rather than only writing to it reactively. Both push memory from a passive lookup table toward something closer to an ongoing, self-maintaining part of the agent’s reasoning loop.

from langchain.memory import VectorStoreRetrieverMemory
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
import faiss

embedding_size = 1536
index = faiss.IndexFlatL2(embedding_size)
vectorstore = FAISS(OpenAIEmbeddings().embed_query, index, InMemoryDocstore({}), {})

retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
memory = VectorStoreRetrieverMemory(retriever=retriever)

memory.save_context({"input": "My favorite color is blue"}, {"output": "Got it."})
memory.save_context({"input": "I am working on an AI project"}, {"output": "Sounds interesting."})

print(memory.load_memory_variables({"prompt": "What color should I use for the chart?"}))