Memory: Short-Term & Long-Term

Why Agents Need Memory

A stateless agent is like starting fresh with every interaction. It knows nothing about the user, nothing about what it’s tried before, and nothing about patterns it’s encountered. For short, single-turn tasks, this is fine. For anything longer or more sophisticated, it’s a serious limitation.

Consider a customer support agent: a user comes back with a follow-up question about an issue they raised last week. A stateless agent has no idea who they are or what was resolved. A memory-enabled agent can retrieve the prior conversation and respond in context.

Or consider an agent that runs complex research tasks: if it’s encountered a particular data source that’s consistently unreliable, shouldn’t it remember that and avoid it? That’s experience, a form of long-term memory.

Memory transforms agents from ephemeral tools into systems that improve and personalise over time.

The Three Types of Agent Memory

1. In-Context Memory

This is the simplest: everything currently in the context window. The conversation history, tool results, reasoning traces, all immediately accessible to the model.

Pros: Zero latency, no extra infrastructure, the model can reason directly over everything it sees.

Cons: Limited by context window size. Everything is lost when the session ends. Can become expensive as history grows.

This is suitable for: single sessions, short conversations, tasks with bounded scope.

2. External Short-Term Memory

A database or cache that persists for a session or a few days. The agent reads and writes to it explicitly (via a tool). It’s not in the context window, so the agent needs to actively query it.

Typical implementation: Redis, SQLite, or a simple file. The agent has read_memory(key) and write_memory(key, value) tools.

Pros: Unlimited size, persists across context resets, structured storage.

Cons: Adds latency (a database read on every access), the agent must know what to look up.

This is suitable for: multi-session tasks, storing intermediate work across a long pipeline.

3. Long-Term Episodic Memory

A vector database that stores experiences, past conversations, solutions, patterns, indexed by semantic meaning. When the agent needs to remember something, it embeds its query and retrieves the most semantically similar stored memories.

Pros: Can store thousands of past experiences, automatically surfaces relevant ones, enables continuous learning.

Cons: More infrastructure, requires embedding model, retrieval isn’t always perfect.

This is suitable for: personalisation, customer support agents, any agent that should “get better over time.”

Implementing Vector Memory with Qdrant

Qdrant is a fast, production-grade vector database with a Python client that runs equally well embedded on disk (zero infrastructure, perfect for getting started) or against a hosted server when you’re ready to scale. Here’s how to set up semantic memory:

import uuid
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
from openai import OpenAI

client = OpenAI()

# Local, embedded mode: stored in ./qdrant_data/, no server required.
# Swap to QdrantClient(url="http://localhost:6333") once you outgrow this.
qdrant = QdrantClient(path='./qdrant_data')

# Qdrant needs the collection's vector shape declared up front:
# text-embedding-3-small produces 1536-dimensional vectors, compared by cosine distance.
if not qdrant.collection_exists('agent_memory'):
    qdrant.create_collection(
        collection_name='agent_memory',
        vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
    )

Now define the two fundamental operations: store and retrieve:

def store_memory(content: str, metadata: dict = None):
    """Store a piece of text as a memory with vector embedding."""
    
    # Create the embedding: a vector that represents the meaning
    embed_response = client.embeddings.create(
        model='text-embedding-3-small',
        input=content
    )
    vector = embed_response.data[0].embedding   # list of ~1536 floats
    
    # Store in the vector database. Qdrant point IDs must be an unsigned
    # int or a UUID, so we generate a fresh UUID per memory rather than
    # hashing the content. content + metadata both live in the payload.
    qdrant.upsert(
        collection_name='agent_memory',
        points=[PointStruct(
            id=str(uuid.uuid4()),
            vector=vector,
            payload={'content': content, **(metadata or {})},
        )],
    )
    
    print(f"Stored memory: {content[:80]}...")

Now retrieval: find the most relevant memories for a query:

def retrieve_memories(query: str, n_results: int = 3) -> list[str]:
    """Retrieve the most relevant memories for a given query."""
    
    # Embed the query using the same model
    embed_response = client.embeddings.create(
        model='text-embedding-3-small',
        input=query
    )
    query_vector = embed_response.data[0].embedding
    
    # Find the closest memories by vector distance. Unlike Chroma, Qdrant
    # doesn't error if fewer than n_results points exist yet, so no
    # count() guard is needed, it just returns what's there.
    results = qdrant.query_points(
        collection_name='agent_memory',
        query=query_vector,
        limit=n_results,
    )
    
    # Return just the text content of the memories
    return [point.payload['content'] for point in results.points]

Using Memory in an Agent

Now wire memory into an agent’s tool loop:

memory_tools = [
    {
        "type": "function",
        "function": {
            "name": "store_memory",
            "description": "Save an important insight, fact, or decision to long-term memory for future reference.",
            "parameters": {
                "type": "object",
                "properties": {
                    "content": {"type": "string", "description": "What to remember."},
                    "category": {"type": "string", "description": "Category tag: 'user_preference', 'solution', 'fact', 'failure'"}
                },
                "required": ["content"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "retrieve_memories",
            "description": "Search your long-term memory for information relevant to the current task.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "What to search for."},
                    "n_results": {"type": "integer", "default": 3}
                },
                "required": ["query"]
            }
        }
    }
]

Now your agent can use these tools in its normal loop. Before starting a task, it can retrieve relevant past experience. After completing a task, it can store what it learned.

A practical pattern: prepend relevant memories to the context automatically:

def agent_with_memory(user_message: str) -> str:
    """Run agent with automatic memory injection."""
    
    # Retrieve relevant past memories
    relevant_memories = retrieve_memories(user_message, n_results=3)
    
    memory_context = ''
    if relevant_memories:
        memory_context = "## Relevant past experiences:\n" + \
                         '\n'.join(f"- {m}" for m in relevant_memories) + \
                         "\n\n"
    
    # Inject memories into the system prompt
    system = AGENT_SYSTEM_PROMPT + "\n\n" + memory_context
    
    # Run the agent normally
    result = run_subagent(system, user_message, tools=all_tools)
    
    # Store what we learned this session
    store_memory(
        content=f"Task: {user_message[:200]}\nOutcome: {result[:200]}",
        metadata={'type': 'session_summary'}
    )
    
    return result

Over time, as the agent completes more tasks, it builds up a rich library of past experiences that get automatically surfaced when similar tasks arise.

Exercise: Implement a simple memory-enabled agent that helps with Python debugging. After each debugging session, store: the bug description, what caused it, and how it was fixed. After 5-10 debugging sessions, ask the agent a new bug question and check whether it surfaces relevant past solutions. Does it get better with experience?