Long-Term Memory with Qdrant

Every enterprise agent operates with a fixed context window, typically 128K to 200K tokens: but enterprise relationships, workflows, and institutional knowledge accumulate over months and years. An agent that forgets everything between sessions cannot build trust, learn from past mistakes, or maintain continuity across a customer journey. This lesson builds a production-grade long-term memory system using Qdrant, giving your agent network episodic recall that persists indefinitely and retrieves selectively.

Why Context Windows Are Not Enough

The fundamental mismatch is between context window capacity (hundreds of thousands of tokens, costing money per request) and enterprise memory requirements (millions of past interactions spanning years). You cannot stuff all prior conversations into every prompt, the cost would be prohibitive and the noise would overwhelm the signal. What you need is semantic retrieval: given the current context, find the five most relevant past memories and inject only those.

flowchart TD
    S["New Session Start\ncurrent context embedding"] --> VS["Vector Search\ncosine similarity in Qdrant"]
    VS --> R["Top-5 Relevant Memories\nepisodic + semantic"]
    R --> P["Prompt Assembly\nsystem prompt + memories + user message"]
    P --> AG["Agent LLM Call"]
    AG --> RE["Agent Response"]
    RE --> MW["Async Memory Write\n(non-blocking)"]
    MW --> MS["Memory Store\nQdrant"]

    style S fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style VS fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style R fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style P fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style AG fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style RE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style MW fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style MS fill:#f0fdf9,stroke:#0D9488,color:#0F172A

Qdrant Setup and HNSW Index

Create a collection to hold the three classical memory types: episodic (specific past events), semantic (generalized knowledge derived from episodes), and procedural (learned workflows and tool-use patterns). Each memory is stored as a Qdrant point: a vector plus a JSON payload carrying everything else.

from qdrant_client import AsyncQdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff, PayloadSchemaType,
)

client = AsyncQdrantClient(url="http://localhost:6333")

await client.create_collection(
    collection_name="agent_memories",
    vectors_config=VectorParams(
        size=1536,              # text-embedding-3-small dimensions
        distance=Distance.COSINE,
    ),
    # HNSW is Qdrant's default index. m=16 / ef_construct=64 are solid
    # defaults up to ~10M vectors; tune m upward for larger datasets.
    hnsw_config=HnswConfigDiff(m=16, ef_construct=64),
)

# Payload indexes make agent_id / memory_type / created_at / importance
# filters fast, the same job Postgres partial indexes would do.
await client.create_payload_index("agent_memories", "agent_id", PayloadSchemaType.KEYWORD)
await client.create_payload_index("agent_memories", "memory_type", PayloadSchemaType.KEYWORD)
await client.create_payload_index("agent_memories", "created_at", PayloadSchemaType.DATETIME)
await client.create_payload_index("agent_memories", "importance", PayloadSchemaType.FLOAT)

# Each memory point's payload shape:
#   agent_id: str, session_id: str, memory_type: "episodic"|"semantic"|"procedural",
#   content: str, importance: float (0.0-1.0, set by the scoring pipeline),
#   access_count: int, created_at: ISO datetime, last_accessed_at: ISO datetime,
#   metadata: dict

Memory Retrieval with Importance and Recency Scoring

Raw cosine similarity ranks memories by semantic closeness but ignores whether the memory is important or recent. A composite score, similarity * importance * recency_decay, produces far more useful retrieval results. Recency decay follows an exponential curve: a memory from yesterday decays little; one from two years ago decays significantly unless it was accessed frequently (indicating it is repeatedly relevant).

import numpy as np
from datetime import datetime, timezone
from qdrant_client.models import Filter, FieldCondition, MatchValue

DECAY_RATE = 0.0002  # Controls how fast importance decays; tune per domain

def recency_score(created_at: datetime, last_accessed: datetime) -> float:
    """
    Exponential recency decay based on days since last access.
    Frequent access resets the clock: a memory that keeps being
    retrieved stays fresh even if it was first created years ago.
    """
    now = datetime.now(timezone.utc)
    days_since_access = (now - last_accessed).days
    # e^(-rate * days) gives 1.0 for brand-new memories, approaching 0 over time
    return float(np.exp(-DECAY_RATE * days_since_access))

async def retrieve_memories(
    client: AsyncQdrantClient,
    agent_id: str,
    query_embedding: list[float],
    top_k: int = 5,
    memory_type: str | None = None,
) -> list[dict]:
    """
    Retrieve the top-k most relevant memories using a composite score.
    query_points uses the HNSW index to get a candidate set; we re-rank
    in Python with the composite score. This is the standard two-stage
    retrieval pattern: fast approximate search, then precise re-rank.
    """
    must = [FieldCondition(key="agent_id", match=MatchValue(value=agent_id))]
    if memory_type:
        must.append(FieldCondition(key="memory_type", match=MatchValue(value=memory_type)))

    result = await client.query_points(
        collection_name="agent_memories",
        query=query_embedding,
        query_filter=Filter(must=must),
        limit=50,               # fetch 50 candidates, re-rank to top_k
        with_payload=True,
    )

    # Re-rank candidates with the composite importance/recency score.
    # point.score is the cosine similarity Qdrant already computed.
    scored = []
    for point in result.points:
        p = point.payload
        recency = recency_score(
            datetime.fromisoformat(p["created_at"]),
            datetime.fromisoformat(p["last_accessed_at"]),
        )
        composite = point.score * p["importance"] * recency
        scored.append({"id": point.id, **p, "composite_score": composite})

    scored.sort(key=lambda r: r["composite_score"], reverse=True)
    top = scored[:top_k]

    # Update access bookkeeping for retrieved memories (tracks usage for
    # decay). set_payload patches just these fields without re-uploading
    # the vector. last_accessed_at is identical across the batch, so it
    # goes in one call; access_count differs per point.
    now_iso = datetime.now(timezone.utc).isoformat()
    await client.set_payload(
        collection_name="agent_memories",
        payload={"last_accessed_at": now_iso},
        points=[r["id"] for r in top],
    )
    for r in top:
        await client.set_payload(
            collection_name="agent_memories",
            payload={"access_count": r["access_count"] + 1},
            points=[r["id"]],
        )

    return top

Async Memory Writes to Avoid Blocking Agents

Writing a memory, embedding generation plus an upsert, takes 200–500ms. If you do this synchronously after every agent response, you add that latency to every user-facing turn. Instead, fire-and-forget to a background task queue. The agent returns its response immediately; the memory write happens asynchronously.

sequenceDiagram
    participant U as User
    participant AG as Agent
    participant Q as Async Queue
    participant MW as Memory Writer
    participant DB as Qdrant

    U->>AG: Send message
    AG->>DB: Retrieve top-5 memories (fast read)
    AG->>U: Return response (fast path)
    AG->>Q: Enqueue memory write (non-blocking)
    Note over AG,U: User sees response immediately
    Q->>MW: Dequeue write task
    MW->>MW: Generate embedding
    MW->>DB: Upsert new memory point + score importance

    style U fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style AG fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style Q fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style MW fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style DB fill:#f0fdf9,stroke:#0D9488,color:#0F172A

Memory Consolidation Pipeline

Left unchecked, episodic memory grows without bound and retrieval quality degrades as low-signal memories crowd out important ones. Consolidation is the process of compressing many related episodic memories into a single semantic memory, analogous to how human long-term memory generalizes from specific experiences to abstract knowledge.

Run a nightly consolidation job per agent: cluster episodic memories from the past 30 days by semantic similarity (k-means on embeddings), then for each cluster with more than ten members and average importance below 0.4, summarize the cluster content using an LLM and store the summary as a single new point with memory_type: "semantic" and recalculated importance. Mark the source episodic memories with a consolidated: true payload field via set_payload, and exclude them from future retrieval with a filter condition. This keeps your active memory set lean and highly relevant without destroying history.

Importance scoring itself warrants explicit logic: assign high importance (0.8–1.0) to memories that contained explicit user corrections, novel information not seen in prior sessions, or decisions that changed agent behavior. Assign low importance (0.1–0.3) to routine confirmations, repeated queries, and small talk. A simple classifier, even a rules-based one, applied at write time is sufficient for most enterprise domains.

In the next lesson, we leave the memory layer and tackle enterprise API integration: connecting your agent network to Salesforce CRM, SAP ERP, and other enterprise platforms using typed clients, OAuth 2.0 credential management, and resilient retry logic.