Context & Memory for Long Sessions

A voice call doesn’t reset between turns the way a fresh chat session might feel like it does. A caller who mentioned their account number at minute two expects the agent to still know it at minute eleven, and every turn of this course’s pipeline has been built around a latency budget that gets harder to protect the longer a call runs. This final lesson of Module 3 is about the tension between those two facts: conversations need memory, and memory costs both time and money in a system billed and budgeted per turn.

Why context growth hurts more in real-time voice than in text chat

In a text chat product, a growing context window is mostly a cost concern, users tolerate a slightly slower response on message 40 of a long thread without much complaint, since they’re already reading at their own pace. In a voice pipeline, context growth is a latency concern first and a cost concern second, because every extra thousand tokens of conversation history re-sent on each turn adds real, felt milliseconds to time-to-first-token, directly eating into the sub-second budget established back in Lesson 2. A call that opens with 200ms LLM latency on turn one can drift to 600-800ms by turn thirty if context is left to grow unbounded, and that drift happens gradually enough that it’s easy to miss without the tracing infrastructure Module 7 covers.

Cost compounds the same way. Most LLM APIs bill per input token, and re-sending the full conversation history on every turn means a 30-turn call’s total token spend grows roughly quadratically with turn count, not linearly, since each new turn’s request includes every prior turn’s content again. For a high-volume support line handling thousands of calls a day, unmanaged context growth turns a call that should cost a few cents in LLM spend into one that costs several times that, purely from re-transmitted history.

Sliding window plus summarization

The standard fix is a two-tier memory structure: keep the most recent N turns verbatim (they’re the most likely to be referenced directly, and keeping them exact preserves nuance), and compress everything older into a compact running summary that gets prepended to every request instead of the full history.

flowchart TB
    NEW["New caller turn"] --> CTX["Assemble context:\nsummary + last N turns"]
    CTX --> LLM["Dialogue manager\nclaude-sonnet-5 / GPT-5"]
    LLM --> RESP["Response to caller"]
    RESP --> APPEND["Append turn to\nrecent-turns buffer"]
    APPEND --> CHECK{"Buffer exceeds\nN turns?"}
    CHECK -->|yes| SUMMARIZE["Summarize oldest turns\ninto running summary"]
    CHECK -->|no| STORE["Persist state\n(session store)"]
    SUMMARIZE --> STORE

    style NEW fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CTX fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style LLM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style RESP fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style APPEND fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style CHECK fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SUMMARIZE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style STORE fill:#fff7ed,stroke:#f59e0b,color:#0F172A

The summarization step itself is a small, cheap LLM call (a short prompt like “compress this conversation segment into 2-3 sentences preserving any account details, stated issues, and decisions made”) run off the critical path, typically triggered asynchronously once the buffer crosses its threshold rather than blocking the caller’s next turn. This is the detail that makes the pattern viable in real time: summarization happens in the background between turns, never in the foreground of the turn the caller is waiting on.

MAX_VERBATIM_TURNS = 6

async def assemble_context(session: SessionState) -> list[dict]:
    messages = []
    if session.running_summary:
        messages.append({
            "role": "system",
            "content": f"Conversation so far: {session.running_summary}",
        })
    messages.extend(session.recent_turns[-MAX_VERBATIM_TURNS:])
    return messages


async def maybe_summarize(session: SessionState):
    if len(session.recent_turns) <= MAX_VERBATIM_TURNS:
        return
    to_compress = session.recent_turns[:-MAX_VERBATIM_TURNS]
    session.recent_turns = session.recent_turns[-MAX_VERBATIM_TURNS:]

    summary_response = await llm.messages.create(
        model="claude-sonnet-5",
        max_tokens=150,
        messages=[{
            "role": "user",
            "content": (
                f"Existing summary: {session.running_summary or 'none'}\n"
                f"New turns to fold in: {format_turns(to_compress)}\n\n"
                "Update the summary in 2-3 sentences, preserving any "
                "account details, stated issues, and decisions made."
            ),
        }],
    )
    session.running_summary = summary_response.text
    await session_store.save(session)  # off critical path, fire-and-forget

Losing information by dropping old turns outright, instead of summarizing them, is the failure mode to watch for here. An account number given in turn 3 has to still be recoverable in turn 40, or the caller experiences the single most trust-eroding moment a long support call can produce: being asked to repeat something they already said.

Persisting state across reconnects

Mobile networks drop. A caller’s call can survive a brief network interruption at the transport layer (Module 5 covers reconnect logic for LiveKit and Twilio Media Streams specifically), but only if the conversation’s actual state, the running summary, recent turns, any tool results already gathered, caller or account identifiers, outlives the dropped connection. Keeping session state as an in-memory object tied to a single WebSocket connection means a reconnect is functionally a new, memoryless call; keeping it in an external store (a fast key-value store like Redis, keyed by a stable session or call ID issued at call start) means a reconnected client can rehydrate the exact same context and continue as if nothing happened.

This has a direct cost implication worth planning for: session state needs a retention and cleanup policy, since a support line handling meaningful call volume will otherwise accumulate an unbounded number of stale session records. A time-to-live on the order of the longest realistic hold-and-reconnect window (typically a few minutes) after the last activity on a session is usually enough, paired with a hard cleanup once a call is marked complete.

What this sets up

Everything built across this module, spoken-language prompting, tool calling with filler acknowledgments, and now bounded, persistent memory, forms the complete conversational core the rest of this course assumes is working. Module 4 picks up right where this leaves off, turning the dialogue manager’s streamed text output into natural-sounding, low-latency speech.