Context Window Management

The Context Window Is Like RAM

Your computer’s RAM holds currently-active data. When RAM fills up, the computer slows to a crawl (swapping to disk) or crashes. The context window is similar: it’s the agent’s working memory, holding everything it’s currently thinking about.

A long-running agent task generates a lot of text:

• System prompt: ~500 tokens
• User goal: ~100 tokens
• Tool call 1 and result: ~500 tokens
• Tool call 2 and result: ~2,000 tokens (web search results are verbose)
• Tool call 3 and result: ~1,500 tokens
• … after 20 tool calls: easily 30,000-50,000 tokens

For a 128k context window, this is manageable. But for longer tasks, multi-day pipelines, or models with smaller windows, context overflow is a real failure mode.

The solution is context management, actively monitoring and trimming context before it hits the limit.

Tracking Token Usage

First, always track how much context you’re using:

import tiktoken
from openai import OpenAI

client = OpenAI()
enc = tiktoken.encoding_for_model('gpt-5.6-sol')

def count_tokens(messages: list[dict]) -> int:
    """Count tokens in a messages array."""
    total = 0
    for msg in messages:
        # Each message has overhead tokens (role, formatting)
        total += 4
        
        content = msg.get('content', '')
        if isinstance(content, str):
            total += len(enc.encode(content))
        elif isinstance(content, list):
            # Multi-modal content (text + images)
            for block in content:
                if block.get('type') == 'text':
                    total += len(enc.encode(block['text']))
    
    return total

Use this before every API call to check headroom:

MAX_TOKENS      = 128_000
SAFETY_MARGIN   = 10_000   # always leave room for the response

def check_context_headroom(messages: list[dict]) -> int:
    used = count_tokens(messages)
    available = MAX_TOKENS - SAFETY_MARGIN - used
    print(f"Context: {used:,} / {MAX_TOKENS:,} tokens ({available:,} remaining)")
    return available

When available drops below a threshold (say, 20,000 tokens), trigger compression.

Strategy 1: Conversation Summarisation

The most effective technique: periodically compress older messages into a summary.

def summarise_history(messages: list[dict], keep_last: int = 4) -> list[dict]:
    """
    Compress older messages into a summary.
    Keeps the system prompt + last N messages intact.
    Summarises everything in between.
    """
    if len(messages) <= keep_last + 1:
        return messages   # nothing to summarise
    
    system_msg = messages[0]       # always keep system prompt
    recent = messages[-keep_last:] # always keep recent messages
    to_compress = messages[1:-keep_last]  # compress the middle
    
    if not to_compress:
        return messages

Now ask the LLM to summarise the middle section:

    compress_text = '\n'.join([
        f"{m['role'].upper()}: {m.get('content', '')[:500]}"
        for m in to_compress
        if isinstance(m.get('content'), str)
    ])
    
    summary_response = client.chat.completions.create(
        model='gpt-5.6-luna',   # use cheaper model for summarisation
        messages=[{
            'role': 'user',
            'content': f"""Summarise this conversation history concisely.
Preserve all key facts, decisions, tool results, and progress made.
Be specific: include numbers, names, and conclusions.

HISTORY:
{compress_text}

Provide a 3-5 sentence summary:"""
        }],
        max_tokens=300,
        temperature=0.0,
    )
    
    summary_text = summary_response.choices[0].message.content
    
    summary_message = {
        'role': 'system',
        'content': f'[Compressed history summary]\n{summary_text}'
    }
    
    return [system_msg, summary_message] + recent

Now integrate this into your agent loop:

def agent_with_context_management(user_message: str) -> str:
    messages = [
        {'role': 'system', 'content': AGENT_SYSTEM_PROMPT},
        {'role': 'user',   'content': user_message},
    ]
    
    while True:
        # Check context before each call
        if check_context_headroom(messages) < 20_000:
            print("⚠ Context getting full, compressing history...")
            messages = summarise_history(messages, keep_last=6)
        
        response = client.chat.completions.create(
            model='gpt-5.6-sol',
            messages=messages,
            tools=tools,
        )
        
        msg = response.choices[0].message
        messages.append(msg)
        
        if not msg.tool_calls:
            return msg.content
        
        # Handle tool calls...

Strategy 2: Selective Retrieval Instead of Full History

For agents that accumulate large amounts of past context (multi-session, knowledge-base agents), instead of loading everything, load only what’s relevant:

def build_selective_context(query: str, all_sessions: list[dict]) -> str:
    """Retrieve only relevant past sessions for this query."""
    
    # Embed the current query
    query_embedding = client.embeddings.create(
        model='text-embedding-3-small',
        input=query
    ).data[0].embedding
    
    # Score each past session by relevance
    def relevance_score(session: dict) -> float:
        session_embedding = session.get('embedding', [])
        if not session_embedding:
            return 0.0
        # Cosine similarity between query and session
        import numpy as np
        a, b = np.array(query_embedding), np.array(session_embedding)
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
    
    # Sort by relevance and take top 5
    scored = sorted(all_sessions, key=relevance_score, reverse=True)
    top_sessions = scored[:5]
    
    # Build a compact context string
    context_parts = []
    for s in top_sessions:
        context_parts.append(f"[Past session {s['date']}]: {s['summary']}")
    
    return '\n'.join(context_parts)

Strategy 3: Chunking Large Tool Results

Sometimes a single tool returns enormous results (reading a 10,000-word article, scraping a large page). Truncate or chunk before adding to context:

def trim_tool_result(result: str, max_tokens: int = 2000) -> str:
    """Trim a tool result to fit within a token budget."""
    tokens = enc.encode(result)
    if len(tokens) <= max_tokens:
        return result
    
    # Truncate and add a note
    trimmed = enc.decode(tokens[:max_tokens])
    return trimmed + f"\n\n[Result truncated. Full content was {len(tokens)} tokens.]"

Apply this when adding tool results to the messages array:

messages.append({
    'role':         'tool',
    'tool_call_id': tool_call.id,
    'content':      trim_tool_result(json.dumps(result), max_tokens=2000)
})

A web search result with 50 articles is probably 30,000 tokens. Trimming to 2,000 per result keeps the most relevant content while keeping your context budget under control.

Exercise: Implement the token tracking function and add it to an agent you built in a previous module. Run a long task (ask the agent to research 10 topics in sequence) and log the token count before each API call. At what step does it start getting close to the limit? Now add the summarisation strategy and compare: can you run the same 10-topic task without hitting the limit?