LLM Fundamentals for Agent Builders

The Bridge Between AI Research and Your Code

AI agents are built on top of Large Language Models (LLMs), models like GPT-5.6, Claude, or Gemini. To build reliable agents, you don’t need to understand the deep mathematics of transformers, but you do need to understand the interface between your code and the model: what you send in, what comes out, and the constraints that shape both.

This module covers the four fundamentals that every agent builder must know cold.

1. Tokens: The Currency of LLMs

LLMs don’t read text the way you do. They operate on tokens, chunks of text that are smaller than words but larger than individual characters. The word “learning” might be 1 token. “Supercalifragilistic” might be 4 tokens. “AI” is 1 token. A space followed by a word often counts differently from the word alone.

Why does this matter?

Cost. Every LLM API charges per token, usually in units of “per million tokens.” If you send a huge system prompt on every request, that cost adds up.

Context limits. Every model has a maximum number of tokens it can process at once, the context window. Exceed it and the API returns an error.

Speed. Generating more tokens takes more time. Long outputs = slower responses.

As a rough guide: 1,000 tokens ≈ 750 words. A typical Python function is 20-50 tokens. A code file is hundreds to thousands of tokens.

You can count tokens for OpenAI models using tiktoken:

import tiktoken

enc = tiktoken.encoding_for_model('gpt-5.6-sol')

text = "Hello, I am an AI agent that can search the web and write code."
tokens = enc.encode(text)

print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")

Run this on your system prompt and typical user messages to understand your real token budget.

2. The Context Window: A Fixed-Size Working Memory

The context window is the total amount of text the model can “see” at once, both the input you send and the output it generates. It’s a hard ceiling, not a soft suggestion.

Think of it like a whiteboard. You can write notes, history, and the current problem on the whiteboard. But the whiteboard has a fixed size. Once it’s full, you must erase some old content to write new content.

For short agent tasks, you’ll rarely hit this limit. For long-running tasks with many tool calls and large tool results, you absolutely will. A single web search result can be 2,000-5,000 tokens. After 30-50 tool calls, you can easily exhaust even a 128k context window.

This is why context management (covered in module 7) is one of the critical challenges in agent design.

3. System Prompts: Your Agent’s Identity Card

Every LLM API call has a messages array. The special message with role "system" is the system prompt, instructions that apply to the entire conversation and shape how the model responds.

Think of the system prompt as the “job description” you give the model before it starts working. A well-written system prompt makes your agent dramatically more reliable.

Here’s a system prompt for a research agent:

RESEARCH_AGENT_SYSTEM_PROMPT = """
You are a research assistant that finds information and produces clear summaries.

## Your capabilities:
- Search the web for current information
- Read URLs and extract relevant content
- Write structured summaries in Markdown

## How to behave:
- Always verify claims with at least 2 sources before stating them as facts
- If you cannot find reliable information, say so clearly
- Present conflicting information from different sources when it exists

## Output format:
- Use bullet points for key findings
- Always cite your sources at the end
- Flag unverified claims with "(unverified)"

## Constraints:
- Do not make up information you cannot find
- Do not provide medical, legal, or financial advice
"""

A good system prompt answers: who are you, what can you do, how should you behave, what format should you use, and what are your limits?

Making an API call with this system prompt:

from openai import OpenAI

client = OpenAI()   # reads OPENAI_API_KEY from environment

response = client.chat.completions.create(
    model='gpt-5.6-sol',
    messages=[
        {'role': 'system',  'content': RESEARCH_AGENT_SYSTEM_PROMPT},
        {'role': 'user',    'content': 'What are the top 3 AI agent frameworks in 2024?'},
    ],
    temperature=0.1,   # lower = more consistent, less creative
)

print(response.choices[0].message.content)

temperature=0.1 is important for agents. Higher temperatures (like the default 0.7) introduce more randomness: good for creative writing, terrible for code execution and structured output. Keep agents at 0.0-0.2 for predictable, reliable behaviour.

4. Structured Output: From Free Text to Parseable Data

When an agent calls a tool, it needs to produce a specific function name and specific parameter values. Free-form text is ambiguous and hard to parse reliably.

Modern LLM APIs offer structured output (or “JSON mode”), a way to guarantee the model produces valid JSON that follows a specific schema.

Here’s how to use structured output with the OpenAI API using Pydantic schema enforcement:

from pydantic import BaseModel
from openai import OpenAI

class SearchDecision(BaseModel):
    should_search: bool
    query: str
    reasoning: str

The BaseModel class from Pydantic defines the exact structure you expect back. OpenAI’s API will enforce this schema.

client = OpenAI()

response = client.beta.chat.completions.parse(
    model='gpt-5.6-sol',
    messages=[
        {'role': 'system', 'content': 'You are a search planning assistant.'},
        {'role': 'user',   'content': 'Should I search the web to find the capital of France?'},
    ],
    response_format=SearchDecision,   # enforce this structure
)

decision = response.choices[0].message.parsed
print(f"Should search: {decision.should_search}")
print(f"Query: {decision.query}")
print(f"Reasoning: {decision.reasoning}")

The .parsed attribute gives you a proper Python object with type-checked fields, not a string you need to parse.

Building Your First Simple LLM Call

Let’s put it all together: a minimal but complete example:

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])

def ask_llm(user_message: str, system_prompt: str = '') -> str:
    """Simple wrapper around the OpenAI API."""
    messages = []
    
    if system_prompt:
        messages.append({'role': 'system', 'content': system_prompt})
    
    messages.append({'role': 'user', 'content': user_message})
    
    response = client.chat.completions.create(
        model='gpt-5.6-luna',   # cheaper model for development
        messages=messages,
        temperature=0.1,
        max_tokens=1000,
    )
    
    return response.choices[0].message.content

# Test it
result = ask_llm(
    user_message="List 3 benefits of using Python for AI development.",
    system_prompt="You are a concise technical educator. Use bullet points."
)
print(result)

This is the foundation that every agent is built on. Everything else, tool calling, memory, multi-agent orchestration, is built around this core pattern of sending messages and receiving responses.

Exercise: Write a system prompt for an agent that helps you review code for security vulnerabilities. What should the agent’s persona be? What constraints should it have? What format should its output take? Test your system prompt on a piece of code that has an obvious SQL injection vulnerability, does the agent catch it and explain it clearly?