Inside the Architecture: Orchestration, Memory, and the Learning Loop

The Subsystems, at a Glance

Every later module in this course goes deep on one piece of this diagram. Before that, it’s worth seeing how the pieces fit together, because Hermes’ behavior only makes sense as a system, not as a list of features.

flowchart TB
    REQ["User request\n(CLI, TUI, or messaging platform)"] --> ORCH

    subgraph ORCH["Orchestration Layer"]
        direction TB
        ROUTE["Decide: answer directly,\ncall a tool, or delegate\nto a sub-agent"]
    end

    MEM[("Memory store\nFTS5 cross-session recall\n+ user model")] -.retrieves.-> ORCH
    SKILLS[("Skills directory\n~/.hermes/skills/")] -.matches.-> ORCH

    ORCH --> MODEL["Configured model\n(any provider, Module 2)"]
    MODEL --> ORCH

    ORCH --> TOOLGW["Tool Gateway\n60+ built-in tools + MCP servers"]
    TOOLGW --> ORCH

    ORCH --> SUBAGENT["Contained sub-agent\n(isolated, short-lived)"]
    SUBAGENT --> ORCH

    ORCH --> RESULT["Response"]
    ORCH -.writes.-> MEM
    ORCH -.writes new skills.-> SKILLS

    style REQ fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style ORCH fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style MEM fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style SKILLS fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style MODEL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TOOLGW fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SUBAGENT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style RESULT fill:#EEF0F7,stroke:#6366F1,color:#0F172A

Tracing One Request

Say you type: “Check if the staging API is returning 500s, and if so, look at the last deploy’s diff.”

1. Memory retrieval. The orchestration layer queries the memory store for anything relevant, maybe it recalls that “the staging API” refers to a specific URL you mentioned three sessions ago.
2. Skill matching. It checks whether an existing skill already covers part of this (a check-staging-health skill you built earlier, say). If one matches closely enough, its instructions get injected as extra context.
3. Model call. The model, whichever provider you configured in config.yaml, receives the request plus the retrieved memory and skill context, along with the current tool definitions.
4. Tool routing. The model decides it needs to make an HTTP request and read a git diff. It emits tool calls; the orchestration layer routes each to the Tool Gateway (built-in HTTP tool, shell/git tool) or an MCP server if one is configured for it.
5. Sub-agent delegation (optional). If the diff is large, the orchestration layer might spin up a contained sub-agent just to read and summarize it, keeping the main conversation’s context lean.
6. Response, and a slightly smarter agent. The final answer comes back to you. If this turned out to be a repeatable workflow, you could /learn it into a permanent skill, covered in Module 4, so next time this whole chain runs as one command.

Why the Split Between Orchestration and Model Matters

Because the orchestration layer, not the model, owns memory, skills, and tool routing, you can swap the underlying model without losing any of the agent’s accumulated capability. Switch from Claude to a local Qwen checkpoint mid-project (Module 2 covers exactly how), and your skills directory, memory store, and MCP server configuration all carry over unchanged. The model is a replaceable component, not the seat of the agent’s identity.

Common pitfall: assuming a bigger or newer model automatically makes the agent “smarter” in every way that matters. A stronger model improves reasoning and tool-call accuracy on a given turn, but it does not, by itself, give the agent memory of your last conversation or a growing library of skills. Those come from the orchestration layer accumulating state over time, which is why a well-used Hermes installation on a mid-tier model can outperform a fresh installation on a frontier model for tasks it has already learned.

What You’ll Build in Each Module

Exercise before continuing: Run hermes tools in your terminal and read through the list. For each category of tool you see (file operations, web, code execution, and so on), guess which subsystem in the diagram above it belongs to. You’ll confirm your guesses in Module 3.