Choosing and Switching Models: Context, Cost, and Fallback Chains

Context Length: How Hermes Figures It Out

Hermes needs to know a model’s usable context window to manage memory retrieval, tool-result truncation, and conversation history correctly. It resolves this through a fixed priority order:

1. model.context_length in config.yaml (highest priority, always wins if set)
2. Custom provider per-model settings
3. Persistent cache from a previous session
4. The endpoint’s own /models API response
5. The models.dev community registry
6. A 128K fallback default

model:
  default: "qwen3.5:9b"
  context_length: 131072   # explicit override: skips detection entirely

Set this explicitly whenever you self-host (Lesson 5) and know the real figure, auto-detection against a local endpoint is the least reliable link in that chain, and a mismatch here is a common source of the “agent seems to forget things mid-task” symptom that looks like a memory bug (Module 4) but is actually a context-budgeting one.

Matching Model to Task

Not every request needs your strongest, most expensive model. A useful mental model:

Switch mid-session with /model (Lesson 4) rather than restarting: start a task on a fast model, and escalate only the sub-question that actually needs deeper reasoning.

Building a Fallback Chain

Manual switching handles “I want a different model.” Fallback handles “my provider just went down and I’m not watching.” Configure it with:

hermes fallback

which walks you through an ordered list of backup providers. A resulting config looks like:

model:
  provider: "zai"
  default: "glm-5.2"

fallback:
  - provider: "anthropic"
    model: "claude-sonnet-5"
  - provider: "nous-portal"
    model: "gpt-5.4"

If the primary Z-AI GLM 5.2 connection errors out (rate limit, outage, expired key), Hermes automatically retries through Anthropic’s Claude Sonnet 5, then Nous Portal, in order, before surfacing a failure to you. Anthropic is a reasonable first fallback precisely because it’s a fully independent stack from Z-AI, a provider-level outage or rate limit on one has no bearing on the other, which is the entire point of a fallback chain: each hop should be able to fail for a different reason than the one before it. This matters most for the unattended deployments you’ll build in Module 5 (serverless backends) and Module 6 (messaging gateways), where nobody is at the keyboard to run /model when something breaks at 2am.

Handling Retired or Renamed Models

Providers periodically retire model names. Rather than manually hunting down every config reference, Hermes provides a migration helper:

hermes migrate xai   # example: migrate retired xAI model configs to current names

This walks your config.yaml and any custom_providers entries, flagging references to deprecated model identifiers and offering current replacements. Run it after a provider’s model deprecation announcement, before your fallback chain silently starts routing to a model that no longer exists.

Exercise: configure a two-provider fallback chain (your Module 4 primary plus one backup), then temporarily set an invalid API key for the primary provider in .env and confirm hermes doctor flags it, then confirm a chat request still succeeds by falling through to the backup. Restore the correct key when you’re done.