Self-Hosting Models: Ollama, vLLM, llama.cpp, and LM Studio

Why Self-Host at All

Cloud providers are the fastest path to a working agent, but self-hosting matters once you care about data never leaving your infrastructure, predictable fixed costs at high volume, or running Hermes somewhere with no reliable internet access. Every option below speaks the OpenAI-compatible /v1 API, which is what makes them all straightforward to wire into Hermes.

Ollama: the Quick Path

Best for trying a model locally with minimal setup.

ollama pull qwen2.5-coder:32b
OLLAMA_CONTEXT_LENGTH=64000 ollama serve

# ~/.hermes/config.yaml
model:
  default: qwen2.5-coder:32b
  provider: custom
  base_url: http://localhost:11434/v1
  context_length: 64000

Note the explicit OLLAMA_CONTEXT_LENGTH and context_length settings, Ollama’s default context window is smaller than the 64K floor Hermes Agent needs for reliable multi-turn tool use (Lesson 2). Skipping this is the single most common cause of “my local model keeps forgetting earlier tool results” reports.

vLLM: Production-Grade Local Serving

vLLM is built for throughput and concurrent requests, the right choice once you’re serving more than one interactive session at a time.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --port 8000 \
  --max-model-len 65536 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The two tool-calling flags matter more than they look. --enable-auto-tool-choice turns on vLLM’s ability to let the model decide when to call a tool (versus forcing it every turn). --tool-call-parser hermes tells vLLM which output format to expect and parse into structured tool calls, this is the same Hermes function-calling format you’ll study in depth in Module 3, and it is why models trained with Hermes-style tool-call formatting (including the Hermes model family from Lesson 1) work especially cleanly here.

Point Hermes at it:

hermes model   # choose "Custom endpoint" → http://localhost:8000/v1

or directly in config:

model:
  default: your-model-name
  provider: custom
  base_url: http://localhost:8000/v1

llama.cpp: Maximum Portability

llama-server runs GGUF-quantized models with minimal dependencies, useful on constrained or unusual hardware (covered further in Module 5’s lesson on RTX and DGX Spark deployment).

llama-server \
  --jinja -fa \
  -c 64000 \
  -ngl 99 \
  -m models/qwen2.5-coder-32b-instruct-Q4_K_M.gguf \
  --port 8080 --host 0.0.0.0

--jinja applies the model’s actual chat template rather than a generic fallback, essential for correct tool-call formatting. -ngl 99 offloads as many layers as possible to GPU. -c 64000 again satisfies the context floor.

LM Studio: GUI-First Local Serving

For a point-and-click option that still exposes an OpenAI-compatible endpoint:

lms server start
lms load qwen2.5-coder --context-length 64000

Hermes finds it at the default http://localhost:1234/v1 via hermes model → “LM Studio”.

Naming Multiple Local Endpoints

Running more than one local server at once (say, a fast small model for quick lookups and a larger one for hard reasoning)? Give each a name in custom_providers instead of overwriting the single default endpoint:

custom_providers:
  - name: local
    base_url: http://localhost:8080/v1
  - name: work
    base_url: https://gpu-server.internal.corp/v1
    key_env: CORP_API_KEY

Switch between them mid-session:

/model custom:local:qwen-2.5
/model custom:work:llama3-70b

Choosing Between Them

Exercise: pull a small model with Ollama, serve it with an explicit 64K context length, and point Hermes at it as a named custom provider. Ask it to call a simple built-in tool (Lesson 8 covers exactly what’s available). If the tool call fails to parse, that’s your first hands-on encounter with why --tool-call-parser and chat templates matter, revisit this lesson’s vLLM section for the fix.