Semantic Routing

Semantic routing is an inference-serving technique that reads what a prompt is actually asking before deciding how expensively to answer it. A lightweight classifier sits in front of the model pool, scores each incoming request for difficulty and intent, and sends “what’s 12% of 340” down a fast, cheap path while sending “walk through the tax implications of this multi-entity merger step by step” down a slower path that enables chain-of-thought reasoning. It matters because reasoning modes are not free: enabling extended chain-of-thought or inference-time scaling on every request buys accuracy on hard problems but burns latency and tokens on easy ones that never needed it. Semantic routing treats “should this prompt get the expensive treatment” as its own classification problem, one that a small model can answer in milliseconds, rather than a decision made once, globally, for an entire deployment.

The Problem: Reasoning Is a Blunt Instrument by Default

Most production LLM deployments face a binary choice that’s really a false one: turn reasoning mode on for every request (accurate, slow, expensive) or off for every request (fast, cheap, wrong on the hard cases). Neither extreme fits real traffic, because real traffic is a mix. A customer-support endpoint might see thousands of “where’s my order” lookups for every genuinely ambiguous billing dispute; a coding assistant might see hundreds of one-line autocomplete requests for every multi-file refactor. Applying test-time compute uniformly means paying the reasoning tax on the easy 90% to get the accuracy gain that only the hard 10% actually needed.

The vLLM Semantic Router project’s own benchmark makes the cost of the “always reason” default concrete: on MMLU-Pro, selectively routing only the prompts that benefit from reasoning achieved a 10.2 percentage point accuracy improvement over always-fast direct inference, while cutting response latency by 47.1% and token consumption by 48.5% compared to a policy that reasons on every request. That’s not a small tuning gain, it’s evidence that most of the accuracy benefit of reasoning modes comes from a minority of prompts, and paying for it on the rest is pure waste.

Architecture: Where the Router Sits

A semantic router typically sits as a proxy layer in front of the model-serving pool rather than as a change to any individual model. The reference implementation built around vLLM uses Envoy as the traffic layer with an External Processing (ext_proc) filter that hands each request to a separate classification service before it reaches a backend:

flowchart LR
    A[Client request<br/>'/v1/chat/completions']:::process --> B[Envoy proxy]:::process
    B -->|ext_proc gRPC call| C[Classifier service:<br/>ModernBERT + Candle]:::data
    C --> D{Difficulty score<br/>vs threshold}:::process
    D -->|below threshold| E[Fast-path pool:<br/>small/no-reasoning model]:::output
    D -->|above threshold| F[Reasoning pool:<br/>large/CoT-enabled model]:::output
    E --> G[Response to client]:::output
    F --> G

    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

The classifier itself is the interesting part: it has to be fast enough to run on every single request without becoming the new bottleneck. The reference design uses ModernBERT, an encoder that incorporates RoPE and FlashAttention to stay lightweight while supporting long context, wrapped in a Rust core (via Hugging Face Candle) for zero-copy tensor handling and CPU-only inference at high concurrency. Running the classifier on CPU rather than GPU is a deliberate choice: it keeps the classification step from competing with the expensive models for scarce accelerator capacity, and a 100M-parameter encoder doesn’t need one anyway.

# Simplified sketch of what the ext_proc classifier does per request,
# run on CPU so it never competes with GPU-bound model serving.
from modernbert_router import DifficultyClassifier

classifier = DifficultyClassifier.load("modernbert-routing-v1")

def classify_request(prompt: str) -> dict:
    # Multi-stage: category, difficulty, plus safety-adjacent checks
    # that fall out of the same forward pass almost for free.
    result = classifier.predict(prompt)
    return {
        "category": result.category,          # e.g. "math", "creative_writing", "code"
        "difficulty": result.difficulty,       # float in [0, 1]
        "pii_detected": result.pii_flags,
        "jailbreak_score": result.jailbreak_score,
    }

The Threshold Is the Whole Trade-off

Everything about semantic routing’s cost/accuracy behavior collapses into one number: the difficulty threshold above which a request gets promoted to the reasoning path. Set it near 0 and almost everything gets the expensive treatment, you approach “always reason” accuracy at “always reason” cost. Set it near 1 and almost nothing does, you’re back to cheap-and-fast with the accuracy gaps that implies. The useful operating point, per the MMLU-Pro results above, is well below the midpoint: most of the accuracy gain comes from routing a relatively small, genuinely-hard slice of traffic, which is exactly why the technique beats both extremes rather than just averaging them.

Semantic Routing vs. Other Ways to Manage the Cost/Accuracy Trade-off

Routing isn’t the only lever for controlling reasoning cost, and it’s worth placing next to its closer relatives:

Semantic routing is complementary to most of these rather than a replacement. A production stack might run semantic routing to pick fast vs. reasoning pools, prefill-decode disaggregation within each pool to separate the compute-bound and memory-bound phases of serving, and a semantic cache in front of the whole thing to skip repeat questions before they ever reach a classifier.

# Combining semantic routing with a cache lookup: check for a near-duplicate
# question first, and only pay for classification + generation on a miss.
async def handle_request(prompt: str):
    cached = await semantic_cache.lookup(prompt, similarity_threshold=0.92)
    if cached:
        return cached.response

    decision = await router.classify(prompt)
    response = await router.route(prompt, decision)
    await semantic_cache.store(prompt, response)
    return response

Beyond On/Off: What Else Falls Out of the Same Classifier

Because the classifier already has to read every prompt before routing it, production implementations tend to have it do more than one job in the same forward pass. The vLLM Semantic Router’s multi-stage design runs category classification (math, code, creative writing, and so on), PII detection, and jailbreak-attempt scoring alongside the difficulty score that drives the reasoning decision. That’s a genuine efficiency win, not just convenience: a single ModernBERT pass that flags a request as both “high difficulty” and “contains PII” lets the proxy layer make a routing decision and a data-handling decision (redact, log differently, route to a compliance-audited model pool) without a second network hop. It also means the same infrastructure investment that buys latency and cost savings from routing doubles as a lightweight, always-on safety net that runs before a single token reaches an expensive model, a useful property distinct from, but complementary to, guardrails implemented via prompt injection detection further downstream.

What’s New (2025-2026)

Semantic routing went from a research idea to a documented, benchmarked open-source component in a short window. The vLLM Semantic Router project’s initial release and blog post landed in September 2025, and the accompanying “When to Reason” paper formalizing the approach and its MMLU-Pro results followed in October 2025. Through 2026, the pattern has been folded into the broader “inference-aware serving” conversation alongside dynamic prefill-decode disaggregation and quantization-aware training, three techniques that together define what production LLM serving stacks look like once teams move past “just point everything at one model” and start treating a request’s own content as a signal for how it should be served. The project’s stated roadmap extends the same classifier to semantic-cache-aware routing and deeper integration with Envoy Gateway API Inference Extension (GIE) and llm-d, pushing routing decisions further into the standard Kubernetes-native serving stack rather than keeping them as a bespoke sidecar.

When Not to Reach for It

Semantic routing adds a classification hop and a second (or third) model pool to operate, that’s real infrastructure and on-call surface for a team to own. It pays off when traffic is genuinely mixed (a real spread of easy and hard requests) and when the reasoning-capable model is meaningfully more expensive or slower than the fast path, enough that avoiding it on the easy majority is worth the classifier’s overhead. It’s the wrong tool for a single-purpose endpoint that only ever receives one kind of request (there’s nothing to route between), or for a deployment where the “expensive” and “cheap” paths don’t actually differ much in cost, in which case the classifier adds latency without buying anything back.

# A support-ticket API that wants cheap, fast answers for routine
# tickets but full chain-of-thought reasoning for genuinely hard ones.
from semantic_router_client import SemanticRouter

router = SemanticRouter(
    classifier="modernbert-routing-v1",
    fast_model="llama-3-8b-instruct",
    reasoning_model="deepseek-r1",
    reasoning_threshold=0.35,   # route to reasoning if difficulty score > 0.35
)

async def handle_ticket(prompt: str) -> str:
    decision = await router.classify(prompt)
    # decision.difficulty in [0, 1], decision.category e.g. "billing_dispute"
    response = await router.route(prompt, decision)
    log_metrics(category=decision.category, routed_to=response.model_used,
                latency_ms=response.latency_ms, tokens=response.total_tokens)
    return response.text