Exact-Match & Semantic Caching

Lesson 4’s request lifecycle placed a cache lookup at stage 4, before any provider is even contacted. This lesson builds that cache for real, in two layers: exact-match, for genuinely repeated requests, and semantic, for requests that are different in wording but ask effectively the same thing, which is where most of a cache’s real savings usually come from in production.

Exact-Match Caching

The simple case: identical requests should never pay for a second LLM call. The key detail is normalization, two requests that are logically identical but serialized differently (key order, whitespace) need to hash to the same cache key:

import hashlib
import json
import redis

r = redis.Redis(host="localhost", port=6379)

def cache_key(model: str, messages: list[dict], temperature: float) -> str:
    # Normalize before hashing: sort keys, fixed separators, so two
    # logically identical requests always produce the same hash
    # regardless of how the client happened to serialize the JSON.
    payload = json.dumps(
        {"model": model, "messages": messages, "temperature": temperature},
        sort_keys=True, separators=(",", ":"),
    )
    return "exact:" + hashlib.sha256(payload.encode()).hexdigest()

def get_cached_or_none(model, messages, temperature) -> str | None:
    key = cache_key(model, messages, temperature)
    cached = r.get(key)
    return cached.decode() if cached else None

def store_in_cache(model, messages, temperature, response_text: str, ttl_seconds=3600):
    key = cache_key(model, messages, temperature)
    r.setex(key, ttl_seconds, response_text)

This alone handles a real and common case, retry storms where a client sends the exact same request multiple times, repeated automated checks, or multiple users independently asking a truly identical FAQ, but it does nothing for two users who ask “what’s your refund policy” and “how do refunds work,” which are the same question in different words and would each pay for a full LLM call under exact-match alone.

Semantic Caching

Semantic caching closes that gap by comparing meaning rather than exact text, using embedding similarity:

import numpy as np
from openai import OpenAI

embed_client = OpenAI(api_key="sk-...")

SIMILARITY_THRESHOLD = 0.95   # cosine similarity; tuned per note below

def embed(text: str) -> np.ndarray:
    resp = embed_client.embeddings.create(model="text-embedding-3-small", input=text)
    return np.array(resp.data[0].embedding)

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

class SemanticCache:
    def __init__(self):
        self._entries: list[tuple[np.ndarray, str, str]] = []  # (embedding, prompt, response)

    def lookup(self, prompt: str) -> str | None:
        query_embedding = embed(prompt)
        best_score, best_response = 0.0, None
        for cached_embedding, _cached_prompt, cached_response in self._entries:
            score = cosine_similarity(query_embedding, cached_embedding)
            if score > best_score:
                best_score, best_response = score, cached_response
        if best_score >= SIMILARITY_THRESHOLD:
            return best_response
        return None

    def store(self, prompt: str, response: str):
        self._entries.append((embed(prompt), prompt, response))

A production deployment backs the _entries list with a real vector store (Redis with a vector index, Faiss, or Qdrant, GPTCache, the open-source library from Zilliz built specifically for this, supports all three) rather than a Python list scanned linearly, but the logic is identical: embed the incoming prompt, find the closest cached entry, and serve it if similarity clears the threshold.

The Threshold Is a Real Tuning Decision

SIMILARITY_THRESHOLD is not a constant to set once and forget. Set it too low (permissive) and prompts that are merely topically related but actually ask different things get served the wrong cached answer, silently, since a semantic cache hit looks identical to a real response to whoever receives it. Set it too high (conservative) and the cache rarely fires, capturing little of the savings semantic caching exists for. Published guidance and common production practice lands cosine-similarity thresholds around 0.92-0.97 for most conversational use cases, with a common tuning approach: start conservative (e.g. 0.95), monitor false-positive hits for a period, and adjust in small increments rather than guessing a final value up front.

Two-Stage Lookup, in Order

flowchart LR
    Req["Incoming request"] --> Exact{"Exact-match\ncache hit?"}
    Exact -->|Yes, cheap hash lookup| Return1["Return cached response\n(near-zero cost)"]
    Exact -->|No| Semantic{"Semantic cache\nhit >= threshold?"}
    Semantic -->|Yes| Return2["Return cached response\n(one embedding call, no LLM call)"]
    Semantic -->|No| Provider["Call provider\n(Module 4 routing)"]

    style Req fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Return1 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Return2 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Provider fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px

Checking exact-match first is a pure cost optimization: a hash lookup is essentially free, while a semantic lookup pays for an embedding call and a similarity search, so the cheap, common case (truly repeated requests) is captured before paying for the broader, more expensive check. Both layers only ever save cost and latency for requests the gateway would otherwise have paid a provider for in full; Lesson 17 covers the complementary mechanism, prompt caching, that providers themselves offer for content that isn’t identical across requests but is reused within them, like a long system prompt repeated on every call.