Prefix Caching and Position-Independent Caching (PIC)

Prefix caching is the practice of keeping the KV cache computed for the beginning of a prompt (a system prompt, a set of tool definitions, a long document) and reusing it for later requests that start with the same text, so the model skips the expensive prefill step for everything it has already seen. It is why a 10,000-token system prompt can cost a tenth as much on the second call as on the first, and why “keep the start of your prompt stable” became the most valuable single rule in production LLM engineering. Position-independent caching (PIC) is the research frontier that removes the rule’s biggest limitation: prefix caching only helps when everything before the reused text is identical, while real RAG and agent workloads reorder, insert, and edit content constantly. PIC lets a cached chunk be reused even when it shows up somewhere new.

Why Only a Prefix?

The restriction is not an implementation shortcut; it follows from how Transformers compute. In a causal model, the key and value vectors for token i depend on the tokens before it. After the first layer, every token’s hidden state has already mixed in information from earlier tokens through attention, so the K and V computed for the words “return policy” are different when those words follow a refund conversation than when they follow a recipe. On top of that, rotary position embeddings rotate each key by an angle that depends on its absolute position, so the same tokens at a different offset produce a different key even before context is considered.

The consequence is a strict rule: a cached KV entry for a span of text is valid only if every token before that span, and therefore the span’s position, is identical to when it was computed. The first token that differs invalidates that token and everything after it, even if the rest of the prompt is byte-for-byte the same.

%%{init: {'theme': 'base'}}%%
graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph Req1["Request 1 (cold)"]
        A1[System prompt + tools]:::process --> A2[Policy manual]:::process --> A3[Question A]:::process
    end
    subgraph Req2["Request 2 (warm)"]
        B1[System prompt + tools]:::data --> B2[Policy manual]:::data --> B3[Question B]:::process
    end
    A1 -. "identical: reuse cached KV" .-> B1
    A2 -. "identical: reuse cached KV" .-> B2
    A3 -. "first difference: recompute from here" .-> B3
    B3 --> OUT[Only the suffix is prefilled]:::output

How Serving Engines Implement It

Two designs dominate open-source serving, and both build on the block-based memory layout from PagedAttention.

Block hashing (vLLM). The KV cache is stored in fixed-size blocks, and each full block gets an identity: a hash of its own tokens combined with the hash of the block before it, plus any extra keys that change the meaning of the block (LoRA adapter IDs, multimodal input hashes, a cache salt). Chaining the parent hash means a block’s identity encodes the entire prefix leading to it, so a lookup is just “is this hash in the table?” repeated block by block until the first miss. Only full blocks are cached, so a prompt’s trailing partial block is always recomputed. When memory is needed, blocks are evicted least-recently-used from a doubly-linked free queue (constant-time updates), with a tie-breaker that evicts the last block of a request first, since the tail hashes the most tokens and is the least likely to be shared. vLLM’s V1 engine enabled prefix caching by default, reporting less than 1% throughput loss even at a 0% hit rate.

# Scenario: the identity scheme behind block-level prefix caching, simplified.
# A block's hash covers its tokens AND the hash of the block before it, so
# equal hashes mean an equal prefix, not just equal local text.
import hashlib

BLOCK = 16

def block_hashes(tokens, salt=""):
    hashes, parent = [], salt.encode()
    for i in range(0, len(tokens) - len(tokens) % BLOCK, BLOCK):   # full blocks only
        block = tokens[i:i + BLOCK]
        parent = hashlib.sha256(parent + repr(block).encode()).digest()
        hashes.append(parent.hex()[:8])
    return hashes

system = list(range(100, 148))              # 48-token system prompt = 3 full blocks
req_a = system + [1, 2, 3, 4, 5, 6, 7, 8]   # question A
req_b = system + [9, 9, 9, 9, 9, 9, 9, 9]   # question B, same system prompt
req_c = [7] + system                        # same system prompt, shifted by one token

block_hashes(req_a) == block_hashes(req_b)  # True: all three blocks match, so 48 tokens are reused
block_hashes(req_c)                         # all three hashes differ: a one-token shift misses everything

Radix tree (SGLang). RadixAttention keeps every cached token sequence in a radix tree: a compressed trie whose edges can be labeled with runs of tokens, mapping token sequences (the keys) to their KV tensors (the values), stored in a paged layout on the GPU. A new request walks the tree to find its longest matching prefix, reuses those tensors, and inserts its own suffix. Eviction is LRU over leaf nodes, applied recursively, and a cache-aware scheduler orders waiting requests to raise hit rates. The SGLang paper reports up to 6.4x higher throughput than state-of-the-art inference systems across agent control, few-shot, JSON decoding, RAG, and multi-turn chat workloads.

# Scenario: the lookup side of a radix-style cache, as a token-level trie.
# A real radix tree compresses runs of tokens into single edges; the matching
# logic (walk until the first miss) is the same.
class TrieCache:
    def __init__(self):
        self.root = {}

    def insert(self, tokens):
        node = self.root
        for t in tokens:
            node = node.setdefault(t, {})

    def match_prefix(self, tokens):        # how many leading tokens are already cached
        node, n = self.root, 0
        for t in tokens:
            if t not in node:
                break
            node = node[t]
            n += 1
        return n

cache = TrieCache()
cache.insert(req_a)
cache.match_prefix(req_b)   # 48: shares the system prompt, diverges at the question
cache.match_prefix(req_c)   # 0: the shifted copy shares nothing

Interactive: Where Does the Prefix Break, and What Does It Cost?

Whether a cache saves money depends almost entirely on where the first changed token sits. Change the last line of a long prompt and nearly everything is reused. Put a timestamp on the first line and nothing is. Cache pricing adds a second wrinkle: some providers charge a premium to write a cache entry, so a prefix that never gets reused can cost more than no caching at all.

Start with the default (a change 90% of the way in) and the cost drops to a small fraction of the uncached bill. Drag the change position to 0% and the strip goes entirely indigo: nothing is reusable, and under write-premium pricing the bill rises above the uncached one, because every request pays the write premium on a prefix that never gets read. Now set the request count to 1 and note that even a perfect prefix cannot pay back a write premium on a single call. Finally switch the granularity to 128 tokens and set the change position to a few percent in: coarse blocks throw away the partial block before the change, which matters most for short prompts. The model assumes 8,000 prefill tokens per second and a 200-token fresh question on every request, so read it as a picture of the mechanism, not a benchmark.

Provider-Level Prompt Caching

Everything above runs inside your own serving stack. Hosted APIs expose the same mechanism as a product feature, with three different interfaces:

Prices, lifetimes, and thresholds change often, so treat the table as a snapshot from September 2026 and check the providers’ docs before budgeting. The economics are stable, though. With a 1.25x write and a 0.1x read (Anthropic’s multipliers, and OpenAI’s on GPT-5.6 and later), writing once and reading once costs 1.25 + 0.1 = 1.35 prompt-equivalents against 2.0 uncached, so a cache entry earns back its premium on the second request. An explicit cache also has a lookback limit: a breakpoint only checks a bounded number of content blocks backward for an earlier write, so a very long conversation needs its breakpoint advanced as it grows.

# Scenario: the same support bot on OpenAI's API. Caching is on by default, so
# the work is (1) keeping the prefix byte-identical and (2) steering related
# requests to the same cache with a stable key.
from openai import OpenAI

client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-6-astra",
    prompt_cache_key="support-bot-policy-v3",   # stable key so related requests route together
    messages=[
        {"role": "system", "content": POLICY_MANUAL},   # stable prefix first
        {"role": "user", "content": "Is the extended warranty transferable?"},
    ],
)
print(resp.usage.prompt_tokens_details.cached_tokens)   # how much of the prompt was served from cache

Designing Prompts for Cache Hits

Because a single changed token invalidates everything after it, prompt layout is cache design. The Manus team called the KV-cache hit rate “the single most important metric for a production-stage AI agent,” pointing out that an agent’s input-to-output token ratio can reach 100:1 and that cached tokens cost roughly a tenth of uncached ones. Their rules generalize:

• Put stable content first and volatile content last. Tool definitions, system instructions, and reference documents go at the top; the user’s question, timestamps, and per-request IDs go at the bottom.
• Make context append-only. Do not edit earlier turns or observations; add to the end. Rewriting history voids the cache from the edit onward.
• Serialize deterministically. Two JSON objects with the same fields in a different key order are different token sequences. Sort keys, fix number formatting, and avoid unordered sets in prompts.
• Never put the current time at the top. A timestamp with second resolution guarantees a miss on every request.
• Mark breakpoints deliberately where the API needs them, and include the end of the system prompt.

# Scenario: assembling an agent's prompt. The "bad" version looks harmless but
# defeats prefix caching on every single call.
import json, datetime

def build_prompt_bad(tools, docs, question):
    return (
        f"Current time: {datetime.datetime.now().isoformat()}\n"     # changes every call: cache miss at token 0
        f"Tools: {json.dumps(tools)}\n"                              # key order may vary between processes
        f"Docs: {docs}\nQuestion: {question}"
    )

def build_prompt_good(tools, docs, question):
    stable = (
        f"Tools: {json.dumps(tools, sort_keys=True)}\n"              # deterministic serialization
        f"Docs: {docs}\n"
    )
    volatile = (
        f"Current time: {datetime.datetime.now().date().isoformat()}\n"   # coarse, and placed last
        f"Question: {question}"
    )
    return stable + volatile                                         # everything reusable comes first

Agents make this harder because their context grows and gets managed. A related finding from the LMCache paper is that context truncation, a common industry practice for staying inside the window, can cut the prefix cache hit ratio by half, since dropping early turns shifts every later token. This is a real tension with context engineering techniques like summarization and compaction, which help against context rot but rewrite the prefix. The usual compromise is to compact rarely and in big steps, so the cache is invalidated once per compaction rather than on every turn.

Keeping the Cache Warm in Multi-Turn Chats and Agents

A conversation is close to an ideal prefix-caching workload, because each turn’s prompt is the previous turn’s prompt plus a little more: the whole history is a reusable prefix, and only the newest user message and the model’s last reply are new. The catch is time. Caches are not permanent. Anthropic’s default lifetime is five minutes, refreshed at no charge each time the entry is read, and OpenAI’s in-memory entries typically survive five to ten minutes of inactivity and up to an hour. Anthropic measures that lifetime from the start of the request that wrote or read the entry, not from the end of the response, so a slow four-minute generation leaves only about a minute for the next turn to arrive before the entry expires.

An agent that pauses for a human, a slow tool, or a queue can therefore come back to find its whole prefix gone and pay a full rewrite. Two levers help: a longer lifetime where the provider offers one (Anthropic’s 1-hour option costs 2x to write, and OpenAI offers extended 24-hour retention), and a tight loop that refreshes entries before they lapse. The longer lifetime is worth its higher write price only when the prefix will actually be read again after the short lifetime would have expired. The only way to know is to measure, and the metric worth tracking is hit rate per prompt template:

# Scenario: a dashboard metric for a multi-turn agent. Track the hit rate per
# prompt template, so a regression (someone added a timestamp at the top of
# the prompt) shows up as a step change instead of a mystery bill.
from collections import defaultdict

stats = defaultdict(lambda: {"cached": 0, "total": 0})

def record(template: str, usage):
    # Anthropic: input_tokens counts only the tokens after the last cache breakpoint,
    # so the full prompt is cache_read + cache_creation + input_tokens.
    # (On OpenAI, use usage.prompt_tokens_details.cached_tokens / usage.prompt_tokens.)
    cached = usage.cache_read_input_tokens or 0
    total = cached + (usage.cache_creation_input_tokens or 0) + usage.input_tokens
    stats[template]["cached"] += cached
    stats[template]["total"] += total

def hit_rate(template: str) -> float:
    s = stats[template]
    return s["cached"] / s["total"] if s["total"] else 0.0

Routing: The Cache Lives on One Machine

In a fleet, a prefix cache is a property of one replica’s GPU memory, not of the service. A request only hits if the load balancer sends it to a replica that already holds its prefix. A round-robin balancer spreads a shared prefix across every replica, so each replica has to warm it independently and hit rates drop as the fleet grows, especially for prefixes with modest traffic. The remedies are affinity and sharing.

Affinity sends requests with the same prefix or conversation to the same replica. OpenAI’s guidance for earlier models is to supply a stable prompt_cache_key precisely to optimize cache routing, and the same idea applies inside a single engine, where SGLang’s cache-aware scheduling orders waiting requests to raise hit rates. Sharing goes the other way: layers like LMCache store KV blocks outside the GPU (in CPU memory, on disk, or in a remote store) so that multiple queries, and even multiple engines, can reuse them, and disaggregated designs such as prefill-decode disaggregation move the cache between machines by design. Affinity is cheap but can create hot spots when one popular prefix dominates traffic; sharing spreads the load but adds transfer time, so it pays off when fetching a prefix is faster than recomputing it.

Where Prefix Caching Breaks

Four common workloads defeat an exact-prefix match even though most of their text repeats:

1. RAG with variable retrieval. The same document chunk is retrieved for many queries, but it lands at a different position and after different neighbors each time. See retrieval-augmented generation and agentic RAG.
2. Agent tool calls. A tool result is inserted mid-conversation, a failed call is removed, or an outdated observation is replaced; the tokens after the edit are unchanged but shifted.
3. Few-shot selection. Examples are chosen per query from a pool, so the same example appears in different orders.
4. Multi-agent handoffs. Sub-agents receive overlapping context assembled in different orders. See subagent orchestration.

In each case the content is stable and the arrangement is not. That is the gap PIC targets.

Cache-Friendly Retrieval Ordering

Before reaching for PIC, there is a cheaper fix for RAG: make the arrangement deterministic. If the retriever returns chunks in relevance order, a small change in scores reshuffles them and voids the prefix. Assembling the prompt in a stable order instead (sorted by document ID, say), and placing a fixed block of always-relevant reference material first, turns a shifting arrangement into a stable one, so that overlapping retrievals share a longer prefix.

# Scenario: a RAG endpoint over a product manual. Top-k retrieval returns the
# same chunks in a different score order on almost every query. Pin a fixed
# reference block first (a guaranteed shared prefix), then sort the rest by a
# stable key instead of by score.
PINNED = ["intro", "safety"]      # always included, always first, always in this order

def build_context(hits, corpus):
    pinned_text = [corpus[doc_id] for doc_id in PINNED]
    rest = sorted(
        (h for h in hits if h.doc_id not in PINNED),
        key=lambda h: (h.doc_id, h.chunk_index),     # stable order, not score order
    )
    return "\n\n".join(pinned_text + [h.text for h in rest])

The catch is that order is not free for the model. Relevance order and cache-friendly order are different orderings, and answers can change depending on where evidence sits in a long context (see context rot), so treat the reordering as a change to check against your own quality tests, not a guaranteed win. Where the order genuinely cannot be stabilized, that is the case PIC exists for.

Position-Independent Caching

The goal of PIC is to cache a chunk of text by its content and reuse it wherever it appears. Two things stand in the way, and they call for different fixes.

Problem 1: position. Because rotary embeddings rotate keys by absolute position, a key cached at position 1,200 is wrong at position 3,400. Fortunately rotations compose, so a cached key can be moved by rotating it by the position difference. This is exact for the positional part:

# Scenario: a tool-output chunk was cached when it sat at position 1,200; a new
# request places it at position 3,400. Instead of recomputing the key, rotate
# the cached one by the delta.
import numpy as np

def rope(x, pos, base=10000.0):
    d = x.shape[-1]
    theta = base ** (-np.arange(0, d, 2) / d)
    ang = pos * theta
    cos, sin = np.cos(ang), np.sin(ang)
    out = np.empty_like(x)
    out[0::2] = x[0::2] * cos - x[1::2] * sin
    out[1::2] = x[0::2] * sin + x[1::2] * cos
    return out

k = np.random.default_rng(0).standard_normal(128)
cached = rope(k, pos=1200)               # what the cache holds
moved = rope(cached, pos=3400 - 1200)    # re-rotate by the position delta
np.allclose(moved, rope(k, pos=3400))    # True: identical to recomputing at the new position

Problem 2: context. Rotation cannot fix the fact that the chunk’s keys and values were computed while attending to different preceding text. A chunk cached after one set of neighbors carries the wrong cross-attention for another. Ignoring this is the fast, lossy option. The research approaches differ mainly in how they handle it:

The approaches fall into three families. Declared modules (Prompt Cache, MiniPIC) hand part of the problem to the developer: you mark which spans are reusable and the system keeps their positions and dependencies consistent, which is exact about what it caches but changes how prompts are written. Automatic selective recompute (CacheBlend, EPIC) needs no change to the prompt and instead spends a small, targeted amount of compute repairing each reused chunk, choosing tokens by how far their cached values deviate from a full computation (CacheBlend) or by neutralizing the attention-sink effect at each chunk’s start (EPIC). Model-aware approaches (COMB, Irminsul) go a step further, either training the model so chunk representations are order-independent or exploiting an architecture whose cached representation is already largely position-free, as in the compressed latent of Multi-Head Latent Attention, leaving only a small rotary component to correct. Each puts the burden somewhere different: declared modules on prompt authors, selective recompute on the GPU, and model-aware methods on training or on a particular model family.

The main idea to take from the table is that PIC is a spectrum of approximations. Reusing a chunk without recomputing anything is fastest and least faithful; recomputing everything is the exact baseline; methods like CacheBlend and EPIC slide along that line by choosing which few tokens to recompute. That recompute fraction is the quality dial.

Interactive: Prefix Caching Versus PIC on a RAG Workload

The widget below compares the two on a retrieval workload. Each query retrieves several chunks. Some of those chunks were retrieved by earlier queries (the chunk reuse rate), and some fraction of those show up in the same position after the same neighbors (the order stability). Prefix caching can only reuse the unbroken run of matching chunks from the start; PIC reuses every previously seen chunk but recomputes a fraction of it to repair cross-attention.

With the defaults, prefix caching recovers only a sliver of the reusable content, because it stops at the first chunk that is missing or out of order, while PIC reuses most of the seen chunks. Raise order stability to 100% and the two converge, and once chunk reuse is high the verdict flips to prefix caching: when everything is cached and in order, exactness is free and PIC’s repair step is pure overhead. Drag the recompute fraction toward 0% and PIC’s saving grows, but that is the direction that risks quality, which is why published methods tend to land near 15%.

Prefix Cache, Semantic Cache, and PIC

These three are often confused, but they sit at different layers and can be stacked:

A production stack can use all three: a semantic cache answers repeated questions without calling the model, a prefix cache makes the calls that do happen cheaper, and PIC recovers the reuse that reordered context would otherwise lose.

Security: Shared Caches Leak Through Timing

A cache hit is faster than a miss, and that difference is observable. If two users share a cache, one user can probe with guessed prompts and use the response time to infer whether another user recently sent that text. Gu et al. audited hosted APIs and detected global cache sharing across users in seven API providers, including OpenAI, and used timing variations to learn architectural details of a model. The fix on your own infrastructure is to scope the cache to the trust boundary. vLLM supports a cache_salt that is injected into the hash of the first block, so only requests with the same salt can reuse each other’s blocks:

# Scenario: a multi-tenant deployment where each customer's documents are
# confidential. Per-tenant salts make cross-tenant cache hits impossible, at
# the cost of not sharing common prefixes (like a public system prompt) across tenants.
from openai import OpenAI

client = OpenAI(base_url="http://vllm-host:8000/v1", api_key="unused")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": tenant_prompt}],
    extra_body={"cache_salt": f"tenant-{tenant_id}"},   # only same-salt requests can hit these blocks
)

The vLLM docs also warn that non-cryptographic hash options trade a higher theoretical collision risk for speed, which in a multi-tenant setting could leak data, so the default SHA-256 is the safer choice there.

What’s New (2025-2026)

• Prefix caching became a default, not a feature flag. vLLM V1 turned it on by default with under 1% throughput cost at a 0% hit rate, and Anthropic, OpenAI, and Google all offer some form of it as an API feature, with reads priced around a tenth of normal input on the first two.
• Agents made the cache the main cost lever. With input-to-output ratios around 100:1 (the figure Manus reported for its agent), hit rate now sets both latency and bill for agent products, which is why context engineering advice increasingly reads as cache-preservation advice.
• PIC moved from idea to engines. CacheBlend’s selective recompute ships in the open-source LMCache, EPIC’s LegoLink was published at ICML 2025, and LMCache packaged KV reuse across queries and engines, reporting up to 15x higher throughput with vLLM on multi-round Q&A and document analysis.
• PIC is being built into models and engines directly. COMB (2026) argues for training the model to support PIC natively via an encoder. Irminsul (May 2026) targets MLA-based models (it was evaluated on DeepSeek-V2-Lite and Kimi Moonlight, among others), citing operator reports of TTFT spikes of 10 to 16 seconds on unchanged content after cache-hit regressions in agentic workloads. MiniPIC (June 2026) showed a usable PIC path in under 100 lines of vLLM changes.
• Editing the cache became a primitive. Leyline (May 2026) lets an agent actively remove or replace a span of cached content and repairs positions with a rotary correction, reporting +11.2 percentage points on replay cache-hit and +14.3 points on agent solve rate in debug-gym, a step toward treating KV state as something agents manage rather than something they merely hit.
• Caches are being audited as attack surface. Timing-based leakage through shared caches went from theory to measured findings against production APIs, and per-tenant cache isolation is now a standard deployment concern.

Practical Guidance

The through-line is that the KV cache is a data structure with a very specific contract: it rewards text that is identical to what came before, in the same order and place, and it charges you for everything else. Prefix caching is what you get when you design your prompts to honor that contract. Position-independent caching is what you get when you cannot, and instead teach the serving stack to reuse content wherever it lands, with a controlled loss of fidelity as the price.

import anthropic

client = anthropic.Anthropic()

# Scenario: a support bot whose system prompt embeds a ~9,000-token policy
# manual. Every request shares that manual and differs only in the customer's
# question, so the manual is the prefix worth caching.
POLICY_MANUAL = open("support_policy.md").read()

def answer(question: str):
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=500,
        system=[
            {"type": "text", "text": "You are a support agent for Acme. Follow the policy below."},
            {
                "type": "text",
                "text": POLICY_MANUAL,
                # Breakpoint at the END of the stable content: everything up to
                # here becomes a reusable prefix. Volatile text goes after it.
                "cache_control": {"type": "ephemeral"},
            },
        ],
        messages=[{"role": "user", "content": question}],  # the only part that changes
    )
    u = resp.usage
    # First call: cache_creation_input_tokens is large (the write).
    # Later calls within the TTL: cache_read_input_tokens is large (the hit).
    print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)
    return resp.content[0].text

answer("Can I return an opened item after 20 days?")
answer("Is the extended warranty transferable?")   # should read the manual from cache