Kimi K3

Released by Moonshot AI on July 16, 2026, Kimi K3 is a 2.8 trillion parameter open-weight Mixture of Experts model, described by Moonshot as the world’s first open model in the 3-trillion-parameter class. It succeeds Kimi K2.7 Code with a substantially different attention stack built around Kimi Delta Attention (KDA), a hybrid linear/full attention mechanism, alongside Attention Residuals (AttnRes) and an extremely sparse Stable LatentMoE routing layer. Together, Moonshot reports these changes deliver roughly 2.5x the overall scaling efficiency of Kimi K2, while natively supporting a 1-million-token context window and unified text, image, and video understanding.

Architecture Overview

graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    IN([Text / Image / Video tokens]):::data --> EMB[Unified Multimodal Embedding]:::process
    EMB --> BLK[Transformer Block, repeated Nx]:::process
    BLK --> LM[LM Head]:::process
    LM --> OUT([reasoning_content + content, up to 1M tokens]):::output

    subgraph BLOCK["Inside One Transformer Block"]
        R1([Residual Stream In]):::data --> ARES[Attention Residuals: gate selects which earlier layer's output to reuse]:::process
        ARES --> ATT{Attention Sublayer}
        ATT -->|3 of every 4 layers| KDA[Kimi Delta Attention: linear, delta-rule state]:::process
        ATT -->|1 of every 4 layers| MLA[Gated MLA: full attention]:::process
        KDA --> MOE[Stable LatentMoE FFN]:::process
        MLA --> MOE
        MOE --> ROUTE{Quantile-Balanced Router}
        ROUTE -->|16 of 896 experts active| EXP[Weighted Expert Sum]:::output
        EXP --> R2([Residual Stream Out]):::data
    end

Four architectural pieces work together inside every block:

• Kimi Delta Attention (KDA): a hybrid linear attention mechanism that handles most layers cheaply.
• Gated MLA: a full-attention layer, interleaved periodically, that preserves global lookup capability KDA alone would lose.
• Attention Residuals (AttnRes): a learned gate that selectively pulls representations from earlier layers instead of accumulating them uniformly.
• Stable LatentMoE: a feed-forward layer that routes each token to a tiny fraction of 896 available experts.

Kimi Delta Attention: A Hybrid Linear/Full Attention Stack

Standard (“full”) self-attention compares every token to every other token, which costs O(n^2) compute and, more painfully at long context, an O(n)-growing KV cache that has to be held in memory for every generated token. KDA replaces most layers with a linear attention variant based on a delta-rule recurrent state update (the same family of ideas behind state-space models like Mamba), which compresses the entire attention history into a fixed-size state that updates in constant time per token:

# Simplified illustration of the complexity gap KDA is built to close.
# Full attention: every new token attends to all n previous tokens,
# and the KV cache grows linearly with sequence length.
def full_attention_step(query, keys, values):
    scores = query @ keys.T          # O(n) work THIS step, O(n) cache growth
    weights = softmax(scores)
    return weights @ values

# Linear/delta-rule attention: a fixed-size recurrent state is updated
# in O(1) work per token, independent of how long the sequence already is.
def kda_step(query, key, value, state):
    state = state + outer(key, value) - decay(state, key)  # delta-rule update
    return query @ state, state       # O(1) work, no growing KV cache

KDA interleaves these two mechanisms in a fixed 3:1 ratio: three KDA (linear) layers handle local sequence structure cheaply, followed by one Gated MLA (full-attention) layer that preserves global information flow across the whole context. Moonshot reports this combination delivers up to a 75% reduction in KV-cache memory and up to roughly 6x faster decoding at million-token context lengths, while matching full-attention baselines on quality, since the periodic full-attention layers catch the long-range dependencies the linear layers are structurally unable to represent on their own.

Attention Residuals (AttnRes)

Ordinary transformer residual connections accumulate every layer’s output into the residual stream uniformly, layer after layer, regardless of whether that layer’s contribution is still useful many layers later. AttnRes replaces this with a learned gate that selectively retrieves representations from specific earlier layers instead:

# Scenario: layer 40 needs information computed at layer 12, not the
# uniformly-blended residual stream every intermediate layer has touched.
def attn_residual_gate(layer_outputs: list, current_depth: int, gate_weights):
    # gate_weights, learned per depth, decide which earlier layers'
    # outputs are still worth retrieving at this depth
    retrieved = sum(w * out for w, out in zip(gate_weights, layer_outputs))
    return retrieved  # replaces naive sum(layer_outputs), which dilutes signal

Moonshot reports AttnRes delivers roughly 25% higher training efficiency at under 2% additional inference cost, a favorable trade against the alternative of simply making the model deeper to get the same effective capacity.

Stable LatentMoE and Quantile-Balanced Routing

K3’s feed-forward layers activate only 16 of 896 experts per token, a sparsity ratio (about 1.8%) considerably more extreme than earlier large MoE models. Pushing sparsity that far typically destabilizes training: a small number of experts tend to dominate early, starving the rest of gradient signal and collapsing most of the 896 experts into dead weight. Stable LatentMoE addresses this with Quantile Balancing, which derives expert allocation from the quantile of each expert’s router score across the current batch rather than from raw router logits, removing the need for the heuristic load-balancing auxiliary losses and hand-tuned balancing hyperparameters earlier MoE architectures relied on.

# Simplified illustration of quantile-based routing vs raw top-k on logits.
# Raw top-k can let a few experts dominate every batch if their logits
# happen to run high; quantile balancing routes based on each expert's
# *relative standing within the current batch* instead.
def quantile_balanced_route(router_logits, k=16, num_experts=896):
    # rank each expert's score against its own recent score distribution,
    # not just against the other experts in this one batch
    quantiles = to_quantile(router_logits, per_expert_history=True)
    return top_k_indices(quantiles, k=k)

Two supporting mechanisms round out the stack: Per-Head Muon, which extends the Muon optimizer to update each attention head’s parameters independently rather than as one fused block, and SiTU (Sigmoid Tanh Unit), an activation function used in place of standard SwiGLU to improve gradient control inside the expert layers.

Interactive: How Sparse Is 16-of-896?

MoE sparsity, the fraction of experts active per token, is a real architectural trade-off: sparser routing means more total model capacity for the same per-token compute cost, but pushes harder against the training instability Quantile Balancing exists to solve. The chart below computes the sparsity ratio for any active-expert count k against K3’s fixed pool of 896 experts, and marks where a few other well-known MoE models sit for comparison.

At the slider’s default of 16, K3 sits near the sparse end of the scale, well past DeepSeek-V3’s already-sparse 8-of-256 (about 3.1%) and far past Mixtral’s 2-of-8 (25%). That gap is exactly why Quantile Balancing exists: routing reliably to 16 out of 896 options, without a handful of experts absorbing most of the gradient signal, is a materially harder balancing problem than routing to 2 out of 8.

Multimodal Input, Context, and Caching

Kimi K3 processes text, images, and video through one native multimodal architecture rather than a bolted-on vision adapter. Image and video inputs are supplied as base64 data or an ms://<file-id> reference after upload; the vision path does not accept public image or video URLs. The model supports up to 1,048,576 completion tokens, and context caching is automatic, requiring no separate configuration to benefit from cached prefixes on repeated calls.

Reasoning Mode and Agentic Tool Use

K3 always runs with thinking mode enabled; there is no non-reasoning mode to switch to. The reasoning_effort parameter (low, high, or max, the default) controls how much of that reasoning budget the model spends per request, and the response separates reasoning_content from the final content, so an application can log or hide the chain of thought without string-parsing it out of the answer.

# Scenario: an agentic coding task where tool calls must always be used,
# never answered from the model's own text, and JSON output must be strict.
response = client.chat.completions.create(
    model="kimi-k3",
    reasoning_effort="max",
    tool_choice="required",
    tools=[{
        "type": "function",
        "function": {
            "name": "run_tests",
            "parameters": {"type": "object", "properties": {"path": {"type": "string"}}},
            "strict": True,
        },
    }],
    messages=[{"role": "user", "content": "Run the test suite in ./src and report failures."}],
)

Additional agentic features include dynamic tool loading via system messages, structured output through JSON Schema with strict: true, and a partial mode that lets a generation continue from a supplied text prefix rather than starting from scratch, useful for enforcing a fixed response scaffold.

Kimi K3 vs. Kimi K2.7 Code

Benchmark Snapshot

Third-party coverage of Moonshot’s release reports the following results, which should be read as vendor- and outlet-reported figures rather than independently reproduced numbers:

Pricing and Deployment

Kimi K3 uses flat, pay-as-you-go pricing with no tiering by context length: input and output tokens are billed at a uniform per-token rate, with separate rates for cache hits and cache misses. API access unlocks after a minimum $1 account top-up, and account tier and rate limits scale with cumulative top-up amount. The model is available as "kimi-k3" through an OpenAI SDK-compatible endpoint at https://api.moonshot.ai/v1, and is live on Kimi.com, Kimi Work, and Kimi Code in addition to the API.

What’s New (2025-2026)

• Open-weight models keep pushing past the trillion-parameter mark. K3’s 2.8T parameter count, following K2’s 1T, continues a 2025-2026 trend of open-weight MoE models closing the scale gap with proprietary frontier models, largely by pushing MoE sparsity further rather than growing active (and therefore inference-cost) parameters proportionally.
• Hybrid linear/full attention is becoming the default for long context, not a research curiosity. KDA’s 3:1 linear-to-full attention ratio sits alongside a broader 2025-2026 shift (Kimi Linear, MiniMax’s lightning attention, Qwen3-Next) toward hybrid attention stacks as the practical way to reach million-token context windows without the KV-cache cost of pure full attention.
• MoE sparsity keeps getting more extreme. K3’s 16-of-896 routing (about 1.8% active) pushes past DeepSeek-V3’s already-aggressive 8-of-256, and required a dedicated stabilization mechanism (Quantile Balancing) specifically because routing collapse gets harder to avoid as the active fraction shrinks.
• Native multimodality as a first-class design goal, not an add-on. Rather than pairing a language model with a separately trained vision encoder, K3 handles text, image, and video within one architecture, reflecting the same direction as Moonshot’s K2.7 Code and most other 2025-2026 frontier releases.

Practical Guidance

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["MOONSHOT_API_KEY"],
    base_url="https://api.moonshot.ai/v1",
)

# Kimi K3 always runs with thinking mode on; reasoning_effort tunes
# how much of that budget it spends: "low", "high", or "max" (default).
response = client.chat.completions.create(
    model="kimi-k3",
    reasoning_effort="high",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What failure does this screen recording show?"},
                # Vision inputs use base64 or an ms://<file-id> reference;
                # Kimi K3's vision path does not accept public image/video URLs.
                {"type": "video_url", "video_url": {"url": "ms://uploaded-clip-4471"}},
            ],
        }
    ],
)

# reasoning_content and content are returned separately, so the chain
# of thought can be logged or hidden without parsing it out of the reply.
print("Reasoning:", response.choices[0].message.reasoning_content)
print("Answer:", response.choices[0].message.content)