Llama 4

Llama 4 is the fourth generation of Meta’s open-weight large language model family, released on 5 April 2025. It is the first Llama generation to be built as a mixture-of-experts model and the first to be natively multimodal, meaning text and vision are fused into one backbone during pretraining rather than bolted together afterward. The launch covered three models Meta calls a “herd”: two that shipped with downloadable weights, Llama 4 Scout and Llama 4 Maverick, and one preview-only teacher model, Llama 4 Behemoth, that the smaller two were distilled from. All three share the same core design: a sparse MoE transformer with early-fusion multimodality, trained in FP8 on more than 30 trillion tokens across 200 languages.

The Llama 4 Herd: Scout, Maverick, Behemoth

The three models differ almost entirely in how many experts they carry and what they are tuned for. The active parameter count, the part that actually runs per token, is 17 billion for both Scout and Maverick.

Llama 4 Scout is the small, wide-context member. With only 16 experts it fits on a single NVIDIA H100 when quantized to int4 (Transformers can do this quantization on the fly), and its instruction-tuned build was stretched to a 10 million token context window, the largest of any widely available model at release. That makes it the one to reach for when the job is “read this entire repository” or “summarize these 400 filings,” not “be the smartest per token.”

Llama 4 Maverick is the general-purpose flagship. It keeps the same 17B active parameters but spreads them across 128 experts for 400B total, so it holds far more knowledge while costing the same per generated token as Scout. Meta positions it as the chat, coding, and image-understanding workhorse, deployable on a single H100 host (FP8 weights are published alongside the BF16 ones) or a smaller multi-GPU setup. Its instruct context is 1M tokens.

Llama 4 Behemoth is the teacher. At roughly 2 trillion total parameters (288B active, 16 very large experts) it was used to codistill Scout and Maverick and was previewed as “still training.” It was repeatedly delayed through 2025 and never shipped with public weights.

Mixture of Experts: Why Total and Active Parameters Diverge

Every Llama 4 layer replaces the single feed-forward block of a dense transformer with a set of expert feed-forward blocks plus one shared expert. A lightweight router picks the top-1 routed expert for each token; that expert and the shared expert run, and the other experts sit idle. The consequence is the number that confuses people about MoE models: Maverick is a “400B model” you can serve at the speed and compute cost of a 17B one, because 383B of those parameters are asleep for any given token.

# Scenario: the routing math, stripped down. Total parameters scale with the
# expert count; compute per token does not, because only two experts fire.
def moe_layer(token_hidden, shared_expert, routed_experts, router):
    scores = router(token_hidden)                 # one logit per expert
    top1 = scores.argmax(dim=-1)                   # Llama 4 routes to a single expert
    routed_out = routed_experts[top1](token_hidden)
    return shared_expert(token_hidden) + routed_out  # the other 15 or 127 experts: untouched

# Scout:    16 experts  -> ~109B stored, ~17B active per token
# Maverick: 128 experts -> ~400B stored, ~17B active per token

The design knob is the expert count. Adding experts grows the weight file (and the memory to hold it) close to linearly, while per-token compute stays flat. That gap is the entire reason Maverick can be much more capable than Scout without being slower to run. The widget below makes the split concrete.

Native Multimodality via Early Fusion

Earlier Llama vision work (Llama 3.2) attached a separate image encoder to a finished text model through cross-attention adapters. Llama 4 uses early fusion: image patches and text are turned into tokens and concatenated into one sequence that the transformer attends over jointly, and the whole thing is pretrained together on text, image, and video data. Meta still trains the vision encoder (a MetaCLIP-derived model) but does so alongside a frozen Llama backbone so the encoder adapts to the language model rather than the other way round.

# Scenario: the processor builds a single interleaved token sequence. The
# model never sees a "this part is an image" flag, only positions in a stream.
messages = [{"role": "user", "content": [
    {"type": "text",  "text": "Compare the two charts and give the delta."},
    {"type": "image", "url": "q1.png"},
    {"type": "image", "url": "q2.png"},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
)
# inputs["input_ids"] now contains text tokens AND image tokens in one row

Practically, this is what lets Maverick handle multi-image prompts (Meta reports testing with up to eight images per prompt) and do grounded reasoning over a document and its figures at once, without a separate captioning hop that would throw away layout and detail.

iRoPE and the 10M-Token Context

Scout’s 10M-token window comes from an attention design Meta calls iRoPE. The “i” is for interleaved: most layers use rotary positional embeddings over a fixed local chunk (8,192 tokens in Scout), and roughly every fourth layer uses no positional embedding at all, attending over the full sequence with the causal mask. The no-position layers are what carry information across millions of tokens; the local RoPE layers keep short-range precision cheap.

The second piece is inference-time temperature scaling of attention. Softmax over a very long sequence flattens toward uniform, so long-range signal washes out. Llama 4 scales the attention logits in the no-position layers as the sequence grows, sharpening them back up. Because this is applied at inference, Scout can be pushed well past the 256K context it was pretrained on.

# Conceptual sketch of the interleaved layer schedule (illustrative, not the
# exact Transformers config keys):
attention_schedule = {
    "chunk_size": 8192,          # local RoPE layers attend within this window
    "global_layer_every": 4,     # every 4th layer: no positional embedding, full attention
    "temperature_tuning": True,  # scale attention logits up as the sequence lengthens
}
# Pretrained at 256K; instruct-tuned to 10M (Scout) / 1M (Maverick).

A caveat that surfaced quickly after release: independent long-context evaluations found real retrieval quality degrading well before the 10M mark, so treat that number as an architectural ceiling rather than a promise of uniform accuracy across the whole window.

Training: FP8, 30T Tokens, Codistillation from Behemoth

Llama 4 was pretrained in FP8 precision, which is what made a 30-trillion-token run (more than double Llama 3’s data) practical; Meta reports 390 TFLOPs per GPU across 32,000 H100s for Behemoth. Scout and Maverick were then codistilled from Behemoth during that teacher’s own pretraining, using a loss that dynamically blends the teacher’s soft predictions with hard ground-truth labels, so the distillation targets were computed once as a side effect of training Behemoth rather than in a separate expensive pass.

graph LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    C[(30T tokens: text,<br/>image, video)]:::data --> B[Llama 4 Behemoth<br/>~2T params, teacher]:::process
    B -->|codistillation loss:<br/>soft + hard targets| M[Llama 4 Maverick<br/>400B / 128 experts]:::output
    B -->|codistillation| S[Llama 4 Scout<br/>109B / 16 experts]:::output
    B -.->|previewed, never<br/>publicly released| X[ ]:::data

Post-training added supervised fine-tuning, online reinforcement learning, and direct preference optimization, with Meta noting it deliberately kept SFT light to leave room for the RL stage to explore.

Benchmarks and the LMArena Controversy

On paper, Maverick launched strong: Meta reported it beating GPT-4o and Gemini 2.0 Flash on several reasoning, coding, and multilingual benchmarks at a fraction of the inference cost, and cited an LMArena ELO of 1417.

That 1417 figure became the story. The model Meta submitted to LMArena was Llama-4-Maverick-03-26-Experimental, a chat-tuned variant “optimized for conversationality” that produced long, emoji-heavy answers human raters tend to prefer, and it was not the same as the weights posted to Hugging Face. When the public build was tested on the same leaderboard it landed far lower, around 32nd. LMArena updated its policies within days, saying Meta’s reading of the submission rules “did not match what we expect,” and the episode became a widely cited example of benchmark gaming through a non-representative eval checkpoint. It is the main reason Llama 4’s launch reputation never quite recovered, independent of the models’ actual merits.

Running Llama 4 Yourself

# Scenario: serve Scout for long-context RAG on a single 8x H100 node with vLLM.
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 1000000 \        # 10M is the ceiling; most deployments cap lower for KV-cache RAM
  --kv-cache-dtype fp8

# Maverick ships FP8 weights directly:
vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --tensor-parallel-size 8 --quantization fp8

The weights are open under the Llama 4 Community License, not a standard open-source license. Two clauses matter in practice: companies with more than 700 million monthly active users must request a separate license from Meta, and the license as written prohibits use by organizations domiciled in the EU (end users in Europe can still reach the models through a hosted product built elsewhere). You also may not use Llama 4 outputs to train a non-Llama model.

What’s New (2025-2026)

• Behemoth never shipped. The ~2T teacher was previewed in April 2025, slipped through the year, and was never released with public weights. Meta has not formally cancelled it, but it is effectively shelved.
• Meta reorganized around it. In June 2025 Mark Zuckerberg announced Meta Superintelligence Labs, led by new Chief AI Officer Alexandr Wang (following Meta’s ~$14B investment in Scale AI) and co-led by former GitHub CEO Nat Friedman. Frontier model work after Llama 4 moved under MSL, and Meta’s public messaging shifted from “open by default” toward keeping its most capable models closed.
• Scout and Maverick remain the current open Llama. As of 2026 there is no Llama 4.1 or Llama 5 with public weights; the April 2025 Scout and Maverick releases are still Meta’s latest open models.
• Ecosystem support matured. Day-one gaps (chunked attention in some inference engines, int4 paths, multi-image handling) were largely closed over 2025 in vLLM, TGI, llama.cpp, and Transformers, so the practical experience of running Scout and Maverick is much smoother now than at launch.

Llama 4 vs. Llama 3

Llama 4’s lasting contribution is architectural: it moved the open-weight mainstream to MoE and to native multimodality in one release, and Scout’s iRoPE showed a practical route to million-plus-token context. The launch execution, the LMArena submission and the missing Behemoth, is why it is remembered as much for its rollout as for its design.

# Scenario: a product-catalog tool that takes two photos of the same
# item and a question, and answers in text. Llama 4 is natively
# multimodal, so there is no separate caption or OCR model in front.
from transformers import AutoProcessor, Llama4ForConditionalGeneration
import torch

model_id = "meta-llama/Llama-4-Scout-17B-16E-Instruct"  # 16 experts, int4 on the fly
processor = AutoProcessor.from_pretrained(model_id)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    attn_implementation="flex_attention",  # needed for the chunked-attention path
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

messages = [{"role": "user", "content": [
    {"type": "image", "url": "https://example.com/chair-front.jpg"},
    {"type": "image", "url": "https://example.com/chair-side.jpg"},
    {"type": "text", "text": "Are these the same product? What colour is the frame?"},
]}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
).to(model.device)

out = model.generate(**inputs, max_new_tokens=256)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[-1]:])[0])