Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for fine-tuning large language models that estimates how good a response was by comparing it against a group of other responses to the same prompt, rather than against a learned value function. Introduced by DeepSeek in the DeepSeekMath paper and popularized by DeepSeek-R1, GRPO strips out the critic network that Proximal Policy Optimization (PPO) normally requires, roughly halving the GPU memory needed for RL fine-tuning while matching or beating PPO on reasoning benchmarks. That efficiency gain is a large part of why DeepSeek-R1 could be trained and released as open weights, and why GRPO (or a close variant of it) has since become the default RL algorithm behind most open reasoning models trained with Reinforcement Learning from Human Feedback-style pipelines in 2025-2026.

Why GRPO Exists: PPO’s Critic Problem

Standard PPO, the algorithm behind the original ChatGPT-era RLHF pipeline, needs two models on top of the policy being trained: a reward model that scores a response, and a critic (value network), roughly the same size as the policy itself, that predicts the expected future reward from any partial response so PPO can compute a low-variance advantage estimate. Training and holding that critic in memory alongside a multi-billion-parameter policy is expensive, and the critic itself has to be trained and can be inaccurate early in training, injecting noise into every advantage estimate it produces.

GRPO’s core idea is to replace the critic with a purely statistical baseline computed on the fly: sample several outputs for the same prompt, and use their own average reward as the “expected value” a single output is compared against. If a candidate response scores above the group’s mean, it gets a positive advantage and is reinforced; if it scores below, it gets a negative advantage and is suppressed. No separate network ever has to learn what “expected reward” looks like, because the group supplies its own estimate every step.

How GRPO Works

%%{init: {'theme': 'base'}}%%
graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    P([Prompt q]):::data --> OLD[Old Policy pi_theta_old]:::process
    OLD -->|sample G outputs| O1[o_1]:::data
    OLD -->|sample G outputs| O2[o_2]:::data
    OLD -->|sample G outputs| O3[...]:::data
    OLD -->|sample G outputs| OG[o_G]:::data
    O1 --> RM{Reward: rule-based verifier or reward model}:::process
    O2 --> RM
    O3 --> RM
    OG --> RM
    RM -->|"r_1 ... r_G"| NORM[Normalize: mean, std within group]:::process
    NORM -->|advantage A_i per token| CLIP[Clipped surrogate objective]:::process
    REF([Reference policy pi_ref]):::data -->|KL penalty, coefficient beta| CLIP
    CLIP --> UPD[Policy Update]:::output
    UPD -.->|becomes new old policy| OLD

Each GRPO update runs four steps for a batch of prompts:

1. Group sampling. For each prompt q, the current policy (frozen as π_old for this step) generates G candidate outputs o_1 ... o_G, typically by sampling with temperature rather than greedy decoding, so the group actually varies in quality.
2. Scoring. Each output gets a scalar reward r_i. For math and code, this is often a cheap, deterministic rule-based verifier (did the final answer match, did the unit tests pass) rather than a trained reward model, since reasoning tasks are one of the few domains where correctness is mechanically checkable, which is exactly what makes GRPO so effective there.
3. Group-relative advantage. Rewards are normalized within the group: A_i = (r_i - mean(r_1...r_G)) / std(r_1...r_G). This one line is what replaces the entire critic network from PPO.
4. Clipped policy update with KL penalty. The policy is updated using the same clipped, importance-weighted surrogate objective PPO uses, plus a KL-divergence penalty (coefficient β) that keeps the updated policy from drifting too far from a fixed reference policy in a single step.

The Objective Function

The formal GRPO objective, as defined in the DeepSeekMath paper, is:

J_GRPO(θ) = E[q, {o_i}_{i=1..G} ~ π_old]
    (1/G) Σ_i (1/|o_i|) Σ_t
        min( ratio_t · A_i,  clip(ratio_t, 1-ε, 1+ε) · A_i )
        − β · D_KL[π_θ || π_ref]

where ratio_t = π_θ(o_i,t | q, o_i,<t) / π_old(o_i,t | q, o_i,<t)

Three quantities control its behavior:

The KL term itself uses an unbiased, always-non-negative estimator rather than the naive log-ratio, so it can be estimated from samples without needing the full distribution:

D_KL[π_θ || π_ref] = π_ref(o_i,t) / π_θ(o_i,t) − log(π_ref(o_i,t) / π_θ(o_i,t)) − 1

GRPO vs. PPO

# PPO: requires training and running a separate critic (value) network
# alongside the policy, adding a second multi-billion-parameter model
# to every training step just to estimate a baseline.
def ppo_advantage(reward, value_estimate):
    # value_estimate comes from a trained critic network V(s)
    return reward - value_estimate  # single-sample advantage vs learned baseline

# GRPO: the baseline comes from the group itself, no critic required.
def grpo_advantage(rewards_in_group):
    return (rewards_in_group - rewards_in_group.mean()) / (rewards_in_group.std() + 1e-4)

Interactive: Group Size and Baseline Stability

G is the one knob in GRPO with a direct, computable cost/benefit trade-off: sampling more outputs per prompt makes the group’s mean a more reliable stand-in for the true expected reward (its standard error shrinks as 1/√G), but every extra sample is an extra forward pass through the policy at training time. The widget below draws a fresh group of sampled rewards and shows how the baseline’s uncertainty shrinks as G grows.

Notice the shaded band around the dashed mean line, that’s the baseline’s standard error, and it visibly tightens as G increases from 2 toward 32. This is precisely the trade-off DeepSeek and follow-on labs tune in practice: DeepSeekMath used G = 64 for its RL stage, while later reasoning-focused runs often use smaller groups (8-16) to fit more RL steps into the same compute budget.

Known Failure Modes and 2025-2026 Fixes

Two structural issues in the original GRPO formulation surfaced once labs beyond DeepSeek started training on it at scale, and both now have named fixes that ship in most open RL frameworks:

• Length bias. Dividing the objective by response length |o_i| systematically rewards longer wrong answers over shorter ones in some regimes, because the per-token penalty gets diluted across more tokens. Dr. GRPO (Understanding R1-Zero-Like Training: A Critical Perspective, Liu et al.) removes both the length normalization and the group standard-deviation normalization, showing this alone improves token efficiency without hurting accuracy.
• Instability in MoE policies and degenerate groups. When every sampled output in a group gets the same reward, std(r) = 0 and the naive advantage formula divides by zero (handled above with a small eps, but the underlying signal is genuinely uninformative in that case: nothing in the group distinguishes good from bad). DAPO (An Open-Source LLM Reinforcement Learning System at Scale, Yu et al.) addresses this with dynamic sampling that discards prompts whose group is entirely right or entirely wrong, plus decoupled upper/lower clip bounds to preserve exploration. GSPO (Group Sequence Policy Optimization, Zheng et al.) moves the importance ratio from the token level to the full sequence level, which Alibaba’s Qwen team reported was necessary to stabilize GRPO-style training for large Mixture of Experts policies, where token-level ratios interact badly with expert routing changes between the sampling and update steps.

What’s New (2025-2026)

GRPO and its variants have effectively become the shared RL substrate underneath the current wave of open reasoning models, not a DeepSeek-specific technique:

• Verifiable-reward RL as the default post-training stage. Most 2025-2026 open-weight reasoning models (the Qwen3 series, GLM-5 line, MiniMax-M2, and Kimi’s K2/K3 line) run a GRPO-family stage on math, code, and structured-output tasks where a rule-based verifier can score outputs cheaply, extending the recipe DeepSeek-R1 popularized rather than reinventing it.
• Sequence-level over token-level importance ratios. GSPO’s shift to sequence-level clipping (mid-2025) has been adopted specifically to keep RL stable when the policy being trained is a large MoE model, since expert routing can differ between the policy that generated a sample and the policy being updated.
• Removing the normalization terms that caused bias. Dr. GRPO’s finding that dividing by response length and group std introduces measurable bias has propagated into most production RL frameworks (verl, OpenRLHF) as an optional or default flag, rather than remaining a research footnote.
• Dynamic and adaptive group construction. Rather than a fixed G for an entire run, 2025-2026 systems increasingly vary group size or filter degenerate groups per-prompt (DAPO’s dynamic sampling), treating group construction itself as a tunable part of the training loop instead of a fixed hyperparameter.

Practical Guidance

GRPO’s real contribution wasn’t a new idea in reinforcement learning, group-relative baselines are a decades-old variance-reduction trick, it was recognizing that for reasoning tasks with mechanically checkable answers, that old trick is enough to drop an entire critic network out of the RLHF pipeline without losing training signal. That single simplification is a large part of why frontier-level reasoning models became trainable outside a handful of labs with PPO-scale infrastructure.

import numpy as np

# Scenario: a math-reasoning RL step. The old policy samples G=8
# candidate solutions for one prompt; a rule-based verifier checks
# each against the known answer and returns 1.0 or 0.0.
def grpo_advantages(rewards: np.ndarray, eps: float = 1e-4) -> np.ndarray:
    """rewards: shape (G,), one scalar reward per sampled output o_i."""
    mean_r = rewards.mean()
    std_r = rewards.std()
    # Group-relative baseline: no critic network, no value function.
    # eps guards the degenerate case where every sample in the group
    # gets the same reward (std_r == 0), which would otherwise divide by zero.
    return (rewards - mean_r) / (std_r + eps)

rewards = np.array([1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0])  # G=8 outputs
advantages = grpo_advantages(rewards)
print(advantages)
# Outputs that beat the group's own average reward get positive
# advantage (reinforced); outputs below it get negative advantage
# (suppressed), all without ever training a separate value model.