Reinforcement Learning with Verifiable Rewards (RLVR)

Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training technique in which a language model’s reward comes from a deterministic program that checks correctness, not from a neural reward model trained on human preferences. If the model solves a math problem, a parser extracts the final answer and compares it to the known one. If it writes code, a test suite runs. The reward is 1 or 0, and nothing about it can be flattered. The term was introduced in AI2’s Tülu 3 report in late 2024, and within months DeepSeek-R1 demonstrated that this deceptively simple substitution was enough to induce long chain-of-thought reasoning in a base model with no reasoning supervision at all. RLVR is now the standard final stage in essentially every open reasoning model, and the reason Group Relative Policy Optimization could drop PPO’s critic network without losing training signal.

The Problem With Learned Reward Models

Classic RLHF trains a reward model on pairwise human preference data, then optimizes the policy against that reward model’s scores. It works, and it produced the first genuinely helpful assistants, but it carries a structural flaw: the reward model is itself a neural network with its own errors, and RL is an extremely effective search procedure for finding them.

This is reward hacking. The policy discovers that longer answers score higher, or that confident phrasing scores higher, or that a particular formatting quirk scores higher, and it drifts toward those artifacts rather than toward being correct. Because the reward model is frozen while the policy keeps improving, the gap between “what the reward model rewards” and “what humans actually wanted” widens throughout training. Practitioners handle this with KL penalties toward a reference policy, early stopping, and periodic reward model retraining, all of which are damage control rather than a fix.

RLVR sidesteps the whole failure mode for the subset of tasks where correctness is mechanically checkable. A unit test does not care how confident the model sounds. An answer-matching function does not prefer long responses. The reward is grounded in something outside the model’s influence.

How an RLVR Loop Works

%%{init: {'theme': 'base'}}%%
graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    DS([Prompt plus ground-truth artifact]):::data --> POL[Policy: sample G rollouts]:::process
    POL -->|"chain of thought plus final answer"| ROLL[Rollouts o_1 ... o_G]:::data
    ROLL --> VER{Verifier: parse, execute, compare}:::process
    GOLD([Gold answer or unit tests]):::data --> VER
    VER -->|"r_i in 0 or 1"| ADV[Advantage: normalize within group]:::process
    ADV --> UPD[Policy gradient update, GRPO or PPO]:::output
    UPD -.->|next step| POL

The loop has four moving parts, and only one of them is unusual.

1. A dataset where every prompt carries a checkable artifact. Not just the question, but the gold answer, the test file, the expected schema, or the constraint to satisfy. Building this dataset is most of the work in an RLVR project.
2. Sampling. The policy generates G rollouts per prompt at nonzero temperature, so the group actually varies. This is where the model explores different reasoning paths.
3. Verification. Each rollout is scored by running the verifier. This is cheap and embarrassingly parallel, though code execution needs sandboxing.
4. A policy-gradient update. Tülu 3 used PPO with a value model. DeepSeek-R1 used GRPO, and the field followed, because a binary verifiable reward pairs naturally with a group-relative baseline.

# Scenario: the sampling-and-scoring half of one RLVR training step,
# the part that differs from a standard RLHF loop.
def rlvr_step(policy, batch, G=8, temperature=1.0):
    all_rollouts, all_rewards = [], []
    for item in batch:
        # Explore: several independent attempts at the same problem.
        rollouts = policy.sample(item["prompt"], n=G, temperature=temperature)
        # Verify: a program decides, not a model.
        rewards = [item["verifier"](r, item["gold"]) for r in rollouts]
        all_rollouts.append(rollouts)
        all_rewards.append(rewards)
    return all_rollouts, all_rewards

# Contrast with the RLHF equivalent, where this line is the whole difference:
#   rewards = reward_model.score(item["prompt"], rollouts)   # learned, gameable

Reward Design Beyond Pass or Fail

A pure correctness bit is the canonical case, but production RLVR runs almost always score a small composite. Each component still has to be mechanically checkable, which is what keeps it RLVR rather than RLHF.

# Scenario: an RLVR run for a model that must emit structured tool calls,
# where format compliance matters as much as the answer being right.
def composite_reward(completion, task):
    reward = 0.0
    # Correctness dominates: everything else is a small shaping term.
    reward += 1.0 * verify_answer(completion, task["gold"])
    # Format: did it use the required <think>...</think> then answer structure?
    reward += 0.2 * float(has_required_sections(completion))
    # Schema: does the emitted JSON validate against the tool signature?
    reward += 0.2 * float(validates_against(completion, task["schema"]))
    # Anti-hack penalty: punish restating the gold answer without reasoning,
    # a shortcut policies find quickly when the prompt leaks the answer.
    reward -= 0.5 * float(answer_leaked_without_work(completion, task))
    return reward

Three families of shaping term recur: format rewards (the <think> block structure DeepSeek-R1 used), constraint rewards (verifiable instruction following, for example “answer in exactly three bullet points”, which a parser can check), and penalty terms for known shortcuts. What you will not find in a well-designed RLVR reward is anything requiring judgment, because the moment you add a judge model you have reintroduced the gameable component RLVR was built to remove.

The Group Composition Problem

RLVR has a distinctive efficiency failure that RLHF does not. When all G rollouts for a prompt are correct, or all are wrong, the group carries no comparative information: every advantage is zero after normalization, and the entire batch of rollouts contributed nothing to the gradient. If the model has already mastered a prompt, or cannot solve it at all, sampling it is wasted compute.

Assume each rollout succeeds independently with probability p (the model’s per-attempt pass rate on that prompt). The chance a group of size G is informative is:

P(informative) = 1 - p^G - (1 - p)^G

This peaks sharply at p = 0.5, which is why curriculum design in RLVR is really about keeping prompts near the edge of the model’s current ability.

Interactive: Which Prompts Actually Teach the Model Anything

The curve is a dome that collapses steeply at both edges, and it flattens across the middle as G grows. At G = 8, a prompt the model already solves 95 percent of the time produces a usable gradient in only about 34 percent of groups, and one it solves 2 percent of the time in only about 15 percent. This is the observation behind DAPO’s dynamic sampling: filter out prompts whose group came back all-correct or all-wrong, and keep resampling until the batch is full of informative ones. Raising G widens the useful band (compare the faded curves), which is the real argument for large groups beyond variance reduction.

Known Failure Modes

Verifier loopholes. RLVR removes reward hacking against a model and replaces it with reward hacking against a program. Policies have learned to write code that detects the test harness, to special-case the exact inputs a test file checks, and to emit answer formats that a loose regex accepts without doing the work. A weak verifier is worse than an honest reward model, because its exploits are invisible in the reward curve. Sandboxing, held-out tests, and adversarial verifier review are the mitigations.

The base-model dependence problem. The Spurious Rewards paper is the most important cautionary result in the area. Training Qwen2.5-Math-7B with random rewards improved MATH-500 by 21.4 points, and with deliberately incorrect labels by 24.1 points, against 29.1 points for ground-truth rewards. The same spurious rewards produced no gains on Llama3 or OLMo2. The interpretation is that for some base models, RLVR is partly surfacing latent capabilities the pretraining already installed rather than teaching new ones, and any RL signal that nudges the model toward its own better-formatted reasoning distribution will show gains. The practical takeaway: an RLVR benchmark improvement is not by itself evidence the method worked. Run a random-reward control.

Entropy collapse and diversity loss. Optimizing hard against a binary reward narrows the output distribution. Models converge on one solution template, pass@1 rises while pass@k falls, and exploration dies. This is the central tension the field is still working on, and it connects RLVR to continual learning concerns about capability loss during post-training.

Domain ceiling. RLVR simply does not apply where correctness is not mechanically checkable. Summarization quality, tone, empathy, and open-ended writing all still need preference-based methods.

What’s New (2025-2026)

• RLVR became a standard stage, not a technique. Every major open-weight reasoning release since DeepSeek-R1 (the Qwen3 line, GLM-5 series, MiniMax-M2, Kimi’s K2 and K3 models) runs a verifiable-reward RL stage on math, code, and structured output. The recipe stabilized fast enough that the interesting work moved from the algorithm to the environments.
• RL environments as the scarce resource. The bottleneck shifted from “which policy-gradient variant” to “what can we verify.” Building sandboxed, resettable environments (repositories with test suites, browser tasks with checkable end states, tool APIs with assertable side effects) is now where labs concentrate effort, and a small industry of RL environment providers formed around it.
• Extension into agentic RL. Applying RLVR to multi-step agent trajectories requires verifiers over outcomes rather than single answers: did the pull request pass CI, did the booking actually get made, did the retrieved evidence support the claim. Intermediate or context rewards that credit correct evidence selection are used to densify the otherwise extremely sparse end-of-trajectory signal.
• Rubric-based rewards at the boundary. To push past the verifiability ceiling, several 2025-2026 systems score open-ended answers against detailed, itemized rubrics. This is genuinely a hybrid: more structured and auditable than a preference model, but not deterministic, so it reintroduces some gaming surface. Treat “rubric rewards” as adjacent to RLVR rather than a subtype of it.
• Stronger controls and better skepticism. Post-Spurious-Rewards, credible RLVR papers now report random-reward and format-only-reward baselines as a matter of course, and evaluate pass@k alongside pass@1 to catch diversity collapse. The field’s evaluation hygiene improved substantially in response to a single negative result.

Practical Guidance

The strategic point about RLVR is narrower than the hype suggests but more durable. It did not discover a better reinforcement learning algorithm. It observed that a large and commercially important class of tasks has ground truth sitting right there, unused, in the form of tests, compilers, parsers, and known answers, and that plugging that ground truth directly into the RL loop removes the least trustworthy component of the entire post-training stack. The frontier now is not the algorithm at all, it is how far the boundary of “verifiable” can be pushed before it stops being verifiable.

import re
import subprocess

# Scenario: an RLVR run mixing math and code prompts. Every prompt in
# the dataset ships with a ground-truth artifact the verifier can check
# against, which is what makes the reward "verifiable" rather than learned.

def verify_math(completion: str, gold_answer: str) -> float:
    """Extract the boxed final answer and compare it to the gold answer."""
    match = re.search(r"\\boxed\{([^}]*)\}", completion)
    if match is None:
        return 0.0                        # no parseable answer, no credit
    return 1.0 if match.group(1).strip() == gold_answer.strip() else 0.0

def verify_code(completion: str, test_file: str) -> float:
    """Run the task's unit tests against the generated program."""
    code = extract_code_block(completion)
    result = subprocess.run(
        ["pytest", test_file, "-q"],
        input=code, capture_output=True, text=True, timeout=30,
    )
    return 1.0 if result.returncode == 0 else 0.0

def score_rollouts(prompt, completions, task):
    """Returns one scalar per sampled completion, ready for GRPO."""
    if task["kind"] == "math":
        return [verify_math(c, task["gold"]) for c in completions]
    return [verify_code(c, task["tests"]) for c in completions]

# These scalars feed straight into a policy-gradient update. No reward
# model was trained, and no human labeled any of these completions.