Off-Policy Evaluation: Testing Policies Without Shipping Them

You have a new decision policy: a retrained uplift model, a different exploration rate, a new offer, a changed budget price. Is it better than the current one? The obvious answer is an A/B test. But A/B tests are slow (90-day retention outcomes), expensive (half the traffic gets the worse policy), and you can only run a few at once.

Off-policy evaluation (OPE) estimates how a new policy would have performed using only the logs of decisions made by the old one. Done well, it lets you screen dozens of candidates offline and A/B test only the most promising. Done carelessly, it produces confident nonsense. This lesson covers both sides.

Why naive replay is biased

The logs contain, for each decision: context x, the action a the logging policy chose, the reward r, and (if you followed the advice of the last two lessons) the propensity p = π₀(a | x), the probability the logging policy chose a.

The tempting evaluation: for each logged row, ask the new policy what it would do; where it agrees with the logged action, keep the reward; average the kept rewards. This is biased. The old policy chose actions based on context, so the rows where it chose action a are not a random sample of contexts. If it gave discounts mainly to easy-to-retain customers, the “discount” rows look great regardless of what discounts actually do.

Four estimators

Let π be the new policy you want to evaluate, and π₀ the logging policy.

Direct method (DM). Fit a reward model r̂(x, a) on the logs, then predict the reward of π’s choice for every context: V̂ = mean[r̂(x, π(x))]. Low variance, but inherits every bias of the reward model, including the confounding problems of Module 4. It’s also the only option where π takes actions the logs never contain.

Inverse propensity scoring (IPS). Reweight matched rows by the inverse of the logging propensity:

V̂_IPS = mean[ 1{π(x) = a} · r / π₀(a | x) ]

If π₀ chose an action with probability 0.1, every matching row stands in for 10 rows. This is unbiased when propensities are correct and every action π might take had non-zero logging probability. It can have very high variance when propensities are small.

Self-normalized IPS (SNIPS). Divide by the sum of weights instead of n. Slightly biased, often much lower variance; a good default.

Doubly robust (DR). Use the reward model as a baseline and add an importance-weighted correction where actions match:

V̂_DR = mean[ r̂(x, π(x)) + 1{π(x) = a} · (r − r̂(x, a)) / π₀(a | x) ]

Accurate model → small corrections, low variance. Correct propensities → the correction removes model bias. It’s the workhorse estimator in practice.

All four in code, for a deterministic target policy (stochastic policies replace the indicator with π(a | x)):

import numpy as np

def ope(logs, target_action, reward_model, clip=None):
    """logs: dict of arrays x, a, r, p (logged propensity of a).
    target_action: array of the new policy's chosen action per row.
    reward_model(x, actions) -> predicted reward per row."""
    x, a, r, p = logs["x"], logs["a"], logs["r"], logs["p"]
    match = (target_action == a).astype(float)
    w = match / p
    if clip is not None:
        w = np.minimum(w, clip)        # trades a little bias for variance

    r_hat_target = reward_model(x, target_action)
    r_hat_logged = reward_model(x, a)

    dm = r_hat_target.mean()
    ips = np.mean(w * r)
    snips = np.sum(w * r) / np.sum(w)
    dr = np.mean(r_hat_target + w * (r - r_hat_logged))

    ess = w.sum() ** 2 / np.sum(w ** 2)   # effective sample size
    return {"dm": dm, "ips": ips, "snips": snips, "dr": dr, "ess": ess}

Report confidence intervals, not just point estimates: bootstrap the logs and recompute. OPE estimates are noisy, and two policies whose intervals overlap heavily are not distinguishable from logs.

When OPE can’t be trusted

OPE fails in predictable ways. Check for each before believing a number:

Missing support. If the new policy chooses actions the old policy never (or almost never) took for those contexts, importance weights are undefined or enormous, and only the direct method’s extrapolation remains. Measure the share of the new policy’s decisions with logged propensity below, say, 1%. If it’s large, OPE can’t evaluate this policy; you need exploration or an online test.

Extreme weights and small effective sample size. The effective sample size, (Σw)² / Σw², says how many rows are really doing the work. A million logged rows with an effective sample size of 300 is a 300-row evaluation. Weight clipping helps at the cost of some bias.

Wrong or missing propensities. If propensities were reconstructed after the fact, or the logging policy was deterministic, IPS-type estimators are unreliable. Deterministic logging (always the top-scoring action) gives propensities of exactly 0 or 1 and no information about alternatives. This is the strongest argument for randomized exploration and for logging propensities at decision time.

Non-stationarity. Logs from last year evaluate how the new policy would have done last year. If the world has changed, so has the answer.

Interference and long-term effects. OPE assumes each decision’s reward depends only on that decision. If offers change future behaviour (customers learning to wait for discounts) or affect each other (shared inventory), single-step OPE misses it.

OPE as a gate, not a verdict

The right role for OPE in a decision platform:

1. Generate candidates: retrained models, different exploration settings, new budget prices, variants of business rules.
2. Screen offline with DR and SNIPS, with confidence intervals, support checks and effective sample sizes reported for each.
3. Discard candidates that are clearly worse or that OPE can’t evaluate reliably.
4. A/B test the one or two survivors online (Lesson 20) to confirm.
5. Compare the OPE prediction with the online result over time. If OPE consistently overestimates, find out why: missing confounders in the reward model, stale propensities, drift.

Over time, the comparison in step 5 tells you how much to trust OPE for each type of change. Teams that calibrate their OPE this way end up running far fewer, far more successful online tests.

This completes the learning side of the stack: effects from experiments and logs, bandits that keep learning, and offline evaluation for new policies. Module 6 turns to the parts of decisions that aren’t learned at all: rules, policies, and where language models fit.