Simulation & Stress Testing Decision Policies

Every evaluation method so far looks at the past. Holdouts and experiments measure what happened. Off-policy evaluation reweights what happened. But decision owners also ask about things that haven’t happened: What if churn doubles? What if finance cuts the budget by half mid-quarter? What if customers learn to game the cancel-page offer? What happens after a year of this policy, not just a month?

Those are questions for simulation: a model of the world, not just of one outcome, that you can run policies against under conditions you choose.

What simulation adds

Simulation earns its place in three situations:

• Counterfactual conditions. Shocks that never occurred in the logs (a demand collapse, a fraud attack, a budget cut), so neither OPE nor past experiments cover them.
• Long-horizon dynamics. Effects that build over many decisions: offer fatigue, customers learning to wait for discounts, inventory carrying over between weeks, credit exposure accumulating. Single-step methods miss these.
• Rare, high-cost events. Tail scenarios you’d never want to discover in production.

Simulation does not replace experiments. A simulator is itself a model, built from assumptions, and its conclusions are only as good as those assumptions. Treat it as a stress-testing and screening tool.

Kinds of simulators

• Replay with perturbation. Take historical data and modify it: scale base rates, remove a share of budget, inject a shock. The cheapest option, limited to changes that don’t require modeling behaviour.
• Agent-based simulators. Model each customer (or driver, applicant, store) as an agent with a state and a response model: how likely they are to churn, how they respond to each offer, how their state changes over time. Run many agents through many periods under each policy.
• Digital twins. Detailed operational simulators of a supply chain, warehouse or network, often built with discrete-event simulation. Common in operations decisions.

For most customer-facing decisions, a modest agent-based simulator is the sweet spot.

A simple customer simulator

Here’s the skeleton of an agent-based simulator for the retention example. Each month, every customer has a churn hazard depending on their state; offers reduce it according to an uplift model; repeated discounts lose effect (fatigue); and a share of customers learn to visit the cancel page to get offers.

import numpy as np
from dataclasses import dataclass

@dataclass
class World:
    base_hazard_scale: float = 1.0      # stress: >1 means more churn
    offer_effect_scale: float = 1.0     # stress: <1 means offers work less
    fatigue_per_offer: float = 0.35     # each past discount cuts effect by 35%
    gaming_rate: float = 0.0            # share of offered customers who learn to game

def simulate(policy, customers, world: World, months=12, seed=0):
    """customers: dict of arrays (features, monthly_margin, baseline hazard
    and per-offer uplift from the fitted models). policy(state) -> action array.
    Returns total margin net of offer costs, for one simulated year."""
    rng = np.random.default_rng(seed)
    n = len(customers["margin"])
    active = np.ones(n, bool)
    past_offers = np.zeros(n)
    gamer = np.zeros(n, bool)
    total = 0.0
    for m in range(months):
        state = {**customers, "active": active, "past_offers": past_offers, "gamer": gamer}
        action = policy(state)                          # 0 = none, else offer id
        offered = (action > 0) & active
        effect = customers["uplift"][np.arange(n), action] * world.offer_effect_scale
        effect *= (1 - world.fatigue_per_offer) ** past_offers   # fatigue
        hazard = customers["hazard"] * world.base_hazard_scale - np.where(offered, effect, 0)
        hazard = np.where(gamer & ~offered, hazard * 1.5, hazard)  # gamers churn if ignored
        churn = active & (rng.random(n) < np.clip(hazard, 0, 1))
        active &= ~churn
        total += customers["margin"][active].sum()
        total -= customers["offer_cost"][np.arange(n), action][offered & active].sum()
        past_offers += offered
        gamer |= offered & (rng.random(n) < world.gaming_rate)
    return total

Every number in this simulator is an assumption: the hazard model, the uplift model, the fatigue rate, the gaming behaviour. The first two come from the fitted models of Modules 2 and 4. The others come from evidence where it exists (past repeat-offer data for fatigue) and from explicit, documented guesses where it doesn’t.

Calibrate, then validate

A simulator must reproduce reality before it can say anything about counterfactuals:

1. Calibrate to history: simulate the historical policy over a past period and tune unknown parameters until simulated churn rates, offer acceptance and margin match what actually happened, overall and by key segments.
2. Validate out of sample: simulate a period, or better a past experiment, the simulator wasn’t calibrated on. If a real A/B test showed policy B beat A by 4%, does the simulator show something similar? Reproducing known policy differences is the most convincing evidence that the simulator can rank new ones.
3. Document every assumption, its source, and how sensitive results are to it.

A simulator that can’t reproduce a known experiment shouldn’t be used to pick policies.

Stress testing

With a validated simulator, run each candidate policy across a grid of scenarios, with many random seeds per scenario:

scenarios = {
    "base": World(),
    "churn_shock_x2": World(base_hazard_scale=2.0),
    "offers_weaken": World(offer_effect_scale=0.5),
    "fast_fatigue": World(fatigue_per_offer=0.6),
    "gaming": World(gaming_rate=0.15),
}
results = {
    (pname, sname): np.mean([simulate(p, customers, w, seed=s) for s in range(30)])
    for pname, p in policies.items()
    for sname, w in scenarios.items()
}

Read the results as a policy × scenario table. What you’re looking for:

• Fragile winners: policies that win in the base case but lose badly in a plausible stress scenario. A 3% base-case gain isn’t worth a 20% loss if offers decay faster than expected, unless you can detect that scenario quickly and switch.
• Robust choices: policies that are rarely best but never bad. Often the right choice under uncertainty, in the same spirit as Lesson 9.
• Breaking points: at what churn shock, budget cut or gaming rate does a policy stop being profitable? Those thresholds become monitoring alerts (Lesson 21).
• Long-term divergence: policies that look similar over one month but diverge over twelve, typically because of fatigue or gaming. Exactly what single-step OPE can’t see.

The sim-to-real gap

The temptation, once a simulator exists, is to optimize policies inside it, perhaps with reinforcement learning. Be careful. An optimizer trained against a simulator will exploit the simulator’s inaccuracies, exactly as Lesson 8’s optimizer exploited prediction errors. The policy learns to win in the simulated world.

Use simulation to generate and screen candidates and to stress test them, then validate with OPE on real logs and with real experiments. Keep the simulator honest by re-validating it against every new experiment result: each one is a free test of whether its predictions hold.

Synthetic customers generated by LLMs (“simulate how this customer persona responds to the offer email”) are sometimes proposed as a shortcut. They can help brainstorm scenarios and message variants, but their responses aren’t calibrated to your customers’ real behaviour. Never use them as evidence of effect size.

With simulation, the toolkit is complete. The capstone puts every piece together into one system.