Contextual Bandits for Personalized Decisions

A multi-armed bandit learns which action is best overall. A contextual bandit learns which action is best for each situation: at each decision it observes a context x (customer features, time, channel), chooses an action, gets a reward for that action only, and updates a model of how reward depends on context and action.

It sits between the uplift models of Module 4 and full reinforcement learning. Like uplift modeling, it estimates action effects per context. Unlike uplift modeling, it collects its own data as it goes, exploring exactly the (context, action) combinations it’s least sure about. Unlike full RL, it assumes actions don’t change future contexts, which holds well enough for many business decisions (one offer per customer per month).

LinUCB

The classic algorithm (Li et al., 2010, from personalized news at Yahoo!) models each arm’s expected reward as linear in the context, and adds an exploration bonus based on uncertainty:

import numpy as np

class LinUCB:
    """Disjoint LinUCB: one ridge regression per arm."""
    def __init__(self, arms, dim, alpha=1.0):
        self.alpha = alpha
        self.A = {a: np.eye(dim) for a in arms}    # X^T X + I
        self.b = {a: np.zeros(dim) for a in arms}  # X^T y

    def scores(self, x, eligible):
        out = {}
        for a in eligible:
            A_inv = np.linalg.inv(self.A[a])
            theta = A_inv @ self.b[a]               # ridge coefficients
            mean = theta @ x
            bonus = self.alpha * np.sqrt(x @ A_inv @ x)  # uncertainty for THIS x
            out[a] = mean + bonus
        return out

    def choose(self, x, eligible):
        s = self.scores(x, eligible)
        return max(s, key=s.get)

    def update(self, a, x, reward):
        self.A[a] += np.outer(x, x)
        self.b[a] += reward * x

The bonus term is large for contexts unlike the ones this arm has seen, so the algorithm tries arms on unfamiliar customer types, and small for familiar contexts, where it exploits. The alpha parameter controls optimism. In production you’d cache A_inv and update it with the Sherman-Morrison formula instead of inverting every time.

LinUCB is deterministic: the same context always gets the same action, until the model changes. That makes propensities degenerate (0 or 1), which is a problem for offline evaluation. Many teams add a small epsilon of randomization for that reason alone.

Thompson sampling with context

Linear Thompson sampling replaces the UCB bonus with a posterior sample: draw coefficients θ̃ from a Gaussian centred on the ridge estimate with covariance proportional to A⁻¹, and pick the arm with the highest θ̃ᵀx. It’s randomized by design, so propensities are well defined.

Linear models are often too simple. For richer models:

• Bootstrapped Thompson sampling: train an ensemble of models (say 10) on bootstrap resamples of the data; at each decision, pick one ensemble member at random and act greedily on it. Works with gradient-boosted trees or neural nets.
• Neural-linear: learn a representation with a neural network, then run linear Thompson sampling on its last layer.
• Epsilon-greedy over a strong model: often surprisingly competitive. A well-tuned gradient-boosted reward model with 5% uniform exploration among eligible actions is a respectable production baseline, and easy to explain and evaluate.

Action features

So far each arm has its own model. That’s fine for five fixed offers, poor for a catalogue that changes. When a new offer launches, its model knows nothing.

The fix is to describe actions with features too (discount depth, duration, channel, offer type) and learn one shared model of reward as a function of context and action features together, including interactions. Now a new “15% for 2 months” offer borrows strength from existing 10% and 25% offers, and needs far less exploration. Vowpal Wabbit’s contextual bandit mode with action-dependent features (--cb_explore_adf) is a mature, fast implementation of this pattern.

Reward design

The bandit optimizes exactly what you reward. This is where most bandit projects go wrong.

• Reward the business objective, net of cost. For retention, the reward is not “accepted the offer” but something like retained margin minus offer cost. Otherwise the bandit learns to give the biggest discount to everyone.
• Prefer incremental rewards when you can. Rewarding retention directly rewards giving offers to sure things, because they’re retained. With a “none” arm in the bandit and enough exploration, the model learns that “none” also retains sure things, and stops paying them. That only works if “none” is genuinely an arm.
• Use proxies carefully. Delayed rewards push teams to fast proxies (click, open, 7-day engagement). Validate that the proxy ranks actions the same way the real outcome does, and periodically retrain against real outcomes.
• Watch for gaming. If the reward can be inflated by an action without real value (spammy subject lines that get opens), the bandit will find it.

The bandit as a service

A production contextual bandit is a small system:

flowchart LR
    R["Decision request\ncontext x"] --> E["Eligibility rules"]
    E --> P["Policy: score eligible actions\nsample with exploration"]
    P --> L["Log: x, eligible set,\naction, propensity, model version"]
    L --> ACT["Serve action"]
    ACT -.->|"reward arrives later"| J["Join rewards to logs"]
    J --> T["Retrain / update model\nhourly or daily"]
    T --> P
    J --> OPE["Offline evaluation\nof candidate policies"]

    style R fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style E fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style P fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style L fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style ACT fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style J fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style T fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style OPE fill:#f0fdf9,stroke:#0D9488,color:#0F172A

The essential pieces:

• The log is the product. Context, eligible set, chosen action, propensity, model version and timestamp for every decision. Rewards are joined later by decision ID. This log is what you’ll train and evaluate on forever.
• Features at decision time. Train on the features as they were when the decision was made, not as they are now. Point-in-time correct feature retrieval prevents leakage.
• Separate update from serving. Serving reads a model snapshot; training produces a new one on a schedule. That keeps latency predictable and makes rollback trivial.
• Bounded exploration. Keep each eligible action’s probability within agreed limits, and respect budget prices from Lesson 7 by subtracting λ × cost from each action’s score.

Contextual bandits vs reinforcement learning

If your actions change the future context (today’s discount changes how the customer responds to tomorrow’s offer, a price change shifts future demand), you’re in reinforcement learning territory. Full RL is powerful but data-hungry, hard to evaluate and risky to run on live customers. Most business decisions are well served by contextual bandits with constraints that encode the dynamics you care about (offer frequency caps, fatigue features in the context). Reach for RL when the sequential effects are large and you have a good simulator (Lesson 24) to train and test against.

Bandits keep trying new policies. Before putting one in front of customers, you want to know whether it’s any good. The next lesson shows how to evaluate a new policy using only the logs of the old one.