Predict-then-Optimize and Its Pitfalls

The obvious architecture for a data-driven decision is two steps. Predict the uncertain inputs (demand, value, risk) with ML. Optimize the decision with those predictions plugged in as if they were true. It’s called predict-then-optimize, and it’s how most production systems work, including the one you built in the last lesson.

It works. But it has failure modes that surprise teams, because the two halves were built separately and each one assumes the other is perfect. This lesson is about where it breaks and what to do.

The optimizer’s curse

Suppose you estimate the value of 1,000 possible marketing campaigns. Each estimate is unbiased: on average it’s right. You fund the 20 with the highest estimated value. When results come in, those 20 disappoint, consistently, by a lot. Why?

Because you didn’t pick 20 at random. You picked the 20 with the highest estimates, and an estimate can be high for two reasons: the campaign is good, or the noise was large and positive. Selecting the top preferentially selects positive noise. The chosen campaigns’ estimates are biased upwards because you chose them by maximizing.

import numpy as np
rng = np.random.default_rng(0)

true_value = rng.normal(100, 10, size=1000)       # what each option is really worth
estimate = true_value + rng.normal(0, 20, 1000)    # unbiased, noisy estimates

top = np.argsort(estimate)[-20:]
print("estimated value of chosen:", estimate[top].mean())    # around 150
print("realized value of chosen: ", true_value[top].mean())  # around 110

This is the optimizer’s curse (Smith and Winkler, 2006). It isn’t a bug in your model. It’s what happens whenever you maximize over noisy estimates, and an optimizer is a very effective maximizer. The noisier the estimates and the more options you choose among, the worse it gets.

In the retention example, the customers with the highest estimated uplift are partly customers whose uplift is high and partly customers whose estimate is noisy (small segments, unusual features). The plan will overpromise, and finance will notice.

The solver exploits errors

A related problem: optimizers are drawn to the places where predictions are wrong in their favour.

If a pricing model slightly overestimates demand at high prices for one product, the optimizer will price that product high. If a routing model underestimates travel time on one road, every route will use it. The model’s average error might be tiny, but the optimizer doesn’t care about the average. It goes looking for the most favourable inputs, and those are disproportionately the erroneous ones.

That’s why “the model has a low MSE” is not reassuring on its own. MSE weights all errors equally. The decision cares enormously about errors on the items, regions and directions where it acts, and not at all about the rest.

Evaluate decisions, not just predictions

The first fix is measurement. Evaluate the whole pipeline by the quality of the decisions it produces, on held-out data with realized outcomes:

def regret(plan_from_predictions, outcomes, solve, evaluate):
    """solve(inputs) -> plan; evaluate(plan, outcomes) -> realized objective."""
    hindsight_plan = solve(outcomes)                 # best plan with perfect info
    best = evaluate(hindsight_plan, outcomes)
    actual = evaluate(plan_from_predictions, outcomes)
    return best - actual                             # >= 0; lower is better

# Compare two forecasting models by the regret of the plans they produce,
# averaged over many held-out weeks, not by MSE alone.
for name, model in [("gbm", gbm), ("gbm_quantile", gbm_q)]:
    r = [regret(solve(model.predict(X_w)), y_w, solve, evaluate)
         for X_w, y_w in held_out_weeks]
    print(name, np.mean(r))

Regret is the gap between the value of the decision you made and the value of the best decision in hindsight. It’s the decision-level counterpart of prediction error. Model selection, feature work and hyperparameter tuning should all look at it.

Two cautions. First, evaluate the plan on outcomes it didn’t see, never on the same data used to estimate the values; otherwise the optimizer’s curse inflates your evaluation exactly as it inflates your plan. Second, regret evaluation assumes you can score a plan on realized outcomes. For decisions where the action changes the outcome (offers, prices), you need the causal and off-policy tools of Modules 4 and 5 to do that honestly.

Practical mitigations

Before reaching for anything sophisticated, these simple fixes go a long way:

Shrinkage. Pull noisy estimates towards a sensible prior before optimizing, more strongly for estimates with more uncertainty. Empirical Bayes shrinkage, hierarchical models, or simply blending a segment-level estimate with the global mean. Shrinkage directly counteracts the optimizer’s curse.

Penalize uncertainty. Optimize a pessimistic estimate: the mean minus k standard deviations, or a lower quantile. Options with uncertain value need to look better to be chosen. This is the same idea as the robust methods in the next lesson.

Constrain the plan to where the model is trustworthy. Limit how far prices can move from historical ranges, how much any single item’s quantity can change week to week, how concentrated the allocation can be. These “trust region” constraints keep the optimizer from wandering into regions with no data.

Hold out the chosen plan’s evaluation. Use one sample to estimate values and select, and an independent sample to estimate the value of what you selected. The second estimate is unbiased; report that number to finance.

Decision-focused learning

The more ambitious fix is to train the predictive model for the decision: instead of minimizing prediction error, minimize the regret of the decisions made from its predictions. This is called decision-focused learning or “smart predict-then-optimize”.

The challenge is that the optimizer’s output is a piecewise-constant function of its inputs, so gradients through it are zero almost everywhere. Approaches include:

• SPO+ loss (Elmachtoub and Grigas): a convex surrogate for regret in linear optimization problems that gives useful gradients.
• Differentiable optimization layers: solve a smoothed (regularized) version of the optimization so it can be differentiated.
• Perturbation methods: add noise to the inputs and average solutions, which smooths the decision as a function of the predictions.
• Weighted losses: a simple approximation that weights each training example’s prediction error by how much it matters to decisions, for example by the item’s margin or the sensitivity of the plan to it.

Decision-focused learning pays off when the model can’t be accurate everywhere (limited features, misspecified model) and should spend its accuracy where decisions are sensitive. If a flexible model already predicts well, two-step training is close to optimal and much simpler to operate. The weighted-loss approximation is a sensible first step: it fits into any existing training pipeline.

A checklist for predict-then-optimize systems

• Report regret on held-out periods next to MSE, MAPE or AUC.
• Shrink noisy estimates, especially for small segments and new items.
• Add trust-region constraints around historical behaviour.
• Estimate the value of the chosen plan on independent data.
• Watch for the optimizer concentrating on a few items, segments or routes; it’s often chasing errors.

So far the predictions going into the optimizer have been point estimates. The next lesson brings in the uncertainty from Lesson 6 and optimizes against it directly.