Decision Services in Production

The previous lessons built components: calibrated models, cost-based thresholds, optimizers, uplift estimators, bandits, rules, LLM extractors, guardrails. In production, they have to run together as one reliable system that answers “what should we do for this unit?” quickly and consistently, and that remembers every answer it gave and why.

The architecture

A decision service has a real-time path and a batch path that share the same policy definition and write to the same decision log:

flowchart TB
    subgraph Batch["Batch path (daily / monthly)"]
        BF["Point-in-time features"] --> BS["Score all units x actions"]
        BS --> OPT["Optimizer\nallocate budget\npublish lambda"]
    end
    subgraph RT["Real-time path (p99 < 100 ms)"]
        REQ["Request"] --> RF["Online features"]
        RF --> RS["Score eligible actions"]
        RS --> SEL["Select: value - lambda x cost\n+ exploration"]
    end
    PB["Policy bundle vN\nrules, models, calibrators,\nobjective, guardrails"] --> BS
    PB --> RS
    OPT -->|"lambda"| SEL
    OPT --> GR["Guardrails"]
    SEL --> GR
    GR --> LOG["Decision log\n(append-only)"]
    LOG --> EVAL["Evaluation, OPE,\nmonitoring, audit, retraining"]

    style PB fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style LOG fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style OPT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SEL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style GR fill:#fef2f2,stroke:#ef4444,color:#0F172A
    style EVAL fill:#EEF0F7,stroke:#6366F1,color:#0F172A

The batch path handles global allocation: score every unit, solve the optimization, publish the decisions and the budget price λ. The real-time path handles one unit at a time within a latency budget, using λ to stay consistent with the global plan (Lesson 7). Both load the same policy bundle, so eligibility rules and objectives can’t drift apart between paths.

The policy bundle

A decision depends on everything: rules version, model version, calibrator, objective weights, cost parameters, exploration rate, guardrail limits, λ. Version them together as one immutable bundle:

policy: retention_offer
version: 2026-10-14.3
rules: eligibility_v7                      # decision table, Lesson 16
models:
  uplift: uplift_xlearner@sha256:9f2c...   # Lesson 11
  calibrator: isotonic@sha256:41aa...      # fit with that exact model, Lesson 4
signals:
  ticket_extractor: prompt_v12 + model pinned to a dated version   # Lesson 17
objective:
  horizon_months: 12
  contact_annoyance_cost: 0.80             # agreed with CX, Lesson 2
exploration:
  floor_per_action: 0.02
  holdout_share: 0.05                      # Lesson 10
budget:
  monthly_cap: 120000
  lambda_source: batch_job/retention_lambda   # refreshed daily
guardrails:
  max_share_25pct: 0.15
  max_daily_spend: 6000
fallback: rules_only_no_discount

Every decision records the bundle version. A rollback is switching the active bundle pointer. A new model can’t accidentally ship with an old calibrator, because they’re released as a unit.

The decision log

The decision log is the most important table you’ll build. It powers evaluation (Lesson 19), off-policy evaluation (Lesson 15), retraining, fairness audits (Lesson 22), monitoring, appeals and incident investigation. Design its schema carefully:

from dataclasses import dataclass, field
from datetime import datetime

@dataclass(frozen=True)
class DecisionRecord:
    decision_id: str              # idempotency key: same request -> same record
    unit_id: str
    decided_at: datetime
    path: str                     # "batch" | "realtime"
    policy_version: str           # the bundle
    features_ref: str             # pointer to the exact feature snapshot used
    eligible_actions: list[str]
    scores: dict[str, float]      # value per eligible action, pre-selection
    chosen_action: str
    propensity: float             # P(chosen_action | context) under the policy
    in_holdout: bool
    reason_codes: list[str]       # rule hits + top model drivers
    guardrail_flags: list[str] = field(default_factory=list)
    fallback_used: str | None = None

Key properties:

• Append-only. Records are never edited. Outcomes and reviewer overrides are joined later from separate tables keyed by decision_id.
• Point-in-time features. Store the features (or an exact pointer to them) as they were at decision time. Training on features as they are today leaks the future, and it’s one of the most common reasons offline results don’t hold up.
• Scores for all eligible actions, not just the chosen one. Off-policy evaluation and debugging both need them.
• Propensity and holdout flag, for unbiased evaluation.
• Idempotency. Callers retry. The same request ID must return the same decision, not a fresh random draw.
• Retention set by regulatory requirements, which for credit and insurance can be years.

Deterministic replay

Given the log, you should be able to replay any past decision: load the bundle version and feature snapshot, rerun, and get the same result. Replay supports:

• Audits and appeals: show exactly why a decision was made.
• Debugging: reproduce a strange decision locally.
• Regression testing: replay last week’s traffic through a candidate bundle and diff the decisions before any rollout.

Make replay deterministic by seeding any randomness (exploration draws) from the decision ID and pinning model versions, including any LLM model version used for signals. Where an external component can’t be made deterministic, log its outputs so replay can reuse them.

Safe rollout

Policy changes are deployments with real-money consequences. Roll them out in stages:

1. Offline: OPE on historical logs (Lesson 15); replay diff on recent traffic.
2. Shadow mode: the candidate runs on live traffic in parallel, logging what it would decide, without acting. Compare action mix, spend rate, latency and guardrail hits against the live policy. No customer impact, so you can run it for as long as you need.
3. Canary: a small share of live decisions (1 to 5%) uses the candidate, with tight guardrails and automatic rollback on anomalies.
4. Experiment: the A/B test from Lesson 20.
5. Rollout, with the holdout continuing.

Latency and fallbacks

Real-time decisions have budgets (often 50 to 100 ms for the whole call). Spend them deliberately:

• Precompute anything you can in batch: LLM-extracted signals, customer embeddings, segment-level values.
• Keep online models small enough to score all eligible actions within budget; per-action scoring multiplies latency by the number of actions, so batch the inference.
• Set timeouts on every dependency, with a predefined fallback: the batch decision for this unit if one exists, else a rules-only policy, else “no action”.
• Record fallbacks in the log (fallback_used) and monitor their rate. A creeping fallback rate is an outage in slow motion, and fallback decisions must be excluded or handled separately in evaluation.

Ownership

Finally, the organizational piece. A decision service needs a named decision owner (business, accountable for outcomes and trade-offs), an engineering owner (accountable for the service), and clear processes for changing rules, objectives and models. Without that, even a well-built service slowly turns back into the “score and walk away” anti-pattern from Lesson 1.

One question remains before the capstone: how do you test a policy against futures that haven’t happened yet? That’s simulation, next.