The calibration backtest loop
Predictions are now scored against real outcomes, calibration quality is a trended first-class metric, and corrections feed back into the personas — the loop that makes simulations provably predictive.
Plausible is not the bar — predictive is. The calibration loop closes today: every predicted behavior carries a numeric likelihood, and when reality answers (a conversion rate, an A/B result, imported survey responses), the outcome is matched to the prediction and scored. The score is derived, never asserted; both raw values stay on the auditable record.
- Outcome scoring: match a real observation to one prediction; the Brier score and hit flag follow mechanically from predicted likelihood vs observed reality.
- The reliability curve: per likelihood level, predicted mean vs observed frequency — a calibrated cohort tracks the diagonal.
- Trend as a first-class metric: every calibration report persists into a series; the trend shows the Brier delta over time and whether the loop is improving.
- The correction round: gather the misses with their evidence trails, recalibrate the personas (trait patches, new grounding from the observed reality), and stamp the round — the next reports show whether it worked.
Improvements
The hypothesis scorecard’s decisive hit rate joins the calibration report, so metric bets and behavioral predictions share one quality view.
prediction.scored and calibration.round_recorded ride the lifecycle-hook transport — wire alerts or recurring jobs to either.