Forecast Calibration & the Brier Score

A forecaster is calibrated when probabilities mean what they say across comparable cases. Among many forecasts near 70%, the event should occur about 70% of the time. Calibration evaluates the language of probability itself.

Reliability and sharpness #

Group forecasts into ranges, then compare average forecast $\bar q_b$ with observed frequency $\bar y_b$ in each bin $b$. A reliability diagram plots

$$(\bar q_b,\bar y_b).$$

Perfect calibration lies on $y=x$. Bins above the diagonal were underconfident; bins below it were overconfident.

Calibration alone is not enough. Always predicting the 60% base rate can be calibrated but not sharp. Sharpness means making meaningfully different probabilities when evidence supports them, subject to staying calibrated.

Brier score #

For forecast $q\in[0,1]$ and outcome $y\in\{0,1\}$,

$$BS=(q-y)^2, \qquad \overline{BS}=\frac1n\sum_{i=1}^n(q_i-y_i)^2.$$

Lower is better. A 0.70 forecast scores 0.09 when YES occurs and 0.49 when NO occurs. The Brier score is strictly proper in this binary loss convention: truthful probabilities minimize expected loss when the belief is correct.

Running 65% example #

If $q=0.65$ is issued 100 times, calibration suggests about 65 YES outcomes. One 100-case batch can deviate through sampling variation; use uncertainty bands and multiple periods rather than demanding exactly 65.

Application to no-hedge #

Calibrate the probabilities available at execution time on filled orders. A reference price may be calibrated overall but miscalibrated precisely when maker orders fill. Report reliability by price band, venue, staleness, time-to-resolution, and fill condition; the adverse-selection slice is the economically relevant one.