Probability Calibration for Trading

A trading model that says 70% should resolve YES about 70% of the time across comparable cases. This is calibration. Trading adds a crucial qualifier: evaluate the probability that existed at decision or execution time on the subset of positions actually taken.

Reliability by trading slice #

For forecast bin $b$,

$$forecast_b=\frac1{|b|}\sum_{i\in b}q_i, \qquad outcome_b=\frac1{|b|}\sum_{i\in b}Y_i.$$

Plot $(forecast_b,outcome_b)$ with uncertainty bands. Useful slices include venue, price band, time to resolution, reference age, maker/taker role, fill status, order size, and news regime.

Probability-sensitive loss #

For each binary forecast,

$$BS_i=(q_i-Y_i)^2.$$

Lower mean Brier score is better on the same set of cases. It distinguishes 0.51 from 0.99 even when both become the same YES classification.

Running 65% example #

If $q=0.65$ is issued 100 times, about 65 YES outcomes are expected. Suppose all opportunities show 65%, but the 40 that fill resolve YES only 20 times. Overall calibration can look good while fill-conditioned calibration is 50%, revealing adverse selection.

Life example #

A doctor may diagnose accurately in routine appointments but poorly in the difficult cases referred by specialists. Evaluating only the combined population hides selection into the treated group.

Application to no-hedge #

Store immutable execution-time inputs: reference VWAP, age, mapping, fair value, $\delta$, maker price, fees, and model version. Then compute reliability and Brier score on fills and rejected opportunities separately. No-hedge edge exists only if the fill-conditioned forecast remains calibrated enough after costs.