Goodness-of-Fit Tests: Chi-Square & Kolmogorov–Smirnov

A goodness-of-fit (GOF) test asks whether data are compatible with a specified probability model. It does not certify that the model is true. Every finite dataset is compatible with many alternatives, and every imperfect model will eventually be rejected with enough data. The useful question is whether the discrepancy measured by the chosen statistic is too large to attribute to sampling variation under the null model.

The chi-square and Kolmogorov–Smirnov (KS) tests look similar only at the decision layer. They compare different objects and are sensitive to different departures.

The common testing mechanism #

A GOF procedure needs four explicit pieces:

1. A null distribution $F_0$ or category probabilities $p_1,\ldots,p_k$.
2. A statistic $T$ that compresses a discrepancy into one number.
3. The sampling distribution of $T$ when the null is true, including any parameters estimated from the same data.
4. A predeclared decision or diagnostic use.

The p-value is $P(T\geq T_{obs}\mid H_0)$ for a statistic where larger means worse fit. It is not $P(H_0\mid\text{data})$, nor is $1-p$ the probability that an alternative is true.

Chi-square: discrepancy in counts #

For $k$ mutually exclusive categories, let $O_i$ be observed counts and $E_i=np_i$ expected counts under the null. Pearson's statistic is

$$X^2=\sum_{i=1}^{k}\frac{(O_i-E_i)^2}{E_i}.$$

Why divide by $E_i$? A deviation of ten counts is surprising when ten were expected but ordinary when ten thousand were expected. Under independent sampling and sufficiently informative expected counts, $X^2$ is approximately chi-square distributed.

If all category probabilities are specified in advance, the degrees of freedom are $k-1$: once $n$ and $k-1$ counts are known, the last count is fixed. Estimating $m$ independent parameters from the same observations usually reduces this to $k-1-m$. Ignoring that reduction makes the reference distribution wrong.

Worked count example #

Sixty throws of a proposed fair die give counts $[8,9,10,11,12,10]$. The expected count is 10 in every cell:

$$X^2=\frac{(8-10)^2+(9-10)^2+0^2+(11-10)^2+(12-10)^2+0^2}{10}=1.0.$$

With five degrees of freedom, this is far below the usual 5% critical value. The data do not provide evidence against fairness—but they also do not prove exact fairness. A die whose probabilities differ by tiny amounts could easily produce these counts.

Sparse expected cells undermine the chi-square approximation. The remedy is not blind adherence to a universal “five per cell” slogan: combine categories only when that preserves the scientific question, or compute an exact/Monte Carlo null distribution. Binning a continuous variable also discards information and makes the result depend on arbitrary cut points.

Kolmogorov–Smirnov: the largest CDF gap #

For a continuous sample $X_1,\ldots,X_n$, the empirical cumulative distribution function is

$$F_n(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbf 1(X_i\leq x).$$

The one-sample KS statistic is

$$D_n=\sup_x|F_n(x)-F_0(x)|.$$

It measures the largest vertical separation between the empirical and hypothesized CDFs. Because a CDF integrates density, KS avoids bins and detects shifts in location, scale, or shape. It is usually most sensitive near the center and less sensitive in the tails.

The familiar distribution-free KS critical values apply to a continuous, fully specified $F_0$. If a normal mean or variance is estimated from the tested sample, the ordinary KS calibration no longer applies; use an appropriate correction or a parametric bootstrap that repeats parameter estimation inside every simulated sample. Ties or discrete outcomes also change the null distribution.

flowchart TD
  H["State the complete null model"] --> S["Choose a discrepancy that sees the failure of interest"]
  S --> C["Calibrate it under the actual fitting procedure"]
  C --> P["Compute statistic and p-value"]
  P --> D["Inspect where and how the model misses"]

Choosing a test by the failure you care about #

Always pair a test with an informative plot: observed versus expected counts with residuals, an empirical-versus-model CDF, a Q–Q plot, and tail diagnostics. The plot locates the failure that the p-value only ranks.

Counterexamples and misconceptions #

• Large samples reject harmless imperfections. If $F_0$ is a convenient approximation rather than a physical law, effect sizes and consequences matter more than eventual rejection.
• Small samples fail to detect consequential alternatives. “Not significant” can mean insufficient resolution.
• Trying many distributions and reporting the best p-value invalidates the nominal interpretation. Model selection has consumed the same data.
• A fitted histogram is not an independent null. Choosing bins after seeing the data changes the question.
• Passing normality does not validate a downstream model. Independence, variance structure, functional form, and measurement process may still fail.

A GOF test is therefore a calibrated diagnostic for one discrepancy—not a universal model-quality score.