Overfitting, the Bias–Variance Trade-off & Regularization

A model overfits when the choices that reduce error on observed data capture noise, leakage, or accidental sample structure rather than a repeatable relationship. Zero training error is not itself a failure; it becomes a failure when unseen error is worse than an appropriately simpler or constrained procedure.

Empirical risk is only an estimate #

Training minimizes empirical risk

$$\hat R(f)=\frac1n\sum_{i=1}^{n}L(y_i,f(x_i)),$$

but the goal is expected risk on future cases

$$R(f)=\mathbb E_{(X,Y)\sim P_{deploy}}[L(Y,f(X))].$$

The training sample is finite. A flexible procedure can exploit chance correlations that lower $\hat R$ without lowering $R$. Model selection intensifies this effect: searching thousands of features or hyperparameters gives noise thousands of chances to look useful.

Bias and variance are behaviors of a procedure #

For squared-error prediction at input $x$, expected error can be decomposed as

$$\mathbb E[(Y-\hat f(x))^2] =\mathop{\mathrm{Bias}}[\hat f(x)]^2 +\mathop{\mathrm{Var}}[\hat f(x)] +\sigma^2,$$

where $\sigma^2$ is irreducible outcome noise. Bias measures systematic error across hypothetical training samples; variance measures how much the fitted predictor changes with the sample. A rigid straight line has high bias on a curved relationship but low variance. A high-degree polynomial can chase every observed point and vary wildly between samples.

This is not a claim that every modern model follows a simple U-shaped curve in parameter count. Optimization and implicit regularization can produce more complex behavior. The durable principle is that capacity and selection must be evaluated as a procedure on unseen data.

Regularization changes what counts as a good fit #

Ridge regression minimizes

$$\sum_i(y_i-x_i^Tw)^2+\lambda\lVert w\rVert_2^2.$$

The penalty makes a small reduction in residual error not worth an enormous coefficient. This stabilizes collinear directions where many coefficient combinations fit similarly. Larger $\lambda$ increases shrinkage.

Lasso uses $\lambda\lVert w\rVert_1$ and can set coefficients exactly to zero. That sparsity is useful, but with correlated features lasso may select one arbitrarily; a zero coefficient is not proof of irrelevance. Elastic net combines $L_1$ sparsity with $L_2$ stability.

Regularization appears in many forms:

• limiting tree depth or pruning weak branches;
• averaging perturbed fits in bagging;
• early stopping before an iterative learner fits residual noise;
• weight decay, dropout, and data augmentation in neural networks;
• smoothing, parameter sharing, and prior distributions;
• restricting features using domain knowledge.

All encode a preference among functions that fit the training data. Hyperparameters controlling that preference must be chosen within validation, not on the test set.

Worked mechanism: unstable polynomial interpolation #

With six noisy observations, a fifth-degree polynomial can pass through every point, so training error is zero. Near the boundary, its coefficients may cancel to fit the points and then produce huge oscillations between or beyond them. A quadratic may miss each observation slightly but recover the underlying curve.

Adding an $L_2$ penalty discourages large cancelling coefficients. The fitted curve no longer interpolates, training error rises, and validation error can fall. The improvement is not because shrinkage discovers truth directly; it reduces sensitivity to sample noise.

Learning and validation curves diagnose different failures #

A learning curve plots training and validation performance against training-set size.

• Both poor and close: likely high bias, weak features, label noise, or an objective mismatch.
• Training strong, validation weak: high variance, leakage, split mismatch, or insufficient representative data.
• Validation improves steadily with more data: collecting similar data may help.
• Both plateau with a gap across sites or time: distribution shift, not ordinary finite-sample overfit, may dominate.

A validation curve varies one capacity parameter. Interpret it only inside a sound split; leakage can make the most complex model look best.

More data is not a universal remedy #

More representative independent data reduces estimation variance. More duplicated rows, repeated measurements treated as independent, stale labels, or examples from the wrong population can reinforce the wrong relationship. If the target is unidentifiable from available features, sample size cannot create information.

Data augmentation helps when transformations preserve the label and match real variation. Horizontally flipping a traffic sign or changing a medically meaningful color may violate that assumption.

Common misconceptions #

• “The validation score is high, so there is no overfit.” Repeated tuning can overfit validation, and a mismatched split can hide leakage.
• “Regularization prevents overfitting.” It reduces certain freedoms; it cannot fix leaked labels or distribution shift.
• “A simpler model always generalizes better.” Too much constraint underfits. Simplicity is relative to data, objective, and invariances.
• “Training longer is always better.” Iterative models can first learn stable signal and later fit residual noise; monitor validation under a fixed protocol.
• “Feature selection makes the final p-values valid.” Selection changes their distribution unless inference accounts for it.

The practical target is not the smallest gap between training and validation scores. It is the lowest credible deployment loss under the constraints of the actual decision.