Discriminant Analysis & Canonical Discriminant Functions

Discriminant analysis connects two views of classification. The generative view models how each class produces feature vectors and applies Bayes' rule. The geometric view searches for projections on which class means are far apart relative to within-class scatter. Under the linear discriminant model, these lead to the same directions.

From class-conditional densities to a decision rule #

For class $k$, assume

$$x\mid y=k\sim\mathcal N(\mu_k,\Sigma_k),\qquad P(y=k)=\pi_k.$$

Bayes' rule assigns $x$ to the class with greatest posterior probability. Terms common to every class can be removed, giving the quadratic discriminant score

$$\delta_k(x)=-\frac12\log|\Sigma_k| -\frac12(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k) +\log\pi_k.$$

The Mahalanobis term measures distance from the class mean after whitening directions by their within-class variance and correlation. A ten-unit deviation along a naturally noisy direction counts less than the same deviation along a stable direction.

Quadratic discriminant analysis (QDA) estimates a different $\Sigma_k$ for every class, so quadratic terms remain and boundaries curve. Linear discriminant analysis (LDA) assumes a shared covariance $\Sigma_k=\Sigma$. The $x^T\Sigma^{-1}x$ term then cancels between classes:

$$\delta_k(x)=x^T\Sigma^{-1}\mu_k-\frac12\mu_k^T\Sigma^{-1}\mu_k+\log\pi_k,$$

which is linear in $x$. LDA has fewer covariance parameters and can outperform QDA with limited data even when the true boundary is somewhat curved; QDA buys flexibility with much higher estimation variance.

Worked one-dimensional boundary #

Let two classes have means $\mu_1=0$ and $\mu_2=2$, shared variance 1, and equal priors. Their LDA scores are equal at $x=1$, the midpoint. If class 1 has prior $0.75$ and class 2 prior $0.25$, the boundary solves

$$-\frac12x^2+\log0.75=-\frac12(x-2)^2+\log0.25,$$

giving $x=(2+\log3)/2\approx1.55$. More evidence is now required to predict the rarer class 2. This is why silently using sample class frequencies as deployment priors can be wrong under case-control sampling or changing prevalence.

Misclassification cost is a separate decision layer. Even calibrated posteriors should be thresholded according to consequences, not automatically at the most probable class.

Fisher's canonical direction #

For two classes, let $S_W$ be within-class scatter and $d=\mu_1-\mu_2$. Projecting onto vector $w$ produces between-class separation $(w^Td)^2$ and within-class spread $w^TS_Ww$. Fisher's criterion is

$$J(w)=\frac{(w^Td)^2}{w^TS_Ww}.$$

Its maximizing direction is proportional to

$$w\propto S_W^{-1}(\mu_1-\mu_2).$$

This is the whiten-then-separate mechanism: directions with large within-class noise are discounted, and correlated measurements are not double-counted.

For $K$ classes, define between-class scatter

$$S_B=\sum_k n_k(\mu_k-\mu)(\mu_k-\mu)^T.$$

Canonical discriminant directions solve the generalized eigenproblem

$$S_Bw=\lambda S_Ww.$$

Because $K$ class means occupy an affine subspace of dimension at most $K-1$, there are at most $K-1$ discriminant coordinates. The eigenvalues rank directions by between-class separation relative to within-class variation.

flowchart TD
  X["Correlated measurements"] --> W["Whiten by within-class covariance"]
  W --> M["Find directions separating class means"]
  M --> C["Project to at most K−1 canonical coordinates"]
  C --> D["Classify with priors and decision costs"]

LDA is not PCA #

PCA ignores labels and preserves directions of greatest total variance. LDA uses labels and preserves directions with high between-class relative to within-class scatter. If most variation is caused by an irrelevant operating condition, PCA may retain it while discarding a low-variance but perfectly discriminative signal. Conversely, LDA can overstate separation when labels are noisy or reused for both fitting and visualization.

Canonical plots are fitted supervised views. Apparent group separation in the same data used to choose the axes is optimistic; assess classification and separation on held-out observations.

Covariance is the fragile component #

With $p$ features, an unrestricted covariance matrix has $p(p+1)/2$ parameters. When $p$ is large relative to class sample sizes, empirical covariance becomes unstable or singular. Remedies include shrinkage toward a diagonal or spherical target, dimensionality reduction fitted inside validation, or a model with stronger structure. Explicitly inverting covariance is numerically unnecessary; stable solvers use matrix factorization or solve linear systems.

Check class-conditional shape, covariance differences, outliers, prior shift, and performance by class. Gaussian densities can yield useful linear boundaries even when normality is imperfect, but probability estimates and extrapolation are more sensitive than the boundary itself.

Misconceptions and limits #

• “Linear” describes the LDA decision surface, not necessarily a causal relationship.
• QDA is not automatically better because it is more flexible; small classes make their covariance estimates especially noisy.
• A discriminant loading is not an independent feature effect. Correlation lets coefficients change sign or magnitude when other variables enter.
• Separation in canonical coordinates does not prove natural clusters; the labels constructed the coordinates.
• Equal training accuracy under different priors does not imply equal operational decisions.

Discriminant analysis is most useful when its distributional structure is plausible, covariance can be estimated responsibly, and the posterior is kept separate from the cost-sensitive action taken afterward.