Principal Components & Factor Analysis: Two Different Questions

Principal component analysis (PCA) and factor analysis (FA) both replace many correlated variables with fewer linear combinations. That superficial similarity causes a persistent mistake: treating rotated “factors” as though PCA had discovered latent causes. PCA is principally a geometric compression; FA is a probabilistic latent-variable model.

PCA asks which subspace preserves variation #

Let $X\in\mathbb R^{n\times p}$ contain $n$ centered observations. The first principal direction is the unit vector $v_1$ maximizing projected sample variance:

$$v_1=\arg\max_{\lVert v\rVert=1}v^TSv,$$

where $S=X^TX/(n-1)$ is the sample covariance matrix. A Lagrange multiplier gives

$$Sv_1=\lambda_1v_1,$$

so $v_1$ is the eigenvector of $S$ with largest eigenvalue. Later components maximize remaining variance while staying orthogonal. The score matrix for $q$ components is $Z=XV_q$; reconstruction is $\hat X=ZV_q^T$.

The same solution minimizes squared reconstruction error among rank-$q$ linear projections:

$$\min_{V_q^TV_q=I}\lVert X-XV_qV_q^T\rVert_F^2.$$

This equivalence explains PCA's utility: variance kept is squared reconstruction error avoided. It does not imply that high variance is scientifically important. A low-variance direction may carry a class boundary or a safety signal.

In practice, compute PCA by singular value decomposition $X=U\Sigma V^T$, not by explicitly forming and diagonalizing $X^TX$ when numerical conditioning matters. Component signs are arbitrary: $(v,z)$ and $(-v,-z)$ encode the same axis.

A two-variable example #

For centered observations $(-1,-1),(0,0),(1,1)$, the sample covariance matrix is

$$S=\begin{bmatrix}1&1\\1&1\end{bmatrix}.$$

Its eigenvectors are $v_1=(1,1)/\sqrt2$ with eigenvalue 2 and $v_2=(1,-1)/\sqrt2$ with eigenvalue 0. Projecting onto $v_1$ preserves all observed variance; the discarded direction contains none. This is an idealized exact line. With noise, the second eigenvalue would be small rather than zero, and selecting one component would trade some reconstruction fidelity for compression.

Whether to use covariance or correlation PCA is substantive. A variable measured in thousands can dominate covariance PCA merely because of units. Standardizing every variable gives each equal marginal variance, which may also be wrong when physical scale is meaningful or high-noise variables should not receive equal influence.

Factor analysis asks what latent variables generated covariance #

The common factor model is

$$x=\mu+\Lambda f+\varepsilon,$$

where $f\in\mathbb R^q$ contains latent common factors, $\Lambda$ is the loading matrix, and $\varepsilon$ contains variable-specific noise. A standard Gaussian model sets

$$f\sim\mathcal N(0,I),\qquad \varepsilon\sim\mathcal N(0,\Psi),$$

with diagonal $\Psi$. Therefore

$$\mathop{\mathrm{Cov}}(x)=\Lambda\Lambda^T+\Psi.$$

FA separates covariance shared across variables from their uniquenesses $\Psi_{jj}$. PCA instead decomposes total observed variance and has no variable-specific error term. Probabilistic PCA is a restricted latent model in which the noise is isotropic, $\Psi=\sigma^2I$; ordinary FA allows different uniqueness for each variable.

Rotation helps interpretation but exposes non-identifiability #

If $Q$ is orthogonal, then

$$\Lambda\Lambda^T=(\Lambda Q)(\Lambda Q)^T.$$

Thus the same covariance can be explained by infinitely many rotated loading matrices. Varimax and other rotations choose a convenient orientation, often making loadings look simpler; they do not discover a uniquely true coordinate system. Oblique rotations allow correlated factors and change the interpretation of pattern and structure loadings.

Factor scores are estimates, not directly observed truths, and different scoring rules can disagree. Naming a factor from two large loadings is a hypothesis requiring construct validity, not a consequence of the eigendecomposition.

Choosing dimension without storytelling #

Explained-variance thresholds are convenient for PCA but arbitrary. A scree plot reveals eigenvalue decay; cross-validated reconstruction or downstream validation connects dimension to use. For FA, compare held-out likelihood, residual correlations, stability across samples, and substantive interpretability. Parallel analysis can help distinguish eigenvalues exceeding those expected from random data, but no single rule replaces model checking.

Fit preprocessing and the decomposition on training data only. Computing scaling or components on the complete dataset leaks held-out distributional information into evaluation.

Counterexamples and failure modes #

• Two variables can share a strong nonlinear relation while linear PCA needs several directions or misses the geometry.
• An outlier can rotate the leading component because covariance is not robust.
• A component explaining little overall variance can be highly predictive of a rare outcome.
• A visually clean two-factor loading pattern can arise from selection, correlated errors, or an underidentified model; it is not evidence of two real causal entities.
• Loadings are not regression effects. A high loading says a variable co-moves with a component/factor under the fitted model, not that intervening on it changes the latent construct.

Use PCA when the claim is geometric and predictive. Use FA when a latent measurement model is scientifically defensible—and keep the causal interpretation proportional to the evidence.