Dimensionality Reduction: PCA, t-SNE & UMAP

Dimensionality reduction maps $x\in\mathbb R^p$ to $z\in\mathbb R^q$ with $q<p$. The purpose must be explicit: faster downstream learning, denoising, compression, visualization, or discovery of local neighbourhoods require different guarantees.

Principal component analysis is the prerequisite linear method. It chooses an orthogonal subspace minimizing squared reconstruction error. This chapter does not re-derive PCA; it focuses on how nonlinear neighbourhood embeddings differ and how all reductions should be used in a learning pipeline.

Decide what relation must survive #

A reduction cannot preserve every pairwise distance, density, topology, label boundary, and reconstruction detail at once. Ask what the downstream claim requires:

PCA has an explicit inverse approximation. Standard t-SNE is primarily a visualization of a fitted sample and has no native parametric inverse. UMAP implementations can transform new points using a fitted graph/embedding, but that does not turn visualization coordinates into lossless features.

t-SNE matches neighbour probabilities #

t-SNE converts high-dimensional pairwise distances into conditional neighbourhood probabilities. Each point gets a local bandwidth chosen so that the distribution has a user-controlled effective neighbourhood size called perplexity. The probabilities are symmetrized into $p_{ij}$.

In the low-dimensional map, similarities $q_{ij}$ use a heavy-tailed Student distribution. The embedding minimizes

$$KL(P\lVert Q)=\sum_{i\ne j}p_{ij}\log\frac{p_{ij}}{q_{ij}}.$$

Because this KL direction heavily penalizes a high-dimensional neighbour placed far apart, t-SNE prioritizes local neighbourhood recall. It penalizes unrelated points placed together less strongly. The heavy tail reduces the crowding that would otherwise force moderately distant neighbours into the map center.

Consequences:

• distances between separated islands are not reliable global distances;
• island area and visual density do not estimate original density;
• cluster count and spacing can change with perplexity, initialization, learning rate, and random seed;
• a continuous manifold can appear as separated blobs.

UMAP builds and lays out a fuzzy neighbour graph #

UMAP constructs a weighted nearest-neighbour graph using locally adjusted distance scales, interprets it as a fuzzy topological representation, and optimizes a low-dimensional graph with similar edge membership. n_neighbors controls the local-versus-broader neighbourhood scale; min_dist controls how tightly embedded points may pack.

The method is often faster than t-SNE and can preserve more broad organization in useful datasets, but no parameter setting guarantees faithful global metric geometry. Stochastic optimization and approximate neighbour search add variability. A transform for new data depends on the fitted reference sample and can be unreliable under shift.

A responsible visualization workflow #

flowchart TD
  X["Raw held-out-aware features"] --> P["Scale; optionally PCA-denoise"]
  P --> E["Fit several seeds and neighbourhood scales"]
  E --> N["Inspect neighbour preservation and raw examples"]
  N --> C["Form hypotheses"]
  C --> V["Validate in original space or independent data"]

PCA before t-SNE/UMAP can remove numerical noise and accelerate neighbour search, especially from thousands of sparse/noisy dimensions. It can also discard a low-variance signal. Select the retained dimension against the actual use, not a universal explained-variance percentage.

Colouring a map by labels is diagnostic, but fitting supervised UMAP with those labels and then celebrating separation is circular. Likewise, clustering the same two-dimensional map can amplify layout artifacts. Prefer clustering in a representation validated for clustering, then use the map only to inspect results.

Leakage and out-of-sample use #

If reduction feeds a predictor, fit scaling and the reducer on each training fold only. Fitting PCA or UMAP on the full dataset exposes validation covariances/neighbourhoods. Even an unsupervised transform can leak distributional information.

For production, version the reducer together with feature schema and model. Monitor whether new points are remote from the fitted reference distribution. A two-dimensional coordinate is rarely a stable semantic feature across retraining because axes can rotate, flip, or reorganize.

Counterexamples #

• Two equal spherical clouds with different sample counts may appear with similar visual areas because neighbourhood objectives equalize local presentation.
• A Swiss-roll manifold is globally far in Euclidean ambient distance but locally continuous; PCA flattens it poorly, while neighbourhood methods can unfold it—but noise can create false graph shortcuts.
• A rare class separated only along a low-variance direction can disappear after variance-based PCA.
• A clean island can be driven by batch, site, or missingness rather than the phenomenon of interest.

Treat nonlinear maps as instruments for asking better questions. The validation of those questions must happen outside the picture.