Machine Learning & Pattern Recognition

A learning problem is defined by an operational decision, prediction unit and time, observable inputs, target, loss, and deployment population—not by choosing an algorithm. Good framing exposes whether labels answer the real question, whether prediction can change the outcome, and whether machine learning is needed at all.

Features are the geometry and information boundary presented to a learner. Encoding, scaling, missing-value handling, aggregation windows, and learned representations determine which distinctions are easy, impossible, or dangerously leaky for the model.

A distance function decides which differences matter before a nearest-neighbour, clustering, retrieval, or anomaly algorithm sees the data. Choosing one means specifying invariances, feature trade-offs, and missing-data policy—not selecting a formula by name.

Pairwise supervision says which examples should be treated as alike or different without requiring a complete class label. It can constrain a partition, learn a geometry, or train a predictor—but noisy constraints, transitive closure, and evaluation leakage decide whether it helps.

A validation split is a simulation of the model's future use, not a random percentage ritual. The split must preserve entity, time, group, and preprocessing boundaries so that model selection cannot consume information reserved for final assessment.

Overfitting occurs when a learning procedure exploits sample-specific variation that does not repeat. Capacity, search, and noisy labels raise variance; regularization, simpler representations, more information, and honest validation trade training fit for lower expected error.

Linear models combine features into an additive score; the loss function turns that score into a regression estimate, probability, or decision boundary. Their value lies in explicit geometry, stable regularization, and strong baselines—not in an assumption that the world itself is linear.

kNN predicts from nearby stored examples, so its behavior is controlled by representation, distance, neighbourhood size, and local data density. It is flexible without fitting a parametric boundary, but becomes unstable or meaningless when proximity no longer corresponds to task similarity.

Naive Bayes combines class priors with feature likelihoods under conditional independence. The assumption is often false, yet the additive log-evidence rule can classify well in sparse high-dimensional settings; its probability estimates and correlated evidence require particular caution.

A decision tree recursively partitions feature space into regions with simpler targets. Greedy impurity reduction makes nonlinear interactions legible, but deep trees are unstable estimators; stopping and cost-complexity pruning control whether a split represents repeatable structure or sample noise.

Ensembles improve prediction by combining models whose errors are not identical. Bagging and random forests average unstable fits to reduce variance; gradient boosting builds an additive model that repeatedly attacks residual loss, reducing bias but demanding careful regularization and validation.

An SVM chooses a separating surface with maximum margin, trading margin violations against model norm when data overlap. Kernel functions express nonlinear similarity through inner products, but scaling, kernel validity, regularization, and probability calibration remain explicit responsibilities.

Clustering algorithms impose different definitions of a group: compactness around centroids, nested linkage, or density connectivity. A useful clustering is stable and actionable under a justified representation; an attractive partition alone is not evidence that natural classes exist.

Dimensionality reduction compresses features for reconstruction, modelling, or visualization, but each method preserves a different relation. PCA is the linear prerequisite; t-SNE and UMAP construct neighbourhood maps whose global distances, densities, and apparent clusters must not be read literally.

Association-rule mining finds itemsets that recur across transactions and turns them into directional co-occurrence rules. Support controls frequency, confidence measures a conditional rate, lift compares it with the consequent's baseline, and generalized rules repeat the search across an item taxonomy without mistaking hierarchy for evidence.

Anomaly detection ranks observations by deviation from a reference notion of normality; novelty detection assumes the training reference is clean enough to learn that notion. Scores become decisions only after thresholds, prevalence, investigation capacity, and evolving context are specified.

The perceptron learns a linear threshold from mistakes; a multilayer perceptron composes differentiable affine maps and nonlinearities. Backpropagation is the efficient chain-rule computation that assigns each parameter its contribution to loss, not a separate learning objective or a biological explanation.

Deep-network training succeeds when activation and gradient scales remain usable across layers and optimization noise is controlled. Initialization, residual paths, normalization, learning-rate schedules, clipping, regularization, and diagnostics work as a coupled system rather than independent magic switches.

Architectures encode assumptions about how information should be shared: convolutions reuse local pattern detectors, recurrence updates a persistent state, and transformers route content through attention. Their inductive biases determine data efficiency, compute shape, and failure modes more than their brand names do.

A recurrent model reuses one state transition across time, so its memory and its gradient are both governed by repeated Jacobian products. LSTM and GRU create gated, partly additive state paths that make useful time scales easier to learn, while masking, state resets, teacher forcing, decoding, and truncation determine what information the deployed model can actually use.

GPT-style language models turn a prompt and its continuation into one token stream, process each prefix with causally masked Transformer blocks, and generate by repeatedly sampling the next-token distribution. This chapter separates that public mechanism from instruction post-training, tools, and the undisclosed internals of current ChatGPT models.

An energy-based model assigns a scalar compatibility score to configurations and derives prediction or probability from relative energies. The freedom to use an unnormalized model is powerful, but it moves the hard work into normalization, sampling, and proving that negative-phase samples represent the model rather than a short, biased chain.

Symbolic music generation models notes, timing, dynamics, controls, and instrument roles rather than raw air pressure. Piano rolls, event streams, and factored tracks preserve different musical relationships, so representation and decoding choices determine sequence length, polyphony, timing error, controllability, and what an evaluation can honestly claim.

How data, model, and pipeline parallelism distribute training; how all-reduce and parameter servers coordinate state; and how synchronization, stragglers, placement, sparse updates, and checkpoints determine correctness and speed.

Classification metrics summarize different consequences of a thresholded score. Precision depends on prevalence, recall ignores false-alert volume, ROC-AUC ranks across thresholds, and no scalar metric can choose an operating point without costs, capacity, and the deployment population.

A calibrated probability matches observed frequency among comparable predictions; it is distinct from ranking skill and from uncertainty about the model itself. Calibration requires independent evaluation, proper scoring rules, representative prevalence, and explicit coverage claims when intervals or sets are reported.

Generative evaluation asks whether samples are useful, plausible, diverse, non-memorized, and produced by a reproducible sampling policy. Likelihood, distributional distances, precision/recall-style measures, downstream utility, and human judgments answer different questions; no single scalar proves the whole claim.

Interpretation methods answer narrow questions about a fitted predictor, such as sensitivity, reliance, or local additive attribution. Their results depend on the reference population and feature-dependence assumptions; they do not identify causes, validate the model, or explain the world by themselves.

Production monitoring separates pipeline integrity, input drift, score behavior, delayed outcome quality, and decision impact. Distribution change is a diagnostic signal rather than proof of model failure; retraining is justified only when a validated response improves the current decision under governed data and labels.