Linear & Logistic Models

A linear model computes a score

$$\eta(x)=w_0+w^Tx.$$

What the score means depends on the observation model and loss. Ordinary least squares treats it as a conditional mean and penalizes squared residuals. Logistic regression passes it through a sigmoid to model a binary probability. The shared additive score is why both are “linear,” even though logistic probability is nonlinear in the score.

Linear regression as prediction #

For quantitative $y$, least squares minimizes

$$\sum_i(y_i-w_0-w^Tx_i)^2.$$

Squaring makes positive and negative residuals comparable, penalizes large misses strongly, and yields convenient optimization. Under independent Gaussian errors with constant variance, it is also maximum likelihood. Those assumptions matter for standard errors and probabilistic intervals; a linear predictor can still be useful without exact normality.

Linearity means additive in chosen features. Adding $x^2$, $\log x$, splines, or interactions lets a linear-in-parameters model represent curved or conditional effects. The feature map determines the geometry.

Logistic regression models log-odds #

For binary $y\in\{0,1\}$,

$$p(y=1\mid x)=\sigma(\eta)=\frac{1}{1+e^{-\eta}},$$

so

$$\log\frac{p}{1-p}=w_0+w^Tx.$$

The sigmoid maps every real score into $(0,1)$. Maximum likelihood is equivalent to minimizing binary cross-entropy

$$-\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)].$$

The gradient contribution is proportional to $(p_i-y_i)x_i$: confident wrong cases push strongly, while already-correct confident cases contribute little. A classification threshold is applied after probability estimation and should reflect costs, capacity, and prevalence.

Odds example #

Suppose $\eta=-3+0.8x$. At $x=0$, $p=\sigma(-3)\approx0.047$. Increasing $x$ by one multiplies the odds by $e^{0.8}\approx2.23$, regardless of the starting value. It does not add a constant 0.8 to probability: the same odds change has a smaller probability effect near 0 or 1 than near 0.5.

The boundary at threshold 0.5 is $\eta=0$, so here $x=3.75$. If false negatives are costly, a lower probability threshold moves the operational boundary without refitting the model.

Multiclass scores #

Softmax regression assigns one score per class:

$$p(y=k\mid x)=\frac{e^{w_k^Tx}}{\sum_j e^{w_j^Tx}}.$$

Only score differences matter; adding the same constant to all scores changes no probability. Each pairwise boundary is linear. One-vs-rest instead fits independent binary models and may require normalization or calibration for coherent multiclass probabilities.

Regularization is part of the model #

With correlated or numerous features, many coefficient vectors fit similarly. An $L_2$ penalty stabilizes coefficients by preferring smaller norms; $L_1$ can produce sparsity; elastic net combines them. Because penalties act on coefficient magnitude, continuous features should usually be scaled first. The intercept is commonly left unpenalized.

Regularization changes coefficient interpretation. It trades unbiased estimation under an ideal specified model for lower predictive variance. A coefficient shrunk toward zero is not evidence that the feature has no causal effect.

Complete separation is a specific logistic failure: if a feature perfectly separates classes, unregularized likelihood keeps improving as coefficient magnitude tends to infinity. A finite optimizer result is then numerical truncation, not a stable estimate. Regularization or a prior makes the problem well posed.

Coefficients are conditional associations #

$w_j$ describes the score change for a one-unit change in $x_j$ while encoded other features stay fixed. This can be meaningless when features cannot vary independently, when interactions are omitted, or when a one-unit change is outside the data range. Correlated variables distribute credit unstably; rescaling changes magnitude; target leakage can produce beautifully interpretable nonsense.

Plot partial predictions over supported ranges, inspect uncertainty and correlations, and compare with a baseline. Never read causality from a predictive coefficient without an identification design.

Failure modes and diagnostics #

• Curved residual patterns reveal missing nonlinear structure in regression.
• Changing variance requires a variance model or robust uncertainty, not merely more features.
• Outliers can dominate squared loss; examine leverage and residual influence.
• A linear boundary cannot solve XOR in its original two features, but an interaction $x_1x_2$ can expose the needed geometry.
• High classification accuracy can coexist with poor probabilities under class imbalance; inspect proper scoring rules and calibration.
• Extrapolation extends the same linear relationship beyond observed support, where no data validate it.

Linear and logistic models are valuable precisely because their constraints are legible. They provide a strong baseline, a debugging surface, and often the right final model when representation captures the important structure.