Naive Bayes Classifiers

Naive Bayes is a family of generative classifiers. It models how features would be distributed inside each class, combines that likelihood with a class prior, and applies Bayes' rule. Its “naive” assumption is not that features are independent overall, but that they are independent conditional on the class.

The decision rule #

For features $x=(x_1,\ldots,x_p)$ and class $k$,

$$P(y=k\mid x)\propto P(y=k)P(x\mid y=k).$$

Conditional independence gives

$$P(x\mid y=k)=\prod_{j=1}^{p}P(x_j\mid y=k).$$

Products of many small probabilities underflow numerically, so compute log-scores:

$$s_k(x)=\log\pi_k+\sum_j\log P(x_j\mid y=k),$$

then choose the largest score. Each observed feature adds evidence to the prior log-odds.

Variants encode different observation models #

• Bernoulli Naive Bayes models binary feature presence/absence.
• Multinomial Naive Bayes models nonnegative counts, commonly word or event counts.
• Complement Naive Bayes estimates weights from the complement of each class and can help imbalanced text problems.
• Gaussian Naive Bayes models each continuous feature with a class-specific univariate Gaussian, equivalent to diagonal class covariance.
• Categorical Naive Bayes estimates a categorical distribution per feature and class.

Choose the likelihood from the data-generating representation. Feeding standardized negative values to a multinomial count model violates its assumptions even if software accepts the array.

Worked evidence calculation #

Suppose 20% of messages are spam. In spam, “free” appears with probability 0.6 and “offer” with probability 0.5; in ham, probabilities are 0.05 and 0.1. For a message containing both words, Bernoulli presence scores are

$$s_{spam}\propto0.2\cdot0.6\cdot0.5=0.060,$$

$$s_{ham}\propto0.8\cdot0.05\cdot0.1=0.004.$$

Normalizing gives about 0.94 for spam under this tiny model. If “free offer” is a phrase whose words almost always co-occur, multiplying their likelihoods double-counts related evidence. The ranking may remain useful, but 0.94 can be seriously overconfident.

Bernoulli models also use absence evidence: not seeing a word contributes $P(x_j=0\mid y=k)$. Multinomial models focus on counts and document length. Confusing these changes the model.

Smoothing prevents impossible evidence #

If a word never appeared in class $k$ during training, maximum-likelihood probability is zero and one occurrence makes the entire class likelihood zero. Additive smoothing estimates, for categorical outcome $v$,

$$\hat P(x_j=v\mid y=k)=\frac{N_{jvk}+\alpha}{N_{jk}+\alpha V},$$

where $V$ is the number of possible values. $\alpha=1$ is Laplace smoothing; smaller positive values are common. Smoothing expresses finite-sample uncertainty and must be validated, not used to hide broken tokenization or unseen-domain shifts.

Why a false assumption can still classify well #

Classification needs the correct ordering of class scores, not a perfect joint density. Errors shared across classes can cancel in likelihood ratios. Sparse text features also make independent additive evidence a strong baseline with little data and fast updates.

Probability estimation is less forgiving. Correlated features can multiply the same evidence repeatedly, producing probabilities too close to zero or one. Evaluate proper scoring rules and calibration on representative held-out data before treating scores as risk.

Failure modes #

• Wrong priors shift every decision. Training prevalence may differ from deployment because of resampling or real prevalence change.
• Continuous Gaussian likelihoods are sensitive to extreme values and badly modeled multimodality.
• Near-zero estimated variance can produce enormous likelihoods; variance smoothing is a numerical/modeling safeguard.
• Feature selection using all labels leaks information just as it does for any classifier.
• Duplicated words, correlated sensors, or nested indicators count related evidence multiple times.
• A good classifier can be a poor generator: its likelihood need not produce realistic samples.

Naive Bayes is not “Bayes-optimal despite bad assumptions.” It is a deliberately constrained evidence aggregator whose low variance and computational simplicity can outweigh misspecification. Treat its posterior as a model output to validate, not a literal degree of belief by construction.