Classification Metrics: Precision, Recall, ROC & Class Imbalance

Classification metrics summarize different consequences of a thresholded score. Precision depends on prevalence, recall ignores false-alert volume, ROC-AUC ranks across thresholds, and no scalar metric can choose an operating point without costs, capacity, and the deployment population.

budding working of working confidence: likely planted 2026-08-20 tended 2026-08-25 classificationmetricsprecision-recallrocclass-imbalance

A classifier usually emits a score. A threshold and policy turn that score into an action. Metrics describe different parts of this chain, so selecting “the best metric” without naming prevalence, costs, and action capacity is under-specification.

Begin with the confusion matrix #

For a declared positive class:

	Predicted positive	Predicted negative
Actually positive	true positive (TP)	false negative (FN)
Actually negative	false positive (FP)	true negative (TN)

From it:

\text{recall}=\text{sensitivity}=\frac{TP}{TP+FN},

\text{precision}=\frac{TP}{TP+FP},

\text{specificity}=\frac{TN}{TN+FP},\qquad \text{FPR}=1-\text{specificity}.

Recall asks what fraction of real positives were found. Precision asks what fraction of alerts were positive. Accuracy counts all correct decisions, but a classifier predicting “negative” for every case is 99.9% accurate when prevalence is 0.1%.

Worked imbalanced example #

In 1,000 cases, 20 are positive. A threshold gives TP=16, FN=4, FP=49, TN=931. Then:

accuracy $=(16+931)/1000=94.7\%$ ;
recall $=16/20=80\%$ ;
precision $=16/65\approx24.6\%$ ;
FPR $=49/980=5\%$ .

“95% specificity” sounds strong, yet roughly three quarters of alerts are false because negatives are abundant. If reviewers can inspect only 20 cases per day, this threshold is operationally impossible despite respectable recall.

Precision changes with prevalence even when class-conditional score behavior stays fixed. Recall and FPR can transfer under certain prior-shift assumptions; precision must be recomputed for deployment prevalence.

Threshold curves answer ranking questions #

An ROC curve plots TPR (recall) against FPR as threshold varies. ROC-AUC has a ranking interpretation: the probability that a randomly drawn positive receives a higher score than a randomly drawn negative, with ties handled appropriately. It is invariant to strictly monotone score transforms and does not measure calibration.

ROC-AUC averages over thresholds, including regions with an unacceptable number of false positives. Under extreme imbalance, a small FPR can still create many alerts. Partial AUC over the feasible FPR range or a cost curve can be more relevant.

A precision–recall (PR) curve shows the trade-off between finding positives and keeping alerts useful. Its no-skill precision baseline is the positive prevalence, so PR curves expose rarity directly. Average precision summarizes ranked precision but depends on implementation/interpolation conventions and deployment prevalence.

Two models can have crossing curves: neither dominates at every operating point. Choose in the region the system can actually use.

Threshold selection is a decision problem #

If scores are calibrated probabilities and costs are stable, predict positive when expected action cost is lower. With false-positive cost $C_{FP}$ and false-negative cost $C_{FN}$ , a simple binary threshold is

p(y=1\mid x)>\frac{C_{FP}}{C_{FP}+C_{FN}},

under the corresponding zero costs for correct actions. Capacity constraints instead select the top $k$ or a threshold yielding a bounded alert rate. Estimate threshold performance on data separate from final testing.

$F_1$ is the harmonic mean of precision and recall:

F_1=\frac{2PR}{P+R}.

It ignores TN and implicitly weights precision and recall symmetrically. $F_\beta$ changes that balance but still does not encode actual costs or capacity.

Multiclass and multilabel aggregation #

Compute per-class metrics before averaging. Macro averaging gives each class equal weight; weighted macro weights by class support; micro pools all decisions and can be dominated by common classes. In single-label multiclass classification, micro precision/recall can collapse to accuracy and hide rare-class failure.

For multilabel tasks, also decide whether errors are averaged per label, per example, or as exact-set matches. Those are different products.

Uncertainty and comparison #

Metrics are estimates from a finite, often grouped sample. Use intervals or resampling at the independent unit—patient, user, day, site—not row-level bootstrap when rows are dependent. Compare models on the same cases and examine paired differences.

Segment metrics can reveal concentrated harm, but repeated slicing creates multiple-comparison noise and tiny denominators. Predeclare important segments, show counts, and treat exploratory extremes as hypotheses.

Common misconceptions #

AUC 0.9 does not mean 90% accuracy or 90% probability quality.
Balancing classes in evaluation destroys the deployment precision estimate unless prevalence is restored.
Setting a threshold on the test set makes the test part of selection.
A confusion matrix without a declared positive class and population is ambiguous.
Improving offline recall can reduce real outcomes if extra alerts overload reviewers.

Report the score distribution, operating threshold, confusion counts, prevalence, uncertainty, and action volume. A metric earns meaning from the decision it represents.