Ensembles: Bagging, Random Forests & Gradient Boosting

An ensemble combines several predictors to produce one output. Merely adding models does not guarantee improvement: if they make the same error, averaging preserves it. The gain comes from controlling error correlation or from making later models correct structure missed earlier.

Bagging reduces variance by averaging #

Bootstrap aggregating (bagging) repeatedly samples training observations with replacement, fits a base learner to each sample, then averages predictions or votes. Deep decision trees are useful base learners because small data changes create diverse trees.

If $B$ predictors each have error variance $\sigma^2$ and pairwise correlation $\rho$, the variance of their average is approximately

$$\rho\sigma^2+\frac{1-\rho}{B}\sigma^2.$$

Increasing $B$ removes the uncorrelated part but cannot remove the shared correlated error. This is why diversity matters more than an arbitrary number of near-identical models.

Bootstrap samples leave out about 36.8% of unique training observations on average. Predictions from trees that did not include a case form an out-of-bag estimate. It is useful during development but does not replace a deployment-shaped temporal, group, or external test.

Random forests decorrelate trees #

A random forest adds random feature subsampling at each split. A strong predictor otherwise dominates the root of nearly every bootstrap tree, making trees similar. Restricting candidate features forces alternative partitions, slightly weakening individual trees while lowering their correlation. The average can therefore improve.

Key controls include number of trees, number of candidate features, tree depth/minimum leaf size, class/sample weights, and bootstrap scheme. More trees usually stabilize the Monte Carlo average rather than overfit in the ordinary sense, but they increase latency and memory and cannot fix a biased representation.

Boosting learns an additive correction sequence #

Gradient boosting constructs

$$F_M(x)=F_0(x)+\eta\sum_{m=1}^{M}h_m(x),$$

where each weak learner $h_m$ is fitted to reduce the current loss and $\eta$ is the learning rate. For squared error, the negative loss gradient is the residual $y-F_{m-1}(x)$, so each tree predicts remaining error. For logistic loss, it fits pseudo-residuals on the score scale.

The name “gradient” comes from functional gradient descent: rather than directly moving a finite parameter vector, the procedure adds a new function in a direction that reduces empirical loss.

Shrinkage ($\eta<1$), shallow trees, row/column subsampling, leaf penalties, and early stopping regularize the sequence. Smaller learning rate usually requires more trees; choose the combination jointly on validation data.

A correction example #

Suppose the initial regression prediction is the global mean, 10. Residuals for three cases are $[-3,+1,+5]$. A shallow tree partitions the cases and predicts approximate residuals $[-2,+3,+3]$. With learning rate 0.2, the new predictions shift by $[-0.4,+0.6,+0.6]$, not by the full tree. Later trees see new residuals and focus on what remains.

This stagewise mechanism can fit intricate interactions. It can also chase mislabeled points: persistent large gradients repeatedly attract later trees unless depth, loss, sampling, and stopping control them.

Bagging and boosting solve different problems #

Stacking is different again: a meta-model learns how to combine base-model predictions. Its training inputs must be out-of-fold predictions; in-sample predictions leak each base model's training fit into the combiner.

Evaluation and interpretation #

Tree ensembles handle nonlinear tabular structure well, but default class probabilities can be miscalibrated. Validate probabilities and decision thresholds separately. Feature importance based on accumulated split impurity is biased and unstable with correlated/high-cardinality features; permutation importance or held-out ablation asks a more operational question, though correlations still complicate interpretation.

Failure modes #

• Thousands of trees average leakage just as efficiently as real signal.
• Random row bootstraps are invalid when rows belong to dependent groups or time periods.
• Boosting on separable noisy labels can continue increasing score magnitude after ranking stops improving.
• Extrapolation is poor: tree predictions remain combinations of learned leaf values.
• Rare categories and unseen regions may land in leaves supported by little relevant data.
• A higher validation score after extensive search can be selection noise; preserve an untouched final test.

Ensembling is not wisdom of crowds by model count. It works when the construction deliberately changes the error structure and the full procedure is validated under the conditions where its combined prediction will be used.