To Bag is to Prune

TL;DR

This paper proposes that Random Forests and similar ensemble methods inherently perform an automatic form of pruning and early stopping, explaining their robustness and overfitting behavior without explicit tuning.

Contribution

It introduces a new explanation for RF's behavior, showing that bootstrap aggregation and perturbation implicitly prune models and enable automatic early stopping.

Findings

Overfitting ensembles perform comparably or better than tuned models.

Randomized ensembles implicitly perform optimal early stopping.

Novel variants of Boosting and MARS can be automatically tuned.

Abstract

It is notoriously difficult to build a bad Random Forest (RF). Concurrently, RF blatantly overfits in-sample without any apparent consequence out-of-sample. Standard arguments, like the classic bias-variance trade-off or double descent, cannot rationalize this paradox. I propose a new explanation: bootstrap aggregation and model perturbation as implemented by RF automatically prune a latent "true" tree. More generally, randomized ensembles of greedily optimized learners implicitly perform optimal early stopping out-of-sample. So there is no need to tune the stopping point. By construction, novel variants of Boosting and MARS are also eligible for automatic tuning. I empirically demonstrate the property, with simulated and real data, by reporting that these new completely overfitting ensembles perform similarly to their tuned counterparts -- or better.