Is interpolation benign for random forest regression?

TL;DR

This paper investigates whether random forest regression models can interpolate training data without sacrificing predictive consistency, revealing conditions under which interpolation and consistency coexist.

Contribution

It provides the first theoretical analysis demonstrating that interpolating Random Forests can be consistent, emphasizing the role of feature randomization and averaging.

Findings

Breiman's RF are consistent when interpolating without bootstrap.

Interpolation area size converges quickly to zero, enabling consistency.

Adaptive RF can achieve both interpolation and consistency.

Abstract

Statistical wisdom suggests that very complex models, interpolating training data, will be poor at predicting unseen examples.Yet, this aphorism has been recently challenged by the identification of benign overfitting regimes, specially studied in the case of parametric models: generalization capabilities may be preserved despite model high complexity.While it is widely known that fully-grown decision trees interpolate and, in turn, have bad predictive performances, the same behavior is yet to be analyzed for Random Forests (RF).In this paper, we study the trade-off between interpolation and consistency for several types of RF algorithms. Theoretically, we prove that interpolation regimes and consistency cannot be achieved simultaneously for several non-adaptive RF.Since adaptivity seems to be the cornerstone to bring together interpolation and consistency, we study interpolating Median RF which are proved to be consistent in the interpolating regime. This is the first result conciliating interpolation and consistency for RF, highlighting that the averaging effect introduced by feature randomization is a key mechanism, sufficient to ensure the consistency in the interpolation regime and beyond.Numerical experiments show that Breiman's RF are consistent while exactly interpolating, when no bootstrap step is involved.We theoretically control the size of the interpolation area, which converges fast enough to zero, giving a necessary condition for exact interpolation and consistency to occur in conjunction.