Consistency of Honest Decision Trees and Random Forests

TL;DR

This paper proves the consistency of honest decision trees and random forests in regression, using elementary arguments that clarify their relationship with kernel methods and simplify existing analyses.

Contribution

It provides a simplified, elementary proof of the consistency of honest trees and forests, unifying and extending previous results with clearer insights.

Findings

Weak and almost sure convergence to the true regression function

Uniform convergence over compact domains

Framework accommodates ensemble variants like subsampling and bootstrap

Abstract

We study various types of consistency of honest decision trees and random forests in the regression setting. In contrast to related literature, our proofs are elementary and follow the classical arguments used for smoothing methods. Under mild regularity conditions on the regression function and data distribution, we establish weak and almost sure convergence of honest trees and honest forest averages to the true regression function, and moreover we obtain uniform convergence over compact covariate domains. The framework naturally accommodates ensemble variants based on subsampling and also a two-stage bootstrap sampling scheme. Our treatment synthesizes and simplifies existing analyses, in particular recovering several results as special cases. The elementary nature of the arguments clarifies the close relationship between data-adaptive partitioning and kernel-type methods, providing an accessible approach to understanding the asymptotic behavior of tree-based methods.