Large Scale Prediction with Decision Trees

TL;DR

This paper proves that decision trees constructed with CART and C4.5 are consistent for regression and classification tasks under certain sparsity constraints, and these properties extend to random forests, with a focus on theoretical guarantees.

Contribution

The paper provides a theoretical proof of consistency for decision trees and random forests under high-dimensional settings with sparsity constraints, broadening understanding of their asymptotic behavior.

Findings

Decision trees are consistent under sub-exponential predictor growth.

Consistency applies to various models including additive and Borel measurable functions.

Random forests inherit the consistency properties of individual trees.

Abstract

This paper shows that decision trees constructed with Classification and Regression Trees (CART) and C4.5 methodology are consistent for regression and classification tasks, even when the number of predictor variables grows sub-exponentially with the sample size, under natural 0-norm and 1-norm sparsity constraints. The theory applies to a wide range of models, including (ordinary or logistic) additive regression models with component functions that are continuous, of bounded variation, or, more generally, Borel measurable. Consistency holds for arbitrary joint distributions of the predictor variables, thereby accommodating continuous, discrete, and/or dependent data. Finally, we show that these qualitative properties of individual trees are inherited by Breiman's random forests. A key step in the analysis is the establishment of an oracle inequality, which allows for a precise characterization of the goodness-of-fit and complexity tradeoff for a mis-specified model.