# Analyzing CART

## TL;DR

This paper investigates the bias and adaptivity of CART decision trees, revealing their relationship with variable importance measures and establishing the consistency of random forests under broad conditions.

## Contribution

It provides a theoretical analysis linking bias, impurity reduction, and variable importance in CART, and proves the consistency of random forests.

## Key findings

- Decision trees with CART have small bias and are adaptive to signal strength.
- The mean decrease in impurity (MDI) correlates exponentially with the probability content of terminal nodes.
- Random forests are shown to be consistent under general conditions.

## Abstract

Decision trees with binary splits are popularly constructed using Classification and Regression Trees (CART) methodology. For binary classification and regression models, this approach recursively divides the data into two near-homogenous daughter nodes according to a split point that maximizes the reduction in sum of squares error (the impurity) along a particular variable. This paper aims to study the bias and adaptive properties of regression trees constructed with CART. In doing so, we derive an interesting connection between the bias and the mean decrease in impurity (MDI) measure of variable importance---a tool widely used for model interpretability---defined as the sum of impurity reductions over all non-terminal nodes in the tree. In particular, we show that the probability content of a terminal subnode for a variable is small when the MDI for that variable is large and that this relationship is exponential---confirming theoretically that decision trees with CART have small bias and are adaptive to signal strength and direction. Finally, we apply these individual tree bounds to tree ensembles and show consistency of Breiman's random forests. The context is surprisingly general and applies to a wide variety of multivariable data generating distributions and regression functions. The main technical tool is an exact characterization of the conditional probability content of the daughter nodes arising from an optimal split, in terms of the partial dependence function and reduction in impurity.