Consistency of Random Forest Type Algorithms under a Probabilistic Impurity Decrease Condition

TL;DR

This paper presents a unifying theorem that establishes the consistency of various tree-based algorithms, including Random Forest variants, under a probabilistic impurity decrease condition, broadening theoretical understanding.

Contribution

It introduces a general consistency theorem applicable to diverse tree algorithms with added randomness and larger function classes, extending prior results.

Findings

Proves consistency for Extremely Randomized Trees, Interaction Forests, and Oblique Regression Trees.

Extends the class of functions for which Random Forests are consistent.

Utilizes a probabilistic impurity decrease condition to unify analysis.

Abstract

This paper derives a unifying theorem establishing consistency results for a broad class of tree-based algorithms. It improves current results in two aspects. First of all, it can be applied to algorithms that vary from traditional Random Forests due to additional randomness for choosing splits, extending split options, allowing partitions into more than two cells in a single iteration step, and combinations of those. In particular, we prove consistency for Extremely Randomized Trees, Interaction Forests and Oblique Regression Trees using our general theorem. Secondly, it can be used to demonstrate consistency for a larger function class compared to previous results on Random Forests if one allows for additional random splits. Our results are based on the extension of the recently introduced notion of sufficient impurity decrease to a probabilistic sufficient impurity decrease condition.