Statistical Advantages of Oblique Randomized Decision Trees and Forests

TL;DR

This paper introduces oblique Mondrian trees and forests that use linear combinations of features for data partitioning, providing theoretical analysis of their statistical properties, convergence rates, and robustness in dimension reduction tasks.

Contribution

It develops a theoretical framework for oblique Mondrian trees and forests, analyzing their generalization error, convergence, and robustness, and compares their optimality to axis-aligned methods.

Findings

Oblique Mondrian estimators achieve minimax optimal rates under certain conditions.

Risk bounds depend on feature estimation accuracy.

Axis-aligned Mondrian trees are suboptimal for ridge functions.

Abstract

This work studies the statistical implications of using features comprised of general linear combinations of covariates to partition the data in randomized decision tree and forest regression algorithms. Using random tessellation theory in stochastic geometry, we provide a theoretical analysis of a class of efficiently generated random tree and forest estimators that allow for oblique splits along such features. We call these estimators \emph{oblique Mondrian} trees and forests, as the trees are generated by first selecting a set of features from linear combinations of the covariates and then running a Mondrian process that hierarchically partitions the data along these features. Generalization error bounds and convergence rates are obtained for the flexible function class of multi-index models for dimension reduction, where the output is assumed to depend on a low-dimensional relevant feature subspace of the input domain. The results highlight how the risk of these estimators depends on the choice of features and quantify how robust the risk is with respect to error in the estimation of relevant features. The asymptotic analysis also provides conditions on the consistency rates of the estimated features along which the data is split for these estimators to obtain minimax optimal rates of convergence with respect to the dimension of the relevant feature subspace. Additionally, a lower bound on the risk of axis-aligned Mondrian trees (where features are restricted to the set of covariates) is obtained, proving that these estimators are suboptimal for general ridge functions, no matter how the distribution over the covariates used to divide the data at each tree node is weighted.