Stabilizing the Splits through Minimax Decision Trees

TL;DR

This paper introduces MinimaxSplit decision trees, a robust alternative to CART that minimizes worst-case child risk, with theoretical guarantees and empirical improvements on structured data.

Contribution

The paper proposes MinimaxSplit trees that optimize worst-case risks, providing new theoretical bounds and demonstrating empirical advantages over traditional methods.

Findings

MinimaxSplit trees outperform CART in structured heterogeneous data.

Theoretical bounds relate global excess risk to local impurities.

Empirical results show improvements in EEG regression and image denoising.

Abstract

By revisiting the end-cut preference (ECP) phenomenon associated with a single CART (Breiman et al. (1984)), we introduce MinimaxSplit decision trees, a robust alternative to CART that selects splits by minimizing the worst-case child risk rather than the average risk. For regression, we minimize the maximum within-child squared error; for classification, we minimize the maximum child entropy, yielding a C4.5-compatible criterion. We also study a cyclic variant that deterministically cycles coordinates, leading to our main method of cyclic MinimaxSplit decision trees. We prove oracle inequalities that cover both regression and classification, under mild marginal non-atomicity conditions. The bounds control the tree's global excess risk by local worst-case impurities and yield fast convergence rates compared to CART. We extend the analysis to a random-dimension forest variant that subsamples coordinates per node. Empirically, (cyclic) MinimaxSplit trees and their forests improve over baselines on structured heterogeneous data such as EEG amplitude regression over fixed time horizons and image denoising, framed as non-parametric regression on spatial coordinates.