Breiman meets Bellman: Non-Greedy Decision Trees with MDPs

TL;DR

This paper introduces DPDT, a novel decision tree framework that combines Markov Decision Processes with heuristic split generation to produce near-optimal trees efficiently, outperforming traditional greedy and existing optimal methods.

Contribution

DPDT is a new approach that bridges greedy and optimal decision trees using MDPs and heuristics, achieving near-optimal solutions with reduced computational complexity.

Findings

DPDT achieves near-optimal training loss on multiple datasets.

DPDT outperforms CART and existing optimal decision trees in generalization.

DPDT is effective in boosting applications, outperforming baselines.

Abstract

In supervised learning, decision trees are valued for their interpretability and performance. While greedy decision tree algorithms like CART remain widely used due to their computational efficiency, they often produce sub-optimal solutions with respect to a regularized training loss. Conversely, optimal decision tree methods can find better solutions but are computationally intensive and typically limited to shallow trees or binary features. We present Dynamic Programming Decision Trees (DPDT), a framework that bridges the gap between greedy and optimal approaches. DPDT relies on a Markov Decision Process formulation combined with heuristic split generation to construct near-optimal decision trees with significantly reduced computational complexity. Our approach dynamically limits the set of admissible splits at each node while directly optimizing the tree regularized training loss. Theoretical analysis demonstrates that DPDT can minimize regularized training losses at least as well as CART. Our empirical study shows on multiple datasets that DPDT achieves near-optimal loss with orders of magnitude fewer operations than existing optimal solvers. More importantly, extensive benchmarking suggests statistically significant improvements of DPDT over both CART and optimal decision trees in terms of generalization to unseen data. We demonstrate DPDT practicality through applications to boosting, where it consistently outperforms baselines. Our framework provides a promising direction for developing efficient, near-optimal decision tree algorithms that scale to practical applications.