Branches: Efficiently Seeking Optimal Sparse Decision Trees with AO*

TL;DR

This paper introduces Branches, an AO*-based algorithm for optimal sparse decision tree learning that improves efficiency and supports non-binary features, outperforming existing methods both theoretically and empirically.

Contribution

It presents a novel AO*-type algorithm called Branches for optimal decision tree learning, with proven guarantees and enhanced efficiency over prior approaches.

Findings

Branches is more efficient than state-of-the-art methods.

Supports non-binary features, leading to larger efficiency gains.

Proven optimality and complexity guarantees.

Abstract

Decision Tree (DT) Learning is a fundamental problem in Interpretable Machine Learning, yet it poses a formidable optimisation challenge. Practical algorithms have recently emerged, primarily leveraging Dynamic Programming and Branch & Bound. However, most of these approaches rely on a Depth-First-Search strategy, which is inefficient when searching for DTs at high depths and requires the definition of a maximum depth hyperparameter. Best-First-Search was also employed by other methods to circumvent these issues. The downside of this strategy is its higher memory consumption, as such, it has to be designed in a fully efficient manner that takes full advantage of the problem's structure. We formulate the problem within an AND/OR graph search framework and we solve it with a novel AO*-type algorithm called Branches. We prove both optimality and complexity guarantees for Branches and we show that it is more efficient than the state of the art theoretically and on a variety of experiments. Furthermore, Branches supports non-binary features unlike the other methods, we show that this property can further induce larger gains in computational efficiency.