Witty: An Efficient Solver for Computing Minimum-Size Decision Trees

TL;DR

This paper empirically evaluates an efficient algorithmic paradigm for minimum-size decision trees, demonstrating significant speedups over existing methods and providing improved theoretical bounds for the problem.

Contribution

It implements and enhances the witness trees paradigm for MSDT, achieving substantial empirical speedups and offering improved worst-case theoretical bounds.

Findings

Achieved a mean 324-fold speedup over naive implementation

Outperformed state-of-the-art solvers with a mean 32-fold speedup

Provided an improved worst-case running-time bound for MSDT

Abstract

Decision trees are a classic model for summarizing and classifying data. To enhance interpretability and generalization properties, it has been proposed to favor small decision trees. Accordingly, in the minimum-size decision tree training problem (MSDT), the input is a set of training examples in $\mathbb{R}^d$ with class labels and we aim to find a decision tree that classifies all training examples correctly and has a minimum number of nodes. MSDT is NP-hard and therefore presumably not solvable in polynomial time. Nevertheless, Komusiewicz et al. [ICML '23] developed a promising algorithmic paradigm called witness trees which solves MSDT efficiently if the solution tree is small. In this work, we test this paradigm empirically. We provide an implementation, augment it with extensive heuristic improvements, and scrutinize it on standard benchmark instances. The augmentations achieve a mean 324-fold (median 84-fold) speedup over the naive implementation. Compared to the state of the art they achieve a mean 32-fold (median 7-fold) speedup over the dynamic programming based MurTree solver [Demirovi\'c et al., J. Mach. Learn. Res. '22] and a mean 61-fold (median 25-fold) speedup over SAT-based implementations [Janota and Morgado, SAT '20]. As a theoretical result we obtain an improved worst-case running-time bound for MSDT.