Optimal Classification Trees for Continuous Feature Data Using Dynamic Programming with Branch-and-Bound

TL;DR

This paper introduces a novel dynamic programming algorithm with branch-and-bound for constructing optimal classification trees directly on continuous features, significantly improving scalability and accuracy over existing methods.

Contribution

It presents a new algorithm that optimizes classification trees on continuous data without coarse binarization, enhancing scalability and performance.

Findings

Runtime improved by one or more orders of magnitude.

Test accuracy increased by 5% over greedy heuristics.

Effective pruning techniques reduce the search space.

Abstract

Computing an optimal classification tree that provably maximizes training performance within a given size limit, is NP-hard, and in practice, most state-of-the-art methods do not scale beyond computing optimal trees of depth three. Therefore, most methods rely on a coarse binarization of continuous features to maintain scalability. We propose a novel algorithm that optimizes trees directly on the continuous feature data using dynamic programming with branch-and-bound. We develop new pruning techniques that eliminate many sub-optimal splits in the search when similar to previously computed splits and we provide an efficient subroutine for computing optimal depth-two trees. Our experiments demonstrate that these techniques improve runtime by one or more orders of magnitude over state-of-the-art optimal methods and improve test accuracy by 5% over greedy heuristics.