Optimal Sparse Regression Trees

TL;DR

This paper introduces a dynamic programming method with bounds for constructing provably optimal sparse regression trees, enabling fast solutions even for complex datasets with many samples and correlated features.

Contribution

It presents a novel approach combining dynamic programming and bounds, leveraging 1D k-Means solutions to efficiently find optimal sparse regression trees.

Findings

Often finds optimal trees in seconds

Handles large datasets with many samples

Effective with highly-correlated features

Abstract

Regression trees are one of the oldest forms of AI models, and their predictions can be made without a calculator, which makes them broadly useful, particularly for high-stakes applications. Within the large literature on regression trees, there has been little effort towards full provable optimization, mainly due to the computational hardness of the problem. This work proposes a dynamic-programming-with-bounds approach to the construction of provably-optimal sparse regression trees. We leverage a novel lower bound based on an optimal solution to the k-Means clustering algorithm in 1-dimension over the set of labels. We are often able to find optimal sparse trees in seconds, even for challenging datasets that involve large numbers of samples and highly-correlated features.