Anytime Optimal Decision Tree Learning with Continuous Features

TL;DR

This paper introduces an anytime, complete algorithm for learning optimal decision trees with continuous features, improving early solution quality and balancing computational effort compared to existing methods.

Contribution

It proposes a novel limited discrepancy search approach that enhances anytime performance for optimal decision tree learning with continuous features.

Findings

Outperforms existing algorithms in early solution quality

Provides high-quality trees at any interruption point

Balances computational effort across the tree structure

Abstract

In recent years, significant progress has been made on algorithms for learning optimal decision trees, primarily in the context of binary features. Extending these methods to continuous features remains substantially more challenging due to the large number of potential splits for each feature. Recently, an elegant exact algorithm was proposed for learning optimal decision trees with continuous features; however, the rapidly increasing computational time limits its practical applicability to shallow depths (typically 3 or 4). It relies on a depth-first search optimization strategy that fully optimizes the left subtree of each split before exploring the corresponding right subtree. While effective in finding optimal solutions given sufficient time, this strategy can lead to poor anytime behavior: when interrupted early, the best-found tree is often highly unbalanced and suboptimal. In such cases, purely greedy methods such as C4.5 may, paradoxically, yield better solutions. To address this limitation, we propose an anytime, yet complete approach leveraging limited discrepancy search, distributing the computational effort more evenly across the entire tree structure, and thus ensuring that a high-quality decision tree is available at any interruption point. Experimental results show that our approach outperforms the existing one in terms of anytime performance.