Adaptively Pruning Features for Boosted Decision Trees

TL;DR

This paper introduces an adaptive pruning algorithm for boosted decision trees that significantly reduces training costs on large datasets by leveraging multi-arm bandit strategies, achieving near-optimal performance.

Contribution

It presents a novel, efficient algorithm for exact greedy decision tree construction inspired by bandit theory, outperforming existing methods like Quick Boost.

Findings

Our algorithm outperforms Quick Boost in speed and accuracy.

Empirical results show near-optimal performance within a broad algorithm family.

Theoretical bounds demonstrate the algorithm's closeness to the best possible performance.

Abstract

Boosted decision trees enjoy popularity in a variety of applications; however, for large-scale datasets, the cost of training a decision tree in each round can be prohibitively expensive. Inspired by ideas from the multi-arm bandit literature, we develop a highly efficient algorithm for computing exact greedy-optimal decision trees, outperforming the state-of-the-art Quick Boost method. We further develop a framework for deriving lower bounds on the problem that applies to a wide family of conceivable algorithms for the task (including our algorithm and Quick Boost), and we demonstrate empirically on a wide variety of data sets that our algorithm is near-optimal within this family of algorithms. We also derive a lower bound applicable to any algorithm solving the task, and we demonstrate that our algorithm empirically achieves performance close to this best-achievable lower bound.