Acceleration Techniques for Learning Optimal Classification Trees with Integer Programming

TL;DR

This paper introduces advanced integer programming techniques such as valid inequalities, cutting planes, and heuristics to improve the scalability of optimal classification tree training using mixed-integer programming.

Contribution

It integrates ideas from dynamic programming into MIP models to enhance their scalability for learning optimal classification trees.

Findings

Significant improvement in finding provably optimal solutions.

Enhanced scalability of MIP formulations for larger datasets.

Effective integration of DP ideas into MIP models.

Abstract

Decision trees are a popular machine learning model which are traditionally trained by heuristic methods. Massive improvements in computing power and optimisation techniques has led to renewed interest in learning globally optimal decision trees. Empirical evidence shows that optimal classification trees (OCTs) have better out-of-sample performance than heuristic methods. The dominant optimisation paradigms for training OCTs are mixed-integer programming (MIP) and dynamic programming (DP). MIP formulations offer flexibility in the objectives and constraints that are modelled, but suffer from poor scaling in the size of the training dataset and the maximum tree depth. DP models represent the state of the art in scaling for OCTs, but lack some of the flexibility of MIP models. In this paper we present progress on using advanced integer programming methods to integrate ideas from DP models into MIP formulations to begin bridging the scaling gap. Using the existing BendOCT model from the literature as a base model, we introduce valid inequalities, cutting planes, and a primal heuristic to improve the scaling of MIP formulations. We show that these techniques significantly improve the ability of BendOCT to find provably optimal solutions over a wide range of datasets.