Output-Constrained Decision Trees

TL;DR

This paper presents new methods for training output-constrained decision trees that produce accurate and feasible predictions, addressing limitations of traditional trees in constrained multi-target regression.

Contribution

It introduces three novel approaches for output-constrained regression trees and a constrained random forest framework, validated on synthetic and real datasets.

Findings

Imposing constraints improves prediction feasibility without sacrificing accuracy

The proposed methods outperform unconstrained decision trees in constrained tasks

The constrained random forest enhances ensemble learning with feasibility guarantees

Abstract

Incorporating domain-specific constraints into machine learning models is essential for generating predictions that are both accurate and feasible in real-world applications. This paper introduces new methods for training Output-Constrained Regression Trees (OCRT), addressing the limitations of traditional decision trees in constrained multi-target regression tasks. We propose three approaches: M-OCRT, which uses split-based mixed integer programming to enforce constraints; E-OCRT, which employs an exhaustive search for optimal splits and solves constrained prediction problems at each decision node; and EP-OCRT, which applies post-hoc constrained optimization to tree predictions. To illustrate their potential uses in ensemble learning, we also introduce a random forest framework working under convex feasible sets. We validate the proposed methods through a computational study both on synthetic and industry-driven hierarchical time series datasets. Our results demonstrate that imposing constraints on decision tree training results in accurate and feasible predictions.