Fast Optimization of Weighted Sparse Decision Trees for use in Optimal Treatment Regimes and Optimal Policy Design

TL;DR

This paper introduces three efficient algorithms for optimizing sparse weighted decision trees, enabling their use in policy design with weighted data, including inverse propensity weighting, with significant speed improvements.

Contribution

The paper proposes novel algorithms that efficiently optimize weighted sparse decision trees, addressing a gap in policy design applications involving weighted data samples.

Findings

Two fast methods are two orders of magnitude faster than direct optimization.

The fast methods maintain accuracy with theoretical error bounds.

Algorithms enable use of weighted data in interpretable decision trees.

Abstract

Sparse decision trees are one of the most common forms of interpretable models. While recent advances have produced algorithms that fully optimize sparse decision trees for prediction, that work does not address policy design, because the algorithms cannot handle weighted data samples. Specifically, they rely on the discreteness of the loss function, which means that real-valued weights cannot be directly used. For example, none of the existing techniques produce policies that incorporate inverse propensity weighting on individual data points. We present three algorithms for efficient sparse weighted decision tree optimization. The first approach directly optimizes the weighted loss function; however, it tends to be computationally inefficient for large datasets. Our second approach, which scales more efficiently, transforms weights to integer values and uses data duplication to transform the weighted decision tree optimization problem into an unweighted (but larger) counterpart. Our third algorithm, which scales to much larger datasets, uses a randomized procedure that samples each data point with a probability proportional to its weight. We present theoretical bounds on the error of the two fast methods and show experimentally that these methods can be two orders of magnitude faster than the direct optimization of the weighted loss, without losing significant accuracy.