Policy Trees for Prediction: Interpretable and Adaptive Model Selection for Machine Learning

TL;DR

This paper introduces Optimal Predictive-Policy Trees (OP2T), a tree-based method for interpretable, adaptive model selection and rejection in machine learning, addressing real-world decision-making challenges.

Contribution

It develops a prescriptive, interpretable tree-based approach for adaptive model selection and rejection, applicable to various data types and model outputs.

Findings

OP2T outperforms baseline methods in accuracy and interpretability

Provides insights into model selection and error-prone scenarios

Works with both structured and unstructured data

Abstract

As a multitude of capable machine learning (ML) models become widely available in forms such as open-source software and public APIs, central questions remain regarding their use in real-world applications, especially in high-stakes decision-making. Is there always one best model that should be used? When are the models likely to be error-prone? Should a black-box or interpretable model be used? In this work, we develop a prescriptive methodology to address these key questions, introducing a tree-based approach, Optimal Predictive-Policy Trees (OP2T), that yields interpretable policies for adaptively selecting a predictive model or ensemble, along with a parameterized option to reject making a prediction. We base our methods on learning globally optimized prescriptive trees. Our approach enables interpretable and adaptive model selection and rejection while only assuming access to model outputs. By learning policies over different feature spaces, including the model outputs, our approach works with both structured and unstructured datasets. We evaluate our approach on real-world datasets, including regression and classification tasks with both structured and unstructured data. We demonstrate that our approach provides both strong performance against baseline methods while yielding insights that help answer critical questions about which models to use, and when.