Limit Regret in Binary Treatment Choice with Misspecified Plug-In Predictors and Decision Thresholds

TL;DR

This paper investigates the worst-case regret of plug-in predictors in binary treatment decisions, especially under model misspecification and the use of x-specific thresholds, providing insights for better model and threshold choices.

Contribution

It offers a detailed analysis of limit maximum regret under misspecified models and x-specific thresholds, guiding optimal model and threshold selection in treatment decisions.

Findings

Limit MR depends on the limit estimate and thresholds.

Misspecified models can lead to high regret.

Jointly choosing models and thresholds improves outcomes.

Abstract

We study the population limit maximum regret (MR) of plug-in prediction when the decision problem is to choose between two treatments for the members of a population with observed covariates x. In this setting, the optimal treatment for persons with covariate value x is B if the conditional probability P(y = 1|x) of a binary outcome y exceeds an x-specific known threshold and is A otherwise. This structure is common in medical decision making and also arises in non-medical contexts such as criminal justice. Plug-in prediction uses data to estimate P(y|x) and acts as if the estimate is accurate. We are concerned that the model used to estimate P(y|x) may be misspecified, with true conditional probabilities being outside the model space. In practice, plug-in prediction has been performed with a wide variety of prediction models that commonly are misspecified. Further, applications often use a conventional x-invariant threshold, whereas optimal treatment choice uses x-specific thresholds. The main contribution of this paper is to shed new light on limit MR when plug-in prediction is performed with misspecified models. We use a combination of algebraic and computational analysis to study limit MR, demonstrating how it depends on the limit estimate and on the thresholds used to choose treatments. We recommend that a planner who wants to use plug-in prediction to achieve satisfactory MR should jointly choose a predictive model, estimation method, and x-specific thresholds to accomplish this objective.