When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments

TL;DR

This paper examines how risk assessment tools trained on observable outcomes, rather than true potential outcomes, can mislead decision-making and cause harm, even under ideal statistical conditions.

Contribution

It highlights the pitfalls of using observable outcomes in RAIs and demonstrates the potential for increased harm despite optimal predictive performance.

Findings

RAIs trained on observable outcomes can lead to worse decisions.

Even with Bayes-optimal predictors and no unmeasured confounding, harm can occur.

Modeling observable outcomes may misrepresent true risks.

Abstract

Risk Assessment Instruments (RAIs) are widely used to forecast adverse outcomes in domains such as healthcare and criminal justice. RAIs are commonly trained on observational data and are optimized to predict observable outcomes rather than potential outcomes, which are the outcomes that would occur absent a particular intervention. Examples of relevant potential outcomes include whether a patient's condition would worsen without treatment or whether a defendant would recidivate if released pretrial. We illustrate how RAIs which are trained to predict observable outcomes can lead to worse decision making, causing precisely the types of harm they are intended to prevent. This can occur even when the predictors are Bayes-optimal and there is no unmeasured confounding.