When accurate prediction models yield harmful self-fulfilling prophecies

TL;DR

This paper reveals that accurate prediction models in healthcare can cause harmful self-fulfilling prophecies, emphasizing the need to rethink validation and deployment practices to prevent adverse outcomes.

Contribution

It provides a formal characterization of harmful self-fulfilling prediction models and demonstrates that well-calibrated models may be ineffective for decision making.

Findings

Prediction models can harm specific patient groups without invalidating their predictive power.

Well-calibrated models before and after deployment are ineffective for decision making.

Current validation practices may be insufficient to prevent harmful outcomes.

Abstract

Prediction models are popular in medical research and practice. By predicting an outcome of interest for specific patients, these models may help inform difficult treatment decisions, and are often hailed as the poster children for personalized, data-driven healthcare. We show however, that using prediction models for decision making can lead to harmful decisions, even when the predictions exhibit good discrimination after deployment. These models are harmful self-fulfilling prophecies: their deployment harms a group of patients but the worse outcome of these patients does not invalidate the predictive power of the model. Our main result is a formal characterization of a set of such prediction models. Next we show that models that are well calibrated before and after deployment are useless for decision making as they made no change in the data distribution. These results point to the need to revise standard practices for validation, deployment and evaluation of prediction models that are used in medical decisions.