The risks of risk assessment: causal blind spots when using prediction models for treatment decisions

TL;DR

This paper highlights the causal blind spots in current prediction models used for treatment decisions, emphasizing the need for causal reasoning to prevent misinterpretation and ensure patient safety.

Contribution

It identifies causal blind spots in common modeling approaches and advocates for integrating causal inference techniques in prediction model development for treatment decisions.

Findings

Models ignoring causal roles can lead to misinterpretation of risks.

Using causal inference improves the accuracy of treatment effect predictions.

Clear communication of model limitations prevents misinformed decisions.

Abstract

Clinicians increasingly rely on prediction models to guide treatment choices. Most prediction models, however, are developed using observational data that include some patients who have already received the treatment the prediction model is meant to inform. Special attention to the causal role of those earlier treatments is required when interpreting the resulting predictions. We identify 'causal blind spots' in three common approaches to handling treatment when developing a prediction model: including treatment as a predictor, restricting to individuals taking a certain treatment, and ignoring treatment. Through several real examples, we illustrate how the risks obtained from models developed using such approaches may be misinterpreted and can lead to misinformed decision-making. Our discussion covers issues attributable to confounding, selection, mediation and changes in treatment protocols over time. We advocate for an extension of guidelines for the development, reporting and evaluation of prediction models to avoid such misinterpretations. Developers must ensure that the intended target population for the model, and the treatment conditions under which predictions hold, are clearly communicated. When prediction models are intended to inform treatment decisions, they need to provide estimates of risk under the specific treatment (or intervention) options being considered, known as 'prediction under interventions'. Next to suitable data, this requires causal reasoning and causal inference techniques during model development and evaluation. Being clear about what a given prediction model can and cannot be used for prevents misinformed treatment decisions and thereby prevents potential harm to patients.