Designing a quasi-experiment to study the clinical impact of adaptive risk prediction models

TL;DR

This paper develops a regression discontinuity framework to evaluate adaptive clinical risk prediction models and decision thresholds, enabling valid causal inference in dynamic healthcare settings.

Contribution

It introduces a novel method that accounts for model and threshold updates, allowing for accurate impact assessment of adaptive risk prediction in clinical trials.

Findings

Accurately recovers treatment effects despite model updates

Handles thresholds that adapt to operational or clinical targets

Restores local exchangeability for valid causal inference

Abstract

Clinical risk prediction is a valuable tool for guiding healthcare interventions toward those most likely to benefit. Yet, evaluating the pairing of a risk prediction model with an intervention using randomized controlled trials presents substantial challenges, making quasi-experimental designs an attractive alternatives. Existing designs, however, assume that both the model and the decision rules used to trigger interventions (typically a risk threshold) remain fixed. This limits their utility in modern healthcare, where both are routinely updated. We introduce a regression discontinuity framework that accommodates adaptation in both the model and the risk threshold. We precisely characterize the form of interference introduced by these adaptations and exploit this structure to establish conditions for identification and thus design estimation strategies. The key idea is to define counterfactual risks-the scores patients would have received under hypothetical reorderings-thereby restoring local exchangeability and enabling valid estimation of the local average treatment effect. Our estimator leverages the fact that, although counterfactual risk vectors grow with time, they typically lie in a low-dimensional space. In simulations of cardiovascular prevention programs, we show that our method accurately recovers treatment effects even as thresholds adapt to meet operational or clinical targets and models are updated to align predicted and observed outcomes or to exclude demographic predictors such as race.