Updating Clinical Risk Stratification Models Using Rank-Based Compatibility: Approaches for Evaluating and Optimizing Clinician-Model Team Performance

TL;DR

This paper introduces a rank-based compatibility measure and a new optimization approach to update clinical risk models, ensuring they remain aligned with clinician expectations while maintaining predictive accuracy.

Contribution

It proposes a novel rank-based compatibility metric and an associated loss function for updating clinical models, addressing limitations of existing threshold-dependent measures.

Findings

The new method improves model compatibility without sacrificing discriminative performance.

Application to MIMIC data shows increased compatibility score by 0.019 with confidence interval 0.005-0.035.

Models updated with the proposed approach outperform existing techniques in maintaining clinician-model team performance.

Abstract

As data shift or new data become available, updating clinical machine learning models may be necessary to maintain or improve performance over time. However, updating a model can introduce compatibility issues when the behavior of the updated model does not align with user expectations, resulting in poor user-model team performance. Existing compatibility measures depend on model decision thresholds, limiting their applicability in settings where models are used to generate rankings based on estimated risk. To address this limitation, we propose a novel rank-based compatibility measure, $C^R$, and a new loss function that aims to optimize discriminative performance while encouraging good compatibility. Applied to a case study in mortality risk stratification leveraging data from MIMIC, our approach yields more compatible models while maintaining discriminative performance compared to existing model selection techniques, with an increase in $C^R$ of $0.019$ ($95\%$ confidence interval: $0.005$, $0.035$). This work provides new tools to analyze and update risk stratification models used in clinical care.