Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs

TL;DR

This paper introduces a new evaluation framework for clinical decision support models that emphasizes calibration, robustness to distributional shifts, and cost sensitivity, addressing limitations of traditional scoring metrics.

Contribution

It develops an adjusted cross-entropy score based on proper scoring rules to better reflect clinical priorities and real-world deployment conditions.

Findings

The adjusted score improves model calibration assessment.

It enhances robustness to class prevalence shifts.

The framework aligns model evaluation with clinical decision-making needs.

Abstract

Machine learning-based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prioritize patient management decisions. However, widely used scoring rules, such as accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness to distributional shifts, and sensitivity to asymmetric error costs. In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for the uncertainty in class prevalences and domain-specific cost asymmetries often found in clinical settings. Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an adjusted variant of cross-entropy (log score) that averages cost-weighted performance over clinically relevant ranges of class balance. The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-world variations.