Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain

TL;DR

This paper introduces a new evaluation framework and a post-hoc calibration method to improve the reliability of probability estimates in medical diagnosis models, especially under label uncertainty.

Contribution

It formalizes metrics for label uncertainty evaluation and proposes alpha-calibration, a novel method for better uncertainty calibration in neural networks.

Findings

Alpha-calibration improves the reliability of uncertainty estimates.

The framework effectively assesses predictions with label disagreement.

Results show enhanced calibration in medical imaging applications.

Abstract

We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called $alpha$-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs.