# Assessing Reliability and Challenges of Uncertainty Estimations for Medical Image Segmentation

## TL;DR

This paper evaluates the reliability of uncertainty estimation methods in medical image segmentation, revealing their limitations at the subject level and proposing auxiliary networks as a promising alternative for failure detection.

## Contribution

It provides a comprehensive evaluation of common uncertainty measures, highlighting their calibration issues at the subject level and introducing auxiliary networks as a novel solution.

## Key findings

- Uncertainty methods are well-calibrated at dataset level but not at subject level.
- Current methods have limited reliability for individual patient failure detection.
- Auxiliary networks are effective and versatile for uncertainty estimation.

## Abstract

Despite the recent improvements in overall accuracy, deep learning systems still exhibit low levels of robustness. Detecting possible failures is critical for a successful clinical integration of these systems, where each data point corresponds to an individual patient. Uncertainty measures are a promising direction to improve failure detection since they provide a measure of a system's confidence. Although many uncertainty estimation methods have been proposed for deep learning, little is known on their benefits and current challenges for medical image segmentation. Therefore, we report results of evaluating common voxel-wise uncertainty measures with respect to their reliability, and limitations on two medical image segmentation datasets. Results show that current uncertainty methods perform similarly and although they are well-calibrated at the dataset level, they tend to be miscalibrated at subject-level. Therefore, the reliability of uncertainty estimates is compromised, highlighting the importance of developing subject-wise uncertainty estimations. Additionally, among the benchmarked methods, we found auxiliary networks to be a valid alternative to common uncertainty methods since they can be applied to any previously trained segmentation model.