Rethinking Uncertainty in Segmentation: From Estimation to Decision

TL;DR

This paper investigates how uncertainty estimates in medical image segmentation can be effectively converted into decision-making policies, demonstrating significant error reduction and robustness improvements.

Contribution

It introduces a decision-focused approach to uncertainty in segmentation, emphasizing the importance of policies over mere uncertainty estimation.

Findings

Up to 80% segmentation errors removed with 25% pixel deferral.

A simple confidence-aware deferral rule improves decision quality.

Calibration does not necessarily enhance decision-making utility.

Abstract

In medical image segmentation, uncertainty estimates are often reported but rarely used to guide decisions. We study the missing step: how uncertainty maps are converted into actionable policies such as accepting, flagging, or deferring predictions. We formulate segmentation as a two-stage pipeline, estimation followed by decision, and show that optimizing uncertainty alone fails to capture most of the achievable safety gains. Using retinal vessel segmentation benchmarks (DRIVE, STARE, CHASE_DB1), we evaluate two uncertainty sources (Monte Carlo Dropout and Test-Time Augmentation) combined with three deferral strategies, and introduce a simple confidence-aware deferral rule that prioritizes uncertain and low-confidence predictions. Our results show that the best method and policy combination removes up to 80 percent of segmentation errors at only 25 percent pixel deferral, while achieving strong cross-dataset robustness. We further show that calibration improvements do not translate to better decision quality, highlighting a disconnect between standard uncertainty metrics and real-world utility. These findings suggest that uncertainty should be evaluated based on the decisions it enables, rather than in isolation.