Conformal Performance Range Prediction for Segmentation Output Quality Control

TL;DR

This paper introduces a novel conformal prediction method for estimating segmentation output quality ranges with statistical guarantees, addressing calibration and uncertainty challenges in neural networks.

Contribution

It develops a sampling-based uncertainty estimation combined with split conformal prediction to produce reliable performance ranges with guarantees.

Findings

Achieves desired coverage with small prediction ranges

Demonstrates effectiveness on retinal vessel segmentation dataset

Validates approach across five uncertainty estimation techniques

Abstract

Recent works have introduced methods to estimate segmentation performance without ground truth, relying solely on neural network softmax outputs. These techniques hold potential for intuitive output quality control. However, such performance estimates rely on calibrated softmax outputs, which is often not the case in modern neural networks. Moreover, the estimates do not take into account inherent uncertainty in segmentation tasks. These limitations may render precise performance predictions unattainable, restricting the practical applicability of performance estimation methods. To address these challenges, we develop a novel approach for predicting performance ranges with statistical guarantees of containing the ground truth with a user specified probability. Our method leverages sampling-based segmentation uncertainty estimation to derive heuristic performance ranges, and applies split conformal prediction to transform these estimates into rigorous prediction ranges that meet the desired guarantees. We demonstrate our approach on the FIVES retinal vessel segmentation dataset and compare five commonly used sampling-based uncertainty estimation techniques. Our results show that it is possible to achieve the desired coverage with small prediction ranges, highlighting the potential of performance range prediction as a valuable tool for output quality control.