Segment Anything with Robust Uncertainty-Accuracy Correlation

TL;DR

This paper introduces RUAC, a method to improve pixel-wise uncertainty estimation in segmentation models under domain shifts by aligning uncertainty with accuracy, even under adversarial perturbations.

Contribution

RUAC adds a lightweight uncertainty head trained with collaborative attacks and applies Uncertainty-Accuracy Alignment to enhance reliability under appearance and deformation shifts.

Findings

RUAC improves segmentation quality across 23 zero-shot domains.

RUAC yields more faithful uncertainty with stronger correlation to accuracy.

RUAC maintains robustness under adversarial texture and geometry perturbations.

Abstract

Despite strong zero-shot performance, SAM is unreliable under domain shift due to Mask-level Confidence Confusion (MCC), where a single IoU-based mask score fails to reflect pixel-wise reliability near boundaries. Motivated by the contrast between texture-biased shortcuts in neural networks and shape-centric processing in human vision, we model out-of-domain variation as appearance shifts and non-rigid deformations that jointly stress calibration. We propose Segment Anything with Robust Uncertainty-Accuracy Correlation (RUAC) for robust pixel-wise uncertainty estimation under appearance and deformation shifts. RUAC adds a lightweight uncertainty head, trains it with a collaborative style-deformation attack that jointly perturbs texture and geometry, and applies Uncertainty-Accuracy Alignment to ensure uncertainty consistently highlights erroneous pixels even under adversarial perturbations. Across 23 zero-shot domains, RUAC improves segmentation quality and yields more faithful uncertainty with stronger uncertainty-accuracy correlation. Project page: https://hongyouzhou.github.io/ruac/.