UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model

TL;DR

UncertainSAM introduces a Bayesian entropy-based uncertainty quantification method for the Segment Anything Model, improving prediction reliability and supporting various applications with a lightweight, post-hoc approach.

Contribution

The paper presents USAM, a novel, efficient post-hoc uncertainty quantification method for SAM, incorporating a new task uncertainty component and demonstrating superior performance.

Findings

USAM outperforms existing UQ methods on multiple datasets.

USAM is computationally efficient and easy to implement.

It effectively identifies model uncertainty sources.

Abstract

The introduction of the Segment Anything Model (SAM) has paved the way for numerous semantic segmentation applications. For several tasks, quantifying the uncertainty of SAM is of particular interest. However, the ambiguous nature of the class-agnostic foundation model SAM challenges current uncertainty quantification (UQ) approaches. This paper presents a theoretically motivated uncertainty quantification model based on a Bayesian entropy formulation jointly respecting aleatoric, epistemic, and the newly introduced task uncertainty. We use this formulation to train USAM, a lightweight post-hoc UQ method. Our model traces the root of uncertainty back to under-parameterised models, insufficient prompts or image ambiguities. Our proposed deterministic USAM demonstrates superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets, offering a computationally cheap and easy-to-use UQ alternative that can support user-prompting, enhance semi-supervised pipelines, or balance the tradeoff between accuracy and cost efficiency.