Towards Integrating Uncertainty for Domain-Agnostic Segmentation

TL;DR

This paper explores how uncertainty quantification can improve the robustness and domain-agnostic generalization of segmentation models like SAM, especially in challenging conditions such as shadows and camouflage.

Contribution

It introduces UncertSAM, a benchmark for stress-testing segmentation models under difficult conditions, and evaluates lightweight uncertainty estimation methods for potential performance enhancement.

Findings

Laplace approximation provides meaningful uncertainty estimates

Uncertainty-guided refinement shows preliminary benefits

Benchmark and code are publicly available

Abstract

Foundation models for segmentation such as the Segment Anything Model (SAM) family exhibit strong zero-shot performance, but remain vulnerable in shifted or limited-knowledge domains. This work investigates whether uncertainty quantification can mitigate such challenges and enhance model generalisability in a domain-agnostic manner. To this end, we (1) curate UncertSAM, a benchmark comprising eight datasets designed to stress-test SAM under challenging segmentation conditions including shadows, transparency, and camouflage; (2) evaluate a suite of lightweight, post-hoc uncertainty estimation methods; and (3) assess a preliminary uncertainty-guided prediction refinement step. Among evaluated approaches, a last-layer Laplace approximation yields uncertainty estimates that correlate well with segmentation errors, indicating a meaningful signal. While refinement benefits are preliminary, our findings underscore the potential of incorporating uncertainty into segmentation models to support robust, domain-agnostic performance. Our benchmark and code are made publicly available.