Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects

TL;DR

This paper investigates the limitations of segmentation foundation models like SAM in segmenting tree-like and low-contrast objects, revealing fundamental challenges and providing a quantitative framework for understanding their performance boundaries.

Contribution

It introduces interpretable metrics for object structure and texture, systematically studies model failures, and demonstrates that fine-tuning cannot fully overcome these fundamental limitations.

Findings

Performance correlates with object structure and texture separability

Models struggle with dense, tree-like, low-contrast objects due to misinterpretation of local structure

Fine-tuning does not fully resolve segmentation challenges for complex structures

Abstract

Image segmentation foundation models (SFMs) like Segment Anything Model (SAM) have achieved impressive zero-shot and interactive segmentation across diverse domains. However, they struggle to segment objects with certain structures, particularly those with dense, tree-like morphology and low textural contrast from their surroundings. These failure modes are crucial for understanding the limitations of SFMs in real-world applications. To systematically study this issue, we introduce interpretable metrics quantifying object tree-likeness and textural separability. On carefully controlled synthetic experiments and real-world datasets, we show that SFM performance (\eg, SAM, SAM 2, HQ-SAM) noticeably correlates with these factors. We attribute these failures to SFMs misinterpreting local structure as global texture, resulting in over-segmentation or difficulty distinguishing objects from similar backgrounds. Notably, targeted fine-tuning fails to resolve this issue, indicating a fundamental limitation. Our study provides the first quantitative framework for modeling the behavior of SFMs on challenging structures, offering interpretable insights into their segmentation capabilities.