Understanding Segment Anything Model: SAM is Biased Towards Texture Rather than Shape

TL;DR

This paper investigates the Segment Anything Model (SAM) and finds that, contrary to expectations, it is more biased towards texture features than shape cues, revealing new insights into its underlying recognition biases.

Contribution

The study introduces a novel experimental setup to disentangle texture and shape cues, revealing SAM's unexpected bias towards texture over shape.

Findings

SAM is biased towards texture-like features rather than shape.

Disentangling texture and shape cues shows texture bias in SAM.

The bias is confirmed through cue conflict experiments.

Abstract

In contrast to the human vision that mainly depends on the shape for recognizing the objects, deep image recognition models are widely known to be biased toward texture. Recently, Meta research team has released the first foundation model for image segmentation, termed segment anything model (SAM), which has attracted significant attention. In this work, we understand SAM from the perspective of texture \textit{v.s.} shape. Different from label-oriented recognition tasks, the SAM is trained to predict a mask for covering the object shape based on a promt. With this said, it seems self-evident that the SAM is biased towards shape. In this work, however, we reveal an interesting finding: the SAM is strongly biased towards texture-like dense features rather than shape. This intriguing finding is supported by a novel setup where we disentangle texture and shape cues and design texture-shape cue conflict for mask prediction.