TextureSAM: Towards a Texture Aware Foundation Model for Segmentation

TL;DR

TextureSAM is a new foundation model that enhances segmentation performance in texture-dominant scenarios by incorporating texture augmentation during training, addressing the shape bias of existing models like SAM.

Contribution

We introduce TextureSAM, a texture-aware segmentation model that mitigates shape bias in SAM through novel texture augmentation techniques and dataset modifications.

Findings

TextureSAM outperforms SAM-2 on natural and synthetic texture datasets.

Texture augmentation improves segmentation accuracy in texture-rich scenarios.

The model's code and dataset will be publicly available.

Abstract

Segment Anything Models (SAM) have achieved remarkable success in object segmentation tasks across diverse datasets. However, these models are predominantly trained on large-scale semantic segmentation datasets, which introduce a bias toward object shape rather than texture cues in the image. This limitation is critical in domains such as medical imaging, material classification, and remote sensing, where texture changes define object boundaries. In this study, we investigate SAM's bias toward semantics over textures and introduce a new texture-aware foundation model, TextureSAM, which performs superior segmentation in texture-dominant scenarios. To achieve this, we employ a novel fine-tuning approach that incorporates texture augmentation techniques, incrementally modifying training images to emphasize texture features. By leveraging a novel texture-alternation of the ADE20K dataset, we guide TextureSAM to prioritize texture-defined regions, thereby mitigating the inherent shape bias present in the original SAM model. Our extensive experiments demonstrate that TextureSAM significantly outperforms SAM-2 on both natural (+0.2 mIoU) and synthetic (+0.18 mIoU) texture-based segmentation datasets. The code and texture-augmented dataset will be publicly available.