An Empirical Study on the Robustness of the Segment Anything Model (SAM)

TL;DR

This paper investigates the robustness of the Segment Anything Model (SAM) across various real-world image perturbations, revealing its vulnerabilities and proposing customization strategies to improve resilience in practical applications.

Contribution

It provides a comprehensive empirical analysis of SAM's robustness under diverse conditions and explores methods to enhance its stability through tailored prompting and domain knowledge.

Findings

SAM's performance declines with image perturbations

Customized prompting improves robustness

Vulnerability varies across different perturbations

Abstract

The Segment Anything Model (SAM) is a foundation model for general image segmentation. Although it exhibits impressive performance predominantly on natural images, understanding its robustness against various image perturbations and domains is critical for real-world applications where such challenges frequently arise. In this study we conduct a comprehensive robustness investigation of SAM under diverse real-world conditions. Our experiments encompass a wide range of image perturbations. Our experimental results demonstrate that SAM's performance generally declines under perturbed images, with varying degrees of vulnerability across different perturbations. By customizing prompting techniques and leveraging domain knowledge based on the unique characteristics of each dataset, the model's resilience to these perturbations can be enhanced, addressing dataset-specific challenges. This work sheds light on the limitations and strengths of SAM in real-world applications, promoting the development of more robust and versatile image segmentation solutions.