Region-Guided Attack on the Segment Anything Model (SAM)

TL;DR

This paper introduces the Region-Guided Attack (RGA), a novel adversarial method targeting the Segment Anything Model (SAM) by manipulating segmented regions, exposing vulnerabilities and highlighting the need for robust defenses in image segmentation.

Contribution

The paper proposes RGA, a region-guided adversarial attack tailored for SAM, leveraging structural segmentation cues for targeted perturbations, a novel approach in segmentation attack strategies.

Findings

RGA achieves high success rates in fooling SAM in white-box scenarios.

RGA effectively fragments large segments and enlarges small ones, causing segmentation errors.

The attack demonstrates robustness in black-box settings, indicating SAM's vulnerability.

Abstract

The Segment Anything Model (SAM) is a cornerstone of image segmentation, demonstrating exceptional performance across various applications, particularly in autonomous driving and medical imaging, where precise segmentation is crucial. However, SAM is vulnerable to adversarial attacks that can significantly impair its functionality through minor input perturbations. Traditional techniques, such as FGSM and PGD, are often ineffective in segmentation tasks due to their reliance on global perturbations that overlook spatial nuances. Recent methods like Attack-SAM-K and UAD have begun to address these challenges, but they frequently depend on external cues and do not fully leverage the structural interdependencies within segmentation processes. This limitation underscores the need for a novel adversarial strategy that exploits the unique characteristics of segmentation tasks. In response, we introduce the Region-Guided Attack (RGA), designed specifically for SAM. RGA utilizes a Region-Guided Map (RGM) to manipulate segmented regions, enabling targeted perturbations that fragment large segments and expand smaller ones, resulting in erroneous outputs from SAM. Our experiments demonstrate that RGA achieves high success rates in both white-box and black-box scenarios, emphasizing the need for robust defenses against such sophisticated attacks. RGA not only reveals SAM's vulnerabilities but also lays the groundwork for developing more resilient defenses against adversarial threats in image segmentation.