ASAM: Boosting Segment Anything Model with Adversarial Tuning

TL;DR

This paper presents ASAM, an adversarial tuning method that enhances the Segment Anything Model's segmentation performance by generating natural, photorealistic adversarial examples using a stable diffusion model, without changing the original architecture.

Contribution

Introduction of ASAM, a novel adversarial tuning approach that improves SAM's segmentation accuracy using natural adversarial examples generated via diffusion models.

Findings

ASAM significantly outperforms baseline SAM across various segmentation tasks.

The method improves robustness and accuracy without additional data or model changes.

Adversarial examples maintain photorealism and alignment with original annotations.

Abstract

In the evolving landscape of computer vision, foundation models have emerged as pivotal tools, exhibiting exceptional adaptability to a myriad of tasks. Among these, the Segment Anything Model (SAM) by Meta AI has distinguished itself in image segmentation. However, SAM, like its counterparts, encounters limitations in specific niche applications, prompting a quest for enhancement strategies that do not compromise its inherent capabilities. This paper introduces ASAM, a novel methodology that amplifies SAM's performance through adversarial tuning. We harness the potential of natural adversarial examples, inspired by their successful implementation in natural language processing. By utilizing a stable diffusion model, we augment a subset (1%) of the SA-1B dataset, generating adversarial instances that are more representative of natural variations rather than conventional imperceptible perturbations. Our approach maintains the photorealism of adversarial examples and ensures alignment with original mask annotations, thereby preserving the integrity of the segmentation task. The fine-tuned ASAM demonstrates significant improvements across a diverse range of segmentation tasks without necessitating additional data or architectural modifications. The results of our extensive evaluations confirm that ASAM establishes new benchmarks in segmentation tasks, thereby contributing to the advancement of foundational models in computer vision. Our project page is in https://asam2024.github.io/.