GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation

TL;DR

GaRA-SAM enhances the robustness of the Segment Anything Model against input degradations by introducing lightweight, input-aware adapters that dynamically adjust the model's effective rank, significantly improving performance on challenging benchmarks.

Contribution

The paper proposes gated-rank adaptation (GaRA), a novel method that maintains SAM's generalization while providing fine-grained, input-aware robustness through dynamic rank adjustment.

Findings

Outperforms prior methods on all robust segmentation benchmarks.

Surpasses previous best IoU score by up to 21.3% on ACDC dataset.

Maintains parameter efficiency and standard training protocols.

Abstract

Improving robustness of the Segment Anything Model (SAM) to input degradations is critical for its deployment in high-stakes applications such as autonomous driving and robotics. Our approach to this challenge prioritizes three key aspects: first, parameter efficiency to maintain the inherent generalization capability of SAM; second, fine-grained and input-aware robustification to precisely address the input corruption; and third, adherence to standard training protocols for ease of training. To this end, we propose gated-rank adaptation (GaRA). GaRA introduces lightweight adapters into intermediate layers of the frozen SAM, where each adapter dynamically adjusts the effective rank of its weight matrix based on the input by selectively activating (rank-1) components of the matrix using a learned gating module. This adjustment enables fine-grained and input-aware robustification without compromising the generalization capability of SAM. Our model, GaRA-SAM, significantly outperforms prior work on all robust segmentation benchmarks. In particular, it surpasses the previous best IoU score by up to 21.3\%p on ACDC, a challenging real corrupted image dataset.