Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning

TL;DR

This paper introduces Self-Perception Tuning (SPT), a novel method to improve SAM's anomaly segmentation in industrial scenarios by addressing domain shift and perception challenges through self-drafting and relational-aware adaptation.

Contribution

The paper proposes SPT, a new tuning approach that enhances SAM's perception for anomaly segmentation using self-drafting and visual-relation-aware adapters.

Findings

SPT significantly outperforms baseline methods on benchmark datasets.

The self-drafting strategy improves initial mask quality.

Relational-aware adapters enhance discriminative perception.

Abstract

Segment Anything Model (SAM) has made great progress in anomaly segmentation tasks due to its impressive generalization ability. However, existing methods that directly apply SAM through prompting often overlook the domain shift issue, where SAM performs well on natural images but struggles in industrial scenarios. Parameter-Efficient Fine-Tuning (PEFT) offers a promising solution, but it may yield suboptimal performance by not adequately addressing the perception challenges during adaptation to anomaly images. In this paper, we propose a novel \textbf{S}elf-\textbf{P}erceptinon \textbf{T}uning (\textbf{SPT}) method, aiming to enhance SAM's perception capability for anomaly segmentation. The SPT method incorporates a self-drafting tuning strategy, which generates an initial coarse draft of the anomaly mask, followed by a refinement process. Additionally, a visual-relation-aware adapter is introduced to improve the perception of discriminative relational information for mask generation. Extensive experimental results on several benchmark datasets demonstrate that our SPT method can significantly outperform baseline methods, validating its effectiveness.