Bridge Feature Matching and Cross-Modal Alignment with Mutual-filtering for Zero-shot Anomaly Detection

TL;DR

FiSeCLIP introduces a training-free, batch-based zero-shot anomaly detection method leveraging CLIP's cross-modal features and mutual filtering, achieving superior results on benchmark datasets.

Contribution

The paper proposes FiSeCLIP, a novel zero-shot anomaly detection approach that combines feature matching, cross-modal alignment, and text-based filtering using CLIP without additional training.

Findings

Outperforms state-of-the-art AdaCLIP by +4.6% in segmentation AU-ROC.

Achieves +5.7% in F1-max for anomaly detection.

Effective in both anomaly classification and segmentation tasks.

Abstract

With the advent of vision-language models (e.g., CLIP) in zero- and few-shot settings, CLIP has been widely applied to zero-shot anomaly detection (ZSAD) in recent research, where the rare classes are essential and expected in many applications. This study introduces \textbf{FiSeCLIP} for ZSAD with training-free \textbf{CLIP}, combining the feature matching with the cross-modal alignment. Testing with the entire dataset is impractical, while batch-based testing better aligns with real industrial needs, and images within a batch can serve as mutual reference points. Accordingly, FiSeCLIP utilizes other images in the same batch as reference information for the current image. However, the lack of labels for these references can introduce ambiguity, we apply text information to \textbf{fi}lter out noisy features. In addition, we further explore CLIP's inherent potential to restore its local \textbf{se}mantic correlation, adapting it for fine-grained anomaly detection tasks to enable a more accurate filtering process. Our approach exhibits superior performance for both anomaly classification and segmentation on anomaly detection benchmarks, building a stronger baseline for the direction, e.g., on MVTec-AD, FiSeCLIP outperforms the SOTA AdaCLIP by +4.6\%$\uparrow$/+5.7\%$\uparrow$ in segmentation metrics AU-ROC/$F_1$-max.