VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer

TL;DR

VisualAD introduces a purely visual, transformer-based approach for zero-shot anomaly detection that eliminates the need for text encoders, achieving state-of-the-art results across multiple benchmarks.

Contribution

It proposes a novel vision transformer framework with learnable tokens and spatial-aware modules for zero-shot anomaly detection, removing reliance on cross-modal text-image alignment.

Findings

Achieves state-of-the-art performance on 13 benchmarks

Works seamlessly with pretrained vision backbones like CLIP and DINOv2

Operates without text encoders, simplifying the ZSAD pipeline

Abstract

Zero-shot anomaly detection (ZSAD) requires detecting and localizing anomalies without access to target-class anomaly samples. Mainstream methods rely on vision-language models (VLMs) such as CLIP: they build hand-crafted or learned prompt sets for normal and abnormal semantics, then compute image-text similarities for open-set discrimination. While effective, this paradigm depends on a text encoder and cross-modal alignment, which can lead to training instability and parameter redundancy. This work revisits the necessity of the text branch in ZSAD and presents VisualAD, a purely visual framework built on Vision Transformers. We introduce two learnable tokens within a frozen backbone to directly encode normality and abnormality. Through multi-layer self-attention, these tokens interact with patch tokens, gradually acquiring high-level notions of normality and anomaly while guiding patches to highlight anomaly-related cues. Additionally, we incorporate a Spatial-Aware Cross-Attention (SCA) module and a lightweight Self-Alignment Function (SAF): SCA injects fine-grained spatial information into the tokens, and SAF recalibrates patch features before anomaly scoring. VisualAD achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains, and adapts seamlessly to pretrained vision backbones such as the CLIP image encoder and DINOv2. Code: https://github.com/7HHHHH/VisualAD