Domain Adaptation of Attention Heads for Zero-shot Anomaly Detection

TL;DR

This paper introduces HeadCLIP, a novel method that adapts vision-language models for zero-shot anomaly detection by dynamically adjusting attention heads and using learnable prompts, significantly improving detection performance across diverse datasets.

Contribution

It proposes HeadCLIP, which effectively adapts both text and image encoders of CLIP for zero-shot anomaly detection through learnable prompts and attention head weights, addressing limitations of prior methods.

Findings

Outperforms existing ZSAD methods on 17 datasets.

Achieves up to 4.9%p improvement in pixel-level detection.

Achieves up to 3.7%p improvement in image-level detection.

Abstract

Zero-shot anomaly detection (ZSAD) enables anomaly detection without normal samples from target categories, addressing scenarios where task-specific training data is unavailable. However, existing ZSAD methods either neglect adaptation of vision-language models to anomaly detection or implement only partial adaptation. This paper proposes Head-adaptive CLIP (HeadCLIP), which effectively adapts both text and image encoders. HeadCLIP employs learnable prompts in the text encoder to generalize normality and abnormality concepts, and introduces learnable head weights in the image encoder to dynamically adjust attention head features for task-specific adaptation. A joint anomaly score is further proposed to leverage adapted pixel-level information for enhanced image-level detection. Experiments on 17 datasets across industrial and medical domains demonstrate that HeadCLIP outperforms existing ZSAD methods at both pixel and image levels, achieving improvements of up to 4.9\%p in pixel-level mean anomaly detection score (mAD) and 3.7%p in image-level mAD in the industrial domain, with comparable gains (3.2%p, 3.2%p) in the medical domain. Code and pretrained weights are available at https://github.com/kiyoonjeong0305/HeadCLIP.