On the Problem of Consistent Anomalies in Zero-Shot Anomaly Detection

TL;DR

This paper investigates the core challenges of zero-shot anomaly detection and segmentation, proposing a graph-based framework and extending it to 3D medical imaging, enabling effective anomaly detection without training data.

Contribution

It formalizes the problem of consistent anomalies, introduces CoDeGraph for filtering them, and extends the approach to 3D medical imaging and vision-language models.

Findings

CoDeGraph effectively suppresses consistent anomalies.

Volumetric tokenization enables zero-shot 3D anomaly detection.

Pseudo-masks can supervise prompt-driven vision-language models.

Abstract

Zero-shot anomaly classification and segmentation (AC/AS) aim to detect anomalous samples and regions without any training data, a capability increasingly crucial in industrial inspection and medical imaging. This dissertation aims to investigate the core challenges of zero-shot AC/AS and presents principled solutions rooted in theory and algorithmic design. We first formalize the problem of consistent anomalies, a failure mode in which recurring similar anomalies systematically bias distance-based methods. By analyzing the statistical and geometric behavior of patch representations from pre-trained Vision Transformers, we identify two key phenomena - similarity scaling and neighbor-burnout - that describe how relationships among normal patches change with and without consistent anomalies in settings characterized by highly similar objects. We then introduce CoDeGraph, a graph-based framework for filtering consistent anomalies built on the similarity scaling and neighbor-burnout phenomena. Through multi-stage graph construction, community detection, and structured refinement, CoDeGraph effectively suppresses the influence of consistent anomalies. Next, we extend this framework to 3D medical imaging by proposing a training-free, computationally efficient volumetric tokenization strategy for MRI data. This enables a genuinely zero-shot 3D anomaly detection pipeline and shows that volumetric anomaly segmentation is achievable without any 3D training samples. Finally, we bridge batch-based and text-based zero-shot methods by demonstrating that CoDeGraph-derived pseudo-masks can supervise prompt-driven vision-language models. Together, this dissertation provides theoretical understanding and practical solutions for the zero-shot AC/AS problem.