Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts

TL;DR

This paper introduces InCTRL, a novel few-shot residual learning approach for generalist anomaly detection that can adapt to diverse datasets without additional training, outperforming existing methods across multiple domains.

Contribution

The paper proposes InCTRL, a residual learning model trained on an auxiliary dataset that enables zero-shot anomaly detection across various domains using few-shot normal samples.

Findings

InCTRL outperforms state-of-the-art methods on nine diverse anomaly detection datasets.

The approach effectively detects industrial, medical, and semantic anomalies without domain-specific training.

InCTRL demonstrates strong generalization capabilities in one-vs-all and multi-class settings.

Abstract

This paper explores the problem of Generalist Anomaly Detection (GAD), aiming to train one single detection model that can generalize to detect anomalies in diverse datasets from different application domains without any further training on the target data. Some recent studies have shown that large pre-trained Visual-Language Models (VLMs) like CLIP have strong generalization capabilities on detecting industrial defects from various datasets, but their methods rely heavily on handcrafted text prompts about defects, making them difficult to generalize to anomalies in other applications, e.g., medical image anomalies or semantic anomalies in natural images. In this work, we propose to train a GAD model with few-shot normal images as sample prompts for AD on diverse datasets on the fly. To this end, we introduce a novel approach that learns an in-context residual learning model for GAD, termed InCTRL. It is trained on an auxiliary dataset to discriminate anomalies from normal samples based on a holistic evaluation of the residuals between query images and few-shot normal sample prompts. Regardless of the datasets, per definition of anomaly, larger residuals are expected for anomalies than normal samples, thereby enabling InCTRL to generalize across different domains without further training. Comprehensive experiments on nine AD datasets are performed to establish a GAD benchmark that encapsulate the detection of industrial defect anomalies, medical anomalies, and semantic anomalies in both one-vs-all and multi-class setting, on which InCTRL is the best performer and significantly outperforms state-of-the-art competing methods. Code is available at https://github.com/mala-lab/InCTRL.