Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models

TL;DR

This paper introduces a new visual dataset and benchmark for zero-shot anomaly detection and reasoning with multimodal large language models, and proposes a specialized model, Anomaly-OneVision, that significantly improves detection and reasoning performance.

Contribution

It establishes the first visual instruction tuning dataset and benchmark for ZSAD and reasoning, and proposes a novel specialist visual assistant model, Anomaly-OneVision, for improved anomaly detection and explanation.

Findings

Current MLLMs struggle with fine-grained anomaly detection.

Anomaly-OneVision outperforms generalist models in detection and reasoning.

The proposed approach extends to medical and 3D anomaly detection.

Abstract

Zero-Shot Anomaly Detection (ZSAD) is an emerging AD paradigm. Unlike the traditional unsupervised AD setting that requires a large number of normal samples to train a model, ZSAD is more practical for handling data-restricted real-world scenarios. Recently, Multimodal Large Language Models (MLLMs) have shown revolutionary reasoning capabilities in various vision tasks. However, the reasoning of image abnormalities remains underexplored due to the lack of corresponding datasets and benchmarks. To facilitate research in AD & reasoning, we establish the first visual instruction tuning dataset, Anomaly-Instruct-125k, and the evaluation benchmark, VisA-D&R. Through investigation with our benchmark, we reveal that current MLLMs like GPT-4o cannot accurately detect and describe fine-grained anomalous details in images. To address this, we propose Anomaly-OneVision (Anomaly-OV), the first specialist visual assistant for ZSAD and reasoning. Inspired by human behavior in visual inspection, Anomaly-OV leverages a Look-Twice Feature Matching (LTFM) mechanism to adaptively select and emphasize abnormal visual tokens. Extensive experiments demonstrate that Anomaly-OV achieves significant improvements over advanced generalist models in both detection and reasoning. Extensions to medical and 3D AD are provided for future study. The link to our project page: https://xujiacong.github.io/Anomaly-OV/