Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models

TL;DR

This paper introduces VELM, a novel LLM-based pipeline for industrial anomaly classification that combines unsupervised detection with large language models, achieving state-of-the-art accuracy and providing new datasets with precise labels.

Contribution

The paper presents VELM, a new approach integrating LLMs for anomaly classification and introduces refined datasets with accurate labels for rigorous evaluation.

Findings

Achieves 80.4% accuracy on MVTec-AD, surpassing baselines by 5%.

Achieves 84% accuracy on MVTec-AC, demonstrating effectiveness.

Introduces MVTec-AC and VisA-AC datasets with precise anomaly labels.

Abstract

Recent advances in visual industrial anomaly detection have demonstrated exceptional performance in identifying and segmenting anomalous regions while maintaining fast inference speeds. However, anomaly classification-distinguishing different types of anomalies-remains largely unexplored despite its critical importance in real-world inspection tasks. To address this gap, we propose VELM, a novel LLM-based pipeline for anomaly classification. Given the critical importance of inference speed, we first apply an unsupervised anomaly detection method as a vision expert to assess the normality of an observation. If an anomaly is detected, the LLM then classifies its type. A key challenge in developing and evaluating anomaly classification models is the lack of precise annotations of anomaly classes in existing datasets. To address this limitation, we introduce MVTec-AC and VisA-AC, refined versions of the widely used MVTec-AD and VisA datasets, which include accurate anomaly class labels for rigorous evaluation. Our approach achieves a state-of-the-art anomaly classification accuracy of 80.4% on MVTec-AD, exceeding the prior baselines by 5%, and 84% on MVTec-AC, demonstrating the effectiveness of VELM in understanding and categorizing anomalies. We hope our methodology and benchmark inspire further research in anomaly classification, helping bridge the gap between detection and comprehensive anomaly characterization.