Customizing Visual-Language Foundation Models for Multi-modal Anomaly Detection and Reasoning

TL;DR

This paper presents a generic multi-modal anomaly detection framework that leverages visual-language foundation models with domain knowledge prompts, enabling robust detection and reasoning across diverse industrial scenarios and data modalities.

Contribution

It introduces a multi-modal prompting strategy and unified input representation to customize foundation models for versatile anomaly detection and reasoning tasks.

Findings

Enhanced anomaly detection performance with visual and language prompts

Effective detection across images, point clouds, and videos

Demonstrated capabilities in multi-object and temporal data scenarios

Abstract

Anomaly detection is vital in various industrial scenarios, including the identification of unusual patterns in production lines and the detection of manufacturing defects for quality control. Existing techniques tend to be specialized in individual scenarios and lack generalization capacities. In this study, our objective is to develop a generic anomaly detection model that can be applied in multiple scenarios. To achieve this, we custom-build generic visual language foundation models that possess extensive knowledge and robust reasoning abilities as anomaly detectors and reasoners. Specifically, we introduce a multi-modal prompting strategy that incorporates domain knowledge from experts as conditions to guide the models. Our approach considers diverse prompt types, including task descriptions, class context, normality rules, and reference images. In addition, we unify the input representation of multi-modality into a 2D image format, enabling multi-modal anomaly detection and reasoning. Our preliminary studies demonstrate that combining visual and language prompts as conditions for customizing the models enhances anomaly detection performance. The customized models showcase the ability to detect anomalies across different data modalities such as images, point clouds, and videos. Qualitative case studies further highlight the anomaly detection and reasoning capabilities, particularly for multi-object scenes and temporal data. Our code is publicly available at https://github.com/Xiaohao-Xu/Customizable-VLM