Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

TL;DR

This paper introduces LAKE, a training-free method to identify sparse anomaly-sensitive neurons in vision-language models, enabling state-of-the-art anomaly detection with interpretability by leveraging latent knowledge within pre-trained models.

Contribution

LAKE uncovers and activates intrinsic anomaly knowledge in pre-trained models without additional training, shifting the paradigm of anomaly detection towards internal neuron-level analysis.

Findings

LAKE achieves state-of-the-art results on industrial anomaly detection benchmarks.

It provides intrinsic interpretability by isolating critical neurons.

The method requires only minimal normal samples for detection.

Abstract

Large-scale vision-language models (VLMs) exhibit remarkable zero-shot capabilities, yet the internal mechanisms driving their anomaly detection (AD) performance remain poorly understood. Current methods predominantly treat VLMs as black-box feature extractors, assuming that anomaly-specific knowledge must be acquired through external adapters or memory banks. In this paper, we challenge this assumption by arguing that anomaly knowledge is intrinsically embedded within pre-trained models but remains latent and under-activated. We hypothesize that this knowledge is concentrated within a sparse subset of anomaly-sensitive neurons. To validate this, we propose latent anomaly knowledge excavation (LAKE), a training-free framework that identifies and elicits these critical neuronal signals using only a minimal set of normal samples. By isolating these sensitive neurons, LAKE constructs a highly compact normality representation that integrates visual structural deviations with cross-modal semantic activations. Extensive experiments on industrial AD benchmarks demonstrate that LAKE achieves state-of-the-art performance while providing intrinsic, neuron-level interpretability. Ultimately, our work advocates for a paradigm shift: redefining anomaly detection as the targeted activation of latent pre-trained knowledge rather than the acquisition of a downstream task.