Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection

TL;DR

This paper introduces AAND, a two-stage anomaly detection framework that enhances feature discrepancy between a pre-trained teacher and a student model through anomaly amplification and normality distillation, achieving state-of-the-art results.

Contribution

The paper proposes a novel Residual Anomaly Amplification module and a reverse distillation paradigm with Hard Knowledge Distillation loss to improve anomaly detection performance.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively amplifies anomalies while preserving feature integrity.

Enhances normal pattern reconstruction with a novel distillation approach.

Abstract

With the wide application of knowledge distillation between an ImageNet pre-trained teacher model and a learnable student model, unsupervised anomaly detection has witnessed a significant achievement in the past few years. The success of this framework mainly relies on how to keep the feature discrepancy between the teacher and student model, in which it has two underlying sub-assumptions: (1) The teacher model can represent two separable distributions for the normal and abnormal patterns, while (2) the student model can only reconstruct the normal distribution. However, it still remains a challenging issue to maintain these ideal assumptions in practice. In this paper, we propose a simple yet effective two-stage industrial anomaly detection framework, termed AAND, which sequentially performs Anomaly Amplification and Normality Distillation to enhance the two assumptions. In the first anomaly amplification stage, we propose a novel Residual Anomaly Amplification (RAA) module to advance the pre-trained teacher encoder with synthetic anomalies. It generates adaptive residuals to amplify anomalies while maintaining the feature integrity of pre-trained model. It mainly comprises a Matching-guided Residual Gate and an Attribute-scaling Residual Generator, which can determine the residuals' proportion and characteristic, respectively. In the second normality distillation stage, we further employ a reverse distillation paradigm to train a student decoder, in which a novel Hard Knowledge Distillation (HKD) loss is built to better facilitate the reconstruction of normal patterns. Comprehensive experiments on the MvTecAD, VisA, and MvTec3D-RGB datasets show that our method achieves state-of-the-art performance.