Knowledge Distillation for Anomaly Detection

TL;DR

This paper introduces a knowledge distillation approach to compress unsupervised anomaly detection models into smaller, supervised models suitable for resource-constrained devices, maintaining performance while reducing size.

Contribution

It proposes a novel knowledge distillation method for unsupervised anomaly detection models and techniques to enhance detection sensitivity.

Findings

Compressed models match larger models in detection performance

Significant reduction in model size and memory usage

Effective deployment on resource-limited devices

Abstract

Unsupervised deep learning techniques are widely used to identify anomalous behaviour. The performance of such methods is a product of the amount of training data and the model size. However, the size is often a limiting factor for the deployment on resource-constrained devices. We present a novel procedure based on knowledge distillation for compressing an unsupervised anomaly detection model into a supervised deployable one and we suggest a set of techniques to improve the detection sensitivity. Compressed models perform comparably to their larger counterparts while significantly reducing the size and memory footprint.