Are generative deep models for novelty detection truly better?

TL;DR

This paper compares generative deep models and classical methods for anomaly detection across various datasets, finding that deep models' performance heavily depends on hyperparameter tuning and often do not outperform simpler methods like kNN.

Contribution

It provides a comprehensive statistical comparison of deep generative models and classical anomaly detection methods, highlighting the importance of hyperparameter selection.

Findings

Deep generative models' performance depends on hyperparameter tuning.

Performance deteriorates with fewer anomalous samples for hyperparameter selection.

Classical methods like kNN often outperform deep generative models in practical scenarios.

Abstract

Many deep models have been recently proposed for anomaly detection. This paper presents comparison of selected generative deep models and classical anomaly detection methods on an extensive number of non--image benchmark datasets. We provide statistical comparison of the selected models, in many configurations, architectures and hyperparamaters. We arrive to conclusion that performance of the generative models is determined by the process of selection of their hyperparameters. Specifically, performance of the deep generative models deteriorates with decreasing amount of anomalous samples used in hyperparameter selection. In practical scenarios of anomaly detection, none of the deep generative models systematically outperforms the kNN.