Unsupervised Anomaly Detection for Tabular Data Using Noise Evaluation

TL;DR

This paper introduces a novel unsupervised anomaly detection method for tabular data that leverages noise evaluation and deep neural networks, achieving high accuracy without using anomalous data during training.

Contribution

The paper presents a new noise-based unsupervised anomaly detection approach with theoretical guarantees, outperforming existing methods on extensive benchmark datasets.

Findings

Achieves 92.27% AUC on average across benchmarks.

Outperforms 12 baseline methods in effectiveness.

Easier to implement than state-of-the-art approaches.

Abstract

Unsupervised anomaly detection (UAD) plays an important role in modern data analytics and it is crucial to provide simple yet effective and guaranteed UAD algorithms for real applications. In this paper, we present a novel UAD method for tabular data by evaluating how much noise is in the data. Specifically, we propose to learn a deep neural network from the clean (normal) training dataset and a noisy dataset, where the latter is generated by adding highly diverse noises to the clean data. The neural network can learn a reliable decision boundary between normal data and anomalous data when the diversity of the generated noisy data is sufficiently high so that the hard abnormal samples lie in the noisy region. Importantly, we provide theoretical guarantees, proving that the proposed method can detect anomalous data successfully, although the method does not utilize any real anomalous data in the training stage. Extensive experiments through more than 60 benchmark datasets demonstrate the effectiveness of the proposed method in comparison to 12 baselines of UAD. Our method obtains a 92.27\% AUC score and a 1.68 ranking score on average. Moreover, compared to the state-of-the-art UAD methods, our method is easier to implement.