A Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data

TL;DR

This paper presents a universal unsupervised framework that refines contaminated training data for anomaly detection, improving performance even when normal data is scarce or mixed with anomalies.

Contribution

It introduces a generic, residual-based anomaly detection refinement method that effectively handles contaminated training data, outperforming naive approaches and approaching ideal anomaly-free training.

Findings

Framework improves anomaly detection accuracy on contaminated data

Outperforms naive contaminated data training methods

Often rivals training with clean, anomaly-free data

Abstract

Anomaly detection (AD) tasks have been solved using machine learning algorithms in various domains and applications. The great majority of these algorithms use normal data to train a residual-based model and assign anomaly scores to unseen samples based on their dissimilarity with the learned normal regime. The underlying assumption of these approaches is that anomaly-free data is available for training. This is, however, often not the case in real-world operational settings, where the training data may be contaminated with an unknown fraction of abnormal samples. Training with contaminated data, in turn, inevitably leads to a deteriorated AD performance of the residual-based algorithms. In this paper we introduce a framework for a fully unsupervised refinement of contaminated training data for AD tasks. The framework is generic and can be applied to any residual-based machine learning model. We demonstrate the application of the framework to two public datasets of multivariate time series machine data from different application fields. We show its clear superiority over the naive approach of training with contaminated data without refinement. Moreover, we compare it to the ideal, unrealistic reference in which anomaly-free data would be available for training. The method is based on evaluating the contribution of individual samples to the generalization ability of a given model, and contrasting the contribution of anomalies with the one of normal samples. As a result, the proposed approach is comparable to, and often outperforms training with normal samples only.