A Flexible Framework for Anomaly Detection via Dimensionality Reduction

TL;DR

This paper introduces DRAMA, a flexible and robust anomaly detection framework based on dimensionality reduction and clustering, suitable for high-dimensional data and adaptable for online and active learning scenarios.

Contribution

The paper presents DRAMA, a versatile Python package implementing a general anomaly detection framework that performs well in high-dimensional settings and allows for optimization with limited anomaly examples.

Findings

DRAMA is robust and competitive with existing algorithms.

It performs effectively on datasets with up to 3000 dimensions.

The framework is adaptable for online, active learning, and unbalanced datasets.

Abstract

Anomaly detection is challenging, especially for large datasets in high dimensions. Here we explore a general anomaly detection framework based on dimensionality reduction and unsupervised clustering. We release DRAMA, a general python package that implements the general framework with a wide range of built-in options. We test DRAMA on a wide variety of simulated and real datasets, in up to 3000 dimensions, and find it robust and highly competitive with commonly-used anomaly detection algorithms, especially in high dimensions. The flexibility of the DRAMA framework allows for significant optimization once some examples of anomalies are available, making it ideal for online anomaly detection, active learning and highly unbalanced datasets.