Anomaly Detection for High-Dimensional Data Using Large Deviations Principle

TL;DR

This paper introduces the Large Deviations Anomaly Detection (LAD) algorithm, which effectively detects anomalies in high-dimensional data by leveraging large deviations theory, outperforming existing methods and enabling online detection in multivariate time series.

Contribution

The paper presents a novel high-dimensional anomaly detection algorithm based on large deviations principles, with demonstrated scalability and effectiveness in real-world COVID-19 data analysis.

Findings

LAD outperforms state-of-the-art anomaly detection methods on benchmark datasets.

The online version of LAD successfully identifies anomalous counties in COVID-19 data.

Several detected anomalies correlate with known poor pandemic responses.

Abstract

Most current anomaly detection methods suffer from the curse of dimensionality when dealing with high-dimensional data. We propose an anomaly detection algorithm that can scale to high-dimensional data using concepts from the theory of large deviations. The proposed Large Deviations Anomaly Detection (LAD) algorithm is shown to outperform state of art anomaly detection methods on a variety of large and high-dimensional benchmark data sets. Exploiting the ability of the algorithm to scale to high-dimensional data, we propose an online anomaly detection method to identify anomalies in a collection of multivariate time series. We demonstrate the applicability of the online algorithm in identifying counties in the United States with anomalous trends in terms of COVID-19 related cases and deaths. Several of the identified anomalous counties correlate with counties with documented poor response to the COVID pandemic.