Copula-based anomaly scoring and localization for large-scale, high-dimensional continuous data

TL;DR

This paper introduces a copula-based anomaly detection method that not only identifies anomalies in high-dimensional data but also localizes their causes, handling large datasets with missing values effectively.

Contribution

It presents a novel, scalable, model-based approach using copula functions for anomaly scoring and localization in high-dimensional, fat-tailed data with missing values.

Findings

Effective localization of anomalies in complex systems

Handles large-scale, high-dimensional data efficiently

Demonstrated on telecommunication network data

Abstract

The anomaly detection method presented by this paper has a special feature: it does not only indicate whether an observation is anomalous or not but also tells what exactly makes an anomalous observation unusual. Hence, it provides support to localize the reason of the anomaly. The proposed approach is model-based; it relies on the multivariate probability distribution associated with the observations. Since the rare events are present in the tails of the probability distributions, we use copula functions, that are able to model the fat-tailed distributions well. The presented procedure scales well; it can cope with a large number of high-dimensional samples. Furthermore, our procedure can cope with missing values, too, which occur frequently in high-dimensional data sets. In the second part of the paper, we demonstrate the usability of the method through a case study, where we analyze a large data set consisting of the performance counters of a real mobile telecommunication network. Since such networks are complex systems, the signs of sub-optimal operation can remain hidden for a potentially long time. With the proposed procedure, many such hidden issues can be isolated and indicated to the network operator.