Online robust covariance matrix estimation and outlier detection

TL;DR

This paper introduces an online method for robust covariance matrix estimation and outlier detection that operates in real-time, effectively handling contaminated data and reducing the masking effect.

Contribution

It presents a novel online approach for jointly estimating the geometric median and variance, enabling real-time outlier detection in streaming data.

Findings

Effective outlier detection demonstrated on simulated data

Robust estimation reduces bias from contaminated observations

Method operates efficiently in real-time scenarios

Abstract

Robust estimation of the covariance matrix and detection of outliers remain major challenges in statistical data analysis, particularly when the proportion of contaminated observations increases with the size of the dataset. Outliers can severely bias parameter estimates and induce a masking effect, whereby some outliers conceal the presence of other outliers, further complicating their detection. Although many approaches have been proposed for covariance estimation and outlier detection, to our knowledge, none of these methods have been implemented in an online setting. In this paper, we focus on online covariance matrix estimation and outlier detection. Specifically, we propose a new method for simultaneously and online estimating the geometric median and variance, which allows us to calculate the Mahalanobis distance for each incoming data point before deciding whether it should be considered an outlier. To mitigate the masking effect, robust estimation techniques for the mean and variance are required. Our approach uses the geometric median for robust estimation of the location and the median covariance matrix for robust estimation of the dispersion parameters. The new online methods proposed for parameter estimation and outlier detection allow real-time identification of outliers as data are observed sequentially. The performance of our methods is demonstrated on simulated datasets.