Detecting Outliers in Data with Correlated Measures

TL;DR

This paper introduces a novel outlier detection method that leverages data correlations and robust regression to identify outliers effectively, outperforming existing techniques on real-world datasets.

Contribution

The paper presents a new outlier detection approach that explicitly models outliers during robust regression, improving detection accuracy and robustness over prior methods.

Findings

Outperforms existing outlier detection methods on real datasets

Effectively detects outliers caused by sensor faults or atypical events

Demonstrates robustness and generality across different data scenarios

Abstract

Advances in sensor technology have enabled the collection of large-scale datasets. Such datasets can be extremely noisy and often contain a significant amount of outliers that result from sensor malfunction or human operation faults. In order to utilize such data for real-world applications, it is critical to detect outliers so that models built from these datasets will not be skewed by outliers. In this paper, we propose a new outlier detection method that utilizes the correlations in the data (e.g., taxi trip distance vs. trip time). Different from existing outlier detection methods, we build a robust regression model that explicitly models the outliers and detects outliers simultaneously with the model fitting. We validate our approach on real-world datasets against methods specifically designed for each dataset as well as the state of the art outlier detectors. Our outlier detection method achieves better performances, demonstrating the robustness and generality of our method. Last, we report interesting case studies on some outliers that result from atypical events.