DOD: Detection of outliers in high dimensional data with distance of distances

TL;DR

This paper introduces new geometric-based outlier detection methods for high-dimensional data that leverage pairwise distances and inner products, providing asymptotic separation guarantees and practical algorithms that outperform existing techniques.

Contribution

The paper presents novel outlier detection statistics based on high-dimensional geometric properties, with theoretical guarantees and practical algorithms that improve detection accuracy.

Findings

Methods outperform existing outlier detection techniques.

Proposed statistics asymptotically separate outliers from non-outliers.

Algorithms demonstrate high detection power with controlled false positives.

Abstract

Reliable outlier detection in high-dimensional data is crucial in modern science, yet it remains a challenging task. Traditional methods often break down in these settings due to their reliance on asymptotic behaviors with respect to sample size under fixed dimension. Furthermore, many modern alternatives introduce sophisticated statistical treatments and computational complexities. To overcome these issues, our approach leverages intuitive geometric properties of high-dimensional space, effectively turning the curse of dimensionality into an advantage. We propose two new outlyingness statistics based on observation's relational patterns with all other points, measured via pairwise distances or inner products. We establish a theoretical foundation for our statistics demonstrating that as the dimension grows, our statistics create a non-vanishing margin that asymptotically separates outliers from non-outliers. Based on this foundation, we develop practical outlier detection procedures, including a simple clustering-based algorithm and a distribution-free test using random rotations. Through simulation experiments and real data applications, we demonstrate that our proposed methods achieve a superior balance between detection power and false positive control, outperforming existing methods and establishing their practical utility in high-dimensional settings.