Robust Subspace Outlier Detection in High Dimensional Space

TL;DR

This paper introduces a robust subspace method for detecting hidden outliers in high-dimensional data by combining local density ratios and neighbor position comparisons across two projections, achieving high precision.

Contribution

It proposes a novel two-step subspace outlier detection approach that effectively identifies hidden outliers in very high dimensional spaces.

Findings

High precision in extremely high-dimensional spaces

Effective detection of hidden outliers within normal clusters

Works well across a range of dimensions from 10 to 10,000

Abstract

Rare data in a large-scale database are called outliers that reveal significant information in the real world. The subspace-based outlier detection is regarded as a feasible approach in very high dimensional space. However, the outliers found in subspaces are only part of the true outliers in high dimensional space, indeed. The outliers hidden in normal-clustered points are sometimes neglected in the projected dimensional subspace. In this paper, we propose a robust subspace method for detecting such inner outliers in a given dataset, which uses two dimensional-projections: detecting outliers in subspaces with local density ratio in the first projected dimensions; finding outliers by comparing neighbor's positions in the second projected dimensions. Each point's weight is calculated by summing up all related values got in the two steps projected dimensions, and then the points scoring the largest weight values are taken as outliers. By taking a series of experiments with the number of dimensions from 10 to 10000, the results show that our proposed method achieves high precision in the case of extremely high dimensional space, and works well in low dimensional space.