Robust Statistical Scaling of Outlier Scores: Improving the Quality of Outlier Probabilities for Outliers (Extended Version)

TL;DR

This paper introduces robust statistical scaling to transform outlier scores into probabilities, enhancing interpretability and accuracy for outliers, especially in critical applications like healthcare and finance.

Contribution

It proposes a robust scaling method that improves outlier probability estimates for outliers, addressing limitations of existing statistical scaling techniques.

Findings

Improved outlier probability accuracy for real-world datasets

Robust scaling outperforms traditional methods in detecting outliers

Enhances interpretability and comparability of outlier scores

Abstract

Outlier detection algorithms typically assign an outlier score to each observation in a dataset, indicating the degree to which an observation is an outlier. However, these scores are often not comparable across algorithms and can be difficult for humans to interpret. Statistical scaling addresses this problem by transforming outlier scores into outlier probabilities without using ground-truth labels, thereby improving interpretability and comparability across algorithms. However, the quality of this transformation can be different for outliers and inliers. Missing outliers in scenarios where they are of particular interest - such as healthcare, finance, or engineering - can be costly or dangerous. Thus, ensuring good probabilities for outliers is essential. This paper argues that statistical scaling, as commonly used in the literature, does not produce equally good probabilities for outliers as for inliers. Therefore, we propose robust statistical scaling, which uses robust estimators to improve the probabilities for outliers. We evaluate several variants of our method against other outlier score transformations for real-world datasets and outlier detection algorithms, where it can improve the probabilities for outliers.