SUOD: Accelerating Large-Scale Unsupervised Heterogeneous Outlier Detection

TL;DR

SUOD is a modular system designed to accelerate large-scale unsupervised heterogeneous outlier detection by optimizing data reduction, model approximation, and task load balancing, while maintaining accuracy.

Contribution

The paper introduces SUOD, a novel acceleration framework that significantly speeds up training and prediction in heterogeneous outlier detection without sacrificing performance.

Findings

Effective acceleration on 20+ benchmark datasets

Maintains high detection accuracy

Demonstrated success in real-world fraud detection case

Abstract

Outlier detection (OD) is a key machine learning (ML) task for identifying abnormal objects from general samples with numerous high-stake applications including fraud detection and intrusion detection. Due to the lack of ground truth labels, practitioners often have to build a large number of unsupervised, heterogeneous models (i.e., different algorithms with varying hyperparameters) for further combination and analysis, rather than relying on a single model. How to accelerate the training and scoring on new-coming samples by outlyingness (referred as prediction throughout the paper) with a large number of unsupervised, heterogeneous OD models? In this study, we propose a modular acceleration system, called SUOD, to address it. The proposed system focuses on three complementary acceleration aspects (data reduction for high-dimensional data, approximation for costly models, and taskload imbalance optimization for distributed environment), while maintaining performance accuracy. Extensive experiments on more than 20 benchmark datasets demonstrate SUOD's effectiveness in heterogeneous OD acceleration, along with a real-world deployment case on fraudulent claim analysis at IQVIA, a leading healthcare firm. We open-source SUOD for reproducibility and accessibility.